Security /

The evanescence of data

By M. E. Kabay
Network World Security Newsletter, 05/22/00

Everyone makes backups and stores them, right? And everyone keeps archives of electronic data in accordance with legal requirements or organizational policy, right?
Well, no.
Many of us are storing records in ways that make it unlikely we will ever be able to read them in the long term. As archivists, we ought to know better.
Storing records is only half the task of records management; making them available and useful is the other essential function.
For short-term storage, there is no problem ensuring that stored information will be usable. Even if a software upgrade changes file formats, the previous versions are usually readable. In a year, technological changes such as new storage formats will not generally make older formats unreadable.
Over the medium term, up to 5 years, difficulties of compatibility increase, although not catastrophically. There are certainly plenty of 5-year-old systems still in use, and it is unlikely that this level of technological inertia will be seriously reduced in the future, if only because of computer users' growing resistance to forced change.
Over the longer term, however, there are serious problems to overcome in maintaining the availability of electronic records. Over the last 30 years, certain forms of storage have become essentially unusable, regardless of the persistence of data. As an example, AES was a powerful force in the dedicated-word-processor market in the 1970s; 8-inch disks held dozens or hundreds of pages of text and could be read in almost any office in North America. Today, it would be extremely difficult to recover data from AES diskettes.
The problems of obsolescence include media degradation, software incompatibilities and hardware incompatibilities.
Media Degradation
Magnetic media degrade over time. Over a period of a few years, thermal disruption of magnetic domains gradually blurs the boundaries of the magnetized areas, making it harder for I/O devices to distinguish between the domains representing ones and those representing zeros. These problems affect tapes, diskettes and magnetic disks and cause parity errors. Specialized equipment and software can compensate for these errors and recover most of the data on such old media.
Tape media suffer from an additional source of degradation: The metal oxide becomes friable and begins to flake off the Mylar backing. Such losses are unrecoverable. They occur within a few years in media stored under inadequate environmental controls and within 5 to 10 years for properly maintained media. Regular regeneration by copying the data long before the underlying medium disintegrates prevents data loss.
Optical disks, which use laser beams to etch bubbles in the substrate, are much more stable than magnetic media. Original predictions in the 1980s suggested that CD-ROMs would remain readable for decades; however, more recent reports suggest that even these relatively stable devices are subject to data degradation over a period of roughly a decade. Archivists predict that a CD ought to last about 25 years when stored at 10 degrees Celsius (50 F) and 30% relative humidity.
Now that you've read all that, I'm sure that the following well-known warning makes a lot of sense: Verify the readability of your backups before storing them.
Software
Data may be readable, but will they be usable by current and future software?
Manufacturers provide backward compatibility, but there are limits. Modern word-processing software can convert files from earlier versions of a variety of products - but only back a few years. Over time, most application programs evolve and drop support of the earliest data formats. Database programs, e-mail, spreadsheets - all of current and future versions may have trouble interpreting old data files correctly. The situation can be even worse for in-house programs, where maintaining compatibility with archival materials can simply drop off the list of priorities when budgets are tight.
In any case, all conversion raises the possibility of data loss, since new formats are not necessarily supersets of old formats. For example, in 1972, Runoff text files on mainframe systems included instructions to pause a daisy-wheel impact printer so the operator could change daisy wheels - but there was no requirement to document the desired daisy wheel. The operator made the choice. What would document conversion do with that in-line instruction?
Operating systems evolve (or, in some cases, degrade - but that's another story). Programs intended for Windows 3.11 of a decade ago do not necessarily function on today's versions of Windows. And the operating systems of yesteryear do not necessarily run on today's hardware. Even emulators can cause problems, because again there is no guarantee of compatibility between the emulated system and the emulator.
Hardware
Finally, even hardware eventually becomes impossible to maintain. As mentioned above, it would be extremely difficult to retrieve and interpret data from word-processing equipment from even 20 years ago. No one outside museums or hobbyist circles can read an 800-bpi 9-track 3/4-inch magnetic tape from a 1980 Hewlett-Packard HP3000 Series III minicomputer. Over time, even parameters such as electrical power attributes may change, making obsolete equipment difficult to run even if it can be located.
The most robust method developed to date for long-term storage of data is Computer Output to Microfilm. Documents are printed to microfilm, appearing exactly as if they had been printed to paper and then microphotographed. Storage densities are high; storage costs are low; and in the worst case, the images can be read with a source of light and a simple lens. In a pinch, the data could theoretically be converted back to digital formats using optical character recognition.
Information security demands that we be able to read old data. It is time for us to pay serious attention to long-term storage technologies.