Regulatory compliance and business intelligence systems have opened the eyes of many companies to a new way of thinking about managing data. But keeping data organised and accessible for a few quarters is one thing — what will happen to it 20 years from now?
As the digital world continues to mature, with digital information reaching a critical mass in all areas of life, that's a question organisations are starting to ask. Despite the fact that digital information has been around for decades, there is still no tried-and-tested way of keeping data intact beyond the next time a medium or file format becomes obsolete — much less of dealing with the surprisingly short physical lifespan of the media.
This year marks a turning point in the digital world, IDC argued in a recent white paper: for the first time, the amount of information created — around 260 exabytes — will surpass the storage capacity available. The figure is symbolic, since much of the information generated doesn't need to be stored, but it underscores that the digital world has matured, something that has far-reaching implications for how companies manage and store their data.
On the management side, the past few years have seen a carrot-and-stick approach to change. Regulatory compliance has forced companies to come up with strategies for dealing with particular types of data — overall, 20 percent of the digital universe is subject to compliance rules and standards, according to IDC's estimates. And, meanwhile, business intelligence (BI) systems have shown companies that, if they are organised enough with their data, it can pay off.
We've reached a critical mass of material that exists only in digital form
Richard Masters, programme manager, British Library's Digital Object Management scheme
"Companies are perceiving a higher value in their information," says IDC analyst Marcel Warmerdam. "The idea is you can capture everything, and then, within the numbers, could be found the solution to profitability, if you can just grab it. BI systems can do that."
The longer-term issues, however, remain more of a mystery. A paradox of the digital world is that, as the ability to store bits increases, the ability to store them over time decreases, something that can be seen in the worryingly short expected lifespans of digital media.
The design life of a low-cost hard drive is five years, while the usable lifespan for magnetic tape could be as short as 10 years, and optical media such as CDs and DVDs may become unusable in just 20 years.
Where digital information is concerned, physical degradation is the least of the conservation problems. The more pressing issues are to do with obsolescence at all levels, including the media, the file formats and the software used to read the files.
All this has been talked about for years, but it's only now that serious efforts are finally getting underway to come up with large-scale, practical answers. Some initiatives are focusing on standards and best practices that can simplify long-term storage, while others, notably those of large university or national libraries, are putting trial systems in place.
Read this
Leader: Lessons in obsolescence
Data integrity over time can, if you wish, be seen as just another task for the overworked IT manager to worry about...
"This is happening now partly because of the realisation that the digital world is really upon us now, in a big way," says Richard Masters, programme manager of the British Library's Digital Object Management scheme. "Until 2002 or 2003, a lot of our digital material was digitised — you could always go back to the original. Now we've reached a critical mass of material that exists only in digital form."
The outcome of all this experimentation should be that, somewhere down the line, there will be tools and a body of knowledge for companies to draw on in dealing with their own stacks of mouldering disk drives and dusty reels of magnetic tape.
Digital longevity
In a famous January 1995 Scientific American article, RAND Corporation computer scientist Jeff Rothenberg noted a disheartening fact about digital objects: the things that make them difficult to preserve are precisely those aspects that make them interesting and attractive in the first place.
In the article, Ensuring the Longevity of Digital Information, an expanded version of which can be found online, Rothenberg argues against the notion that standardised formats can be a solution to preservation problems — a concept underlying, for instance, the current debate over the standardisation of Microsoft's XML-based document formats, and that of the OpenDocument Format (ODF).
Rothenberg says it's an illusion to think that even something as simple as a word-processing document format can be encapsulated in a long-term standard. "The incompatibility of word-processing file formats is a notorious example — nor is this simply an artefact of market differentiation or competition among proprietary products," he wrote. "Rather it is a direct outgrowth of the natural evolution of information technology as it adapts itself to the emerging needs of users."
The same goes for every other type of file format, Rothenberg argues. From the point of view of preservation, this means standards can't save the day — formats will continue to evolve. Moreover, they'll undergo "paradigm shifts" in which the old ways of thinking are, as often as not, swept away.
One answer to this continual change is to continually migrate, or translate, documents into current formats — an approach adopted by the British DOM programme, for one. But paradigm shifts mean that...
