Case study: Using the web to scale a paper mountain

You might imagine that digital technology is quietly doing away with the need to store paper. In fact, there's never been a greater need for physical archives, thanks to an increased compliance burden. Fortunately, although archiving boxes of paper may be distinctly low-tech, there are increasingly high-tech ways to manage the process.

Like most government agencies, the Department for Environment Food and Rural Affairs (Defra) is a very paper-dependent organisation. It has 750,000 files in its archive, covering subjects from agriculture to pollution. Each box can have up to 150 documents, and there's enough of them to stretch end-to-end from London to York, says Mike Kaye, Defra departmental records officer.

"We've all got a duty to manage information," he says. "It's an asset second only to staff, and it's worth investing the time and effort to manage it properly."

But despite the huge volume of paper to sort through, Kaye claims to be able to have any one of those documents on his desk in less than a day, thanks to a system that brings digital management to hard copy storage.

Hoarding dead trees may seem ironic for an organisation spearheading the government's environmental strategy, but it's the reality of life post-Enron. Everything in the private sector must have an audit trail to meet corporate governance regulations such as the Sarbanes-Oxley Act, while in the public sector, freedom of information (FoI) legislation makes similar demands. Older laws such as the Data Protection Act apply to both and, although digital data takes centre stage, all apply just as much to paper.

"Enron and Worldcom sharpened people's minds, and Freedom of Information and Sarbanes-Oxley raised the profile of record management," says Mike Kennedy, director of client services at records management specialist Iron Mountain, which manages Defra's archive. "Significant fines have been levied on organisations that can't produce files on request, or dispose of them inappropriately."

"The volume of paper to be managed is still growing exponentially," says Kennedy, "although electronic data is growing faster. Companies are suffering from record overload."

You can't keep everything though. It's a physical impossibility. The Iron Mountain system Defra uses emerged as a realistic way of meeting the regulations, without getting buried under a mountain of paper. "A retention strategy is a matter of balance," says Kennedy.

Defra looks after its own files for seven years, at which point they are archived at Belvedere in Kent, in one of Iron Mountain's 80 UK warehouses. When the files are 30 years old they are destroyed, except for historically important records that go to the National Archive.

While they are in Iron Mountain's hands, the documents have to be available to civil servants working on new policies, and others serving information requests from the public. Staff expect to be able to call up the information digitally, but digitising Defra's mountain of paper would never be practical.

The way forward was to digitise the metadata. The file labels and the information about what is in each folder are in a database, which staff can access through Iron Mountain's Connect portal. Once they've found the documents they want, they order them and receive them by courier the next day.

"It's like ordering a book from Amazon," says Kaye. "There's a very attractive front end, and the portal is available 24/7." Everyone knows how to use online ordering systems, and there's a Google-style search box to make simple queries, as well as more advanced multi-field searches. Most staff can use the system within an hour's training, says Kaye, plus they can use it when working from home.

Defra has no-one working full time on retrieval. Kaye's staff of seven manages Defra's current records and recent files. Alongside this, they handle about 125 archive requests each day, placed by phone from the rest of Defra's 8,000 staff. "They talk through the request on the phone, and once they find the right file, they press one button, and it's on the way," says Kaye. "The customer gets an email to say it is coming."

Defra started using Iron Mountain's services in 2003. Before that, the department had a basic system, which told staff roughly where a file was, but left them searching the racks to find particular items. "It was time consuming," says Kaye. The archives were kept in a facility shared with the Ministry of Defence, at Hayes in Middlesex.

In 2003, they were moved to Belvedere, a site shared with other Iron Mountain customers. Despite the size of Defra's paper pile, it's not even in the top 10 of the company's customers in the UK, says Kennedy. The big users are international finance companies.

Iron Mountain claims to be the biggest player worldwide in hard copy record management; its biggest competitor is the vaults that many companies keep in-house. By comparison, outsourced archives can offer better access; few bodies are likely develop a web portal for their own records.

With the records moved, Defra and Iron Mountain immediately began planning the way to gain digital access. Iron Mountain has been evolving a records-management portal for some years, and is expanding its digital records-management business through a recent purchase of Connected.

This stage of the process can be painful, says Kennedy. "For organisations that don't have a good records-management strategy, it can bring big internal challenges. For some clients we have to do remedial work on the metadata, particularly at the disposal end of the spectrum."

Fortunately, Defra's files all had useful metadata — the information about the file that is written on its cover. The big challenge was to capture it reliably, a process that took Iron Mountain staff about six months, with a thorough quality-control process. "It's essential to get the data correct," says Kaye.

Things don't stand still in archiving, however. The nature of data is changing, and at some point in the future, the Defra archive will be handling recent documents, where the primary data is online. "The ratio will shift between paper and electronic records," says Kennedy, and Kaye confirms: "We're working with Iron Mountain on a digital archive."

Archivists never make big claims for their work — it's not their style. But Kaye is prepared to agree that improved availability may help form better government policies: "It encourages people to do more and deeper research," he says. "Having papers available should help inform the quality of the papers they are producing themselves."