Dealing with a recent death in my family started me thinking about how we carry entirely too much information around in our heads. Of course the assumption we always make is that we are going to be around when people have questions about critical processes or equipment, but what if we aren't?
For every IT shop out there, there is a system or a process or a program that only one person knows and everyone is afraid to touch because it is so sensitive. Yet we fail to create documentation about these systems because good documentation takes time and effort and has to be revised to stay current. Another challenge is that the people who actually produce or keep these systems running often don't actually like to write or are not even capable of writing good documentation. Yet, clear, concise documentation is critical to disaster recovery/business continuity.
You should be reviewing and updating your disaster recovery/business continuity plan at least yearly. Take this time to make sure that the supporting documentation for your critical systems is also up to date. Several standards organisations have issued disaster recovery guidelines including ASIS, the US-based society for security management professionals. They have issued the free publication, Business continuity guideline: a practical approach for emergency preparedness, crisis management, and disaster recovery.
Prepare for disasters of all shapes and sizes
When many of us work on disaster recovery plans we picture our data centre falling into a big hole or going up in flames. What we usually don't do is to take into account that we and our colleagues might not be around. The assumption is often made that there will be somebody with knowledge of the situation that can step in and take over. But, what if there isn't?
If you think such a situation is unlikely, all you have to do is remember the terrible destruction of 9/11 — not the loss of equipment and systems, but the people who were responsible for them. The stark reality is that government organisations and agencies are bigger targets for terrorists, whether international or home-grown. It's a sobering thought, but one that must underpin your IT strategy when thinking about security measures or disaster recovery efforts. That's a worst-case scenario on a large scale, but even the mundane could turn into a disaster for your IT department.
How often has your management team or a big chunk of your programming staff or network support staff gone to lunch together? Or maybe taken a flight together? I know that there were many times when I had all my management staff with me as we travelled to lunch or a meeting. What would happen if no one came back from lunch? Can your organisation handle that? Is your disaster recovery plan stored where someone can find it? In someone's desk? Who will know where to get it at the worst time possible?
Getting prepared: Create the proper documentation
First you need to acknowledge how important documentation is to your operations. By doing so, you can then justify the time and resources required to create and maintain good documentation to management. Secondly, make it a formal part of your operations. Put someone (whose primary responsibility is documentation) in charge of it and make sure it is getting done properly. Thirdly, spend some resources on an analyst who is good at understanding processes and can put them down in a manner that can make sense to someone that is not in your shop. Lastly, make it part of your disaster recovery/business continuity review process.
While expecting to be around tomorrow makes for a healthy outlook on life, we owe it to our organisations to be a bit more pessimistic and plan for the sudden loss of key personnel. Give your disaster recovery and business continuity plans this test: Pull it out of your drawer, read it, and see if it holds up without input from any of the key personnel you have identified in the document. If you can hand it to a perfect stranger with IT knowledge and he/she can get things going again — pat yourself on the back. If not, you need to set some time aside to update your documentation. And if you don't think it's important, just remember 9/11.
