Towering rebuild: what I learnt from 9/11

Daily Newsletters

Sign up to ZDNet UK's daily newsletter.

When the towers collapsed, Thermo lost its development servers, which were in the WTC office, as well as file and e-mail servers, desktop workstations, and all the rest of the equipment necessary to run an office. We also lost hundreds of hours of work that hadn't yet been backed up, including new staging templates I'd been working on, improvements to the database by our data base administrator, and a host of bug fixes and interface improvements we had been planning to migrate to our co-location facility that week. However, the company had made some fortuitous choices in the past. The most important of those prior to the eleventh was the decision to co-locate most of the company's servers off site. While we lost a great deal, the lion's share of the Web site and its content, representing thousands of hours of labor, remained intact in New Jersey. The original impetus behind co-location was to protect against comparatively trivial problems, like power outages. But it was also economical, according to Wissman. "For most corporations, it makes sense to leave hosting to people who do it best. The systems you need, like truly redundant power and Internet access, are extremely expensive, but relatively cheap as a marginal cost in a co-hosted facility." Co-location reduces some risks, but it just transfers others to a different place. Realising this, Thermo now removes full data backups to an offsite location at least once a week. "We do a much better job than we used to," Wissman told me. "Until 9/11, we made nightly backups, but left the tape there. Now we take it off site-we're much more aware of what it really means to be redundant and backed up. It's cute to back up on a local tape, and it's useful if there's a grain of sand on the hard drive or whatever. But it won't help at all if there's a real event, potentially something as simple as a fire." Off-site backup can be as basic and cheap as having someone bring a tape home once a week, or as complex as having full real-time mirroring of all data between two facilities. While the latter approach, done correctly, eliminates the possibility of data loss, it requires significant throughput, and can be expensive. Leibundgut and Wissman determined that that level of service wasn't required, and that regular incremental backups would suffice. Every organisation needs to balance the costs of continuity strategies versus the cost of possible downtime. For a big company, continuity costs can be massive. According to a recent Network World report, Coca Cola Enterprises, the bottling division of The Coca-Cola Company, will spend over $400 million on a disaster recovery/business continuity plan this year, the largest IT project in the company's history. Thermo was lucky in many ways, because while the Web site was an important business tool, we weren't as reliant on real-time communication as some other companies. The site's primary purpose was to store the huge volumes of information the company has on the thousands of products sold by various Thermo business units. As Thermo increasingly relies on e-commerce, and uses the Web for internal business processes, the disaster recovery and continuity planning will have to change to reflect the changing role of the Web in the organisation. For now, Thermo has chosen to rely on increased planning using mostly equipment and software already on hand -- for instance, the development environment, housed in a separate facility than the production servers, can be converted to a stand-in production environment should the need arise. If a standby system is necessary, it doesn't have to be identical to the production system it's replacing -- fewer or less expensive servers may be suitable, since a lower level of service can be acceptable as long as availability is maintained. This decision will vary greatly depending on the type of business being protected -- for instance, a news organisation might see Web traffic spike in the event of a major catastrophe, and actually require greater capacity with their standby servers than their normal production servers.

Post your comment

In order to post a comment you need to be registered and logged in.

You can also log in with Facebook. Log in or create your ZDNet UK account below

  • Login

Will not be displayed with your comment

By signing up for this service, you indicate that you agree to our Terms and Conditions and have read and understood our Privacy Policy. Questions about membership? Find the answers in the Community FAQ

Get ZDNet UK's daily newsletter

Enter your email address to sign up

ZDNet UK Live

BrownieBoy

> I'm told it's somewhat annoying when people have their Macs stolen > and Apple stores treat the thief as the owner, but there you go. Ouch,...

19 minutes ago by BrownieBoy on AMD Ultrathins to challenge Intel Ultrabooks
Moley

@kevinmchapman. OK, I acknowledge that 'most' was a gratuitous throwaway comment as an afterthought and too presumptuous. As to proof, as you...

5 hours ago by Moley on A tale of two distros: Ubuntu and Linux Mint
Jack Schofield

@BrownieBoy > Works really well for thieves.... >> Nice attempt to deflect the argument by tossing in a point that's totally >> irrelevant, even...

6 hours ago by Jack Schofield on AMD Ultrathins to challenge Intel Ultrabooks
raskolnikof

fantastic that the so called piracy bills have been withdrawn. however, these anti-democracy supporters are still in the shadows so lets be alert...

7 hours ago by raskolnikof on SOPA, Protect IP support wavers in face of online protest
Tony Douglas

Please God no; teach them anything you like - thinking rationally, the uses and misuses of data, what data is and what it's not - but leave the...

9 hours ago by Tony Douglas via Facebook on Kids are the future. Teach ’em to code.
BrownieBoy

@Jack, > Works really well for thieves.... Nice attempt to deflect the argument by tossing in a point that's totally irrelevant, even it were...

23 hours ago by BrownieBoy on AMD Ultrathins to challenge Intel Ultrabooks
bootlegger

Make that 13 people now - I got refused today at Manchester airport. I thought I was up to date on this legislation - I knew of the EU ruling from...

1 day ago by bootlegger on UK airport body scans will not be opt out
tinycg

Don't forget to check out apps like GoodReader or SlideShark either, they're indispensible for people on the go in presentation situations. Best...

1 day ago by tinycg on Four top iPad apps for people on the move
TerryRK

Well it seems there is something a number of us agree on. Why is the Ubuntu Unity launcher so ugly? I thought perhaps it was something to do with...

1 day ago by TerryRK on A tale of two distros: Ubuntu and Linux Mint
Freebies202

Duplicate comments are not made intentionally. Its very good to know that now you are keeping check on this problem because sometimes a commenter...

2 days ago by Freebies202 on Microsoft fixes blog comments, speeds up blogs with open source
kevinmchapman

"the very significant number of users" and "many (most) of us" - you have no evidence for these statements. It is a fact that most users are saying...

2 days ago by kevinmchapman on A tale of two distros: Ubuntu and Linux Mint
Marg Menzies Harrison

Another grammar faux pas is the improper use of "you". When sitting down down in a restaurant, for example, I get cringe when the waitress...

2 days ago by Marg Menzies Harrison via Facebook on 10 flagrant grammar mistakes that make you look stupid
zdnetukuser

And NOW, folks, for Canonical's next trick... Kubuntu is late. Here's a pencil. Draw your own conclusions. cf.:...

2 days ago by zdnetukuser on Linux Minterface
Moley

@kevinmchapman. The discussion here reflects the very significant number of users who really do like the traditional menu system and who wish to...

2 days ago by Moley on A tale of two distros: Ubuntu and Linux Mint
kevinmchapman

Er, no... It is an efficient means of finding the application/file/setting you need in one place. The icons are a simply a fallback for when you...

2 days ago by kevinmchapman on A tale of two distros: Ubuntu and Linux Mint
TerryRK

Isn't the provision of a text based search an admission by the developers that the mass of icons approach does not work? I don't need to use a...

2 days ago by TerryRK on A tale of two distros: Ubuntu and Linux Mint
kevinmchapman

"Unity and GNOME 3 both abandon the old text-based cascading menus in favour of a graphical icon-driven system." Point truly missed. Both use a...

2 days ago by kevinmchapman on A tale of two distros: Ubuntu and Linux Mint
TerryRK

whs001 - Thank you, I'm glad you liked the article. I absolutely agree with you on your first point. I should perhaps have made it clearer that...

2 days ago by TerryRK on A tale of two distros: Ubuntu and Linux Mint
Dennis Nilsson

If we allow corporate interest to dictate the way our government circumvents due process against foreign entities then we should accept the same...

2 days ago by Dennis Nilsson via Facebook on ACTA stumbles in Germany
GHar123

I totally dislike pirating of works, I fear that artists will be deterred from creating works if they think that they are going to get ripped off....

3 days ago by GHar123 on ACTA stumbles in Germany