London hosting firm Redbus Interhouse has blamed a power outage for a meltdown which took many of its customers off the web on Tuesday morning.
The incident underlines the importance of power, which according to other reports this week is a limiting factor in London's data centre capacity.
Redbus has apologised and blamed the fault on a failure that occurred during a routine generator test. The company added that it is going to decommission its existing infrastructure and move customers to new upgraded systems.
The company claimed the underlying electrical fault, which occurred at 0956 on Tuesday, was fixed within five minutes, and the redundant system restored in nine hours. Individual Web sites were restored over the day, for example Shout99 contractors' site appeared to have been restored at around 1550 on Tuesday.
"A routine load-building generator test was being carried out at the Redbus, Harbour Exchange facility," said a Redbus statement. “During the initial stages of the generator testing a fault occurred within the switchgear panel serving the power to the 8th and 9th data floors.”
A coil in the generator was "displaced", causing a short-circuit to the uninterruptible power supply (UPS) modules; the power was dropped, and the Redbus units moved to battery power.
Unfortunately, the fire alarm also went off, meaning the building had to be evacuated. Redbus staff were stranded outside the building while the UPS batteries quietly ran out and the two floors shut down.
Power was restored to the affected racks via a manual bypass onto raw mains at approximately 1001, Redbus explained. "The redundant system was fully restored at approximately 1915."
Talkback
Although ZDNet didn't report on them, this is not the first Redbus outage. There have been 3 in the space of 6 months. As a customer there I am in no way "satisfied" with the service level.
The power failure at RedBus HEX also damaged lots of equipment including power supplies for many servers, routers, switches and power distribution systems. Most power wasn't actually available to customers until around 1pm, some longer. Some of the equipment damaged by the power outage caused problems into Wednesday. Redbus engineers were so tied up with problems they took around 5 hours to respond.
I agree that power took much longer to restore than the 5 minutes Redbus are implying. We and many other customers didn't get back online until after 4pm
The failure also took out the hard drive of our Uk2 server. Replacement of the har drive, reload of systems, application software and data saw us back on line at 00:15 Friday 4 March.