Tens of thousands of small businesses and individuals were forced offline on Thursday, after a power failure at Telehouse Docklands affected several UK Internet service providers.
Telehouse is a major UK co-location facility which provides hosting facilities for many ISPs and larger companies. It offers the promise of a secure, resilient infrastructure, thanks to its multiple power supplies and dedicated connectivity.
However, a power outage on Thursday morning caused significant disruption to several ISPs, including PlusNet and Clara.net.
It appears that problems began just before 8am on Thursday, when one of Telehouse's power-distribution units failed after a cable connection burned out. At noon, Telehouse implemented a workaround, which took some other connections offline temporarily.
"Power was restored to affected customers at approx 12.20pm and this was achieved without the need to isolate the power-distribution unit, thereby avoiding further customer disruption. A post-event review is underway," said Telehouse in a statement.
The problems affected several server suites used by PlusNet. Its broadband connectivity and email and usenet servers were hit, and around half its customers who were connected at the time were knocked offline. Users were unable to connect via Telehouse until the afternoon, which means only half of PlusNet's 200,000 customers could get online at once.
Most services were restored by mid-afternoon on Thursday, although PlusNet's telephone system was still suffering problems.
It appears that those affected will not be eligible for compensation. A PlusNet spokesman said the company had apologised to those who were affected and thanked them for their patience during the engineer repair work.
A Telehouse spokesman added that an inquiry was underway to find out how a single connection failure could have caused such damage.
Talkback
We use plus net. We had no adsl connectivity back until late evening. We have had no apology or explanation from Plusnet.
If the affected servers at Telehouse had been running a city trading system, billions could have been lost. It should not have taken 10 hours to swap out a power unit, and bring everything back online.
I still fail to understand why it happened at all. We have the technology for backup power supplies, redundant systems etc. Allegedly Telehouse implements this type of technology.
However, I have visited Telehouse in the past, and while accessing my servers, I could easily have disconnected other customers' servers, or uploaded malware, had I wanted to. So perhaps Telehouse's reliability claims are just marketing hype.
ISP's should know by now there is no such thing asd 100% availability. Single points of failure should be removed, diversity should be built in and regular testing should test systems recovery. Most problems aren't usually high lighted until the day you need the generator or reboot the server thats not been rebooted for 12 months.
It strikes me as bad management and cutting corners. A good system will degrade, but not fail in the event of failure.
Working as an IT Consultant and Engineer for many years has taught me that people are processes are to blame for most issues. As a technical solution architect I'm very weary of the promises delivered by Data Centers. In my experience their approach is based on "bare bones" meaning that they will cut everything down to the bone in order to save costs in delivering services. This in turn leads to low quality resourcing of technical staff and poor management of processes.
This however is a market problem that is caused by the fact the rack space is a commodity and therefore needs to be aggressively streamlined in order to be profitable for the data centers.
I know that I'm giving out mixed messages in regards to where the blame lies but the truth is that if hosting became too expensive customers would host themselves (like the old days) therefore data centers are stuck in the position of always fighting to keep costs low.
Another analogy I can think of that is part of the problem is that a volvo is a very safe car and will safe your bacon, yet most people would not spend the money on safeguarding their life as they would buy a fiat instead (no offense fiat!!).