The PlusNet outage that hit the internet service provider's users early on Thursday was the result of human error at a third-party data centre, it now appears.
A full report on the incident is yet to be drawn up but, according to PlusNet, initial investigations show that contractors cut all power to the as-yet-unidentified data centre, in the process knocking out PlusNet's internal systems, portals and email.
"We have found out this morning that the owners of the data centre had contractors in doing some maintenance. They failed to notify us of this work," wrote PlusNet's communications manager, Mark Kelly, on the ISP's community page on Thursday.
"During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre," Kelly continued. "That action caused all power to all services to die including the generators and [uninterruptible power supply]. Failover couldn't work as the power to everything was killed. We believe that this was human error; however, we are awaiting a full report from our suppliers."
Kelly added that not all services can be automatically failed over. "[Databases and so on] have to be failed over manually to ensure that data integrity is maintained," he wrote. "As we were unaware that this work was taking place we did not have anyone on standby to react. Thankfully our incident management processes and call out system kicked in very effectively. We had engineers on site within about 20 minutes and 97 percent of services were back up and running in around four hours."
"I'll reiterate that this was a catastrophic failure which we believe was caused by human error," wrote Kelly. "All the resiliency in the world could not have prevented this situation."
However, Kelly admitted that PlusNet's communications team should have become aware of the incident sooner and relayed information to customers more effectively. "I am dealing with that aspect internally this morning," he wrote.
