A large number of subscribers to the Internet service provider (ISP) PlusNet have experienced a serious outage of their email services, which also seems to have led to thousands of emails being erased.
According to a statement from the ISP: "At 10am on Sunday 9 July, upgrade work on the PlusNet PLC email platform inadvertently caused email problems for half of our PlusNet customers and the majority of our Force9, Free-Online and Metronet customers (140,000 customers)".
The statement adds that "any of these customers using IMAP or Webmail and therefore normally only storing mail on our servers will be missing historical messages".
It's not clear how much mail is missing. A Plusnet spokesperson told ZDNet UK on Wednesday that "the affected disks are currently undergoing recovery", and said that the company expects to have a "full view" of the extent of the damage in the next two days.
According to The Register, which first reported the story, over 700GB of data was lost.
It appears the problem was caused when a senior engineer mistook the management interface of a live email server for that of a backup server, and erased all the data on the wrong one.
In the statement, Plusnet said it won't be offering compensation for a break in service, but did "share our customers' concerns and frustration in relation to the matter".
On an unofficial Plusnet user forum on Wednesday, company representative Ian Wild assured disgruntled members: "It was a genuine mistake, an accident, and I'm afraid no measure of change control procedures or anything else would have completely prevented this.
"At the end of the day, all we can do is hold our hands up here and say that although an unfortunate set of events, this should never have happened… you can never remove the risk of a human error entirely."
Talkback
humans are also the weak link in security systems.
we are weak, we need replacing with bots.
As a PlusNet user, formally very enthusiastic,I could not now recommend them to anyone. Apart from the recent email disasters, still not fully resolved, they have made many changes to support, and had other service interruptions and issues that have lowered my satisfaction considerably. Web mail is particularly 'buggy' and the bugs are not fixed over a long time. They need to climb back.
And they have quietly canned their much vaunted upgrade to 8MB for existing customers, blaming BT issues (maybe that's true).
As for the human error, if every proposed action in a crisis situation was checked by an experienced engineer prior tothe first guy 'pressing the button', it could probably have been avoided.