Toolkit

Story: UK VoIP market hots up

• 
• 

Date: Friday 10th March 2006

Dear Customer

Earlier this week there was a serious failure of parts of the AlwaysON network that affected a number of customers.

This was a direct result of the fact that a part of our network was running unprotected during a transition between an interim and a long term solution for managing denial of service attacks. Unforeseen by the engineers implementing the change, there had remained a gap in the protection system during this transition. This gap allowed an overload of a supposedly protected part of the core infrastructure and the overload could only be rapidly resolved by isolating the public internet.

During the recovery from the problem one part of the network did not come fully back on line and this problem was the most difficult to isolate. The underlying cause for this inability to recover is being investigated by our supplier, but as a result, certain customers experienced a longer outage than should have been expected even given the magnitude of the original problem.

We have now audited the technical issues and the communication processes and have started to implement a number of changes. During next week we will issue an information bulletin to describe some key changes to incident communication. This will include robust mechanisms for sending updates to you and for you to contact us in the event of an emergency.

I can assure you that the gap in the protection has now been plugged, and that further actions are planned as a second line of defence.

AlwaysON very much regret the impact of this incident on a number of customers and the resultant inability to maintain adequate communications during the problem period. We apologise for the difficulties that this has caused you and your customers

• 
• 

• 
• 
• 

Full Talkback thread