"We had problems with a routine software upgrade affecting desktop PCs," said a DWP spokesperson on Friday. "There have been delays with dealing with new or amended claims because many staff have not been able to access their systems. We have been working overnight with our suppliers to resolve the PC problems and nearly all users can now access their business systems."
The spokesperson confirmed that 80 percent of computers were brought down, and that its main suppliers are Microsoft and EDS.
Microsoft said that the issue has now been fully resolved, but was unable to provide any information on what caused the crash. EDS did not immediately respond to requests for comment.
"Microsoft is aware of the issue," said Microsoft the spokesperson. "We worked closely with our partners to help rectify the situation and to support the DWP."
The Public and Commercial Services Union (PCS), the union which represents civil and public servants in central government, said that the number of employees is around 130,000 full-time equivalents, with approximately 100,000 terminals, which would put the number of desktop computers affected at 80,000. A DWP spokesperson denied the figure of 100,000, saying that it employs only 80,000 staff, but did not have any figures on the number of desktop computers in operation.
The PCS said that the news of IT problems at the Department for Work and Pensions, calls into question the feasibility of axing 30,000 jobs on the basis of IT improvements.
"There was anecdotal evidence of Giros being written by hand and services in large parts of the country being severely limited as it emerged there had been major IT problems across the department since Monday," said a PCS spokesman.
Work and pensions secretary Alan Johnson has ordered an internal inquiry into the role of Microsoft and EDS, who run the ministry's network as part of a £2bn information technology deal, according to The Guardian.
By Thursday night, 95 percent of desktops were fully functional and the department claims that the impact on customers has been minimal.
This disruption is the latest in a line of government technology failures, which included the failure of the £450m IT project at the Child Support Agency and the collapse of the £500m Electronic Benefits Transfer System in August.
IT consultancy Avanade -- the joint venture between Microsoft and Accenture -- warned companies on Friday that the urgent need to migrate from Windows NT 4.0 before Microsoft withdraws support has caused problems for companies.
"With the imminent retirement of support for Windows NT4.0, the migration space has become increasingly frenetic over the past few months," said an Avanade spokesperson. "Avanade has identified that its customers are under increasing pressure to migrate quickly, but are concerned that with increased speed of migration comes risk."
For more reaction to the DWP debacle, read ZDNet UK's interview with Liberal Democrat IT spokesperson Richard Allan.
Talkback
Sorry but this is unacceptable and as a cockup even surpasses Microsoft's high standards. My mum lost a vital £400 because of this, MS really screwed us this time.
Microsoft causing PCs to crash unknowingly. How incredible!! Seems that if you throw enough money at something, the risks are overlooked and outright ignored.
How can the people - who elect these dolts who made the disasterous decision to go with a known vendor who has a track record of virus-prone, exploitable, unstable, and extremely over-priced software - hold the government accountable? After all, it is the government who serve the people.
Microsoft only serves Microsoft, no matter the cost, no matter who gets hurt.
Blame the driver (EDS) and not the car (MS)
MS NOT. So why is EDS the only people to have the problem ?
Shame on you zdnet. You know where the problem lies. But that did not stop you from using the MS name in the headline.
not an MS fan. But a realist.
the point is that MS software shouldn't just fall over, reguardless of what 3rd party software is installed
Well, there aren't really enough details for comment on this piece. Did the 'update' cause a problem because of bad programming in the 3rd party software? Did the 'update' change some default access settings and cause the failure ? That would be bad -- if access is too lax that's still the sysadmin's problem, not the OS supplier -- the OS shouldn't simply clamp down on an automated update because there's a very good chance that things will break. (Anyone remember the MS article about how XP ServicePack2 will break a large % of apps?) How was the problem resolved -- was it a 'rollback' of the offending update? Sure this is a major cock-up but it's far from clear who's to blame on this one, nor it is really possible to assure that an MS update of any sort won't break something -- there are a lot of bad programs out there. The question that remains is: what sensible sysadmin would use automated updates in this fashion? I only have a handful of pet computers, but I have a test machine which is subjected to updates before they're rolled out to the other machines.
"the point is that MS software shouldn't just fall over, reguardless of what 3rd party software is installed "
Do not know if that is completely fair. I have software that once installed knocks my Linux servers on their butts. Same for my Windows servers, I blame that on the software developer. The thing is this only happens in the "test lab." You know that place where we test new software, hardware, and other configuration changes before installing them in a production environment. To have this happen to 80,000 machines, is in a single word - unforgivable. However I agree with the previous poster, blame the driver (EDS) not the car (MS). This is operator error not OS error. EDS should be dumped!
I find it interesting EDS is involved.
I'm no fan of Microsoft, and I love laughing at them since I personally have migrated to Linux, but I work for a (US) government agency that has a contract with EDS for millions of workstations throughout the world. The migration had been horrible, maintenance is a chore for the average person, and trouble-calls are usually fixed by the user, not the trouble-call desk.
I'm willing to bet that EDS is at fault in this situation, but does EDS even provide non-MS solutions? If I, personally, were running a company, I don't think I could seriously consider EDS for any IT solution. Their track record (from what I've seen) is horrible for cost over-runs and maintenance.
Didn't they test the upgrade first!!!
Why didn't they do a programmed upgrade to batches of computers, after they had tested the upgrade for compatibility?
No one in their right mind upgrades that many computers all at one time.
Basic tenet of change management, apart from testing on a subset, is that the change should be capable of being backed out if it fails. Even ITIL change/release management doesn't mention this fact. At least, the version I've been looking at doesn't.
If it's EDS why was there a surprise that the implementation and maintenance of the PCs-in-question was botched? Having worked on a contract that involved fixing errors in same on EDS' part - they were booted off the job - I can attest to this happening in the past at least once. While on board the contract in 1995-96, I heard about other PC Horror Stories that happened because of technicians from EDS either being incompetent or simply not being interested in what they were doing enough to leave critical setup issues unresolved. It was generally professed by the ex-contractors that it was thought noone would encounter the problems, and thus the cost-saving decision to not prevent them from happening was taken forward. In actuality this cost the clients dearly, as a result of this incomplete or incompetent work, as they had to get a more trusted firm to come in and fix it all up. So much for low bids!
Compound this effect with the "workmens' blight" that the IT sector in the UK is infested with - one aspect of this involving "I don't need to finish the job, they'll have me back and pay me more money" - and voila! Instant domino effect-in-waiting. So why is this a surprise?
To Quote . . ."If it's EDS why was there a surprise that the implementation and maintenance of the PCs-in-question was botched?"
This is exactly the kind of narrow minded idiotic statement expected from people with no current insight into the true overall performance and dedication of EDS technicians or EDS for that matter. I work for EDS and in particular the DWP project (yes, await the response now) and I have seen EDS rise and fall through one reason or another and admittedly we had, have and always will have problems - this is the very nature of I.T. - and surprisingly enough (and I know this will astound some of you) other I.T. service providers such as IBM also have their fair share of problems (honestly, no joke!!!).
I, and the other 120,000 employees of EDS are committed to providing 101% dedication to our customers and yes, we've seen EDS's name in the news through "failed" projects and "cock-ups" although these failures are shared failures with more than EDS involved - they involve everyone (including the customer) from design through to implementation and do not merely mean EDS balls'd it up again!. If we plan for months only to be told by the customer that a change if required at the last minute and our testing time can't be extended, it is inevitable things are going to go wrong at go live!. This is just one example of the pressures EDS are under.
People are always quick to judge, I've done it myself, although when you're in the know you feel compelled to hit back and attempt to put the record straight. EDS are not serving a variety of customers in 60 countries because we're incompetant - we're serving them because we DO know how to offer, plan and implement I.T. solutions - and when things go wrong we do our utmost to put them right, regardless of who caused it in the first place.
I have a feeling that the problem here lies with EDS not doing the necessary testing beforehand, otherwise I am sure Microsoft woudl want us all to know if there is a problem with one of their products so it does not bring down many more Government PCs. They should know better, test, start small (pilot) and then implement with confidence.
This was an EDS error, MS are not at fault (this time). Lib Dem Richard Allen incorrectly says elsewhere the problem was caused by attempts to upgrade 2000 to XP. This is not the case. It was an XP upgrade for a few selected terminals in the London area (already running XP) that that accidently got applied to the whole network. As the majority of terminals run 2000 the upgrade caused them to crash, the results of which are known.
I don't wish to offend the EDS employee who has already posted but as an employee of the Gov department concered I rarely hear a good word about EDS and the work they do for us.
The real problem is that just about everyone considers themselves IT expert enough to make 'informed decisions' based on 'facts' presented by persons who are commercially motivated and consider themselves IT expert enough also.
This means that wannabees, never minds, not there yet and real (multi-platform) professionals get equal voting and the latter ones are greatly outnumbered by the former.
As a result overall quality goes down the tube yet revenues for the commercially motivated increase.
Add to that that absolutely no-one (certainly high level decision makers) will really get axed, trialed and executed no matter how big the mistakes, how large the budget overruns, how empty the promises turn out to be and how huge the clean-up costs are.
Then ask yourself what possible reasons the responsible people (decision makers, managers, advisors, consultants and external commercial companies) actually have to change their behaviour and attitude as long as they're not made responsible and realisticly fully accountable for their actions, decisions and advises.
As history has shown: buying into build-in required extra work with zero liability for the ones providing it will not get you what you need. It'll get you what they told you to ask for. And that's only good for them.
The fact of business life is that they won't do what lowers their profits unless there's a real risk of having to pay a fine of some sort that's significantly larger then the probable revenues they gain from just going ahead as they see fit.
As such certainly governments are advised to require from their solution providers at least two total solutions (otherwise you can say you're dealing with a company that doesn't have a clue enough) along with the requirement that the providing party will specifiy a complete and total roll-back plan beforehand which they will be fully accountable for if the pre-arranged delivery conditions are not completely satisfied in time.
On an individual case level that'll increase the price at first but on an overall scale and in the longer run it'll significantly lower cost because it'll require responsible business behaviour and a greater demand for real professionals that can provide more then one answer (thus improving competition which will automaticly provide a lasting better value for money overall).
Get the Fact's ???
As long we don't know the facts, why bother accusing?