Following the news on Friday that around 80,000 desktop machines may have crashed in the Department of Work and Pensions (DWP) following a routine upgrade, ZDNet UK spoke with the Liberal Democrat IT spokesperson Richard Allan MP about the dangers of outsourcing and over-reliance on technology.
What are your initial thoughts on what went wrong with the DWP desktops?
What I have heard is that they were upgrading Windows 2000 to Windows XP. Initially I thought it must have been XP to a service pack because that is the problem in a lot of places.
To be able to make this kind of mistake with desktops is actually quite tough. The problem was across multiple sites. The fact that they [Microsoft] had to fly people in is an extraordinary state of affairs to get into. It's amazing that anyone could manage to cock it up so successfully.
But how could a relatively straightforward desktop upgrade cause so many problems?
There are some serious architecture problems there. Obviously they had no imaging system or anything like that; even in my office I have a small office network and it has the facility to take a PC image and bring them back to that (working) state if anything goes wrong.
It also throws a question mark over some of these massive outsourcing deals that we have got going on with DEFRA, we have got one going on with MoD. The logic of these deals is that you have one external supplier who manages all your desktops which is sold as low risk: 'they are only managing the desktop, it is just a bit of Windows and Office'. But what you can see is that you no longer have the internal staff to manage these things.
So you think the problem lies with the fact that the DWP has chosen to outsource its desktop management to EDS and Microsoft?
It is interesting it is EDS again. Their record recently has not been good. I can understand problems with server upgrades but killing desktops is gross incompetence. This worries me far more than anything else they have done.
It comes back to the nature of these contracts -– EDS are sitting there at some kind of central control centre and the idea is that its much more efficient to that they have trimmed it all down to have someone sit their and do everything remotely. They have pressed a button to try and update a few machines but have made a mistake a sent out this patch to sixty thousand out of eighty thousand -– they have all received the patch, the patch has killed the machines and there has been no easy way back to revive them.
Talkback
The Lib dem guy is not stupid!!
EDS are a bunch of fools as it is, this just furthers that knowledge.
I, as a windows expert, would NEVER EVER want to do a windows UPGRADE remotely. Id certainly do a windows fresh install remotely. But NEVER EVER an upgrade.
Windows is unstable enough you dont want to be upgrading oses remotely too!
Bad management and poor techicians are one of the causes here!
Everybody has had enough of Microsoft and other incompetent IT staff. They are costing everybody money, time and frustration. Professional organisations are already getting smarter to upgrade their Systems to Linux.
The 'Personal Computer' has no applicability
to large network environments. A PC is designed to do boot up ready to do any one of a thousand things, making it unwieldy, unsteady, slow and generally tiresome. Computers have only just become fast enough to make them bearable even for individual use.
A Govt Dept database should be on a dedicated network of hardwired machines about as big as a sony walkman with software upgrades supplied in a form similar to a Secure Digital chip. No remote reboot, you just get the new chip in the post and throw away the old one.
Essence of the argument _ less to go wrong.
The real problem is that just about everyone considers themselves IT expert enough to make 'informed decisions' based on 'facts' presented by persons who are commercially motivated and consider themselves IT expert enough also.
This means that wannabees, never minds, not there yet and real (multi-platform) professionals get equal voting and the latter ones are greatly outnumbered by the former.
As a result overall quality goes down the tube yet revenues for the commercially motivated increase.
Add to that that absolutely no-one (certainly high level decision makers) will really get axed, trialed and executed no matter how big the mistakes, how large the budget overruns, how empty the promises turn out to be and how huge the clean-up costs are.
Then ask yourself what possible reasons the responsible people (decision makers, managers, advisors, consultants and external commercial companies) actually have to change their behaviour and attitude as long as they're not made responsible and realisticly fully accountable for their actions, decisions and advises.
As history has shown: buying into build-in required extra work with zero liability for the ones providing it will not get you what you need. It'll get you what they told you to ask for. And that's only good for them.
The fact of business life is that they won't do what lowers their profits unless there's a real risk of having to pay a fine of some sort that's significantly larger then the probable revenues they gain from just going ahead as they see fit.
As such certainly governments are advised to require from their solution providers at least two total solutions (otherwise you can say you're dealing with a company that doesn't have a clue enough) along with the requirement that the providing party will specifiy a complete and total roll-back plan beforehand which they will be fully accountable for if the pre-arranged delivery conditions are not completely satisfied in time.
On an individual case level that'll increase the price at first but on an overall scale and in the longer run it'll significantly lower cost because it'll require responsible business behaviour and a greater demand for real professionals that can provide more then one answer (thus improving competition which will automaticly provide a lasting better value for money overall).
LOL....the comments here are amazing....its a terrible, terrible thing that happened to the DWP, it speaks of weak change control and poor management, it has absolutely nothing to do with Linux, Windows, Large organisations or anything else, it's called a MISTAKE, an ACCIDENT! - yeah careless, yeah dumb, imagine how the poor guy feels who pressed the submit button, get off your soapboxes for goodness sake - geez you guys obviously all collectively had never had a car wreck, never tripped over a paving stone, done something you werent supposed to....how lovely to be so angelic
I think youll find, Mr Liberal Democrat MP, that its very likely the people involved with this upgrade previously worked for the DWP, having worked with EDS I know that they dont magcially parachute hundreds of staff in, most EDS people either used to work for a client of EDS (of which there are thousands btw) or still work on there original site doing a similair job.
Yeah, blame this account and its weak change control ,but dont Tar the whole of EDS as incompetent, I have a lot of friends who work for EDS and they are very smart, very dedicated people.
The technologies are available to secure and instantly recover computers with absolute certainty. By using products such as the Valt.X Computer Security subsystem all 80,000 systems could have been recovered instantly by simply restarting the computers.
See www.valtx.com .