When I first began looking at 'crowdsourced' development as a testing methodology (if we can attribute it with a term that mature), I immediately questioned how different this might be from open source community-driven application development.
My immediate suspicion was that it is simply a marketing term being 'bandied' about by vendors who specialise in development disciplines such as application testing or software change management perhaps.
So while I was on the sniff around this week, my inbox received a sprinkling from TCL, the UK partner for uTest - a community of 20,000+ ‘virtual’ testers sourced from all over the world to help companies test their mobile, web, desktop and gaming applications. This arrangement, so says TCL, helps build a test proposition that is cheaper and more effective than an in-house offering.
Now here comes the difference... it's not community driven application refinement for the greater good of all that ultimately leads to free software - these testers are paid per-bug.
The company says that this approach typically saves about 30% for customers undergoing testing procedures. Positioned as an extension to an in-house QA (quality assurance) team rather than a replacement, crowdsourcing attracts and creates, according to TCL, an interconnected community of testers who seek fame and recognition within the marketplace based on the quality and quantity of the bugs they are able to find.
Citing a recent example, uTest set loose 1,100 software testers on four major search engines: Google, Yahoo, Bing and Google's Caffeine update. Google had the fewest number of bugs and the least severe bugs among the competition, while Bing amassed the most bugs yet still scored well in the accuracy of its results.
NB: there are also monthly ’Zapper’ events where testers - both pros and amateurs - are invited to come along and test a new software application.)
Arguably I suppose, any goods-producing organisation exercises a degree of quality control or testing before it goes to market. Web pages, software and mobile phone applications shouldn't really be any exception – should they?
TCL’s chairman and all round grande fromage Stewart Noakes is on the record as saying that, “Not only is software testing being revolutionised by the idea of using a global community of professional testers, but the concept has proven to deliver compelling, real world business benefits at a fraction of the costs of traditional software testing.”
“The testers are customer-rated and motivated by a pay-for-performance model. Unlike hiring additional in-house QA personnel or signing long-term outsourcing contracts, crowdsourcing enables massive software testing coverage. Bespoke virtual QA teams with appropriate skills are constructed by environment and demographics such as access device type, application rules, geographic location etc and deliver real-time responsiveness. In fact, a complete test cycle can be run in 48 hours or less,” adds Noakes.
So what do you think? There's not so much reportage out there on this subject so I thought it deserved a passing mention. Could this break the mould for testing as we know it? Could this be one of the resultant developments made more popular during the recession (due to the money saving opportunities it represents) that lives on? This story is not fully told as yet I feel.
Twitter 
Linkedin 
ZDNet UK App 
Facebook 
RSS Feeds 
Newsletters 
Talkback
In my view this is not a bad approach to this problem at all, given the fact that almost all main game software house's with the exception of independents game developers, just don't really give a toss about releasing untested buggy products.
As long as they make there profit and their financial stats are met they don't really give a toss from that point onwards, to put this into some form of perspective it would be the equivalence of some one buying a new car being given a delivery date, and on that date it turns up in a flat pack box with missing instructions & parts.
One great example of this would be to point to Codemasters recent Christmas release of a game called Operation Flashpoint Dragon Rising on multiple formats ie; PS3, Xbox, and the PC version being the worst of the three.
Weighing in at the usual price of around £35-£40 a pop per game, only to be met with a train wrecked experience with an incredible amount of bugs and crashes, and ill thought out controller foundations simply ported over from another platform type because they where to lazy to make a proper set for that given format.
Although they do release patches after the release date for the products in most cases this is now fast becoming known as DLC "Down Loadable Content", where as the purchaser's are expected to pay an additional fee to effectively have bugs and issues present on release day fixed.
This model was not possible before because such systems where never connected to the internet, so the product makers had to get it right the first time around, although smaller bugs did slip through they could be addressed later on for the desktop machines at least, and by no means where these bugs game breakers for both formats PC's & Consoles.
But in no way back then did developers rely on the net to play catch up on their shoddy workmanship as they do now, and now even worse charge additional fee's to do that work.
It has become so horrific now I seriously think its about time these software products are screened before they are allowed to go on sale, as are cars before they are allowed to go on the roads, the reason being is they have had their chance & trust for the last couple of years now and yet they still choose to do this, its time to revoke that trust.
Interesting article Adrian, and you're right, it is something that deserves more than a passing mention. When I read it, the first thing that came to mind was Donald Knuth's original offer of rewards for finding bugs in TeX and METAFONT (see <a href="http://en.wikipedia.org/wiki/Knuth_reward_check">Knuth Reward Check"</a>, and learned that he has made a similar offer for finding errors in his books. (How I would love to have earned one of those checks from Donald Knuth!)
The point is the same - motivating independent individuals to look carefully and critically for bugs and errors. I think the "independent" part of that is important - experienced developers know that they are not very good "testers" for their own software, because they don't look at it the right way - how many times have I had to track down and fix a bug that resulted from someone doing something that i simply never even thought of trying?
There is an interesting difference in philosophy between this "bounty" approach to testing, and the generally accepted method in the world of PCs today, which is to simply ship a "Beta" or "x.0" release of the software and let the user base spend the first six months debugging it for you. "Bounty hunters" want to find bugs, they are pleased when they do, whereas even when they know they are getting "pre-release" software, people whose intention is to actually use it, not specifically bug-check it, will sooner or later get irritated or frustrated if they repeatedly run into problems.
jw
@jw: "ship a "Beta" or "x.0" release of the software and let the user base spend the first six months debugging it for you."
I think if you can combine this public "I can't believe it's not beta" testing methodology with a *decent* feedback mechanism, then it can work well (though heaven forbid don't get rid of the QA departments!!)
I agree in principle, Jake - but as you say, the key is having a decent feedback mechanism and a competent QA department. With those two pieces in place, you can have an effective and mutually beneficial testing program; without them, the company is simply exploiting the users.
jw