Data mining, the ability to find unexpected patterns in accumulated data, was born during a lunch break.
At a customer conference in the early 1990s, an executive at British department store chain Marks & Spencer was explaining his database woes to Rakesh Agrawal, an information retrieval specialist at IBM. The store was collecting all sorts of data but didn't know what to do with it.
So Agrawal and his team began devising algorithms for asking open-ended queries, eventually authoring a 1993 paper that would become required reading in data-mining science. The report has been cited in more than 650 other studies, making it one of the most widely cited papers of its kind.
"We were not even sure we should send it, because we thought people might think it was too simple-minded," said Agrawal, who holds the prised internal title IBM Fellow.
The anecdote illustrates why IBM invests more than $5bn annually on technological research. With more than 3,000 people in its dedicated research division and nearly two-thirds of its 300,000 employees overall working in some sort of technical capacity, IBM has a scientific backbone that allows it to create -- and lead -- new or developing markets.
Few institutions can rival its scope or depth. Five IBM employees have won Nobel Prizes for, among other achievements, the discovery of electron tunnelling and the invention of a microscope that captures images of individual atoms. Add to that seven National Medals of Technology, five National Medals of Science and four A.M. Turing Awards. In each of the last 10 years, IBM has received more patents than any other company or individual in the world.
IBM creations include the programming language Fortran, the magnetic disk drive, superconductivity and the Data Encryption Standard -- not to mention the Schoolmaster in 1928, the first public address system.
Big Blue's historic achievements have grown out of its culture as a quasi-university, complete with its own system of recognition. The company acknowledges every employee's third patent by elevating him or her to a new plateau. Invitations to become a Master Inventor, Distinguished Engineer or the pinnacle, an IBM Fellow, are coveted.
It is a natural progression for many new recruits from universities, which receive millions of dollars in research funding annually from IBM and serve as a pipeline to future employment both in the United States and abroad. In China, IBM's Beijing labs receive applications from about 1,800 PhD or Master's candidates each year. It hires a dozen.
"Companies that have a day-to-day involvement have a higher probability of landing students," said Jim Plummer, dean of the School of Engineering at Stanford University, adding that recruitment is one of the big benefits for IBM in these programmes. Hundreds of Stanford PhD graduates have joined the company.
Yet one of the most important reasons behind IBM's success in luring the best and the brightest lies on the business side of its operations, far from the laboratory and other academic settings. Big Blue has intentionally blurred the line where research ends and product development begins, understanding that such versatility appeals to technologists at all levels and inspires better work.
Making it happen
Unlike some of the large corporate labs of previous generations, IBM has devised methods to better capitalise on inventions from its labs and to ensure that research projects target what customers want. Along the way, inventors and researchers take more ownership of their projects, being able to follow them through the commercial process.
"They reined in the labs about 10 years ago and made it clear to people that they had to have a strong partnership with people in the product development groups," said Randal Bryant, dean of the School of Computer Science at Carnegie Mellon University. "That 'throwing it over the wall' approach never works well."
As a result, a number of researchers leave their divisions to shepherd their technologies through the development process every year.
Nelson Mattos exemplifies that process. Before coming to IBM in 1992, the Brazilian native was a professor at the University of Kaiserslautern in Germany. Now, with the honorary title of Distinguished Engineer, he is director of the Information Integration software group while continuing to advise IBM interns on PhD projects.
Others have taken similar paths. Anant Jhingran served as director of Computer Science at the IBM Almaden Research Centre and the senior manager for e-commerce and data management at the company's Watson Research Centre before jumping to product development a few years ago.
Today he is working on ways to combine database and Web searches. "The windows between research and development are fairly thin," he said.
In the On Demand Innovation Services programme, for example, researchers from facilities in Silicon Valley, New York and six other locations are assigned to work on thorny aspects within ongoing contracts. Likewise, the Information Integration Leadership Board is made up of corporate customers that advise IBM on features in demand for upcoming database products.
Product designers are often required to watch as customers try beta versions, said Laura Haas, an IBM Distinguished Engineer and development manager of DB2 Information Integrator. Recently, she served as a beta tester for a project created by one of her subordinates.
"I crashed the system," she recalled. The developer cringed, she said.
The commercialisation process isn't perfect. IBM invented the relational database, but Oracle became the first company to successfully take the concept to market. Attempts to unseat Microsoft's dominance in the PC industry with technologies like OS/2 and MicroChannel flopped. And RISC processors, which IBM has promoted heavily, sell in far fewer units than Intel-type chips do.
Still, "they tend to do pretty well," said Steve O'Grady, an analyst at consulting firm RedMonk. "What is different about IBM, as opposed to Bell Labs or PARC, is that they do have a lot of feet on the street in the real world. It will be interesting to see what they do with their search stuff. How do you manage vast quantities of information?"
Talkback
"We were not even sure we should send it, because we thought people might think it was too simple-minded," said Agrawal, who holds the prised internal title IBM Fellow.