Few data-mining companies are willing to discuss real-world examples of how the craft has boosted sales or customers. But Usama Fayyad, a former Microsoft executive, who left the company to create Kirkland, Washington-based DigiMine, said he used data mining to help revamp Microsoft's MSNBC.com Web site and boost readership. Fayyad found that a 22 percent slice of MSNBC readers had nearly identical online behaviour, clicking on exactly the same reports. But these users didn't fit into any of the company's five reader categories, which included political newshounds, sports junkies and weather buffs. Fayyad, who holds a doctorate from the University of Michigan, said his company determined that the glue holding this mysterious group together was vaguely scandalous stories similar to those in gossip tabloids. MSNBC changed its format significantly to appeal to this large group, and now the home page is required to have at least one such feature per day. The research helped turn MSNBC's Living section into the site's most popular destination, Fayyad said. "The lesson is that before data mining, they didn't know what was happening to a quarter of their database," Fayyad said. "If three or four shelves fall over in a brick-and-mortar store, the customers won't walk around them and the clerks will fix them. The equivalent is happening on the Web, but no one knows how to fix the bottlenecks." For decades, utility companies have been using data mining to predict with some accuracy when generators are likely to fail. The technique started making more inroads into the corporate world in the 1990s, catching on as a means to detect fraud in the insurance, health care and credit card industries. By finding patterns and predicting likely behavior, companies can catch people who lie on applications or are likely to engage in dangerous or illegal activities. So far, few general consumer e-tailers and content producers are fully exploiting data mining. That's partly because the practice -- involving algorithms, samplings and parallelisms -- is complicated and poorly understood. But it's starting to find its way into the mainstream. "E-commerce is the newest and hottest use," said Michael Gilman, president and chief executive of Data Mining Technologies of Bethpage, New York. "Anywhere you have historical data, you can use it to get patterns that you can't see with the human eye." One of the oldest and largest data-mining companies is the 25-year-old SAS Institute, based in Cary, North Carolina, which says it had already been working with 98 percent of Fortune 500 companies and is now targeting e-commerce. Retailers that sell products via catalogs and Web sites routinely increase their return on investment by more than 1,000 percent by using data mining, according to SAS statisticians. "A lot of catalogue companies were doing a fine business before, thank you very much," said Anne Milley, analytical strategist for SAS. "Then we came in and they were amazed. You look at who they're targeting, what they're sending and how often, and the frequency of repeat purchasers. You look at marketing mix--who is buying through catalogues, who is buying online -- and figure out what is the optimal way to contact customers." Data mining is likely to penetrate society further as the technology becomes easier to use. San Mateo, California-based Epiphany is one of several Web-based customer relationship-management companies that is deeply involved in data mining and is well known for its relatively easy-to-use tools. George John, who has a doctorate in statistics from Stanford University and is the self-declared "data mining guru" of Epiphany, said the company's controversial simplification of data mining was intentional. He considers it one of Epiphany's biggest attributes when vying for business against other data-mining companies -- which feature software that may be more sophisticated but is usually vastly more elusive to the average business. "In the first generation of data mining at Epiphany, we tried to step back and see what business users would use it for -- we knew they'd be asking lighter questions, where you wouldn't need ten PhDs forecasting profitability down to the penny," said John, an IBM veteran who began the data-mining program at Epiphany. "Every time we tried to make the [user interface] cleaner, we thought 'Now the marketers will use it.' It was just paying attention to what people wanted to do." Though it seems logical, the practice of simplifying data-mining results has its detractors. Fayyad and other experts warn that excessive simplification can skew results and lead executives to make pricing or inventory decisions based on faulty reasoning. A more fundamental controversy is also brewing as data mining moves out of academia and into the corporate world: academic statisticians take pride in their complex analyses, and many snub fellow PhDs who enter corporate environments, calling them turncoats pandering to marketers. John, the Epiphany guru, says he must constantly correct people who use the term "dumbing down" to refer to the company's colour charts and other simple statistical diagrams. He prefers to call it "deeper penetration" of data mining into the ranks of marketers and other nonstatisticians. "We profile a set of customers with nice charting, drawing pictures of what customers are like," John said, almost apologetically. "The key was admitting that was OK. It was OK if the technology behind it wouldn't get you a Nobel Prize." Take me back to Pt I/ The hunt for consumer gold For complete business coverage, see ZDNet UK's Enterprise Channel. Have your say instantly, and see what others have said. Click on the TalkBack button and go to the ZDNet News forum. Let the editors know what you think in the Mailroom. And read other letters.