Google itself has experimented with distributed computing. Last year, the search leader invited 500 people to try out a new version of a toolbar that lets Windows users donate their computers' unused processing power to the Folding@home scientific research project at Stanford University. That experiment resulted in a small success when Stanford published a scientific paper based on the Folding@home calculations last year. However, the idea of using distributed computing to boost search results remains is still in its early stages. Grub has operated under the radar since 2000, when it was founded in Oklahoma by Kord Campbell. Since Grub was acquired, the company's four-person team has moved to LookSmart's San Francisco headquarters. Previous attempts to harness distributed computing models to update search listings have so far failed to produce useful results, according to search experts. For example, Infoseek has a patent on a system under which sites feed their content to a search index, in order to keep it updated and comprehensive -- but the company never did anything with it, said Danny Sullivan, editor of the industry newsletter Search Engine Watch. Sullivan said that setting up a collective effort to catalogue the Web could lead to some improvements in efficiency, but it could open the door to other problems. "If you allow anyone to just send you information, a small number of people will try to manipulate the system," he said. "Suddenly, you'll have someone that says: 'Surprise! I'm an Amazon.com affiliate and I have a million-page Web site, each page duplicates an exact page at Amazon, add me to (your) index.' When it comes to Web search, some people will abuse this, because there are monetary reasons to do it." Many pages on the Web are static and so don't need to be indexed frequently, said Sullivan. Instead, search engines need to be more intelligent about directing people specifically to information relevant to their searches. "If it were just that the system was going to harness the collective computing power of Web users, I think it would be useful. But it comes back to...spammers. When you drop the barriers completely, will the experience (for search consumers) be great? The conventional wisdom would be no." Sullivan speculated that LookSmart might use the Grub system to start a "trusted feed" service for inclusion into its WiseNut index. Marketers could send updated Web pages to the index to refresh it for a fee -- or what's known as paid inclusion. Search engines, including Inktomi and Fast's AlltheWeb, already use such a service to keep indexes of product-related sites and catalogues fresh, and to augment revenue. Pulling in participants
So far, only about 130 volunteers are participating in the test to donate their computer's processing power to crawl the Web. They do so by downloading software to their PC. The company's success will hinge on the number of people it signs up to donate computer resources to the cause. As part of the project, Grub promotes the benefits of "local searching," in which Webmasters can index their own sites and submit changed pages to the Grub directory, a process that can help save on network resources. LookSmart also plans to introduce a Web application programming interface (API), with which Webmasters will be able to query documents contained in the registry. Charles King, research director for the Sageza Group, a Mountain View, California-based information technology analysis firm, said that the real challenge that LookSmart faces is in recruiting devoted volunteers. King said that projects such as Seti@Home, a distributed computing search for extraterrestrial life, have an "inbuilt geek factor" that draws Web surfers to donate their computer resources to the cause. Similarly, Intel hosts a project to research cures for cancer, luring people who have been touched by the disease. Though he doubted whether Web indexing would be a big attractor, he said that it could be a good application for distributed computing. "The Web is growing at such a phenomenal rate that charting an index of it is an ongoing process," said King. "There are so many pages added on a daily basis, that a snapshot today will be inaccurate tomorrow. But this is the kind of thing that will be successful only as long as they can inspire interest and keep it."
For a round-up of the latest tech business coverage, see the Business News Section.
Let the editors know what you think in the Mailroom.
