...academia, where today's most successful search companies got their start. "A big source of new ideas comes out of universities," said Geoff Yang, a venture capitalist at Redpoint Ventures, which has backed such companies as AskJeeves and TiVo.
Google and Yahoo were practically hatched in the same dorm room at Stanford University by two pairs of graduate students roughly six years apart. Lycos, a one-time search leader, came out of Carnegie Mellon University (CMU). Newer projects include Vivisimo, a clustering search tool from CMU professor Raul Valdes-Perez.
The search problems of today are different from those of five years ago. With books, scholarly papers and television programs being digitised and put online, the technology necessary to search through the material needs to be that much better. People need a way to trust the information they find and to ask more-complex questions with search tools so they can extract knowledge or ideas.
Jaime Carbonell, director of CMU's Language Technologies Institute, said his research team is perfecting a technology for personalised search that would solve some of the privacy concerns surrounding the wide-scale collection of sensitive data, such as names and query histories. CMU's project takes an auxiliary approach to software already being tested by commercial players like Yahoo and Google, which are collecting and storing search histories on their own networks.
CMU developed an add-on application that people download to a PC. It allows users to maintain and modify personal information, such as query history, preferences and favoured sites, within a search profile. A search engine would be able to query the profile, along with the user's search term, to deliver a set of tailored results each time, thereby keeping personal information off the network and on the client's desktop.
Carbonell said the technology will be ready within a year, and CMU could either offer it as open source software or license it to industry players.
CMU is also working under a government grant on a longer-term project called Javelin, focused on question-and-answer search technology. Google, MSN, Ask Jeeves and others already help people find quick answers for word definitions or encyclopaedia facts like "What is the population of Los Angeles?" But for complex queries like "What is the cheapest flight from San Francisco to London?" or "Which university has the largest computer science department?" finding answers is still like doing long division.
"This is dynamic information," Carbonell said. "You must parse the question, look for answers in multiple places and do a comparison. There are multiple steps, and we're looking at how to do it in one step and provide a trace for the user."
He said it is likely to take another four of five years to build such functionality that can scale computationally for wide consumer usage and deliver the kind of efficiencies the government and Internet users expect. The universities of Texas and Pennsylvania are also exploring different approaches to the same problem.
Stanford continues in its role as a breeding ground for search projects. Since 2003, Google has purchased...
