The arms race between search engines and traffic-hungry Web sites is headed to a new level. Battles have see-sawed for years between search engines intent on providing unbiased listings and companies seeking top placement in results. But now a controversial experiment aimed at allowing its users to filter out sites could help Google either cement or squander its lead in the competitive search market. Currently, Google's proprietary system ranks sites primarily by words listed on the page, terms used in the title, or similar factors. It also ranks a page's popularity, determined by the number -- and importance -- of sites linking to a page. For example, a page that is linked to 100 times from a reputable newspaper's Web site would rank higher than a page linked to 500 times from a porn site. Now, for the first time, any Google user may be able to help determine the popularity of Web sites ranked by its search engine. Two weeks ago, the search service began testing a Web page voting system that could eventually let Web surfers have a say in what sites rise to the top of its results. Although Google is well regarded for its highly relevant search results, experts who specialise in pushing sites into the top rankings -- a technique known as "search engine optimisation" -- say the company's success has made it a target. "Once a search engine optimiser sets their sights on a search engine, it doesn't take them long to figure out its weaknesses," said Dana Todd, co-founder of interactive agency SiteLab, who says her company uses "white magic" to get top billing for a site. "Google has always tried to be user aware and...known for its relevant results. But now it's starting to get junky. And in order to manage that, they need help from users." Google's quest for unbiased results comes as search engines evolve into one of the most sought-after marketing tools on the Net. Sites such as Yahoo!, America Online and Microsoft's MSN openly incorporate paid listings along with their normal results -- a practice that has been targeted in an investigation by federal regulators. Google also offers special placement on its search page for advertisements related to keywords, although the paid listings are clearly marked. Meanwhile, "spammers," which aim to capitalise on popular search terms by misrepresenting the content of a site -- and their more legitimate search optimisation cousins -- have become a fixture of the Internet search landscape. OneUpWeb Optimization, which helps online properties land prime real estate in search engines' rankings, has a lengthy client list that includes Symantec, Kimberly-Clark and Priceline.com, according to its Web site. In the cloak-and-dagger world of search engines, companies can use various tricks to sway the technology agents, or spiders, that search engines use to catalogue the Web. For example, some site operators try to skew search results by including hidden text or misleading keywords on pages fed to the spiders. In extreme cases, people create hundreds of thousands of "doorway pages" to serve as entry points to a Web site and drive more traffic to the page. Others will create Web pages that are elaborately interlinked to get a higher ranking on Google and other search services that list sites by link popularity. On guard
Search engine experts said that most search engines, including Google, have come up with defences against many techniques for manipulating search results. For example, rogue operators at one time tried to falsely identify a Web site's purpose or description in metatags, or keywords designed to help search engines identify relevant Web sites. "Google is really good at spam detection," said Chris Sherman, editor of SearchDay, a newsletter published by Search Engine Watch. "If anyone tries to spam in the old-fashioned way, like cramming pages with metatags, that's not going to work." But Sherman said search engines are still trying to counter new techniques. One method, called "cloaking," sets up a dummy page including lots of relevant information for keywords hidden through a special link. The cloaked page is fed to the search engine to boost a site's search ranking for specific terms such as games, sports or books. When surfers go to that link, however, they see a page that is different from the one indexed by the crawler. Marketers say cloaking and other tactics can be useful and legitimate tools in certain cases. Some site operators use a cloaked page to get a highly relevant site a top slot in search results and to keep outsiders, or spammers, from understanding how it landed there. "In the wrong hands, search engine optimisation can be the most dangerous weapons for spammers," said Jessie Stricchiola, director of online marketing for Chase Law Group, a Los Angeles-based law firm. "Search engine specialists research every day to figure out how to best promote a Web site...If pornographers see that everyone is searching for 'purple sneakers,' they're going to find a way to link their Web sites to 'purple sneakers.'" Abuse management
Signs that spam has become a greater priority for Google include a service launched several weeks ago that lets people file complaints via email of alleged spamming abuse. A note on the Web site warns that action may not be prompt, however. That service was followed two weeks ago by a new beta version of Google's Web browser toolbar, which adds smiling and frowning face icons to the usual menu of buttons. People can click on either icon to indicate whether they like a Web page. Votes tallied will not immediately influence search results on Google, which is merely evaluating the feature for now, according to a spokesman. "This is at a very early stage; it's definitely an experiment," said Google's David Krane, who added that spam control is just one of the potential uses of the voting feature. The new feature would give Google insight into the popularity of a site based on opinion, not link structure. For example, the search would call up a page that may not have many sites linking to it but is extremely popular with consumers. "This gives ordinary Web users the ability to tell a search engine what they like and don't like and we don't have that interaction now," said Danny Sullivan, editor of Search Engine Watch, a search engine industry newsletter and Web site. But as Google toys with new, nonmathematical checks, it has unleashed concerns that it will simply open itself up to new problems. "There are people that will go to great lengths to try to sway results and this gives them another method of doing that," Sullivan said. "People who like their Web sites are going to go to their own site and punch the happy-face button about a billion times, and people who want to hurt their competitors will go to their competitors' sites and punch the unhappy-face button." He compared the beta technology to that of AskJeeves' DirectHit search service, which ranks Web pages by number of clicks it receives. Some Web site operators claim they have developed technology to boost the site ranking by engineering clicks. Placing greater emphasis on human input could mark a significant turning point for Google, which until now has relied primarily on mathematical formulas and automation. Google's Krane, however, noted that the company has solicited user feedback in the past in the form of focus groups, for example. He added that the voting system takes into account abuses such as repeat clicks that attempt to "noodle" the system. Rather than using the votes to tinker with the specific rankings of particular pages or sites, he said, the feature would most likely be used to bolster the relevance of overall results. "It will most likely have more of an aggregate impact," Krane said. "We have indexed more than 1.6 billion Web pages, so it is extremely inefficient to go after individual pages." For everything Internet-related, from the latest legal and policy-related news, to domain name updates, see ZDNet UK's Internet News Section. Have your say instantly, and see what others have said. Click on the TalkBack button and go to the Telecoms forum. Let the editors know what you think in the Mailroom. And read other letters.