Previous
Search engine crawlers dig up way too much
News Asking Web crawlers not to index a page does not make it inaccessible to the outside world. Google also maintains a site for Webmasters giving them several options for curtailing or turning away search crawlers.
[November 26, 2001, 14:41]
Building Custom Crawlers for Oracle Secure Enterprise Search
White Papers Oracle provides a number of crawlers out-of-the-box, and also provides an API for customers, partners and others to develop secure crawlers to access datasources not easily crawlable by the standard set of crawlers.
[February 24, 2007, 0:00]
Search payments cause a wave of concern for surfers
News Complicating matters, Overture was slated to introduce a paid-inclusion programme that combined its two new crawlers this month, but Yahoo spokeswoman Diana Lee would not say whether that will still happen now that it has the three technologies to...
[October 14, 2003, 15:15]
RIAA apologises for more mistaken warnings
News The RIAA refuses to disclose what techniques its crawlers use, but the group appears to employ companies such as MediaForce and MediaDefender. The errors represent a black eye for the RIAA's latest efforts against piracy, which rely on automated...
[May 14, 2003, 7:33]
Report criticises Google's porn filters
News But the company challenged the methodology of the study, saying that some of the sites are missing because their Webmasters employ a device called the "robots.txt" file, which is designed to limit automated Web crawlers in various ways.
[April 11, 2003, 7:56]
CrawlTrack
Downloads CrawlTrack is a free application (license GNU GPL), which allow to track search-engines crawlers and spiders visits on your website and to follow day after day your position in the main search engines and social bookmarks index.
[September 27, 2007, 8:00]
LookSmart draws on desktop power
News Last year, LookSmart bought WiseNut, an emerging technology company that uses automated crawlers to index the Web, for about $9.25m in stock. LookSmart, a US-based online directory is hoping to spin a small acquisition into a big project that will...
[March 21, 2003, 12:18]
Trend Micro Smart Protection Network - Security Made Smarter
White Papers Trend Micro leverages patent-pending technology to correlate the threat data gathered through a network of proactive email, Web, and file reputation technologies, Web Crawlers, honeypots and global threat sensors of customers, partners, and threat...
[October 2, 2009, 1:23]
mail to guard
Downloads Spammers for several years now have been using spiders/crawlers on the World Wide Web to continually search the contents of sites for the "@" character and collect every e-mail address they find for inclusion in their target address databases.
[September 28, 2004, 8:00]
Open Search Server
Downloads The crawlers go through web sites and file systems to rapidly and easily build your index. Open Search Server (OSS) is a search engine software developed under the GPL v3 open source licence. Built using the best open source technologies available...
[December 14, 2009, 5:32]
Robots.txt Editor
Downloads By means of this program you will be able to visually generate industry standard robots.txt files; identify malicious and unwanted spiders and ban them from your site; direct search engine crawlers to the appropriate pages for multilingual sites...
[August 1, 2005, 18:00]
AOL casts doubts on BT's child-porn protection
Talkback I think that a worm could spread an auto dialler to dial onto compromised computers, to dial some of these sites, in order to get onto web crawlers, and this is probably increasing BT figures. one of the other questions is is it possible to spoof...
[July 21, 2004, 12:38]
Police create online crime busting service
News It is hoped the Bradford Crimebeat scheme at www.bradfordpolice.co.uk will help catch racists, kerb crawlers and drug dealers in Bradford. The public will soon be able to report suspected criminals to the police via a secure Web site.
[May 8, 2000, 11:27]
Google blacklists BMW.de
News Cutts explained that when Google's crawlers visited a BMW page, it saw blocks of text with repeated key search words such as neuwagon, which means new car in German. Google has blacklisted BMW.de after BMW violated its guidelines by using a...
[February 6, 2006, 11:20]
Web hacks using 'evasive' techniques
News Evasive attacks can also identify the IP addresses of crawlers used by URL filtering, reputation services and search engines, and reply to these engines with legitimate content such as news. Hackers whose malicious code is hosted on websites are...
[June 4, 2007, 16:57]
Websense reveals bait for Web 2.0 cybercrooks
News Automated "active HoneyJax" bots solicit users to join networks or reply to requests, and work in a similar way to web-search crawlers, Hubbard added. Websense has revealed one of its methods for monitoring malicious web activity.
[August 8, 2007, 10:01]
FTC cracks international Web porn ring
News or waiting for automatic crawlers to do the work for them. The Federal Trade Commission announced Wednesday it has won a federal injunction against an international porn ring that cloned 25 million Web pages and "hijacked" unsuspecting visitors to...
[September 23, 1999, 9:29]
Net not as interconnected as you think, Part II
News For example, an "origination site" might have to increase its efforts to be easily found by Web crawlers. "Our experimental evidence reveals a rather more detailed and subtle picture: Significant portions of the Web cannot at all be reached from...
[May 15, 2000, 9:04]
RIAA apologises for Penn State copyright warning
News The combination of the word "Usher" and the suffix ".mp3" had triggered the RIAA's automated copyright crawlers. The Recording Industry Association of America apologised on Monday to Penn State University for sending an incorrect legal notice of...
[May 13, 2003, 8:36]
Google releases Web 2.0 security tool
News He added that Ratproxy is intended to complement active crawlers and manual proxies, as well as other passive proxies. Google has released as open source a web application assessment tool, Ratproxy, that was designed to root out potential security...
[July 11, 2008, 13:15]
