* Where the Robot Exclusion Standard falls short and Robotcop fills the gap
If there’s one thing that every commercial Web site wants, it is for the search engine spiders to crawl their sites and make them findable. But sites don’t always want to have their entire contents indexed or possibly be indexed by every spider. This is the reason we have Robot Exclusion Standard (see editorial links below).
The RES is implemented as a file named robots.txt in the server’s root that specifies which spiders can go to which subdirectories. Even though the protocol is pretty simple, there are a number of online services that will generate correctly formed robots.txt files or check existing files for correctness.
Surprisingly, despite the simplicity of the standard some sites apparently still get it wrong. According to the Our Big Crawl site, which examined the robots.txt files from more than 75,000 sites, some 5% used bad style and around 2% were so badly formed that “they would not be recognized by any spider.”
The problem with the standard is that it can’t enforce compliance. This means that spiders used by spammers to harvest e-mail addresses will just ignore any robots.txt file, as will spiders used by any organization that doesn’t care about your wishes. The solution is to use Robotcop (tag: “robots.txt: It’s the Law”).
Robotcop operates by checking all requests to a site – if the requests don’t pass the tests then Robotcop doesn’t allow the request to proceed and can ban the requesting IP address from all access (the IP address is removed from the list when it sends no further requests for a configurable period).
Robotcop is written in C for the Apache 1.3 server for Red Hat and FreeBSD with plans to port the code to IIS and Apache 2.x in the future.
