Scooter

Scooter is the Web "crawler" for the AltaVista public search engine. AltaVista once claimed that it was "the fastest known Web crawler in existence".

Scooter adheres to the rules of politeness for Web spiders that are specified in the Standard for Robot Exclusion (SRE). It asks each server which files should be excluded from being indexed. It does not (or can not) go through firewall . And it uses a special algorithm for waiting between successive server requests so that it doesn't affect response time for other users.

In addition to AltaVista's free public search service, its parent company, Digital Equipment Corporation (DEC) sells AltaVista for individual use (downloaded to your computer, it can search everything on your hard disk) and for enterprise intranet .