Browse Definitions :
Definition

crawler

A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot." Crawlers are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Crawlers apparently gained the name because they crawl through a site a page at a time, following the links to other pages on the site until all pages have been read.

The crawler for the AltaVista search engine and its Web site is called Scooter. Scooter adheres to the rules of politeness for Web crawlers that are specified in the Standard for Robot Exclusion (SRE). It asks each server which files should be excluded from being indexed. It does not (or can not) go through firewalls. And it uses a special algorithm for waiting between successive server requests so that it doesn't affect response time for other users.

This was last updated in April 2005

Continue Reading About crawler

SearchCompliance
  • ISO 31000 Risk Management

    The ISO 31000 Risk Management framework is an international standard that provides businesses with guidelines and principles for ...

  • pure risk

    Pure risk refers to risks that are beyond human control and result in a loss or no loss with no possibility of financial gain.

  • risk reporting

    Risk reporting is a method of identifying risks tied to or potentially impacting an organization's business processes.

SearchSecurity
  • Pretty Good Privacy (PGP)

    Pretty Good Privacy or PGP was a popular program used to encrypt and decrypt email over the internet, as well as authenticate ...

  • email security

    Email security is the process of ensuring the availability, integrity and authenticity of email communications by protecting ...

  • Blowfish

    Blowfish is a variable-length, symmetric, 64-bit block cipher.

SearchHealthIT
SearchDisasterRecovery
  • What is risk mitigation?

    Risk mitigation is a strategy to prepare for and lessen the effects of threats faced by a business.

  • fault-tolerant

    Fault-tolerant technology is a capability of a computer system, electronic system or network to deliver uninterrupted service, ...

  • synchronous replication

    Synchronous replication is the process of copying data over a storage area network, local area network or wide area network so ...

SearchStorage
  • direct access

    In computer storage, direct access is the process of reading and writing data on a storage device by going directly to where the ...

  • kibi, mebi, gibi, tebi, pebi and exbi

    Kibi, mebi, gibi, tebi, pebi and exbi are binary prefix multipliers that, in 1998, were approved as a standard by the ...

  • holographic storage (holostorage)

    Holographic storage is computer storage that uses laser beams to store computer-generated data in three dimensions.

Close