Browse Definitions :
Definition

spider

A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web." Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read.

The spider for the AltaVista search engine and its Web site is called Scooter . Scooter adheres to the rules of politeness for Web spiders that are specified in the Standard for Robot Exclusion (SRE). It asks each server which files should be excluded from being indexed. It does not (or can not) go through firewall . And it uses a special algorithm for waiting between successive server requests so that it doesn't affect response time for other users.

This was last updated in April 2005

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

-ADS BY GOOGLE

File Extensions and File Formats

SearchCompliance

  • regulatory compliance

    Regulatory compliance is an organization's adherence to laws, regulations, guidelines and specifications relevant to its business...

  • privacy compliance

    Privacy compliance is a company's accordance with established personal information protection guidelines, specifications or ...

  • data governance policy

    A data governance policy is a documented set of guidelines for ensuring that an organization's data and information assets are ...

SearchSecurity

SearchHealthIT

  • telemedicine (telehealth)

    Telemedicine is the remote delivery of healthcare services, such as health assessments or consultations, over the ...

  • Project Nightingale

    Project Nightingale is a controversial partnership between Google and Ascension, the second largest health system in the United ...

  • medical practice management (MPM) software

    Medical practice management (MPM) software is a collection of computerized services used by healthcare professionals and ...

SearchDisasterRecovery

  • disaster recovery (DR) test

    A disaster recovery test (DR test) is the examination of each step in a disaster recovery plan as outlined in an organization's ...

  • crisis management plan (CMP)

    A crisis management plan (CMP) is a document that outlines the processes an organization will use to respond to a critical ...

  • pandemic plan

    A pandemic plan is a documented strategy for business continuity in the event of a widespread outbreak of a dangerous infectious ...

SearchStorage

  • megabytes per second (MBps)

    Megabytes per second (MBps) is a unit of measurement for data transfer speed to and from a computer storage device.

  • zettabyte

    A zettabyte is a unit of measurement used by technology professionals and the general public to describe a computer or other ...

  • hybrid flash array

    A hybrid flash array is a solid-state storage system that contains a mix of flash memory drives and hard disk drives.

Close