Browse Definitions:
Definition

spider

A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web." Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read.

The spider for the AltaVista search engine and its Web site is called Scooter . Scooter adheres to the rules of politeness for Web spiders that are specified in the Standard for Robot Exclusion (SRE). It asks each server which files should be excluded from being indexed. It does not (or can not) go through firewall . And it uses a special algorithm for waiting between successive server requests so that it doesn't affect response time for other users.

This was last updated in April 2005

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

-ADS BY GOOGLE

File Extensions and File Formats

Powered by:

SearchCompliance

  • risk map (risk heat map)

    A risk map, also known as a risk heat map, is a data visualization tool for communicating specific risks an organization faces. A...

  • internal audit (IA)

    An internal audit (IA) is an organizational initiative to monitor and analyze its own business operations in order to determine ...

  • pure risk (absolute risk)

    Pure risk, also called absolute risk, is a category of threat that is beyond human control and has only one possible outcome if ...

SearchCloudProvider

  • cloud ecosystem

    A cloud ecosystem is a complex system of interdependent components that all work together to enable cloud services.

  • cloud services

    Cloud services is an umbrella term that may refer to a variety of resources provided over the internet, or to professional ...

  • uncloud (de-cloud)

    The term uncloud describes the action or process of removing applications and data from a cloud computing platform.

SearchSecurity

  • cyberextortion

    Cyberextortion is a crime involving an attack or threat of an attack coupled with a demand for money or some other response in ...

  • Cybercrime

    Cybercrime is any criminal activity that involves a computer, networked device or a network.

  • National Security Agency (NSA)

    The National Security Agency is the official U.S. cryptologic organization of the United States Intelligence Community under the ...

SearchHealthIT

  • Practice Fusion

    Practice Fusion Inc. is a San Francisco-based company that developed a free electronic health record (EHR) system available to ...

  • RHIA (Registered Health Information Administrator)

    An RHIA, or registered health information administrator, is a certified professional who oversees the creation and use of patient...

  • 21st Century Cures Act

    The 21st Century Cures Act is a wide-ranging healthcare bill that funds medical research and development, medical device ...

SearchDisasterRecovery

SearchStorage

  • Random Access Memory (RAM)

    Random Access Memory (RAM) is the hardware in a computing device where the operating system (OS), application programs and data ...

  • floating gate transistor (FGT)

    A floating gate transistor (FGT) is a complementary metal-oxide semiconductor (CMOS) technology capable of holding an electrical ...

  • bad block

    A bad block is an area of storage media that is no longer reliable for storing and retrieving data because it has been physically...

SearchSolidStateStorage

  • hybrid hard disk drive (HDD)

    A hybrid hard disk drive is an electromechanical spinning hard disk that contains some amount of NAND Flash memory.

Close