Browse Definitions :
Definition

crawl depth

Contributor(s): Stan Gibilisco

Crawl depth is the extent to which a search engine indexes pages within a website. Most sites contain multiple pages, which in turn can contain subpages. The pages and subpages grow deeper in a manner similar to the way folders and subfolders (or directories and subdirectories) grow deeper in computer storage.

In general, the further down in the Web site hierarchy a particular page appears, the smaller the chance that it will appear with a high rank in a search engine results page (SERP). A Web site's home page has a crawl depth of 0 by default. Pages in the same site that are linked directly (with one click) from within the home page have a crawl depth of 1; pages that are linked directly from within crawl-depth-1 pages have a crawl depth of 2, and so on.

A crawler -- also known as a spider or bot -- is a program that visits websites and reads their pages and other information in order to create entries for a search engine index.

This was last updated in October 2012

Join the conversation

2 comments

Send me notifications when other members comment.

Please create a username to comment.

Thanks for your answer. I need to know one thing. My site having multiple subdomains and hence i need to calculate the depth of the requested url in the subdomain and i need to skip crawling that subdomain if depth is not 1. I did alll but i would like know reponse.meta["depth"] will return a depth of the url in entire site But how to calculate the depth of the subdomain requested url
Cancel
thank you for this amazing topic informations
Cancel

-ADS BY GOOGLE

File Extensions and File Formats

Powered by:

SearchCompliance

  • privacy compliance

    Privacy compliance is a company's accordance with established personal information protection guidelines, specifications or ...

  • data governance policy

    A data governance policy is a documented set of guidelines for ensuring that an organization's data and information assets are ...

  • risk management

    Risk management is the process of identifying, assessing and controlling threats to an organization's capital and earnings.

SearchSecurity

  • juice jacking

    Juice jacking is a form of hacking that compromises a device's USB charging port or charging cable, allowing hackers to break ...

  • DNS attack

    A DNS attack is an exploit in which an attacker takes advantage of vulnerabilities in the domain name system (DNS).

  • Advanced Encryption Standard (AES)

    The Advanced Encryption Standard, or AES, is a symmetric block cipher chosen by the U.S. government to protect classified ...

SearchHealthIT

SearchDisasterRecovery

SearchStorage

  • M.2 SSD

    An M.2 SSD is a solid-state drive (SSD) that conforms to a computer industry specification written for internally mounted storage...

  • RAID (redundant array of independent disks)

    RAID (redundant array of independent disks) is a way of storing the same data in different places on multiple hard disks or ...

  • cache memory

    Cache memory, also called CPU memory, is high-speed static random access memory (SRAM) that a computer microprocessor can access ...

Close