What is a crawl budget

Dictionary  |  22.10.2016

What is a crawl budget

Crawl budget is a volume of URLs that a crawler can scan in a certain period of time. Its amount is most commonly stated in a quantity of URLs scanned by a specific robot (crawler) in one day. It depends on several factors. The most important are content quality, website load speed and internal linking structure.

How to identify a web crawl budget

To identify a crawl budget of specific crawlers (robots) of search engines, an access server log analysis is used. Access log is a file on a server that records all the requests that are processed by a server. The output data are returned as:

  • User-agent (It is used to identify requests of a robot on a server)
  • IP
  • URL of a request
  • Date and time of a request
  • ...and many others

It enables SEO specialists or anyone else to closely analyze information about requests performed by search engine crawlers.

Crawl budget illustration

Crawl budget optimization

The search engines assign a crawl budget primarily based on its authority (link portfolio) and a volume of unique/quality content that they are able to obtain. In the matter of web crawler passage, a crawl waste has to be considered. Crawl waste is comments and queries that go on non-existent websites or the ones we don't want to index. These are most common problems that occur, regarding log analysis:

  • URL with error response
  • Non-indexable websites
  • Website with “thin content”

Další články

You haven’t decided yet?

That’s ok, you can try us easily for free.