What is crawl budget?
Crawl budget is the number of pages a search engine is willing and able to crawl on a given site within a certain period, shaped by how fast the site responds and how much the engine values its content.
Definition
Crawl budget describes the finite attention a search engine gives to crawling a particular site. It is commonly understood as the product of two factors: crawl capacity, or how many requests a site can handle without slowing down, and crawl demand, or how much the search engine wants to crawl the site based on its size, freshness, and popularity.
For most small sites, crawl budget is not a concern because search engines can easily crawl every page. It becomes meaningful for large sites with many thousands of URLs, sites that generate many URL variations, or sites that update frequently and need new content discovered quickly.
Why it matters
If a search engine spends its limited crawling on low-value or duplicate URLs, important pages may be discovered or refreshed more slowly. On a large site, wasted crawling can delay indexing of new content and let stale versions linger in the index.
Managing crawl budget is really about directing crawler attention to the URLs that deserve it. The goal is not to force more crawling, but to ensure the crawling that happens lands on the pages you actually want indexed and kept fresh.
How to optimize it
Reduce the number of low-value URLs a crawler encounters. Consolidate duplicates with canonical tags, prune or block thin and infinite-variation pages such as endless faceted filters, fix broken links and long redirect chains, and keep your sitemap accurate so it lists only the URLs worth indexing.
Site speed and server health also matter, because a fast, reliable site raises crawl capacity. Strong internal linking helps crawlers reach important pages in fewer hops, and keeping your most valuable content close to the site structure ensures it is found and refreshed efficiently.
Frequently asked questions
Does every website need to worry about crawl budget?
No. Small and medium sites are usually crawled fully without issue. Crawl budget mainly matters for very large sites, sites with many URL variations, or those that publish and update content at high volume.
How can I see how my site is being crawled?
Search engine webmaster tools report crawl statistics such as pages crawled per day and response times, and server log files show exactly which URLs crawlers request and how often.
Keep reading
Want this machine pointed at your site?
This page is one of a swarm we generated for ourselves. We build the same for you.
Get Started - $10,000