Guide

Crawl Budget: When It Matters and How to Fix It

Crawl budget only matters for some sites. If yours is one of them, wasted crawling can leave your best pages unseen. Here is how to check and what to fix.

Crawl budget is the number of URLs Google is willing and able to crawl on your site in a given period. For most sites it is not a problem. It becomes one when you have a very large site, lots of automatically generated URLs, or many pages Google knows about but has not fetched. Then wasted crawling can delay or prevent your important pages from being indexed.

How crawl budget works

Google's documentation describes two parts. Crawl capacity limit is how much Google can fetch without overloading your server. If your server is fast and returns few errors, that limit rises. If it slows or throws errors, Google backs off. Crawl demand is how much Google wants to crawl your pages, based on how popular they are and how often they change.

Put together, they set how much attention your site gets. Anything that makes Google fetch useless URLs spends that attention on the wrong pages.

Does crawl budget matter for your site?

Be honest here. Plenty of owners blame crawl budget for problems that are really about quality. A small site that is slow to rank is rarely short of crawl budget.

It is worth investigating when:

Our short answer on what crawl budget is covers the basics if you want a quicker read.

What wastes crawl budget

Crawl waste

Common ways sites burn crawling

Faceted navigation

Filter and sort combinations that create huge numbers of near identical URLs.

URL parameters

Session IDs, tracking tags and sort options that duplicate the same page.

Duplicate pages

The same content reachable at several URLs with no clear canonical.

Soft errors and 404s

Long chains of dead URLs that links and sitemaps still point to.

Redirect chains

Several hops between the old URL and the final page.

Slow responses

A struggling server tells Google to slow down.

How to check your crawl activity

Use the Crawl Stats report

In Search Console, open Settings, then Crawl stats. It shows total crawl requests, download size and average response time over time, and breaks requests down by response code, file type and purpose. Look for a high share of error codes, redirects or requests to URLs you do not care about.

Compare crawled against indexed

Open the Pages report under Indexing. A large gap between known URLs and indexed URLs, with many in Discovered, currently not indexed, can signal a crawling problem. Our guide on Discovered, currently not indexed explains what that status means and how it differs from crawled but not indexed.

Read your server logs

Server logs show every Google request, not a sample. They are the most reliable evidence of where crawling goes. Group Googlebot requests by folder and URL pattern, then ask whether those patterns deserve the attention.

Step by step fixes

1. Remove the waste at the source

The best fix is to stop creating useless URLs. Remove internal links to filter combinations you do not want indexed. Fix templates that add session or tracking parameters to links.

2. Consolidate duplicates

Use redirects for pages that no longer need to exist as separate URLs, and set clear canonical tags on near duplicates. Canonicals are a hint, not a command, so make sure your internal links and sitemap agree with them.

3. Clean up errors and redirect chains

Fix internal links pointing at 404 pages and redirects. Point each link straight at the final URL. Our guide on fixing 404 errors at scale shows how to handle large numbers without breaking pages that still earn traffic.

4. Block what should never be crawled

For URLs with no search value, such as internal search results or endless calendar pages, a robots.txt disallow can stop the fetching. Be careful. Blocking a URL does not remove it from the index, and a blocked page cannot pass on a noindex instruction because Google cannot read it.

5. Keep your sitemap honest

List only canonical, indexable URLs that return a 200 status. A sitemap full of redirects and errors teaches Google to trust it less.

6. Speed up the server

Faster, more reliable responses raise the crawl capacity limit. Check your hosting under load, not only on a quiet day.

How long until crawling improves

Google recrawls on its own schedule, so changes show gradually. Watch the Crawl stats report and the Pages report over several weeks. If the share of crawl spent on junk falls and key pages move into the index, the fix is working. If nothing moves, the real cause may be content quality rather than crawling.

When to call for help

Call for help when your site has too many URLs to review by hand, when logs show Google ignoring important sections, or when you have blocked or redirected things and indexing got worse. Crawl problems are easy to misread, and a wrong robots.txt rule can hide a whole site.

Our technical SEO fixes work starts with the crawl evidence, then fixes the cause. Tell us about the site through a free SEO triage request.

FAQ

Questions owners ask us

Does every website need to worry about crawl budget?

No. Google says crawl budget is mainly a concern for very large sites with many thousands of pages that change often, or for sites with a large number of pages Google has found but not crawled. Most small and mid sized sites are better served by fixing quality and structure.

Can I increase my crawl budget?

You influence it indirectly. Google describes crawl budget as a mix of crawl capacity, set by how well your server responds, and crawl demand, set by how popular and fresh your pages are. A fast, error free server and valuable pages raise both. Cutting low value URLs at the source helps the crawling you already get go further.

Does blocking pages in robots.txt save crawl budget?

It stops Google fetching those URLs, but Google says it does not automatically move that crawling to your other pages. It also does not remove pages from the index, and it hides them from Google entirely. Use it carefully. Our free SEO triage can check whether a block is helping or hiding something important.

Get started

Tell us where it hurts.

Your site, what changed and when. We reply with a first read on the cause.