Crawl budget: when it matters and how to stop wasting it

Crawl budget is the number of URLs a search engine is willing and able to fetch from your site in a given period. It is one of the most discussed ideas in technical SEO and one of the least relevant for most sites. A 300-page brochure site, a blog with 2,000 posts or a shop with a few thousand products will almost never be limited by it. A marketplace with five million listing URLs, a news site publishing hundreds of articles a day, or a store whose filters generate billions of URL combinations very often is.

This guide explains how the budget is set, how to tell whether it is your actual problem, and which fixes change crawler behaviour rather than just looking tidy in an audit.

What sets the budget

Google describes crawl budget as the combination of two things.

• Crawl capacity limit. How hard Googlebot can hit your server without hurting it. If responses are fast and error-free, the limit rises. If response times climb or the server returns 5xx errors or 429 Too Many Requests, Googlebot backs off, sometimes sharply.
• Crawl demand. How much Google wants to crawl. Popular URLs, URLs that change often, and URLs Google has not seen before get more demand. Stale, low-value or duplicate URLs get less.

The practical consequence: you cannot ask for more budget. You can only make your server easier to crawl and make each fetched URL more worth fetching. Everything below is a variation on those two levers.

Do you actually have a crawl budget problem?

Before changing anything, answer three questions with data rather than instinct.

Question	Where to look	Points to a budget problem if…
How many URLs exist versus how many are indexed?	Search Console Page indexing report, your own database count	Large numbers sit in "Discovered – currently not indexed" and the count keeps growing
How long do new pages take to be crawled?	URL Inspection on recent pages, server logs	Important new URLs wait days or weeks for their first Googlebot hit while junk URLs are crawled daily
Where do Googlebot's requests go?	Crawl Stats report, server logs	A large share of requests hit parameter URLs, redirects, 404s or duplicates

The status "Discovered – currently not indexed" is the one most often cited as proof of a budget issue. It is weaker evidence than it sounds. Google uses it when it knows a URL exists but has not fetched it yet, and on small sites that usually means Google did not think the URL was worth fetching, which is a quality signal. If your site has 800 pages and 300 of them are discovered but not crawled, look at the content and internal linking of those 300 before blaming budget.

The strongest evidence comes from logs. If you can see that Googlebot made 40,000 requests yesterday and 31,000 of them went to sort-order variations of the same category pages, you have a budget problem and you know exactly where it is. The log file analysis guide covers how to get that breakdown.

Where crawl gets wasted

On sites that do have the problem, the waste tends to come from the same handful of patterns.

Infinite or near-infinite URL spaces

Faceted navigation is the usual culprit. A clothing store with 12 sizes, 20 colours, 15 brands, 4 price bands and 6 sort orders can produce millions of filter combinations from a few hundred real categories. Calendars that link to the next month forever, internal search result pages that are linked from somewhere, and session IDs in URLs do the same thing. The faceted navigation guide goes through which combinations deserve to be crawlable.

Redirect chains and internal links to redirects

Each hop in a redirect chain is a separate request. If your navigation still links to old URLs that redirect, every crawl of every page pays that tax. Update internal links to point to the final URL; keep redirects for external links and bookmarks.

Soft 404s and thin duplicates

Empty category pages that return 200, out-of-stock product pages with no content, and tag pages with one post each are all fetched, judged worthless and fetched again later. Return a real 404 or 410 for pages that no longer exist, and consolidate near-duplicates.

Slow responses

If your average response time to Googlebot doubles, the number of pages it fetches in the same time roughly halves, and the capacity limit drops on top of that. A site that serves cached HTML in 150 ms will be crawled very differently from one that renders every page from scratch in 2 seconds. See server response time for how to measure and fix it.

Resources you forgot about

Googlebot also fetches CSS, JavaScript, images and API endpoints used for rendering. A front end that loads 60 separate script files per page, each with a cache-busting query string that changes on every deploy, generates a lot of fetches. Bundling and stable file names help.

The fixes, roughly in order of impact

1. Stop generating junk URLs at the source. The best fix for parameter sprawl is not a robots.txt rule but templates that do not link to useless combinations in the first place. If no internal link points to ?sort=price_desc&view=list&page=1, crawlers will rarely find it.
2. Block infinite spaces in robots.txt. For URL patterns that should never be crawled, such as internal search results, sort orders and session parameters, a Disallow rule is the only tool that actually prevents fetching. Test the pattern carefully; robots.txt rules often match more than you intend.
3. Fix internal links to redirects and errors. Crawl your own site, export every internal link whose target returns 3xx or 4xx, and update the templates.
4. Return honest status codes. 404 or 410 for removed content, 200 only for pages with real content, and 503 with a Retry-After header during planned maintenance.
5. Speed up HTML responses for the URL types Googlebot fetches most. Logs tell you which templates those are.
6. Keep XML sitemaps clean. List only canonical, indexable, 200-status URLs, and keep lastmod honest so Google can prioritise genuinely changed pages.

Things that do not help much, despite their popularity: adding nofollow to internal links, setting a crawl delay (Googlebot ignores the crawl-delay directive), and pinging sitemaps repeatedly. Google retired its sitemap ping endpoint, and resubmitting an unchanged sitemap does nothing.

noindex, canonical and robots.txt are not interchangeable

This is the point where many crawl budget projects go wrong. The three tools do different things, and only one of them reduces crawling.

Tool	Stops the URL being fetched?	Stops it being indexed?	Use it for
robots.txt Disallow	Yes	No — it can be indexed from links, without content	Infinite URL spaces, internal search, sort orders
meta robots noindex	No — the page must be fetched for the tag to be read	Yes, once crawled	Pages users need but search does not, such as thank-you pages
rel=canonical	No	Usually consolidates to the target, but it is a hint	Duplicates that must stay accessible, such as tracking parameters

Noindexed pages are crawled less often over time, so noindex has a small long-term effect on crawl, but it is not a crawl budget tool. And combining Disallow with noindex on the same URL defeats the noindex, because the crawler can never read it.

Monitoring after the change

Crawl behaviour changes slowly. After a robots.txt change or a template fix, give it two to four weeks before judging. Watch three numbers:

• Total crawl requests in the Crawl Stats report. It may not go down; the goal is to redirect requests, not reduce them.
• Share of requests to URLs you care about, from logs. This is the number that should rise.
• Time from publication to first crawl for new important pages. For a news site this should fall to minutes or hours; for a large catalogue, a day or two is healthy.

Also keep an eye on response time and error rates in the same report. A deploy that adds 400 ms to every page will quietly undo the gains. A scheduled check with the DeskSEO checker records server response time on every run, which makes that kind of regression easy to spot in the score history.

A short decision guide

• Under about 10,000 URLs and content changes weekly or less: ignore crawl budget. Work on content quality, internal links and sitemaps.
• Tens of thousands of URLs that change daily, or hundreds of thousands of URLs: check logs once a quarter. Fix parameter sprawl and redirect links.
• Millions of URLs, or faceted navigation without controls: crawl budget is a core part of your technical SEO. Treat crawl share by template as a tracked metric, and review it after every navigation change.