Crawl budget

What crawl budget actually is

Google never publishes a figure called crawl budget. What exists is the intersection of two things its crawling documentation does describe: how much load your server absorbs without degrading, and how much Googlebot wants your URLs. Everything else in the crawl budget literature is inference from server logs. That distinction is operational, not academic, because a good half of the tactics sold as crawl budget optimisation act on the capacity side when the actual constraint sits on the demand side, or the exact reverse.

The vocabulary gets muddled too. Exploration, rendering and indexation are three separate queues with three separate verdicts. Googlebot fetching a URL tells you nothing about whether the page will be rendered, and rendering tells you nothing about whether it will be indexed. A page can be crawled every week for two years and never enter the index, which is not a budget problem at all: it is a quality or duplication verdict. When someone tells you a page is missing because Google ran out of crawl budget, ask to see the log lines before you accept the diagnosis.

If you want the concept laid out in three minutes before the operational part:

The mental model that survives contact with real sites: crawl budget is a rate, not a stock. You do not spend it down and hit zero on the 28th. Googlebot arrives at a certain frequency, spreads its fetches across the URL inventory it knows about, and adjusts both numbers continuously. Optimising means changing the distribution. Raising the total is rarely something you control.

How Google allocates it: the capacity side

The capacity limit is Google protecting your server from itself. Response times climb, 5xx and 429 responses appear, connection errors accumulate, and the crawl rate drops within hours. It recovers slowly, which is the asymmetry that hurts: a bad deployment on a Friday evening can suppress crawling for a fortnight on a site that was perfectly healthy before.

The technical levers are unglamorous and they work. Google updated its crawling infrastructure documentation on 20 November 2025 with explicit guidance on HTTP caching, ETag and Last-Modified headers, HTTP/1.1 versus HTTP/2, compression and crawl-rate management. Serving proper validators so that unchanged pages return 304 instead of a full HTML payload is the single cheapest capacity win most large sites have never implemented. A site with 400,000 URLs and no validators forces a complete transfer on every re-crawl, and Googlebot pays for that in fetches it will not spend elsewhere.

Two operational details worth writing into your runbook. First, the manual crawl rate limiter is gone from Search Console: the only remaining emergency brake is returning 503 or 429 deliberately and briefly, and doing it for days will get URLs dropped. Second, Google moved its crawler IP-range JSON files on 31 March 2026, from the old /search/apis/ipranges/ path to /crawling/ipranges/, because the ranges cover more than Search crawlers now. Any WAF allowlist or automated Googlebot verification script still pointing at the old endpoint is a crawl outage waiting to happen, and it will look exactly like a mysterious capacity collapse in your reporting.

The demand side, where netlinking acts

Crawl demand is the half nobody can buy with better hosting. Google decides how much it wants your URLs based on popularity, perceived freshness, and the inventory it believes exists behind your architecture. A URL that nothing links to, internally or externally, sits at the bottom of the queue no matter how fast your server answers.

This is where the two disciplines meet. A page buried eight clicks deep with a single link from a paginated archive gets crawled on a schedule measured in months. Move it into a hub, give it three contextual links from pages Googlebot already visits daily, and the refresh interval collapses. Internal architecture is the lever most teams own outright and use least.

External links act on the same variable from outside. A page that starts accumulating referring domains gets rediscovered faster, and the host doing the linking matters more than its raw metrics: a site that is itself crawled several times a day passes discovery quickly, one that Googlebot visits monthly does not. That is the practical reason we look at crawl frequency, not just authority scores, when we pick which hosts a campaign actually runs on. Across the French editorial media we operate in-house, the pattern is consistent: a target page linked from a frequently crawled host is rediscovered in days, from a dormant one in weeks. The 2025 Editorial.Link survey of 518 SEO professionals, where 48.6 % rated digital PR the most effective link-building tactic, is measuring the same mechanism from the acquisition angle.

A quick visual recap of the components before we get into scale:

Large sites, ecommerce facets and JavaScript

Here is the stance the top-ranking articles on this topic keep hedging on: crawl budget is not a universal concern, and treating it as one wastes audit hours. Google's own crawl budget documentation frames it as a concern for very large sites and for sites that generate many URLs quickly. A 600-page corporate site does not have a crawl budget issue. It has a content quality issue, a duplication issue, or an internal linking issue, and calling it crawl budget just moves the diagnosis somewhere comfortable.

Above that scale the problem becomes real and it is almost always combinatorial. Faceted navigation on an ecommerce catalogue turns 20,000 products into millions of crawlable filter combinations, each one a URL Googlebot can find and fetch. Session parameters, sort orders, tracking suffixes and calendar pagination do the same thing on other site types. The inventory Google perceives stops matching the inventory that has any commercial reason to exist.

The three controls do different jobs and confusing them is the classic error. A canonical annotation on duplicate variants consolidates signals but does not stop the fetch: Google still has to crawl the URL to read the tag. A noindex directive removes the page from the index and eventually reduces its crawl frequency, but again requires the crawl. Only a robots.txt disallow actually prevents the fetch, and it does so at the price of blinding Google to everything else on that URL. On a facet explosion, disallow is usually correct. On a duplicate product page you want consolidated, it is the worst available choice.

JavaScript adds a second queue. Pages that need rendering wait for a rendering pass that runs on its own schedule, so a client-side rendered catalogue effectively doubles the cost of every URL. Server-side rendering the parts of the page that carry links and content is not a performance nicety at that scale, it is what keeps discovery working.

Measuring it without guessing

Start with the Crawl Stats report in Search Console. Total requests, average response time and bytes downloaded give you the trend line, but the value is in the breakdowns: by response code, by file type, by Googlebot type, and above all by purpose. That last one splits fetches between discovery and refresh. A site pushing new content weekly that shows 90 % refresh has a discovery problem. A stable site showing mostly discovery is being fed URLs it should not be generating.

Google's own walkthrough of the report, worth ten minutes if you have never used it seriously:

Then go to the logs, because Crawl Stats aggregates and aggregation hides the waste. There is no official formula, but there is arithmetic that answers the question people are really asking. Take thirty days of server logs, keep verified Googlebot hits, and divide the number of requests on indexable HTML URLs by the number of indexable URLs in your sitemap. Ten thousand indexable URLs receiving 60,000 verified Googlebot HTML fetches a month gives an average refresh interval of five days. Then look at the distribution rather than the mean: if the top 5 % of URLs absorb half the fetches while a third of your money pages were not crawled at all in thirty days, you have your remediation list, and it is an internal linking and sitemap hygiene list, not a server list.

Tooling is cheap at this level. Screaming Frog's Log File Analyser is free up to 1,000 log lines and one project, and £99 per user per year for unlimited events subject to storage, with bot verification and crawl-frequency views built in (official pricing page, 2026). A proper log analysis pass costs an afternoon and settles arguments that surveys of Search Console screenshots never will.

One 2026 addition to the method: segment AI crawlers separately. Cloudflare measured overall crawler traffic up 18 % between May 2024 and May 2025, with Googlebot up 96 % and GPTBot up 305 %, and reported a June 2025 crawl-to-referral ratio of roughly 14 requests per referral for Google against 1,700:1 for OpenAI and 73,000:1 for Anthropic. Those fetches consume the same server capacity that determines Googlebot's crawl rate. The 2025 Web Almanac, published 15 January 2026, found GPTBot directives in about 4.5 % of robots.txt files overall and 20.9 % on the top 1,000 sites, so the policy question is now mainstream: a blanket block protects infrastructure and reduces exposure in AI search, a blanket allow does the opposite. Decide it deliberately rather than by inaction.

What goes wrong in practice

The most expensive mistake we see in audits is robots.txt used as a cleanup tool. Blocked URLs still appear in the index as bare URLs when something links to them, redirects on blocked paths are never read so the equity never moves, and canonical tags on blocked duplicates are invisible. The file that governs crawler access deserves more care than it usually gets: the 2025 Web Almanac found 84.9 % of desktop and mobile sites returning HTTP 200 for robots.txt, with 404 on 13.3 % of desktop sites and timeouts on about 1 %. A robots.txt that times out is treated far less kindly than one that returns a clean 200 with nothing in it.

Next in frequency: sitemaps stuffed with non-canonical, redirected or noindexed URLs, which is an explicit instruction to Googlebot to spend fetches on pages you do not want. Then redirect chains, where every hop is a separate request. Then soft 404s, which are worse than real 404s because Google keeps re-checking them. None of these will kill a small site, and all of them compound past a few hundred thousand URLs.

Two beliefs to retire. Crawl budget is not a ranking factor: nothing ranks better because it was crawled more often, it simply gets its updates reflected faster, which matters for news, pricing and seasonal inventory and almost nothing else. And llms.txt is not a crawl or indexation control for Google. Google clarified on 15 June 2026 that the file is not used by Google Search and has no positive or negative effect on visibility or rankings. Keep it for other systems if you like, but do not put it in a crawl budget remediation plan.