robots.txt for AI crawlers

Three crawler families, one file

The protocol itself has not moved. The Robots Exclusion Protocol was standardised years ago, and the parsing rules are the same ones we have lived with for two decades. What changed is who sits at the other end of the request. Until roughly 2023, a user-agent token in robots.txt mapped to a search engine that indexed you and, in exchange, might send you a click. That symmetry is gone.

Three populations now read the file, and they have nothing in common beyond the syntax. Training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended ingest text into model weights. They will never send you a visitor, and the value you get back is diffuse: your positions, your brand, your phrasing end up in the model's latent knowledge. Retrieval crawlers such as OAI-SearchBot or PerplexityBot build the live index that generative answers cite, which is the only family with a direct line to a clickable source link. User-triggered agents such as ChatGPT-User or Perplexity-User fetch a page because a human asked for it in that moment, so blocking them is functionally the same as blocking a browser.

Google is the special case that trips up most audits. Google-Extended governs whether your content feeds Gemini and related generative uses. It has no effect on Googlebot, on indexing, or on ranking. AI Overviews and AI Mode are served from the ordinary Search index crawled by Googlebot, which means there is no robots.txt token that removes you from them while keeping you in the ten blue links. The available controls there are snippet-level, nosnippet, max-snippet, data-nosnippet, and they cost you the classic snippet too. Google's own May 2026 guide on optimising for generative features in Search states the position plainly: this is still SEO, not a separate discipline with its own opt-out lever. If you want to understand what that surface actually rewards, the mechanics of generative engine optimisation matter far more than any directive you write in this file.

What the 2026 crawl data actually says

The volume shift is not marginal. Cloudflare measured overall crawler traffic up 18 % between May 2024 and May 2025, with GPTBot up 305 %, Googlebot up 96 % and ChatGPT-User up 2,825 % from a small base. By July 2025, still per Cloudflare, GPTBot's share of AI crawler traffic had gone from 4.7 % to 11.7 % year over year, ClaudeBot from 6 % to roughly 10 %, while Bytespider collapsed from 14.1 % to 2.4 %. Cloudflare's December 2025 year in review put AI bots at 4.2 % of HTML requests against 4.5 % for Googlebot alone, with user-action crawling up more than fifteenfold across the year.

The number that should decide your policy is the ratio, not the volume. Cloudflare's crawl-to-click analysis found approximately 38,000 Anthropic crawls per referred visitor and approximately 194 Perplexity crawls per referred visitor in July 2025, and reported that training accounted for close to 80 % of AI crawling overall. Read that as a pricing signal. A crawler that hits you forty thousand times to send one person is buying your content at a rate you did not agree to. A crawler at two hundred to one is behaving like a search engine with poor conversion. Those are different counterparties and they deserve different lines in the file.

Publisher behaviour has not caught up with the distinction. BuzzStream's study of around 100 leading US and UK news sites, reported in early 2026 and picked up by Search Engine Journal, found 79 % blocking at least one training crawler, 71 % blocking at least one retrieval or live-search crawler, 67 % blocking PerplexityBot and 46 % blocking Google-Extended. Only 14 % blocked every analysed AI crawler and 18 % blocked none. The middle of that distribution is where the damage sits: a majority of these sites shut the door on the crawlers that generate citations while believing they were only refusing to feed model training.

Measurement finally exists to arbitrate this. Google shipped Search Generative AI performance reports in Search Console on 3 June 2026, covering AI Overviews, AI Mode and generative features in Discover, with impressions, pages, countries, devices and dates. The initial release does not break out clicks for AI features, so you can prove exposure and not conversion. That is still a large improvement over the previous state, where a robots.txt change and a traffic change were connected by nothing but conviction.

Where this lands in a netlinking operation

If you place links or brand mentions for a living, the robots.txt of the host site is now part of the deliverable, and almost nobody checks it. A contextual link on a site that blocks OAI-SearchBot and PerplexityBot still passes classic ranking signal, still counts as a referring domain, and is completely invisible to the generative layer. The page cannot be retrieved, so it cannot be quoted, so the brand mention it carries has no path into an answer engine citation. You paid for one channel and got half of it.

This is a simple pre-flight check to add to any media vetting routine: fetch /robots.txt, look for the retrieval tokens, and treat a site that blocks them as classic-SEO-only inventory. It should not disqualify the host, plenty of strong French media block aggressively, but it should change what you expect from the placement. When we assemble a plan on the catalogue of media you can browse before ordering anything, this is one of the technical attributes that separates two sites with identical metrics.

Operating your own media removes the ambiguity. Across the French editorial media Nautilinks runs in-house, robots.txt is a single governed policy rather than a per-site accident of history, which means retrieval and agent crawlers are open everywhere by default. That is a deliberate trade: we accept the training crawl on content we produce ourselves in exchange for a coherent presence in the retrieval indexes that feed generative answer panels. A publisher whose business is selling the content itself should reach the opposite conclusion, and both positions are defensible. What is not defensible is having no position and inheriting one from a copy-pasted file.

The crawl budget angle deserves a sentence, because it gets overstated. On a mid-sized editorial site, AI crawler load is a bandwidth question, not an indexing question. Cloudflare's figures put AI bots at a share of HTML requests comparable to Googlebot's, which is noticeable on a small server and irrelevant to how Google schedules its own crawl of you. If someone justifies a blanket AI block by invoking crawl budget, they are reaching for a technical alibi for a commercial decision. Say the commercial thing instead.

What we see go wrong

The most common failure is the inherited blocklist. A file circulates, someone pastes forty user-agent tokens into production, and half of those tokens are dead, misspelled, or belong to retrieval bots the business actually wants. Bytespider is a good marker: it dropped from 14.1 % to 2.4 % of AI crawler traffic between July 2024 and July 2025 per Cloudflare, yet it still headlines blocklists written in 2023. Meanwhile OAI-SearchBot, the one that feeds ChatGPT's live citations, gets caught in a wildcard nobody re-read.

The second failure is confusing declaration with enforcement. Cloudflare documented in August 2025 that Perplexity used undeclared crawler signatures capable of bypassing robots.txt directives, and subsequently extended AI Crawl Control to fingerprint and block non-compliant crawlers at the edge. The commercial answer moved in the same direction with pay-per-crawl and HTTP 402 responses. The lesson is old and keeps needing repeating: robots.txt is a request addressed to well-behaved software. Paywalled archives, member areas and expensive endpoints need authentication, rate limiting or WAF rules, not a Disallow line.

Third, precedence errors. Robots parsers match the most specific user-agent group and then ignore every other group entirely, so a site with a permissive User-agent: * block and a later named group for GPTBot gives GPTBot only what is in its own group. Crawl-delay directives that Google ignores outright, sitemap lines buried mid-file, and a staging Disallow: / shipped to production are the other three that cost real money. Verify the rendered file on the live host, never the template in the repository.

Fourth, blocking asset paths. Sites that disallow their image or media directories to slow AI crawlers also prevent legitimate retrieval of the images that generative surfaces display next to a citation. The visibility loss is quiet and nobody attributes it to the file.

A directive set that holds up

Start from the business question rather than the file. If your revenue comes from selling access to the content itself, restrict training crawlers and keep retrieval open, then enforce the training restriction at the edge because the honour system will not hold. If your revenue comes from visibility, attention or lead generation, which covers most brands and every link-selling media, open everything and spend your effort on being quotable instead.

Whatever you decide, log user-agents at the server or CDN and review the top fifty every quarter. Token names change, new agents appear monthly, and a policy written in 2025 is already describing a crawler population that no longer exists. Pair that log with the Search Console generative reports and, for the engines Google does not cover, a cross-engine citation monitor. Teams that want the whole picture on that surface usually end up working with us on measuring how often a brand actually surfaces in AI answers, because the internal tooling rarely spans more than one engine.

Two habits close the loop. Diff your robots.txt on every deploy and alert on the change, since this file breaks in silence and nothing in your analytics will point at it. And when you brief anyone on placements, ask what the host allows, because a link that no retrieval crawler can reach is a link with one channel switched off. That check belongs in the same pass as anchor calibration and velocity, which is how we run it when a campaign is calibrated over several months rather than bought in a single batch.