ClaudeBot

Three user-agents, one brand, three separate decisions

The most expensive misunderstanding about ClaudeBot is treating it as «the Claude bot». It isn't. Anthropic runs a small fleet, and on 25 February 2026 it clarified the split in its own crawler documentation, as reported by Search Engine Land. ClaudeBot collects public web content that may be used for model training. Claude-User fetches a page in real time because a user asked Claude something that requires it. Claude-SearchBot indexes content for Claude's search results. Three names, three purposes, three separate lines in your robots file.

That split is the whole subject. A publisher who added a Disallow for ClaudeBot in 2024 and considered the matter closed did not remove themselves from Claude's answers. They removed themselves from a training corpus, which is a copyright and licensing position, and left retrieval-time visibility entirely untouched. The reverse mistake is worse: a wildcard disallow aimed at «AI bots» takes out Claude-SearchBot and Claude-User too, which is the one part of the fleet that can actually surface your brand inside an answer a prospect is reading. Anyone maintaining directives aimed at AI crawlers should be able to name which of the three they are targeting and why.

Anthropic's 2026 model system cards keep describing ClaudeBot as a general-purpose crawler for training data that follows site robots.txt instructions. Worth reading that sentence precisely: it follows the file, it is not bound by it. Compliance with robots.txt is a stated policy of the operator, enforced by nobody. That is not a Claude-specific caveat, it has been true of the exclusion protocol since its inception, but it matters more when the content being collected has commercial value to the collector.

What ClaudeBot actually does on your server

Volume first, because it drives every infrastructure argument you will have with a hosting provider. Cloudflare's July 2025 comparison found ClaudeBot's share of AI-crawler traffic rising from about 6% in July 2024 to nearly 10% in July 2025, while GPTBot went from 4.7% to 11.7% and Bytespider collapsed from 14.1% to 2.4%. Through 2026 the readings are far less stable: 11.69% in April 2026 against GPTBot's 9.84%, then 9.73% in May, then roughly 19.8 to 20% in June after a reported 66% month-over-month surge that made ClaudeBot the second-largest individual AI crawler behind Googlebot, then 16.28% in July. Anyone quoting a single share figure for ClaudeBot is quoting a snapshot with a two-fold error bar.

On shape rather than volume, a 30-day server-log study of 12 production sites published on 26 April 2026 measured around 1,800 ClaudeBot hits per site per day against 4,200 for GPTBot and 980 for PerplexityBot, and found ClaudeBot crawling deeper, an average of 5.2 pages versus 3.8 for GPTBot, with a preference for documentation and API paths. Twelve sites is a directional sample, not an industry benchmark, and the authors say so. Still, the depth signal matches what we see in logs on our own properties: this crawler follows internal paths further than the average AI bot, so a site with weak internal architecture will hand it a lot of low-value pages and pay for the transfer.

Practically, ClaudeBot consumes your server capacity, not Google's crawl budget. The two are unrelated, and conflating them produces bad panic decisions. Identify it in logs by user-agent string, then segment: hits from ClaudeBot tell you about training collection, hits from Claude-User tell you a human asked Claude a question your page was retrieved for, and that second number is the one worth watching week over week.

Crawl-to-refer: a cost line, not an acquisition channel

The ratio that settles the debate is crawl-to-refer, the number of pages a bot takes for every visitor it sends back. Cloudflare's analysis put ClaudeBot at roughly 38,065 crawled pages per referral in July 2025, an improvement on the 286,930:1 measured six months earlier and still an order of magnitude away from anything a publisher would call an exchange. A later analysis of Cloudflare Radar data covering 1 January to 16 March 2026 measured 23,951:1 for ClaudeBot, against 1,276:1 for GPTBot, about 111:1 for Perplexity and about 5:1 for Google. The authors are explicit that these ratios move with the time window, the site sample and the referral attribution method, and that traffic arriving through a native app often carries no Referer header at all, which inflates every AI ratio to some degree.

Even with generous error bars, the conclusion holds: ClaudeBot is a content-acquisition cost, and arguing about it in traffic terms is a category error. Cloudflare's broader July 2025 work found 79% of AI crawling served training, 17% search and 3.2% user-action retrieval, which is the same story from the supply side. If the objective is to be named and linked inside generated answers, the levers are Claude-SearchBot access, retrievable page structure, and the volume of third-party editorial mentions a model can find when it fans a query out across sources. That last one is not a robots.txt setting, it is an off-site programme, and it is what we deliver when a client asks us to make a brand quotable inside AI answers rather than merely crawlable. The mechanics of getting picked up are covered in our entry on how models choose which sources to cite.

Writing a robots policy that matches your actual goal

For most commercial sites, the defensible position in 2026 is asymmetric: decline the training collection, keep the retrieval paths open.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

That file opts out of future training while preserving inclusion in Claude search results and real-time answers, subject to Anthropic honouring its own documented behaviour. It is a stance, and it is the one we would defend for a publisher whose content is the product. For a B2B site whose content exists to generate leads, the calculus flips: the training corpus is where a model learns that your brand is associated with your category, and cutting that off to save bandwidth is a poor trade.

The market has already moved from blanket refusal to targeting. A June 2026 snapshot of 107 prominent sites found 38 of them, 35.5%, explicitly blocking ClaudeBot, 36.4% blocking at least one Anthropic crawler and 44.9% blocking at least one major AI crawler, while only 9 sites, 8.4%, used a blanket wildcard disallow. Cloudflare reported in January 2026 that between July 2025 and January 2026, sites blocking crawlers like ClaudeBot and GPTBot were nearly seven times as numerous as sites blocking Googlebot and Bingbot. The economic evidence is genuinely unsettled: a Rutgers and Wharton publisher study found the top 30 publishers blocking AI crawlers saw a 23% decline in total traffic and 14% in human traffic, then a later revision reportedly produced a much smaller weekly decline of around 7%, and the authors warn throughout that correlation is not causation. Nobody should be quoting that study as proof in either direction.

What we see go wrong

The wildcard disallow written in a hurry is the classic. It usually appears after a bandwidth alert, it blocks the retrieval agents alongside the training crawler, and six months later somebody notices the brand has stopped appearing in AI answers and blames the content team. Second on the list is crawl-delay: it never made it into the original exclusion standard, and support across AI crawlers is inconsistent. If a bot is genuinely hurting your origin, rate-limit it at the edge where the rule is enforced, and keep robots.txt for stating intent.

Third, log trust. Any client can send a ClaudeBot user-agent string, and scrapers do exactly that to inherit a well-behaved reputation. Before you build a policy on log volume, verify a sample against Anthropic's published crawler information rather than taking the header at face value. Fourth, name confusion, which the SERP for this term reflects: assistants and community tools that borrow the Claude name, the ones circulating as Clawd or OpenClaw, are not Anthropic crawlers and blocking a string that resembles them accomplishes nothing.

The last one is structural. Your robots file governs your domain and nothing else. Every mention of your brand sitting on a third-party media property is crawled under that publisher's rules, which is why an off-site programme is more robust to any single site's blocking decision than your own pages are. Across the French editorial media we run in-house, the AI crawler posture is set once at network level and applied uniformly, so a client's mentions do not sit behind a wall somebody put up during a bandwidth incident. If you would rather inspect that before committing, the catalogue of media is public and filterable without an account, and teams who prefer someone else to sequence the whole programme can have us calibrate the campaign over several months instead.

One last operational note: Google Search Console added dedicated generative-AI performance reporting on 3 June 2026, with separate visibility views for AI Overviews, AI Mode and generative features in Discover. That data covers Google surfaces only. Nothing in Search Console will ever tell you what Claude did with your content, so ClaudeBot measurement stays a server-log and edge-analytics job.