robots.txt

A crawl gatekeeper, not an index switch

The most expensive misunderstanding about robots.txt is packed into one sentence people still say in 2026: «I blocked it in robots.txt so it won't be indexed». Wrong on both counts. The file, a plain-text document served at the root of each host, tells compliant crawlers which URL paths they may fetch. It says nothing about what a search engine may keep in its index. Google can index a disallowed URL purely from external links pointing at it, showing it in results with no snippet because it was never allowed to read the page. If your goal is deindexation, the correct tool is a noindex directive Googlebot is actually allowed to crawl. Robots.txt and noindex are not interchangeable; used together on the same URL they cancel each other out, a point we come back to below.

Formally, the Robots Exclusion Protocol was standardized by the IETF as RFC 9309. The RFC matters for one reason: it settled parsing behavior (grouping by user-agent, longest-match precedence between allow and disallow, wildcard handling) that used to differ between crawlers. What it did not settle is enforcement. Robots.txt is a convention, not an access control. Google's own crawler documentation, updated April 23, 2026, states its crawlers «always obey robots.txt rules when crawling automatically», and the reputable engines behave the same. Anything else, scrapers included, complies only if it chooses to.

The operational framing for a working SEO: robots.txt is where you spend or protect crawl capacity, express bot policy, and occasionally shoot yourself in the foot at scale. It is the only file on a site where a one-character edit can silently zero out organic acquisition.

What Google actually parses in 2026

Google's Search Central documentation was clarified during 2025 to state explicitly that any field not listed in the docs is unsupported. That leaves exactly four: user-agent, disallow, allow and sitemap. Crawl-delay has never been honored by Google (Bing still respects it), and the undocumented noindex-in-robots.txt trick is officially dead. If your file carries anything beyond those four fields for Googlebot's benefit, it is dead weight.

Reality matches this minimalism. Google ran a custom parser across 16 million robots.txt files and published the results in April 2026 (summarized by StanVentures, April 24, 2026): usage drops off almost vertically after allow, disallow and user-agent, and the wildcard user-agent * dominates by far. Most site owners write blanket rules for all bots rather than crawler-specific groups. The same study surfaced a less flattering finding: a meaningful number of «robots.txt» files are broken, including entire HTML pages served at the robots.txt URL. A parser reading an HTML document where it expects directives simply finds none, and crawls everything.

Error handling is where the asymmetry bites. A robots.txt that returns 404 is interpreted as no restrictions: Google crawls freely, which is almost always what a small site wants anyway. A robots.txt that returns 5xx is treated as a temporary sitewide disallow. A missing file is a shrug; a broken server response on that one URL freezes crawling of the whole host. The 2025 Web Almanac, as reported by Search Engine Land (April 6, 2026), shows the ecosystem slowly improving: 404 rates on robots.txt fell from 14% to 13% year over year and 5xx responses sit around 0.1%. Slow progress, but the direction is right.

The sitemap field deserves its own note: it is the one place in robots.txt where you point crawlers at your XML sitemap without requiring a Search Console property, which is why it remains the standard hand-off for engines you never registered with, Bing, Seznam and the AI retrieval bots included.

Robots.txt as an AI policy file

The structural shift of the last two years is that robots.txt stopped being a Search-only concern. Search Engine Land's 2026 overview of Web Almanac data describes it plainly as an emerging «policy document» for bot and AI governance. The traffic numbers back this up: GPTBot grew from 2.9% of desktop crawler requests in 2024 to 4.5% in 2025, and ClaudeBot from 1.9% to 3.6%, nearly doubling. Google itself migrated its crawling documentation in late 2025 to a dedicated crawling-infrastructure site, because that infrastructure now serves Search, Shopping, News, Gemini and AdSense alike.

Yet most of the web has not made a decision. Cloudflare's analysis found that by mid-2025 only around 14% of the top 10,000 domains carried any AI-specific robots.txt rules. The other 86% are making the training-versus-retrieval call by accident, through whatever their wildcard rules happen to say.

One token is chronically misunderstood: Google-Extended. Blocking it opts your content out of Gemini and Vertex AI training. It does not touch Googlebot, does not affect Search rankings, and does not remove you from AI Overviews, which are fed by standard Googlebot. Site owners who blocked Google-Extended expecting to disappear from AI answers changed nothing except their training-data status. To actually stay out of AI Overviews you would have to block Googlebot itself, which means leaving Google Search entirely. That is the trade Google engineered, and pretending otherwise is wishful reading of the docs.

Our stance for monetized editorial sites: blanket-blocking AI retrieval bots trades away citation visibility in answer engines for nothing you can invoice. Block training bots if the principle matters to you, but make retrieval access a deliberate line in the file, not a side effect of a wildcard written in 2019.

Where robots.txt matters in a netlinking operation

For a link buyer, robots.txt is a due-diligence artifact. A backlink placed on a URL that Googlebot is disallowed from fetching transmits nothing: no anchor signal, no authority, no indexation of the linking page. The StanVentures write-up of Google's 16-million-file study drew exactly this implication for outreach: a target site whose robots.txt blocks Google turns your placement budget into decoration. The check costs thirty seconds. Fetch the publisher's robots.txt, confirm the section or category where your article will live is crawlable, and confirm the site is not blocking Googlebot globally. From what we see in audits, buyers verify traffic and authority metrics religiously and skip the one file that can nullify both.

On the publisher side, robots.txt is crawl-budget plumbing. On a large media site, disallowing faceted search, internal search results and infinite parameter spaces keeps Googlebot's attention on articles rather than on URL noise. That is a real lever on sites with hundreds of thousands of URLs, and largely a placebo below ten thousand: small sites do not have a crawl budget problem, they have a content problem.

This is also one of the arguments for buying placements inside an owned network rather than through blind intermediaries. Across the 50 French editorial media we operate in-house at Nautilinks, robots.txt is versioned, deliberately minimal and checked at every deploy, because we answer for indexation of the pages we sell links on. When the publisher and the link seller are the same entity, crawlability stops being a leap of faith; you can inspect every medium in the catalog before placing an order, which is exactly the verification step most marketplaces make impossible.

Failure modes we keep seeing in audits

The classic remains the staging block that ships to production: a sitewide Disallow: / written for a preprod environment, copied along with the rest of the build. The site keeps serving 200s, analytics look normal for a while, and organic decay only becomes visible in Search Console weeks later. Any deploy pipeline that can touch robots.txt should diff it on every release.

Second: blocking CSS and JavaScript directories. Google renders pages, and a Googlebot that cannot fetch your assets evaluates a broken layout. This habit dates from an era when blocking «technical» folders felt tidy; in 2026 it only degrades rendering with zero upside.

Third: using robots.txt as a security mechanism. Disallow lines are public, so listing /admin or /export paths hands any attacker a map of exactly what you consider sensitive. Ignoring robots.txt is not illegal in itself, it is the violation of a convention; whatever legal weight exists comes from terms of service, copyright and computer-misuse law, not from the file. Anything genuinely private belongs behind authentication.

Fourth, the self-canceling combo: disallowing a URL and putting noindex on it. The noindex lives in HTML or headers that Googlebot is now forbidden to fetch, so it is never seen and the URL can linger in the index indefinitely. The same logic applies to canonical tags on blocked parameter URLs: a canonical Google cannot crawl consolidates nothing. Deindexation always requires an open door and an explicit instruction behind it.

Finally, validate the file like the production asset it is: curl it, check the HTTP status, run it through Search Console's robots.txt report, and test the specific URLs that earn your money. Google's study finding whole HTML pages served as robots.txt proves that a surprising share of the web has never done even that.