- →X-Robots-Tag and meta robots share the same directive vocabulary; the only real difference is that the header rides on any response, so it is the only option for PDF, images, feeds and API output.
- →A robots.txt disallow makes the header invisible: Google has to crawl the URL to read it, so blocking the path prevents the noindex from ever being applied.
- →Since Google extended nosnippet to AI Overviews and AI Mode (Search Central, documentation updated 24 March 2026), sending nosnippet in the header removes the page as an input for those surfaces, not just its two-line SERP extract.
- →Check the raw HTTP response with curl -sIL, following every redirect hop: the noindex is often set on an intermediate 301, on the CDN edge, or only on GET and not on HEAD.
- →Before paying for a placement, verify the target URL returns no noindex and no nofollow in its headers; a page served X-Robots-Tag: nofollow kills every link on it at once, and nothing in the HTML shows it.
- →During a core update window, audit deployment headers before rewriting the link strategy: a framework or security layer shipping noindex looks exactly like an algorithmic demotion in Search Console.
What the header really controls
X-Robots-Tag is an HTTP response header, which means it travels with anything the server can send: a PDF, a JPEG, an XML feed, a CSV export, a JSON payload from a headless stack. That is its entire reason to exist. A meta robots tag lives inside the HTML head, so it can only speak for HTML documents. Every non-HTML file on the domain is deaf to it, and the header is what fills that hole.
The directive vocabulary is identical on both sides. Google’s robots meta tags specification (Search Central, documentation last updated 24 March 2026) treats the meta tag and the header as two expressions of the same rule set: noindex, nofollow, none, all, noarchive, nosnippet, max-snippet, noimageindex, notranslate, unavailable_after. On the wire it reads X-Robots-Tag: noindex, nofollow. You can scope a rule to a single crawler with a user agent token, X-Robots-Tag: googlebot: noindex, and you can send several header lines on the same response, one per crawler, which is how you keep a page out of Google while leaving it available to Bingbot or to an internal search appliance.
The dependency that catches experienced people is discovery. Google only sees the header if it fetches the URL. If the path is blocked in the crawl directives file at the root of the domain, the request never happens, the header is never read, and the URL can still surface in results from external signals alone. Disallow and noindex are not additive, they are mutually exclusive: use one or the other, never both on the same URL. This is the single most common self-inflicted indexing bug we find, and it applies exactly the same way to the meta-level version of the directive.
The directives that actually matter in 2026
Most of the directive list is legacy trivia. Three of them carry real operational weight today.
noindex is the severe one: it removes the URL from the index entirely. It is also slow to reverse, because the recovery path runs through recrawl, and a low-value file that Google fetches once a quarter will sit out of the index for a quarter after you fix the header. Treat it as a decision, not as a toggle.
nofollow in the header is the underrated one. In HTML you attach nofollow to individual links with a rel attribute. In the header there is no such granularity: X-Robots-Tag: nofollow applies to every link on the page at once, and it leaves no trace in the source code. A publisher can serve a page whose markup shows clean dofollow links while the response header neutralises all of them. Nothing in the rendered page tells you.
nosnippet changed status in the last eighteen months. Google’s documentation now states that nosnippet, including when delivered through X-Robots-Tag, applies to web search, Images, Discover, the generative answer block at the top of the results and AI Mode, and prevents the content from being used as direct input for those two AI surfaces. max-snippet:0 behaves as an equivalent, while a positive value caps how much text those features can draw on. So the header is now an AI-visibility lever, and the arbitrage is uncomfortable. Ahrefs reported in an update using December 2025 data that AI Overviews reduced position-one organic CTR by 58 %, against the 34.5 % relative decline their earlier 300,000-keyword study measured between March 2024 and March 2025. Seer Interactive, studying 3,119 search terms from June 2024 to October 2025, found that sites cited in AI Overviews got 35 % higher organic CTR than uncited ones, while still sitting below their own prior-year absolute CTR. Read together: being cited is better than being ignored, so blanket-nosnippet as a defensive reflex costs you the citation without recovering the click. Our stance is to reserve it for content you genuinely do not want reproduced, paywalled excerpts, proprietary data tables, and to leave the rest visible.
unavailable_after has a narrow but real use for expiring content, event pages and time-boxed offers. Everything else on the list, noarchive in particular, is mostly a relic of a cache feature Google no longer surfaces the same way.
Serving it, and verifying it for real
On Apache the header goes in the vhost or in .htaccess, scoped by file type. A rule wrapped in <FilesMatch "\.(pdf|docx)$"> with Header set X-Robots-Tag "noindex, nofollow" inside covers a document library without touching the HTML pages that link to it. On Nginx it is a location ~* \.(pdf|docx)$ block containing add_header X-Robots-Tag "noindex" always;, and the always keyword matters: without it the header is skipped on error and redirect responses, which are exactly the responses you were trying to control.
On WordPress the picture is messier than the plugin marketing suggests. Yoast SEO and Rank Math primarily write meta robots into the HTML head, so a PDF sitting in wp-content/uploads bypasses them entirely; the media-attachment settings in those plugins govern the WordPress attachment page, not the binary file. To cover the file itself you either add the rule at server level as above, or hook wp_headers in the theme functions file and set the header conditionally. Doing it in PHP has one advantage worth the trouble: you can log which conditions fired, and stale conditional logic is what produces the header nobody can explain six months later.
Verification is where most audits are sloppy. Viewing the page source shows you nothing, because the header is not in the document. Use curl -sIL https://example.com/page/ and read every hop: the noindex is frequently set on a 301 in the middle of the chain rather than on the final 200. Watch for stacks that emit the header on GET but not on HEAD, which makes a -I-only check lie to you; confirm with curl -sD - -o /dev/null https://example.com/page/. Browser devtools work fine for spot checks in the Network tab. At scale, the desktop crawler most technical teams already run exposes response-header directives alongside status codes and redirects, with the free edition reported as capped at 500 URLs and the paid licence in the region of £199 to £259 per year depending on the 2026 listing you consult. The URL Inspection tool in Search Console tells you what Google actually applied, which is the answer that settles arguments.
One more habit: test from outside your network and outside your own CDN account. Edge rules, WAFs and preview environments routinely inject headers that never appear on an origin request.
What a stray header costs a netlinking operation
Link acquisition only pays when the linking page is crawlable and indexable. A placement on a URL served X-Robots-Tag: noindex transfers nothing worth measuring, and a placement on a page served X-Robots-Tag: nofollow is neutralised in a way the buyer cannot see from the HTML. This is not a theoretical loss. A 2025 survey of 518 SEO professionals put the average acceptable price for a high-quality backlink at $508.95, with agencies allocating 32.1 % of SEO budgets to link building, and a 2026 survey of 500 professionals reported that 58 % had increased their link-building budgets while 14 % cut them. At those levels, a header check that takes four seconds per URL is the cheapest control in the whole workflow, and it belongs in the same review pass where you compare what a placement should reasonably cost against what is being asked.
Run the check twice: once before ordering, once about a week after the article goes live. Publishers who apply noindex to their sponsored-content directory rarely announce it, and some apply it retroactively when they clean up their own indexing. Across the French editorial media we operate in-house at Nautilinks, header state is part of the technical baseline of every property rather than a per-order verification, which is one of the structural differences between buying on a catalogue of sites the operator actually owns and buying a slot on an inventory nobody controls end to end.
What we see go wrong in audits
The staging leak is the classic. A pre-production environment ships X-Robots-Tag: noindex globally, the configuration is copied to production during a migration, and a section of the site disappears with no warning in the deployment log. Nothing in the CMS, the sitemap or the HTML gives it away.
Second: the blanket rule on an uploads directory. Someone wants one confidential PDF out of the index and writes a FilesMatch covering every PDF on the domain. Six months later the technical documentation that used to bring qualified long-tail traffic is gone, and the cause is a line in an .htaccess file nobody has opened since.
Third: using the header where a canonical belongs. Duplicate or near-duplicate URLs, faceted pages, print versions and tracking variants are consolidation problems, not exclusion problems. noindex throws away the signals the URL had accumulated; the annotation that designates the preferred version of a URL keeps them. Note that Google supports a Link rel=canonical HTTP header too, and for non-HTML files it is the only way to express it.
Fourth, and the most expensive in consulting hours: misattribution during a volatile period. Google’s status dashboard records the May 2026 core update as running from 21 May 2026 at 08:43 PDT to 2 June 2026 at 05:43 PDT, and a spam update from 24 to 26 June 2026. When traffic drops inside one of those windows, the reflex is to question link quality or content. Check the response headers first, across a real sample of templates: a CDN rule, a framework upgrade, a PDF handler or a security layer that started emitting noindex produces a curve that is indistinguishable from an algorithmic demotion in aggregate reporting, and it is fixable in an afternoon. Interpreting those aggregates has also become harder since Google’s documentation confirmed that AI Mode data counts toward Search Console Performance totals, so read the drop by page type before reading it as a verdict.
Nautilinks operates an owned network of editorial media. In-house written articles, transparency disclosures respected, anchor mix calibrated.
Frequently asked questions
When should I use X-Robots-Tag instead of a meta robots tag?
Whenever the resource is not an HTML document: PDF, images, video files, XML or JSON output, downloadable archives. Those cannot carry a meta tag at all. The header is also the better choice when you need a rule applied by pattern across thousands of URLs, since one server-level directive covers a whole file type or directory, and when the CMS gives you no reliable hook into the HTML head. For a single HTML page managed in a CMS, the meta tag is easier to audit and easier to hand over.
Why does my noindex header get ignored on some URLs?
Almost always because the URL is blocked in robots.txt. Google has to fetch the response to read the header, and a disallowed path is never fetched, so the directive is never applied while the URL can still appear from external signals. Remove the disallow, let the page be crawled, wait for the noindex to be processed, and only then reconsider blocking. Second cause: the header is set on a redirect hop rather than on the final 200 response, which a curl check without -L will miss.
Does X-Robots-Tag: nofollow work like rel=nofollow on a link?
Same intent, different scope. The rel attribute targets one link; the header applies to every link on the page with no way to exempt any of them, and it leaves nothing visible in the markup. That asymmetry matters when you buy placements: a publisher can serve clean-looking dofollow links in the HTML while the response header cancels the lot. It is also why the header is a blunt instrument for internal link sculpting, where you want per-link control.
How do I check the header quickly on a live URL?
Run curl -sIL on the URL and read every hop in the chain rather than only the final response. If your stack answers HEAD and GET differently, confirm with curl -sD - -o /dev/null, which issues a real GET and prints the headers. The Network tab in browser devtools works for one-off checks. For a full site, a desktop crawler that exposes response-header directives alongside status codes gives you the pattern, and Search Console URL Inspection confirms what Google actually applied.
Should I use nosnippet to keep my content out of AI Overviews?
Rarely. Google’s documentation confirms nosnippet blocks the content from being used as direct input for AI Overviews and AI Mode, so it works. The cost is that you also lose the citation, and Seer Interactive’s study of 3,119 terms found cited sites got 35 % higher organic CTR than uncited ones. Losing the snippet does not recover the click. Reserve it for excerpts you have a contractual or editorial reason to protect, and use max-snippet with a positive value when you only want to limit exposure.
Do Bing and other engines honour the header the same way?
Bing supports X-Robots-Tag with the core directives, noindex, nofollow, noarchive, and reads them in response headers as Google does. Support for the finer options, max-snippet in particular, is less consistently documented outside Google, and the AI-surface behaviour is entirely Google-specific. If a non-Google engine matters to your traffic mix, verify with that engine’s own webmaster tools rather than assuming parity, and prefer per-crawler header lines over one global rule.
Test your knowledge
Quiz: X-Robots-Tag
1/3A URL carries X-Robots-Tag: noindex but is also disallowed in robots.txt. What happens?