noindex

What noindex actually controls

Noindex is a page-level directive that tells a search engine: fetch this page if you want, but keep it out of the results. Google honors it in exactly two places, a meta robots tag in the HTML head or an X-Robots-Tag HTTP header, and the header variant is the one auditors forget. A PDF, an image, or an API-rendered page can carry noindex invisibly at the HTTP layer while the HTML looks clean, so any serious check inspects both.

The third historical location is dead. Google no longer supports the noindex directive inside robots.txt, and any audit tool still flagging it is reading a checklist from another decade. Robots.txt and noindex answer two different questions: the robots.txt exclusion layer governs whether a crawler may fetch a URL, while noindex governs whether a fetched URL may appear in search. Googlebot must download the page to read the directive. That single mechanical fact explains half the noindex incidents we untangle: block a URL in robots.txt and Google can never see the noindex you added, so the page lingers in the index as a URL-only result, sometimes for months.

The other distinction worth being precise about is directive versus hint. Noindex is a directive: Google and Bing comply, consistently. A canonical is a hint that Google can and does override. When a page genuinely must leave the index, noindex is the tool. When two pages compete and you want one to absorb the signals of the other, noindex is the wrong tool, because it discards rather than consolidates.

How noindex behaves in 2026

Two documented changes in late 2025 rewrote assumptions that had held for years. First, rendering. Independent testing published by Tame the Bots on December 16, 2025 showed that Google now executes JavaScript on some noindexed pages, and Google updated its documentation the same day: noindex no longer means not rendered. For years, operators relied on the opposite, using noindex as a cheap way to keep Google from executing scripts or seeing client-side redirects. That assumption is gone.

Second, the JavaScript escape hatch closed. Google's documentation updates from November and December 2025 warn explicitly against shipping a noindex in the source HTML and removing it with JavaScript: if you want the page indexed, the directive must not be present in the original response. Single-page applications that inject or strip robots directives during hydration are gambling on render timing, and Google now tells you not to.

Scale-wise, the directive is common but far from universal. The HTTP Archive's 2025 Web Almanac SEO chapter, published in January 2026, measured noindex in roughly 1.4% to 9.1% of robots directives depending on which crawler the rule targets.

The most consequential shift is not about the tag you write but the one Google applies for you. Starting around May 27, 2025, a wave of sites saw their «Crawled, currently not indexed» counts climb in Search Console without any technical defect, a pattern documented by Flyhigh Media in February 2026: Google increasingly declines to index technically accessible pages on quality and uniqueness grounds. Treat that as algorithmic exclusion, a different animal from your declarative noindex. One you fix by removing a tag; the other you fix by making the page worth indexing. Confusing the two wastes weeks. And since May 2026, the stakes include AI surfaces: Google's AI search guidance states that a page must be crawlable, renderable, and free of robots or noindex blocks to be eligible for AI Overviews and AI Mode. A noindexed URL is invisible to Google's AI answers, full stop.

Noindex in a netlinking operation

For anyone buying or selling links, noindex is not a technical footnote, it is the difference between an asset and a write-off. A backlink hosted on a noindexed page passes nothing you can bank on: a URL Google refuses to index is not a stable part of the link graph, and its outbound links cannot be counted as an asset. Yet from what we see in audits, buyers routinely verify a placement once at publication and never again, while the page quietly falls out of the index three weeks later, sometimes because a plugin noindexes ageing posts, sometimes because a publisher deliberately noindexes its sponsored section after invoicing.

The operational answer is verification at three layers, on a schedule: the meta tag in rendered HTML, the X-Robots-Tag header on the live response, and the actual index status via URL Inspection, because a page can be perfectly indexable and still be algorithmically excluded. A site: query is not a verification method; it is a rumor.

This is also a transparency problem, not just a technical one. On the 50 owned French media we operate at Nautilinks, article indexability is checked at every deploy, and a placement that drops out of the index is our problem to fix, not the buyer's to discover. That level of control is only possible because the media are run in-house; when you buy through opaque intermediaries, you inherit whatever robots configuration the publisher ships that week. It is one of the reasons we built a catalogue where every medium can be inspected before you order rather than a blind marketplace: index status is a claim you should be able to check, not take on faith.

Failure modes we see in audits

The classic self-inflicted wound is the noindex plus disallow combination. The logic feels rigorous, belt and braces, and it is exactly backwards: the disallow prevents Googlebot from fetching the page, so the noindex is never read, and the URL can persist in the index stripped of its content. Deindex first with noindex alone, then, once the page is gone, add the disallow if you want to stop the crawl.

The second is the stray directive. Search Engine Journal's June 2026 coverage of the ongoing deindexing complaints put it plainly: a stray noindex, a robots rule, or a server error can pull a page without any algorithmic judgment behind it. Staging configurations that survive a deploy, theme toggles, SEO plugin defaults on archives and pagination: the causes are mundane and the detection is always the same first reflex, fetch the live HTML and headers before theorizing about algorithms. That reflex mattered doubly during the Search Console Page Indexing anomaly that ran from November 2025 to late April 2026, when the report itself misreported data and pushed operators to chase deindexing ghosts. The report is a lagging summary; the live response is the truth.

The third is signal contradiction: a noindexed page carrying a canonical pointing at a stronger URL. Google has to guess which instruction you meant, and its guess changes. Closely related is using noindex to solve duplication at all. Duplicate and near-duplicate clusters should consolidate through canonicals so the signals accumulate on the surviving URL; noindexing the duplicates burns whatever equity they held, including any backlinks pointing at them. In a link-driven operation, that is throwing away inventory.

Crawl budget and what to use instead

On December 2, 2025, Google added two explicit myths to its Search Central crawl-budget documentation: noindex is not a good way to control crawl budget, though it can indirectly free some in the long run, and 4xx pages, except 429, do not waste crawl budget. That closed a debate that should have been closed years ago. Server logs on large faceted sites show Googlebot continuing to fetch noindexed URLs long after the directive is in place, because it has to keep fetching them to confirm the directive still holds. Noindex controls indexing; it does not meaningfully control crawling.

The working decision grid is short. To keep a page out of the results while letting it serve users, noindex. To stop crawlers from fetching entire URL spaces, robots.txt. To consolidate duplicates, canonical. To remove something urgently, the Search Console removal tool backed by a noindex so it stays gone. And to keep AI bots out, none of the above reliably: standalone AI crawlers are inconsistent about robots conventions, and edge-level blocking is the only enforcement that does not depend on politeness.

Monitoring closes the loop. Crawl your own properties with header capture enabled, track the count of indexable URLs per template as a deploy-time metric, and alert on deltas: a noindex regression caught in CI costs nothing, the same regression caught in next month's traffic report costs a quarter. That discipline is cheap, it is boring, and it is worth more than most of what gets sold as technical SEO.