Free tool Technical SEO

Multi-bot robots.txt Tester

Test your robots.txt against 18 user-agents, classic search engines and AI bots, with RFC 9309 compliant rule priority. You decide who can crawl your site.

Your robots.txt

We append /robots.txt automatically if absent.

Also worth exploring

All tools
hreflang Tester Verify your hreflang tags: valid ISO codes, reciprocity between language versions, x-default, conformance with Google's recommendations. HTTP Headers SEO Checker Inspect HTTP response headers: x-robots-tag, cache-control, HSTS, link, vary, server. SEO + security annotations on impactful headers. sitemap.xml Validator Validate your sitemap.xml: structure, URL count, lastmod outliers, sitemap-index recursion. Conforms to sitemaps.org and Google. Bulk Status Code Checker Test up to 200 URLs at once: HTTP code, redirect chain, response time. Ideal for backlink profile audits.

Frequently asked questions

Which user-agents are tested?

This tool tests 18 crawlers across 4 families. Search engines (Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider), image / specialty (Googlebot-Image, Googlebot-News), generative AI (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Perplexity-User, Applebot, Applebot-Extended, Google-Extended, CCBot), and the * wildcard. The AI family evolves quickly, we adjust regularly.

How does rule priority work?

According to the Robots Exclusion protocol (RFC 9309 since 2022), the longest rule matching the path wins, regardless of order in the file. If Allow: /blog/ and Disallow: /blog/draft/ are defined, /blog/draft/2024.html is blocked because the Disallow rule is more specific. On exact length tie, Allow wins at Google.

Why test against AI bots?

Blocking GPTBot, ClaudeBot, PerplexityBot prevents your content from training these models or being cited in their answers. Conversely, allowing these bots helps your brand appear in ChatGPT, Claude, Perplexity. It's a strategic trade-off (AI visibility vs intellectual property) that deserves an explicit decision, not an inherited default.

My robots.txt is not publicly accessible, can I still test it?

Yes. Paste the content directly into the second "Paste content" tab instead of providing a URL. The test runs locally in the browser, with no transmission to a third-party server.

Beyond robots.txt, who is talking about your site in LLMs?

Nautilinks helps brands position themselves in content reused by ChatGPT, Claude and Perplexity through links and mentions on French media.

See the AI visibility service All tools