- →A Spider crawl is a snapshot of what one user agent fetched from one IP at one moment. It is evidence, not truth: pair every crawl-based indexation claim with Search Console data before you act on it.
- →The free version stops at 500 URLs per crawl, which silently truncates any real site. Screaming Frog's official FAQ prices the licence at £199 per user per year, with a May 2025 currency adjustment to $279 / €245.
- →For host vetting, the three columns that decide a placement are outbound link count per template, indexability of the article template, and how deep the blog sits in the internal link graph. Volume of published posts decides nothing.
- →Version 24.0 (19 May 2026) added Model Context Protocol support, so crawls, exports and analysis can be driven from an AI assistant. That moves technical audit from a manual sort-and-filter job toward an agent-operated one.
- →Crawling cannot tell you whether a page is cited in AI Overviews. Search Console's dedicated generative-AI reports, added 3 June 2026, are the only first-party measurement, and Log File Analyser 7.0 (29 April 2026) covers the server-side half by separating genuine bots from spoofed ones.
- →Post-placement verification through list mode plus custom extraction catches link rot, silent nofollow injection and redirect chains months before a quarterly backlink tool refresh would.
What the Spider actually is, beyond « a crawler »
Screaming Frog SEO Spider is a desktop application that requests URLs the way a bot would, stores every response header, every rendered byte and every link it finds, then hands you the result as a sortable table. That last part is the whole product. Every other technical SEO tool decides for you what deserves attention; the Spider decides nothing and lets you filter 40,000 rows on the column you actually care about. For a consultant who already knows what a broken canonical looks like, that neutrality is worth more than any issue score.
The property that matters most is also the one people forget: the Spider knows nothing it has not fetched itself. It is not an index, not a database of the web, not a backlink source. When it reports that a page returns 200 and is indexable, that is a statement about one HTTP request made from your IP, with your user agent, at that minute. It says nothing about whether Google has the page, has kept it, or has decided it deserves a position. Confusing the two produces most of the false positives in audit reports we get asked to review.
It also runs on your machine, which sets a hard physical ceiling. RAM mode is fast and dies on large sites; database storage mode is slower and handles millions of URLs, but the write path is only as quick as your disk. On a laptop with an SSD, a 200,000 URL crawl in database mode is routine. On a spinning disk it is an afternoon. Screaming Frog's official FAQ lists the licence at £199 per user per year, with a currency adjustment recorded in May 2025 to $279 / €245, and the free version capped at 500 URLs per crawl. That 500 URL cap is not a trial inconvenience, it is a truncation: crawl a 3,000 page magazine without a licence and you will confidently analyse the wrong sixth of the site.
How it works in 2026: modes, integrations, recent releases
Three operating modes cover almost everything. Spider mode starts from a seed URL and follows links, which is how you map architecture and find orphans. List mode takes a set of URLs you paste in and fetches only those, which is how you verify placements, check a migration map, or test a redirect list. SERP mode ingests titles and descriptions for pixel-width checks, which matters less than it used to now that Google rewrites titles routinely.
Rendering is the fork in the road. The default HTML-only crawl shows you what a bot sees before JavaScript executes; switching to JavaScript rendering fires headless Chrome and shows the post-render DOM. On sites where the internal link graph is built client-side, the difference between the two crawls is the audit. Run both, diff the internal link counts, and you have your answer in ten minutes rather than in a week of theorising.
The API integrations are what turn a crawl into a decision tool. Connect Search Console and every crawled URL carries its clicks and impressions; connect Ahrefs or Semrush and it carries its referring domain count. A page that is crawlable, internally linked, has backlinks and still records zero impressions is a very different problem from a page nobody links to. Without the joins you are guessing which one you have.
Recent releases have pushed the tool in two directions worth noting. Version 23.0, shipped 20 October 2025, revised the Google-related file-size checks after Google clarified that the relevant limit was 2 MB rather than 15 MB, and added configurable « HTML Document Over 2MB » and « Resource Over 2MB » issues. Version 24.0, shipped 19 May 2026, added Model Context Protocol support so that crawls, analysis and exports can be driven from Claude, LM Studio and other assistants, plus Auto Compare Crawls, crawl-change email notifications and uncrawlable-link detection. The MCP addition is the structurally interesting one: it moves technical SEO from a human sorting columns toward an agent querying a crawl, and the audit questions you can ask cheaply change accordingly.
The companion product deserves a mention because it answers the question the Spider cannot. Log File Analyser 7.0, released 29 April 2026, improved analysis of search-engine and AI-bot crawling, including verification of genuine versus spoofed crawlers. Your crawl tells you a page is reachable. Your logs tell you whether Googlebot actually requested it, and whether the AI crawler in your access log was really who it claimed to be.
Where it earns its keep in a netlinking operation
Vetting a host before you pay is the first use case, and it is the one most buyers skip because a metrics screenshot feels sufficient. Crawl the candidate domain and look at three things. First, outbound external links per article template: a site whose posts carry eight commercial outbound links each is a placement farm regardless of its Domain Rating. Second, indexability of the article template itself, because a noindex directive or a canonical pointing back to the category kills the link before it exists. Third, crawl depth of the blog section: if articles sit five clicks from the homepage behind a paginated archive nobody links to, the page you are buying receives almost nothing to pass on.
This is the check that separates a catalogue you can trust from one you cannot. On the French editorial media we operate in-house, the same crawl runs as a monthly hygiene pass, which is why the list of media we run ourselves carries real crawl-depth and outbound-link figures rather than a vendor-supplied DR. When you buy through an intermediary you generally get the metric and not the crawl, so run your own.
Post-placement verification is the second use case and the one that pays for the licence outright. Keep a CSV of every URL you have a link on, run it through list mode monthly with custom extraction pulling the rel attribute and href of any anchor pointing at your domain. You will catch three failure modes long before a backlink index refresh would: the article quietly moved and now serves a 404, the publisher added a site-wide nofollow, or your link now sits behind a redirect chain that leaks through two hops. The 2025 survey of 518 SEO professionals reported an average acceptable price of $508.95 for a high-quality backlink, so a placement that silently goes nofollow is not a rounding error, and comparing that number against what a placement actually costs when there is no middleman in the chain is a five-minute exercise worth doing annually.
The third use case is your own site. Link equity arriving from external placements only converts if the internal graph moves it toward pages that can rank. Crawl your own domain, sort by inlinks, and check that the pages you are actively building links to are the pages your own internal linking favours. In practice they often are not: the money page collects a handful of internal links while an old blog post nobody monetises collects hundreds. That is a fixable problem you find in one crawl, and fixing it costs nothing, unlike adding another wave of placements from the open catalogue.
What we see go wrong with crawl data
Treating crawlability as indexation is the error that generates the most wasted work. A crawl reports « indexable » when nothing in the response forbids indexing. Google's decision to keep or drop a URL depends on quality, duplication and demand, none of which the Spider can observe. Any audit line that says « these 400 pages are deindexed » based on a crawl alone is a hypothesis, not a finding, and we have seen enough of those turn out to be a broken build or a stale export to insist on a Search Console check first.
Crawling with the default user agent against a host with bot protection is the second. You get 403s across the board, you conclude the site is broken, and the site is fine. Set the user agent to Googlebot when you legitimately need parity, throttle to a few URLs per second, and respect robots.txt unless you have a deliberate reason to ignore it and the site owner's permission.
The third is expecting the Spider to answer questions about backlinks or about AI visibility. It has no backlink index; that is what Ahrefs, Semrush and Majestic exist for. And no amount of crawling reveals whether your page is cited in an AI Overview. Ahrefs' study updated in 2026, covering 300,000 keywords and comparing December 2023 with December 2025, found AI Overviews associated with a 58% lower click-through rate for the top organic result, up from 34.5% in their April 2025 analysis. If that is the stake, crawl data is the wrong instrument: Search Console added dedicated generative-AI performance reports on 3 June 2026, and those plus server logs are the measurement. The crawl only tells you whether the page a generative engine might cite is fetchable and structured, which is necessary and nowhere near sufficient.
Tactical notes for a working SEO
Save configurations per job type rather than reconfiguring each time. A host-vetting config with JavaScript rendering off, external links stored, a 5,000 URL limit and the Ahrefs API connected answers a placement question in four minutes. A full-audit config is a different file. Switching between them beats a single config you keep half-editing.
Use Auto Compare Crawls on the properties that matter to you. A weekly scheduled crawl of your own site, compared automatically against last week's, surfaces the template change nobody told you about, the canonical that flipped after a plugin update, the 40 internal links a redesign quietly dropped. Crawl-change email notifications from the 24.0 release make this passive rather than a calendar reminder you ignore.
Learn custom extraction properly, because it is where the tool stops being generic. XPath against a publisher's article template pulls author names, publication dates, sponsored disclosure labels and outbound link counts across a hundred candidate hosts in a single pass. That is the difference between qualifying placements by eye at ten per hour and qualifying them by data at a thousand.
Finally, keep the crawl and the log file on the same desk. Google's spam-update cadence in 2026 was fast: the March update completed in roughly a day, and the June update ran 49 hours according to Google's official status record, so the window between a manipulative pattern being visible and being acted on is short. Knowing which of your pages Googlebot actually fetches, and whether the AI crawlers hitting you are genuine, is a server-side question your crawler was never built to answer.
Nautilinks operates an owned network of editorial media. In-house written articles, transparency disclosures respected, anchor mix calibrated.
Frequently asked questions
Database storage mode or RAM mode for a large crawl?
Database mode for anything past roughly 100,000 URLs, and for any crawl you want to reopen later. RAM mode is faster on small sites but the crawl dies with the application and large crawls hit your memory ceiling mid-run. The real constraint in database mode is disk speed, so an SSD is not optional. Set the storage directory somewhere with genuine free space, since a multi-million URL crawl with rendered HTML stored will consume tens of gigabytes.
Can Screaming Frog tell me if a backlink I bought is still passing equity?
It can tell you almost everything that matters. Run the placement URL through list mode with custom extraction pulling the anchor's rel attribute and href. That confirms the page returns 200, is indexable, the link is still present, is not nofollow or sponsored, and points at your live URL rather than through a redirect. What it cannot tell you is whether Google counts the link, which no tool can. Monthly runs catch link rot and silent nofollow injection early.
Is the MCP support in version 24.0 actually useful or is it a checkbox feature?
Useful, with a caveat. Driving crawls and exports from an AI assistant removes the sort-and-filter labour that eats most audit time, and it makes ad hoc questions cheap enough to ask. The caveat is that an assistant summarising a crawl will happily assert an indexation conclusion the data does not support. Treat agent output as a first pass over the table, then verify the specific rows yourself before writing anything into a client deliverable.
How do I use a crawl to qualify a host before buying a placement?
Crawl 2,000 URLs of the candidate with external links stored. Check outbound external links per article, indexability of the article template, crawl depth of the blog section, and the ratio of published posts that are actually linked from anywhere other than a paginated archive. A host with clean depth, few commercial outbound links per post and an indexable template is worth paying for. Publication volume and Domain Rating tell you nothing on their own.
Does the 2 MB file-size issue introduced in version 23.0 matter in practice?
It matters more than the old 15 MB threshold suggested, because Google clarified that 2 MB is the relevant limit, and the tool made both the HTML document and resource checks configurable in the October 2025 release. On editorial sites it is rarely triggered by article HTML; it fires on bloated JavaScript bundles and uncompressed images. Treat a hit as a signal to look at what the template is loading, not as a ranking issue in itself.
Why do I need Log File Analyser if the Spider already crawls everything?
Because your crawl is a hypothesis about what bots can reach, and logs are the record of what they did reach. Version 7.0, released 29 April 2026, added verification of genuine versus spoofed search-engine and AI bots, which is the only reliable way to know whether the AI crawler in your access log was authentic. If crawl budget or generative-engine access is the question, the answer lives in the logs, not in the crawl.
Test your knowledge
Quiz: Screaming Frog
1/3What does an « indexable » status in a Screaming Frog crawl actually establish?