This practical runs the corpus crawl from Module 2 by asking Claude, through the MCP server built into Screaming Frog’s SEO Spider. The crawl, the checks and the decisions are the same as in the manual practical; what changes is that each check becomes a prompt you can save and repeat on the next audit.
What this lesson covers
- The requirements and the two connection modes
- Installing the extension in Claude Desktop
- Starting a crawl from a saved configuration
- Keeping exports out of the context window
- The spot-check and exclusions as prompts
- Where human judgement stays in the loop
Model Context Protocol is the standard way to give Claude access to a tool on your own machine, and Screaming Frog ships an official server inside the SEO Spider from version 24. The lesson sets it up step by step: the settings that switch it on, the difference between running it visibly in the normal window and headless in the background, and the extension file Claude Desktop installs. It then starts the crawl with the configuration you already tested by hand, so the selector, exclusions and user agent are identical.
The prompts are written around two warnings from Screaming Frog’s own guide. Large exports overflow a model’s context window, so the crawl data goes to a file and Claude reports only the counts that matter: pages crawled, indexable pages, pages where the main-content extraction came back empty or thin. And letting a model run scripts on your machine is a genuine security decision, so every script is shown before it runs. The spot-check compares extracted text against each page’s visible text for ten random URLs, and Claude proposes an exclusion list that you accept or reject pattern by pattern.
The lesson closes with a simple division of labour. Claude starts, watches, exports, counts and compares; you choose the selector, decide what is out of scope, open pages yourself to check its reading and sign off the corpus before any API spend.
This practical sits in Module 2, Diagnostics: Extracting the Truth About Your Site, in The AI-powered Entity Audit, beside the manual corpus practical it mirrors. It suits learners who already use Screaming Frog with a paid licence and want the crawl and its checks to be repeatable from one quarter to the next.