This lesson builds the input every later number depends on: a clean corpus of your site’s pages with the main content isolated from navigation, footers and other furniture. It covers the crawl set-up, the decisions about what to exclude and sample, and the two-minute check that prevents most bad audits.
What this lesson covers
- The URL, title, meta description, H1 and main-body columns, and why the last one is hard
- Custom extraction in Screaming Frog or any crawler
- Finding and testing a content selector across templates
- An exclusion list you can defend months later
- Sitemap seeding, link discovery and what the difference tells you
- Sampling by template and value, and spot-checking before any API call
Four of the five columns an audit needs come free from any crawler. The fifth — the main body content — is the one that decides whether the audit means anything, because without it every page looks like it is about the same few navigation items. The lesson shows how one CSS selector, set up as a custom extraction that pulls text rather than HTML, fixes that, how to find the selector by walking up the page structure in the browser inspector, and why it must be tested on blog, product and category templates separately.
The rest of the lesson is about corpus decisions that are rarely documented. On the demo site, a run that reported 76 groups of duplicate pages across 174 URLs turned out to be certificate pages, dashboards and tag pagination that never belonged in the audit — a corpus problem, not a content one. Seeding from the sitemaps found 457 URLs where link discovery found 647, and the 190-page gap was itself a finding. For large sites the lesson sets out sampling by template, weighting by value and documenting the choice, and finishes with a five-page spot-check that compares extracted text to the live page.
This lesson opens Module 2, Diagnostics: Extracting the Truth About Your Site, in The AI-powered Entity Audit, and is followed by a practical on setting up custom extraction step by step. It suits anyone about to crawl a site for content analysis, from in-house SEOs on a single site to agencies running audits for several clients.