This practical lesson turns the corpus-building ideas from Lesson 2.1 into a step-by-step procedure you can follow on your own site. By the end you have a crawl export with the main content isolated, a written exclusion list and a corpus you have checked before spending anything on an API.
What this lesson covers
- The columns the corpus needs and where each comes from
- Selector discovery in the browser, with the demo site as the worked example
- Custom extraction set-up and the settings that matter
- Template testing, exclusions and sampling notes
- The spot-check that catches broken extractions
- Hand-off to the Entity Extractor in the next lesson
The procedure follows the order you would actually work in. You start in the browser, inspecting a real article and walking up the page structure until the highlight covers the body copy and nothing else — on the demo site that is the element with the class entry-content, which holds 16,053 characters of article while 1,411 characters of header, navigation and hero sit outside it. You then set up the crawler’s custom extraction with that selector, extracting text rather than HTML, and test it on each template the site uses before committing to a full crawl.
The second half covers the decisions that make the corpus defensible: an exclusion list of cart, account, search, feed and pagination URLs written down with the reasons, a note of any sampling, and the inlinks export that Module 5 needs. It ends with the five-page spot-check — matching extracted text to the live page, checking character counts and looking for the same top entities on unrelated pages — which the course calls the highest-value two minutes in the whole audit.
Screaming Frog is used as the worked example because it is the crawler most SEOs already have, but the lesson sets out the equivalent settings to look for in any crawler, so the same corpus can be produced with another tool or a script as long as it isolates the main content as text.
This practical continues Module 2, Diagnostics: Extracting the Truth About Your Site, in The AI-powered Entity Audit, and leads into entity extraction with the Google NLP API. It suits learners who want written, checkable instructions for their crawler set-up, and anyone handing corpus preparation to a colleague.