Please purchase this course, or sign in if you’re already enrolled, to access the course content.
Lesson overview
This practical turns the warning in Lesson 4.4 into a script: filter every corpus with the same rules, and write down what you filtered. It finds boilerplate text and chrome entities in each competitor dataset, proposes what to remove and keeps a record of every decision.
The script works at two levels. At the text level it finds sentences and lines repeated across a large share of one site’s pages — newsletter boxes, customer stories, partner panels, cookie and footer text — and removes them before extraction so they never become entities. At the entity level it flags button text and bare numbers for removal and lists entities that appear on a large share of pages for a person to review, because a focused site’s real core topics are frequent too. On the course’s frozen competitor corpora, one site’s extracted content turned out to be more than half repeated marketing blocks.
The lesson walks through scanning all four corpora, reading the two review files, applying the decisions, re-extracting the cleaned text with the Entity Extractor and re-running the gap matrix. The review files stay with the audit as the record of what was filtered, so the comparison can be explained and repeated.
The lesson explains how to set the thresholds for your own corpora rather than accepting the defaults, using the same tune-then-trust approach as the chrome filter in Lesson 2.3: start with a threshold, read what it would remove, and adjust until it catches the furniture without catching the subject.
It also shows how to recognise a corpus that needs re-crawling rather than filtering — when the main-content selector captured whole page-builder sections — and how to document the filtering in the audit so a client can see exactly what was compared. The same rules apply to your own site, so the comparison stays like for like.
This practical follows Lesson 4.4 in Module 4 of The AI-powered Entity Audit. It needs the crawl exports for each corpus and runs locally at no cost.
In this lesson you’ll cover: