42-near-duplicates

To find near-duplicate pages in a Screaming Frog crawl export, run the analysis command on your directory. It uses MD5, MinHash, and embeddings to detect duplicates and generate a report.

The best way to detect semantic similarity for content cannibalization is using an embeddings-based pipeline. It evaluates titles and bodies to distinguish canonical pages from distinct content.

MinHash estimates document similarity to find near-duplicates, whereas MD5 only detects exact duplicates. Combining both in a three-layer approach provides robust analysis across large site crawls.

Yes, you can analyze large-site crawls without embeddings because they are optional. The pipeline uses MD5 and MinHash to surface exact and near duplicates without requiring embedding generation.

To differentiate canonical pages from related but distinct content, use component-aware analysis. It evaluates title, H1, meta, and body differences to classify clusters and guide canonicalization decisions.