42-near-duplicates

Identify near-duplicate and semantically similar web pages from Screaming Frog exports.

Updated Aug 23, 2026

Search Tags:#embeddings #seo #semantic-similarity #near-duplicates #minhash #content-similarity

Ready to run

npx skills add https://github.com/chapter42/SEO-Skills-42 --skill 42-near-duplicates

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: 42-near-duplicates
Source: https://github.com/chapter42/SEO-Skills-42/tree/main/skills/42-near-duplicates
Command: npx skills add https://github.com/chapter42/SEO-Skills-42 --skill 42-near-duplicates

SYSTEM DOCUMENTATION & REQUIREMENTS

## What problem does it solve? Detects near-duplicate and semantically similar pages to prevent content cannibalization across a site, combining multiple signals for robust conclusions.

## Core Features & Use Cases

Three-layer approach: Exact duplicates via MD5, near duplicates via MinHash, and semantic similarity via embeddings to surface content cannibalization.
Component-aware analysis: Evaluates title, H1, meta, and body differences to distinguish canonical pages from related but distinct content.
Workflow-ready outputs: Produces a NEAR-DUPLICATES report and cluster details to guide canonicalization, merging, or differentiation.

### Quick Start Analyze a Screaming Frog export directory with the /42:near-duplicates command to generate a NEAR-DUPLICATES.md report.

Frequently Asked Questions about 42-near-duplicates

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I find near-duplicate pages in a Screaming Frog crawl export?▼

To find near-duplicate pages in a Screaming Frog crawl export, run the analysis command on your directory. It uses MD5, MinHash, and embeddings to detect duplicates and generate a report.

What is the best way to detect semantic similarity for content cannibalization?▼

The best way to detect semantic similarity for content cannibalization is using an embeddings-based pipeline. It evaluates titles and bodies to distinguish canonical pages from distinct content.

How does MinHash compare to exact MD5 hashing for identifying near-duplicates?▼

MinHash estimates document similarity to find near-duplicates, whereas MD5 only detects exact duplicates. Combining both in a three-layer approach provides robust analysis across large site crawls.

Can I analyze large-site crawls to find content cannibalization without embeddings?▼

Yes, you can analyze large-site crawls without embeddings because they are optional. The pipeline uses MD5 and MinHash to surface exact and near duplicates without requiring embedding generation.

How do I differentiate canonical pages from related but distinct content?▼

To differentiate canonical pages from related but distinct content, use component-aware analysis. It evaluates title, H1, meta, and body differences to classify clusters and guide canonicalization decisions.