Training Data Dedup + Leakage Guard

What problem does it solve?

This skill solves the critical challenge of deduplicating training data and preventing leakage of holdout/test content into training corpora, ensuring honest evaluation and cleaner training sets.

Core Features & Use Cases

• Multi-stage dedup: canonicalization, exact dedup, fuzzy dedup via MinHash+LSH, and leakage guard against holdout sets.
• Audit-ready: generates a reproducible audit trail and JSON report detailing duplicates removed and leakage detected.
• Use Case: sanitize a large text corpus by removing exact and near-duplicate samples and flagging potential leakage before model training.

Quick Start

Run the dedup-leakage pipeline on your training corpus to remove exact and near-duplicate samples and flag potential leakage against holdout sets.