skills.rest
sovr610
sovr610COMMUNITY
Explore all skills by this author →
WebsiteGitHub

Training Data Dedup + Leakage Guard

Deduplicate training corpora and detect holdout leakage with MinHash+LSH.

Updated Feb 28, 2026
Search Tags:#audit#deduplication#training-data#leakage#minhash#lsh#holdout
One-click install
npx skills add https://github.com/sovr610/refffiy --skill training-data-dedup-leakage-guard
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Training Data Dedup + Leakage Guard
Source: https://github.com/sovr610/refffiy/tree/main/brain-ai-dev/skills/training-data-dedup-leakage
Command: npx skills add https://github.com/sovr610/refffiy --skill training-data-dedup-leakage-guard

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This skill solves the critical challenge of deduplicating training data and preventing leakage of holdout/test content into training corpora, ensuring honest evaluation and cleaner training sets.

Core Features & Use Cases

  • Multi-stage dedup: canonicalization, exact dedup, fuzzy dedup via MinHash+LSH, and leakage guard against holdout sets.
  • Audit-ready: generates a reproducible audit trail and JSON report detailing duplicates removed and leakage detected.
  • Use Case: sanitize a large text corpus by removing exact and near-duplicate samples and flagging potential leakage before model training.

Quick Start

Run the dedup-leakage pipeline on your training corpus to remove exact and near-duplicate samples and flag potential leakage against holdout sets.

Frequently Asked Questions about Training Data Dedup + Leakage Guard

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate training data and prevent holdout leakage before model training?▼

Training data deduplication and leakage prevention involve canonicalization, exact dedup, MinHash+LSH fuzzy dedup, and holdout leakage guarding. This produces sanitized corpora and audit-ready JSON reports.

How does MinHash and LSH fuzzy dedup work on large text corpora?▼

MinHash and LSH fuzzy dedup identifies near-duplicate samples within large text corpora by generating hash signatures and grouping similar documents. This catches non-exact overlaps standard deduplication misses.

Can I generate an audit trail for duplicates removed and leakage detected in my training set?▼

Yes, you can generate an audit trail for duplicates removed and leakage detected. The pipeline outputs a reproducible JSON report detailing exact and fuzzy duplicates removed alongside flagged holdout leakage.

Does this data quality pipeline work for large-scale ML model training corpora?▼

Yes, this data quality pipeline works for large-scale ML model training corpora. It sanitizes large text collections by removing exact and near-duplicate samples while flagging potential leakage against evaluation holdouts.

What is the best way to detect test data leakage in an evaluation holdout set?▼

The best way to detect test data leakage is applying a leakage guard against holdout sets. It flags potential leakage of holdout or test content into training corpora to ensure honest model evaluation.

Related Skills

Explore More Data & Analytics Skills→
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
henrycashe26henrycashe26

ml-training-monitor

Diagnose machine learning training runs by interpreting loss curves and validation metrics.

Community
Advanced
Sheldon-92Sheldon-92
3

synthetic-data

Enforce contamination checks, deduplication, and quality filtering on synthetic datasets.

Community
Advanced
gar-aigar-ai
1

mithril-dedup-agent

Deduplicate large ML training datasets using MinHash and LSH clustering.

Official
Advanced