skills.rest
Sheldon-92
Sheldon-92COMMUNITY
Explore all skills by this author →
GitHub

synthetic-data

Enforce contamination checks, deduplication, and quality filtering on synthetic datasets.

3|Updated Sep 27, 2025
Search Tags:#deduplication#data-quality#synthetic-data#data-curation#pretraining#contamination-detection
One-click install
npx skills add https://github.com/Sheldon-92/TAD --skill synthetic-data-sheldon-92
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: synthetic-data
Source: https://github.com/Sheldon-92/TAD/tree/main/.agents/skills/synthetic-data
Command: npx skills add https://github.com/Sheldon-92/TAD --skill synthetic-data-sheldon-92

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill bundles a set of governance and automation rules to improve synthetic data quality for AI fine-tuning, integrating contamination detection, deduplication, and quality-filtering pipelines to avoid memorization and data leakage.

Core Features & Use Cases

  • Cross-cutting data hygiene rules from Self-Instruct, Evol-Instruct, and contamination literature to audit and cleanse synthetic data.
  • Deduplication and decontamination pipelines including exact, near-duplicate, and semantic dedup to reduce memorization.
  • Guided data curation workflows with references to concrete rules (quality-filtering, deduplication, contamination) to build trustworthy fine-tune datasets.

Quick Start

Feed your dataset to the synthetic-data pack and run the built-in validation to enforce decontamination before scoring and execute near-duplicate filtering.

Frequently Asked Questions about synthetic-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I decontaminate synthetic data before fine-tuning?▼

Decontaminate synthetic data by running automated benchmark contamination checks before scoring. This process enforces decontamination rules to detect and filter leaked evaluation samples, preventing model memorization and data leakage during AI fine-tuning.

What is the best way to deduplicate self-instruction datasets?▼

Deduplicate self-instruction datasets using a combination of exact, near-duplicate, and semantic dedup. This Skill applies MinHash/LSH-based dedup and ROUGE-L filtering to remove redundant samples and improve data quality for fine-tuning.

How does semantic deduplication work for AI training data?▼

Semantic deduplication removes training pairs with high semantic similarity. This process applies configurable pipelines with explicit rules for ROUGE-L filtering and MinHash/LSH-based dedup to reduce memorization and ensure cross-cutting data hygiene.

Can I configure the quality filtering rules for alignment prompts?▼

Yes, you can configure quality filtering for alignment prompts using modular references. The process provides explicit, configurable pipelines referencing quality-filtering-rules.md to build trustworthy fine-tune datasets.

When should I run contamination detection on my dataset?▼

Run contamination detection before scoring to ensure cross-cutting data hygiene. This process automates contamination auditing for self-instruction data and alignment prompts to prevent benchmark leakage before model fine-tuning.

Related Skills

Explore More Data & Analytics Skills→
RachasumanthRachasumanth

data-annotation

Create, format, and validate instruction-tuning and preference datasets for AI model fine-tuning.

Community
Advanced
pmcdowallpmcdowall

fine-tuning-with-trl

Align large language models via reinforcement learning and preference-based fine-tuning workflows.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

dataset-engineering

Create and process AI training datasets with deduplication, synthesis, and formatting.

Community
Advanced
gar-aigar-ai
1

mithril-dedup-agent

Deduplicate large ML training datasets using MinHash and LSH clustering.

Official
Advanced