auditing-sft-dataset

What problem does it solve?

It audits supervised fine-tuning datasets before training so you can catch schema errors, template mismatches, duplicates, leakage into eval, and sensitive-content exposure before a bad dataset reaches production.

Core Features & Use Cases

• Schema and format validation: Checks ShareGPT, Alpaca, OpenAI-style messages, or custom schemas for parse failures and malformed rows.
• Chat-template conformance: Verifies every row renders correctly under the target template and flags cross-template token leakage.
• Dataset integrity checks: Finds exact duplicates, near-duplicates, and train-vs-eval contamination using overlap-based leakage detection.
• PII and policy enforcement: Applies a user-supplied or minimum default PII policy and quarantines or flags sensitive rows with an audit trail.
• Human review support: Produces a label-quality sample for manual inspection instead of guessing at answer correctness.
• Use case: A team scraping support chats for Llama fine-tuning can audit the corpus, quarantine bad rows, and re-run the audit after remediation.

Quick Start

Audit the attached SFT training file against the held-out eval split, the target chat template, and the supplied PII policy, then return a findings report with quarantine files and remediation steps.