What problem does it solve?
It audits supervised fine-tuning datasets before training so you can catch schema errors, template mismatches, duplicates, leakage into eval, and sensitive-content exposure before a bad dataset reaches production.
Core Features & Use Cases
- Schema and format validation: Checks ShareGPT, Alpaca, OpenAI-style messages, or custom schemas for parse failures and malformed rows.
- Chat-template conformance: Verifies every row renders correctly under the target template and flags cross-template token leakage.
- Dataset integrity checks: Finds exact duplicates, near-duplicates, and train-vs-eval contamination using overlap-based leakage detection.
- PII and policy enforcement: Applies a user-supplied or minimum default PII policy and quarantines or flags sensitive rows with an audit trail.
- Human review support: Produces a label-quality sample for manual inspection instead of guessing at answer correctness.
- Use case: A team scraping support chats for Llama fine-tuning can audit the corpus, quarantine bad rows, and re-run the audit after remediation.
Quick Start
Audit the attached SFT training file against the held-out eval split, the target chat template, and the supplied PII policy, then return a findings report with quarantine files and remediation steps.
