auditing-train-test-split

What problem does it solve?

This Skill audits train and test splits to catch hidden leakage, misleading validation, and evaluation setups that overstate real-world performance.

Core Features & Use Cases

• Checks for identifier leakage, target leakage, and preprocessing leakage before metrics are trusted.
• Validates stratification for imbalanced classification and chooses the right split strategy for grouped, panel, and time-indexed data.
• Recommends corrected splitters such as group-aware, stratified-group, time-series, walk-forward, or date-cutoff methods.
• Helps diagnose suspiciously high accuracy or AUC, especially when multiple rows belong to the same person, customer, session, or time sequence.
• Distinguishes supervised split audits from unsupervised clustering, where cluster stability is the better validation approach.

Quick Start

Ask the Skill to audit your split, identify any leakage or temporal violations, and recommend the correct validation strategy for your dataset.