skills.rest
yakeworld
yakeworldCOMMUNITY
Explore all skills by this author →
GitHub

kaggle-leakage-audit

Audits machine learning pipelines for data leakage from preprocessing steps applied before train-test splits.

6|1|Updated May 11, 2026
Search Tags:#audit#model evaluation#cross-validation#data leakage#scikit-learn#class imbalance#smote
One-click install
npx skills add https://github.com/yakeworld/Synthos --skill kaggle-leakage-audit-yakeworld
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kaggle-leakage-audit
Source: https://github.com/yakeworld/Synthos/tree/main/skills/private/research/kaggle-leakage-audit
Command: npx skills add https://github.com/yakeworld/Synthos --skill kaggle-leakage-audit-yakeworld

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, scikit-learn, imbalanced-learn, and includes scripts (resource) components.

What problem does it solve? Data leakage from preprocessing steps (imputation, scaling, SMOTE) applied globally before cross-validation splits silently inflates model metrics, leading to invalid conclusions in Kaggle-style machine learning experiments. This Skill detects and quantifies that leakage. ## Core Features & Use Cases - Leakage Variant Comparison: Benchmarks a correct baseline (all preprocessing inside CV folds) against leakage variants (ImputeLeak, SMOTELeak, SevereLeak) and reports F1 differences in a leakage kill table. - Imbalance and Complexity Analysis: Correlates class imbalance ratio and model complexity (RandomForest/XGBoost) with leakage severity, identifying global SMOTE as the primary leakage source. - Extreme Imbalance Handling: For datasets with positive class below 1%, forbids F1-only conclusions and requires precision/recall/AUC supplementary metrics. - Use Case: Given an imbalanced binary classification dataset, run the audit script to compare fold-internal preprocessing against global SMOTE variants and produce a report showing how much F1 inflation the leakage caused. ## Quick Start Audit my imbalanced Kaggle dataset for preprocessing leakage by comparing fold-internal preprocessing against global SMOTE, impute, and scale variants with RandomForest and XGBoost models.

Frequently Asked Questions about kaggle-leakage-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect data leakage from SMOTE in cross-validation?▼

Apply SMOTE only inside each cross-validation training fold, never on the full dataset before splitting. Compare F1 scores between this baseline and a variant where SMOTE runs globally before the split; the F1 inflation reveals the leakage impact.

Does global standardization before train-test split cause data leakage?▼

Global imputation and scaling before splitting technically leak information but have almost no measurable effect on F1 scores. The audit shows global SMOTE is the primary leakage source, while global standardization is nearly harmless.

Why is F1 unreliable for extremely imbalanced datasets?▼

When the positive class is below 1 percent, all models can score F1 near zero regardless of leakage, making F1-only conclusions invalid. Supplement evaluation with precision, recall, and AUC to draw meaningful conclusions.

Are complex models like XGBoost more affected by data leakage?▼

Yes, higher-complexity models such as RandomForest and XGBoost show larger F1 inflation from leakage than simpler models like logistic regression. The audit annotates RF and XGB leakage bias separately from low-complexity baselines.

When can I skip an imbalance leakage audit?▼

Balanced or uniformly distributed multiclass datasets are not affected by imbalance-related leakage, so the deep audit can be skipped. Preprocessing should still remain inside cross-validation folds as a general correctness practice.

Related Skills

Explore More Data & Analytics Skills→
AlexKapadiaAlexKapadia
6

fetch-dataset-from-kaggle

Fetch Kaggle datasets into a Pixie tool's data folder with credentials.

Community
Advanced
ihmorolihmorol
1

experiment-audit

Audit machine learning experiments for data leakage and reproducibility.

Community
Advanced
MuvonMuvon
3

ai-evals-design

Designs statistically grounded LLM evaluation methodology with paired significance testing and effect sizes.

Official
Advanced
SnakeOSnakeO
9

debug:scikit-learn

Diagnose scikit-learn debugging issues in training and testing workflows.

Community
Intermediate