kaggle-validation

Detect distribution shifts and select fold types for Kaggle validation.

1|Updated Jun 6, 2026

Search Tags:#validation #machine learning #data analysis #cross-validation #kaggle

Ready to run

npx skills add https://github.com/OlixIgnacious/agent-skills --skill kaggle-validation

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: kaggle-validation
Source: https://github.com/OlixIgnacious/agent-skills/tree/main/.claude/skills/kaggle-validation
Command: npx skills add https://github.com/OlixIgnacious/agent-skills --skill kaggle-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, scikit-learn, lightgbm, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a structured approach to design and lock the validation strategy for Kaggle competitions, ensuring comparability and accuracy of experiments.

Core Features & Use Cases

Adversarial Validation: Before fold selection, it checks for distribution shifts to avoid misleading folds.
Fold Type Selection: It guides through choosing the right fold type (KFold, StratifiedKFold, TimeSeriesSplit, GroupKFold) based on data characteristics.
Lock Fold Indices Early: Ensures consistency across experiments by saving fold assignments.
OOF Prediction Framework: A template for out-of-fold predictions to support ensembling.
CV–LB Correlation Verification: After submissions, it verifies the correlation between CV and public LB scores for early detection of validation issues.
Validation Checklist: Offers a comprehensive checklist to ensure a robust validation strategy.

Quick Start

Run the skill with /kaggle-validation to design your validation strategy for a Kaggle competition.

Frequently Asked Questions about kaggle-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I choose the right cross-validation strategy for a Kaggle competition?▼

Cross-validation strategy selection is guided by analyzing data characteristics to choose KFold, StratifiedKFold, TimeSeriesSplit, or GroupKFold. It first detects distribution shifts using adversarial validation to ensure folds remain reliable and representative.

Why does my cross-validation score disagree with the Kaggle public leaderboard?▼

Disagreements between cross-validation and public leaderboard scores happen when folds misrepresent test data. This skill verifies CV and public LB score correlation after submissions to detect validation issues early and ensure local metrics match leaderboard performance.

How do I set up out-of-fold predictions for ensembling with pandas and scikit-learn?▼

Setting up out-of-fold predictions for ensembling requires a template framework using pandas, numpy, scikit-learn, and lightgbm. This skill locks fold indices early to ensure experiment consistency, generating reliable OOF predictions for downstream model blending.

When should I use adversarial validation in my machine learning pipeline?▼

Adversarial validation is used before fold selection to detect distribution shifts between training and test sets. It prevents misleading folds by identifying if training data significantly differs from test data, maintaining robust and reliable cross-validation results.

Does this validation framework work without lightgbm installed?▼

The validation framework requires lightgbm installed alongside pandas, numpy, and scikit-learn for execution. These dependencies are necessary to run scripts handling fold selection, distribution shift detection, and out-of-fold prediction generation effectively.