skills.rest
OlixIgnacious
OlixIgnaciousCOMMUNITY
Explore all skills by this author →
GitHub

kaggle-validation

Detect distribution shifts and select fold types for Kaggle validation.

1|Updated Jun 6, 2026
Search Tags:#validation#machine learning#data analysis#cross-validation#kaggle
One-click install
npx skills add https://github.com/OlixIgnacious/agent-skills --skill kaggle-validation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kaggle-validation
Source: https://github.com/OlixIgnacious/agent-skills/tree/main/.claude/skills/kaggle-validation
Command: npx skills add https://github.com/OlixIgnacious/agent-skills --skill kaggle-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, scikit-learn, lightgbm, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a structured approach to design and lock the validation strategy for Kaggle competitions, ensuring comparability and accuracy of experiments.

Core Features & Use Cases

  • Adversarial Validation: Before fold selection, it checks for distribution shifts to avoid misleading folds.
  • Fold Type Selection: It guides through choosing the right fold type (KFold, StratifiedKFold, TimeSeriesSplit, GroupKFold) based on data characteristics.
  • Lock Fold Indices Early: Ensures consistency across experiments by saving fold assignments.
  • OOF Prediction Framework: A template for out-of-fold predictions to support ensembling.
  • CV–LB Correlation Verification: After submissions, it verifies the correlation between CV and public LB scores for early detection of validation issues.
  • Validation Checklist: Offers a comprehensive checklist to ensure a robust validation strategy.

Quick Start

Run the skill with /kaggle-validation to design your validation strategy for a Kaggle competition.

Frequently Asked Questions about kaggle-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the right cross-validation strategy for a Kaggle competition?▼

Cross-validation strategy selection is guided by analyzing data characteristics to choose KFold, StratifiedKFold, TimeSeriesSplit, or GroupKFold. It first detects distribution shifts using adversarial validation to ensure folds remain reliable and representative.

Why does my cross-validation score disagree with the Kaggle public leaderboard?▼

Disagreements between cross-validation and public leaderboard scores happen when folds misrepresent test data. This skill verifies CV and public LB score correlation after submissions to detect validation issues early and ensure local metrics match leaderboard performance.

How do I set up out-of-fold predictions for ensembling with pandas and scikit-learn?▼

Setting up out-of-fold predictions for ensembling requires a template framework using pandas, numpy, scikit-learn, and lightgbm. This skill locks fold indices early to ensure experiment consistency, generating reliable OOF predictions for downstream model blending.

When should I use adversarial validation in my machine learning pipeline?▼

Adversarial validation is used before fold selection to detect distribution shifts between training and test sets. It prevents misleading folds by identifying if training data significantly differs from test data, maintaining robust and reliable cross-validation results.

Does this validation framework work without lightgbm installed?▼

The validation framework requires lightgbm installed alongside pandas, numpy, and scikit-learn for execution. These dependencies are necessary to run scripts handling fold selection, distribution shift detection, and out-of-fold prediction generation effectively.

Related Skills

Explore More Data & Analytics Skills→
ginaechoginaecho

agentic-labeling

Automate dataset clustering, feature engineering, and cluster interpretation with multi-agent architecture.

Community
Advanced
dianapriordianaprior
5

tabpfn-regress

Generate TabPFN regression baselines with out-of-fold predictions and ensembled submissions.

Community
Advanced
LeandroBenjaminLLeandroBenjaminL

ml-modeling

Train machine learning models with validation and explainability using scikit-learn, lightgbm, xgboost, and shap.

Community
Advanced
nahisahonahisaho
3

scientific-ml-classification

Benchmark multiple classification models with StratifiedKFold cross-validation and report ROC-AUC metrics.

Community
Intermediate