skills.rest
KameniAlexNea
KameniAlexNeaCOMMUNITY
Explore all skills by this author →
𝕏@kamenialexnea22GitHub

adversarial-validation

Detect and quantify distribution shift between training and testing datasets using a LightGBM classifier.

Updated Feb 24, 2026
Search Tags:#machine learning#feature importance#adversarial validation#data drift#distribution shift#sample weighting
One-click install
npx skills add https://github.com/KameniAlexNea/gladius-agent --skill adversarial-validation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: adversarial-validation
Source: https://github.com/KameniAlexNea/gladius-agent/tree/main/gladius/utils/templates/skills/adversarial-validation
Command: npx skills add https://github.com/KameniAlexNea/gladius-agent --skill adversarial-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill identifies and helps mitigate distribution shift between training and testing datasets, which can lead to poor model generalization and a significant gap between cross-validation and leaderboard scores.

Core Features & Use Cases

  • Distribution Shift Detection: Uses a classifier to distinguish between train and test data, providing an AUC score to quantify the shift.
  • Leaking Feature Identification: Highlights features that are most indicative of the shift, suggesting potential data leaks or structural differences.
  • Adversarial Sample Weighting: Generates weights to adjust the training process, emphasizing samples that resemble the test distribution.
  • Use Case: Before training a model, run this skill to ensure your training data is representative of the test data. If a significant shift is detected (AUC > 0.55), use the identified features or sample weights to improve model robustness.

Quick Start

Run the adversarial validation script to detect distribution shift and identify leaking features.

Frequently Asked Questions about adversarial-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect distribution shift between my training and testing datasets?▼

You can identify leaking features by analyzing the feature importance scores from the LightGBM classifier. The classifier highlights features that are most indicative of the distribution shift, revealing potential structural differences or data leaks.

How do I mitigate data drift impact on model performance?▼

Data drift impact is mitigated by generating adversarial sample weights that adjust the training process. These weights emphasize training samples that closely resemble the test distribution, improving model robustness against the shift.

Why does my cross-validation score differ significantly from my leaderboard score?▼

A significant gap between cross-validation and leaderboard scores often indicates distribution shift between training and testing data. Adversarial validation detects this shift by checking if a classifier can distinguish the datasets, revealing poor model generalization.

When should I run adversarial validation on my machine learning data?▼

You should run adversarial validation before training a model to ensure your training data is representative of the test data. It is highly applicable in machine learning competitions and real-world deployment scenarios where data drift is a concern.

Can I use adversarial validation without installing additional dependencies?▼

Yes, you can run the adversarial validation script without installing additional dependencies. The Skill operates standalone with its included scripts, utilizing a LightGBM classifier to detect distribution shift and generate sample weights.

Related Skills

Explore More Data & Analytics Skills→
OlixIgnaciousOlixIgnacious
1

kaggle-adversarial-validation

Detect train/test distribution shift in Kaggle competitions using adversarial classifiers.

Community
Advanced
snoodleboot-iosnoodleboot-io
2

cross-validation-strategies

Provides k-fold and bootstrap cross-validation strategies for ML models to detect overfitting and improve generalization on unseen data.

Official
Intermediate
tomlupotomlupo

shap

Explain model predictions by attributing outputs to input features with SHAP values.

Community
Advanced
rajyeole6rajyeole6

agency-model-qa-specialist

Audit machine learning models by replicating training pipelines and computing PSI, Gini, KS, and SHAP metrics.

Community
Advanced