skills.rest
xjtulyc
xjtulycCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ml-for-research

Build scikit-learn pipelines with nested cross-validation and FLAML AutoML.

33|6|Updated Mar 17, 2026
Search Tags:#machine learning#AutoML#model cards#model calibration#nested cross-validation#SHAP explainability#scikit-learn pipelines
One-click install
npx skills add https://github.com/xjtulyc/awesome-rosetta-skills --skill ml-for-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ml-for-research
Source: https://github.com/xjtulyc/awesome-rosetta-skills/tree/main/skills/00-universal/ml-for-research
Command: npx skills add https://github.com/xjtulyc/awesome-rosetta-skills --skill ml-for-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, flaml, shap, optuna, pandas, matplotlib.

What problem does it solve?

This Skill helps you build trustworthy machine learning models for research by avoiding data leakage, estimating performance fairly with nested cross-validation, and producing explainability and reporting artifacts that meet publication expectations.

Core Features & Use Cases

  • Reproducible research pipelines: scikit-learn Pipeline + ColumnTransformer that correctly impute, scale, one-hot encode, and prevent train/test leakage during cross-validation.
  • Unbiased evaluation: nested cross-validation (outer repeated stratified folds + inner hyperparameter tuning) to reduce optimistic bias in reported metrics.
  • Strong baselines and tuned models: FLAML AutoML with a time budget plus Optuna-based hyperparameter search options, with comparison to simple baselines (e.g., logistic regression / dummy).
  • Explainability and transparency: SHAP TreeExplainer outputs (e.g., summary/waterfall/dependence) to interpret feature contributions.
  • Calibrated probabilities and model cards: calibration curves using CalibratedClassifierCV, and structured model card generation for transparent method/limitations disclosure.

Quick Start

Use the ml-for-research skill to run a nested cross-validation experiment with a scikit-learn preprocessing pipeline, then generate SHAP summaries and a calibrated model-card report for your dataset.

Frequently Asked Questions about ml-for-research

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prevent data leakage during cross-validation in scikit-learn pipelines?▼

To prevent data leakage during cross-validation, use scikit-learn Pipeline and ColumnTransformer to encapsulate preprocessing steps like imputation and scaling. This ensures transformations are fitted only on training folds within each split, keeping test data unseen.

Why use nested cross-validation for machine learning model selection?▼

Nested cross-validation is used to reduce optimistic bias in reported performance metrics. By separating model selection in an inner loop from performance estimation in an outer loop, it provides an unbiased evaluation of hyperparameter tuning and model generalization.

How do I generate SHAP explainability plots for a trained scikit-learn model?▼

You can generate SHAP explainability plots by applying SHAP TreeExplainer to your trained model. This calculates feature attributions, allowing you to produce summary, waterfall, and dependence plots to interpret how each feature contributes to predictions.

Can I use FLAML AutoML with Optuna hyperparameter tuning for tabular research tasks?▼

Yes, you can use time-budgeted FLAML AutoML to establish strong baselines and Optuna-compatible logic for hyperparameter tuning. This combination allows efficient search for optimized models on supervised tabular research tasks.

What is the best way to calibrate predicted probabilities and create model cards?▼

The best way to calibrate predicted probabilities is using CalibratedClassifierCV to generate reliable probability outputs. You can then document the calibrated model's methods, metadata, and limitations by generating a structured model card.

Does this machine learning workflow handle mixed numeric and categorical features automatically?▼

Yes, this machine learning workflow handles mixed numeric and categorical features automatically. It uses a ColumnTransformer within the scikit-learn Pipeline to correctly apply scaling to numeric features and one-hot encoding to categorical features.

Related Skills

Explore More Education & Research Skills→
Estrella-231Estrella-231

scikit-learn

Builds and evaluates scikit-learn classification, regression, clustering and preprocessing pipelines with mixed numeric/categorical data.

Community
Intermediate
andikarachmanandikarachman
14

model-card

Generate standardized model documentation in HuggingFace Model Card and NVIDIA Model Card++ formats.

Community
Advanced
Yezez9Yezez9

shap

Compute SHAP feature attributions and generate explanation plots for machine learning models.

Community
Advanced
pur3v4d3rpur3v4d3r
6

scikit-learn

Train and evaluate classical machine learning models with scikit-learn.

Community
Advanced