skills.rest
kalilurrahman
kalilurrahmanCOMMUNITY
Explore all skills by this author →
Website𝕏@krahmanGitHub

model-evaluation

Evaluate ML model performance with metrics, calibration, fairness analysis, and go/no-go recommendations.

1|Updated Mar 21, 2026
Search Tags:#machine learning#calibration#model evaluation#mlflow#scikit-learn#classification metrics#fairness analysis
One-click install
npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill model-evaluation-kalilurrahman
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-evaluation
Source: https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts/tree/main/04-ai-ml/model-evaluation
Command: npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill model-evaluation-kalilurrahman

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, scikit-learn, matplotlib, mlflow.

What problem does it solve? Choosing the wrong metric or evaluating on leaky data produces models that look good in notebooks but fail in production. This Skill provides a rigorous evaluation framework that aligns metrics with business costs, quantifies uncertainty, and surfaces fairness issues before deployment. ## Core Features & Use Cases - Metric Selection & Computation: Maps task types (binary classification, regression, ranking, survival) to appropriate primary metrics like AUC-ROC, AUC-PR, F1, NDCG@K, and cost-sensitive custom metrics. - Statistical Rigor: Computes bootstrap confidence intervals, calibration curves with Expected Calibration Error, and threshold analysis across decision boundaries. - Fairness & Go/No-Go Decisions: Evaluates performance across demographic subgroups, detects AUC/FNR disparities, and produces evidence-based go/no-go recommendations against requirements. - Use Case: Before promoting a fraud detection model to production, run the evaluator on a held-out test set with cost_fp=1 and cost_fn=10, verify recall meets the 0.65 requirement with confidence intervals, and confirm no fairness disparity across age groups. ## Quick Start Evaluate my trained classifier on the test set with bootstrap confidence intervals, calibration analysis, and a fairness check across customer segments, then give me a go/no-go recommendation.

Frequently Asked Questions about model-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a machine learning model with imbalanced classes?▼

Use AUC-PR (average precision) and F1 instead of accuracy or AUC-ROC, since rare positive classes make ROC overly optimistic. Also stratify your train/test splits so every fold contains positive examples, and tune the decision threshold on validation data.

What metrics should I use for binary classification model evaluation?▼

For balanced classes use F1 and AUC-ROC; for imbalanced classes use AUC-PR and F1; for cost-sensitive problems like fraud detection, compute a custom cost metric weighting false positives and false negatives by their business costs.

How do I compute confidence intervals for model metrics?▼

Use bootstrap resampling: resample the test set with replacement many times (e.g., 1000 iterations), recompute each metric per sample, and take percentiles for the interval bounds. This reveals variance hidden by single-point estimates.

Why is my model's calibration important and how do I check it?▼

Calibration matters when predicted probabilities drive decisions—a model predicting 80% should be correct 80% of the time. Check it with a calibration curve and Expected Calibration Error; ECE below 0.05 indicates good calibration.

How do I check a model for fairness across demographic groups?▼

Compute metrics like AUC, F1, false positive rate, and false negative rate separately for each subgroup with sufficient samples (30+). Flag concerns when AUC disparity exceeds 0.05 or FNR disparity exceeds 0.1 between groups.

When should I not use a random train-test split?▼

Avoid random splits for time-series data, since they leak future information into training. Use a temporal split instead, and always reserve the test set for a single final evaluation after all tuning is complete.

Related Skills

Explore More Data & Analytics Skills→
jvdominojvdomino

domino-experiment-tracking

Track ML experiments and model versions across Domino projects.

Community
Intermediate
jvdominojvdomino

domino-model-monitoring

Monitor deployed Domino models for drift, quality, and alerts.

Community
Advanced
DuqueOMDuqueOM
5

drift-detection

Detect data and concept drift in ML services using PSI and sliced AUC metrics.

Community
Advanced
sfc-gh-miwhitakersfc-gh-miwhitaker
3

tool-cortex-code-costs

Monitor and analyze Cortex Code usage costs with a Snowsight dashboard.

Community
Advanced