skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

evaluation

Evaluate machine learning models on imbalanced datasets with metrics and techniques.

1|Updated Jun 4, 2026
Search Tags:#model evaluation#statistical tests#evaluation metrics#class imbalance#probability calibration#threshold optimization#learning curves
One-click install
npx skills add https://github.com/hung-phan/ml-skills --skill evaluation-hung-phan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/hung-phan/ml-skills/tree/main/skills/ml-review/references/ml-training/evaluation
Command: npx skills add https://github.com/hung-phan/ml-skills --skill evaluation-hung-phan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires imbalanced-learn, scikit-learn, numpy, matplotlib, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a comprehensive toolkit for evaluating models on imbalanced datasets, addressing the limitations of accuracy and offering robust metrics and techniques to ensure better model performance on underrepresented classes.

Core Features & Use Cases

  • Class Imbalance Techniques: Offers techniques like SMOTE, ADASYN, and RandomOverSampler to handle imbalanced datasets.
  • Evaluation Metrics: Computes precision, recall, F1-score, ROC-AUC, PR-AUC, MCC, Kappa, Log Loss, and Brier Score for model evaluation.
  • Multi-Class Metrics and Confusion Matrix: Provides multi-class evaluation metrics and visualization of confusion matrices.
  • Probability Calibration: Calibrates probabilities using Platt scaling and isotonic regression for better reliability.
  • Threshold Optimization: Finds optimal thresholds using Youden's J Statistic, F1-optimal threshold, and cost-matrix optimization.
  • Statistical Model Comparison: Compares models using paired t-test, corrected resampled t-test, Wilcoxon Signed-Rank, McNemar's Test, Friedman + Nemenyi, and Nemenyi post-hoc tests.
  • Learning Curves for Diagnosis: Visualizes learning curves to diagnose model bias and variance.
  • Complete Evaluation Pipeline with Imbalanced-Learn: Integrates with imbalanced-learn for a robust evaluation pipeline.
  • "Too-Good-To-Be-True" — Leakage and Sanity-Check Playbook: Provides a checklist to avoid common pitfalls in model evaluation.

Quick Start

To evaluate a model, use the evaluation skill with the following command: /evaluation I have a binary classification model. How can I evaluate its performance on an imbalanced dataset?

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate model performance on imbalanced datasets?▼

Evaluating model performance on imbalanced datasets requires moving beyond accuracy to metrics like precision, recall, F1-score, ROC-AUC, PR-AUC, MCC, and Kappa. This Skill computes these metrics and integrates imbalanced-learn techniques to handle underrepresented classes effectively.

What is the best way to find optimal classification thresholds for imbalanced data?▼

Finding optimal classification thresholds uses techniques like Youden's J Statistic, F1-optimal threshold selection, and cost-matrix optimization. This Skill applies these threshold optimization methods to maximize performance based on your specific operational costs and class distribution.

Can I use scikit-learn and imbalanced-learn for model comparison and statistical tests?▼

Yes, you can use scikit-learn and imbalanced-learn for model comparison through statistical tests. This Skill supports paired t-tests, Wilcoxon Signed-Rank, McNemar's Test, and Friedman with Nemenyi post-hoc tests to validate performance differences between models.

How does probability calibration improve machine learning model evaluation?▼

Probability calibration improves machine learning model evaluation by aligning predicted probabilities with actual outcomes using Platt scaling and isotonic regression. This Skill applies these methods to ensure model reliability scores reflect true likelihoods, reducing Brier Score and Log Loss.

What learning curve analysis techniques help diagnose model bias and variance?▼

Learning curve analysis techniques diagnose model bias and variance by visualizing training and validation performance across varying dataset sizes. This Skill generates learning curves using matplotlib and scikit-learn to identify whether models need more data or complexity adjustments.

Are there sanity checks to detect data leakage during model evaluation?▼

Yes, sanity checks to detect data leakage during model evaluation are provided through a leakage and sanity-check playbook. This Skill includes a checklist to identify common pitfalls and prevent overly optimistic results caused by improper data handling.

Related Skills

Explore More Data & Analytics Skills→
dkyazzentwatwadkyazzentwatwa
86

classification-helper

Automate classification model training and evaluation with scikit-learn.

Community
Intermediate
human-avatarhuman-avatar
212

probability-base-rate-anchoring

Anchor probability estimates in historical base rates with reference-class reasoning.

Official
Intermediate
snoodleboot-iosnoodleboot-io
2

imbalanced-classification

Apply oversampling techniques to balance imbalanced classification datasets.

Official
Intermediate
tondevreltondevrel
19

pytorch

Develop and train deep learning models using PyTorch with tensor operations and automatic differentiation.

Community
Advanced