skills.rest
HermeticOrmus
HermeticOrmusCOMMUNITY
Explore all skills by this author →
Website𝕏@dbodartmGitHub

Model Evaluation Patterns

Generate classification metrics, calibration analysis, and fairness evaluations for machine learning models.

Updated Feb 25, 2026
Search Tags:#calibration#fairness#model evaluation#LLM evaluation#classification metrics#sliced evaluation
One-click install
npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill model-evaluation-patterns-hermeticormus
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Model Evaluation Patterns
Source: https://github.com/HermeticOrmus/LibreMLOps-Claude-Code/tree/main/plugins/model-evaluation/skills/model-evaluation-patterns
Command: npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill model-evaluation-patterns-hermeticormus

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, fairlearn, evaluate, ragas, datasets, numpy, pandas, matplotlib, scipy, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need to thoroughly evaluate machine learning models beyond simple accuracy, ensuring they are robust, fair, and performant across various scenarios.

Core Features & Use Cases

  • Comprehensive Metrics: Generates a full suite of classification metrics with confidence intervals.
  • Calibration Analysis: Assesses if model probabilities accurately reflect true likelihoods.
  • Sliced Evaluation: Identifies performance disparities across different data subgroups.
  • Fairness Metrics: Quantifies bias related to sensitive attributes.
  • LLM Evaluation: Integrates BERTScore and RAGAS for generative model assessment.
  • Use Case: After training a churn prediction model, use this Skill to generate a complete evaluation report, check its calibration, and verify it doesn't unfairly penalize specific customer segments.

Quick Start

Generate a full evaluation report for the test data using the provided true labels, predicted labels, and predicted probabilities.

Frequently Asked Questions about Model Evaluation Patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate machine learning model fairness and detect bias across sensitive attributes?▼

To evaluate machine learning model fairness, you need to quantify bias related to sensitive attributes using sliced evaluation and fairness metrics. This process identifies performance disparities across different data subgroups to ensure robust validation.

What is the best way to assess LLM evaluation metrics using RAGAS and BERTScore?▼

Assessing LLM evaluation metrics involves integrating tools like RAGAS and BERTScore for generative model assessment. This approach measures performance beyond simple accuracy, addressing the need for robust validation in generative workflows.

How do I perform calibration analysis to check if predicted probabilities reflect true likelihoods?▼

Calibration analysis checks if model probabilities accurately reflect true likelihoods by comparing predicted confidence against actual outcomes. This ensures generated classification metrics reliably represent real-world probabilities.

Does scikit-learn support comprehensive classification metrics with confidence intervals?▼

Scikit-learn supports comprehensive classification metrics, and when combined with libraries like numpy and scipy, you can calculate confidence intervals. This generates a full suite of performance measurements for robust model validation.

Can I generate a sliced evaluation report to identify performance disparities in customer churn prediction?▼

You can generate a sliced evaluation report to identify performance disparities by analyzing model metrics across different data subgroups. This verifies your churn prediction model does not unfairly penalize specific customer segments.

Why does my model evaluation show high accuracy but fail on specific data subgroups?▼

Model evaluation may show high accuracy but fail on specific data subgroups due to unmeasured fairness issues or poor calibration. Sliced evaluation and fairness metrics are required to quantify bias and reveal these hidden performance disparities.

Related Skills

Explore More Data & Analytics Skills→
dkyazzentwatwadkyazzentwatwa
86

classification-helper

Automate classification model training and evaluation with scikit-learn.

Community
Intermediate
Yezez9Yezez9

shap

Compute SHAP feature attributions and generate explanation plots for machine learning models.

Community
Advanced
qinyan-aiqinyan-ai
48

scikit-learn

Develop, evaluate, and tune machine learning models with scikit-learn pipelines.

Community
Advanced
DTMC-marketplaceDTMC-marketplace

huggingface-evaluate

Evaluate machine learning models and datasets using Hugging Face Evaluate metrics.

Official
Intermediate