Model Evaluation Patterns

Generate classification metrics, calibration analysis, and fairness evaluations for machine learning models.

Updated Feb 25, 2026

Search Tags:#calibration #fairness #model evaluation #LLM evaluation #classification metrics #sliced evaluation

Ready to run

npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill model-evaluation-patterns-hermeticormus

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Model Evaluation Patterns
Source: https://github.com/HermeticOrmus/LibreMLOps-Claude-Code/tree/main/plugins/model-evaluation/skills/model-evaluation-patterns
Command: npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill model-evaluation-patterns-hermeticormus

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, fairlearn, evaluate, ragas, datasets, numpy, pandas, matplotlib, scipy, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need to thoroughly evaluate machine learning models beyond simple accuracy, ensuring they are robust, fair, and performant across various scenarios.

Core Features & Use Cases

Comprehensive Metrics: Generates a full suite of classification metrics with confidence intervals.
Calibration Analysis: Assesses if model probabilities accurately reflect true likelihoods.
Sliced Evaluation: Identifies performance disparities across different data subgroups.
Fairness Metrics: Quantifies bias related to sensitive attributes.
LLM Evaluation: Integrates BERTScore and RAGAS for generative model assessment.
Use Case: After training a churn prediction model, use this Skill to generate a complete evaluation report, check its calibration, and verify it doesn't unfairly penalize specific customer segments.

Quick Start

Generate a full evaluation report for the test data using the provided true labels, predicted labels, and predicted probabilities.

Frequently Asked Questions about Model Evaluation Patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate machine learning model fairness and detect bias across sensitive attributes?▼

To evaluate machine learning model fairness, you need to quantify bias related to sensitive attributes using sliced evaluation and fairness metrics. This process identifies performance disparities across different data subgroups to ensure robust validation.

What is the best way to assess LLM evaluation metrics using RAGAS and BERTScore?▼

Assessing LLM evaluation metrics involves integrating tools like RAGAS and BERTScore for generative model assessment. This approach measures performance beyond simple accuracy, addressing the need for robust validation in generative workflows.

How do I perform calibration analysis to check if predicted probabilities reflect true likelihoods?▼

Calibration analysis checks if model probabilities accurately reflect true likelihoods by comparing predicted confidence against actual outcomes. This ensures generated classification metrics reliably represent real-world probabilities.

Does scikit-learn support comprehensive classification metrics with confidence intervals?▼

Scikit-learn supports comprehensive classification metrics, and when combined with libraries like numpy and scipy, you can calculate confidence intervals. This generates a full suite of performance measurements for robust model validation.

Can I generate a sliced evaluation report to identify performance disparities in customer churn prediction?▼

You can generate a sliced evaluation report to identify performance disparities by analyzing model metrics across different data subgroups. This verifies your churn prediction model does not unfairly penalize specific customer segments.

Why does my model evaluation show high accuracy but fail on specific data subgroups?▼

Model evaluation may show high accuracy but fail on specific data subgroups due to unmeasured fairness issues or poor calibration. Sliced evaluation and fairness metrics are required to quantify bias and reveal these hidden performance disparities.