Calibration Analysis

Name: Calibration Analysis
Availability: InStock
Author: trudumb

Compute Brier score decomposition and information ratio across predicted probabilities.

Updated Aug 27, 2026

Search Tags:#calibration #model-evaluation #machine-learning #brier-score #information-ratio #calibration-curve #pnl-attribution

Ready to run

npx skills add https://github.com/trudumb/hyper_make --skill calibration-analysis

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Calibration Analysis
Source: https://github.com/trudumb/hyper_make/tree/main/.claude/skills/foundation/calibration-analysis
Command: npx skills add https://github.com/trudumb/hyper_make --skill calibration-analysis

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Systematic analysis of model predictions vs realized outcomes to identify exactly where the model is wrong and by how much.

Core Features & Use Cases

Brier score decomposition (reliability, resolution, uncertainty) and information ratio to quantify calibration quality across prediction bins.
Calibration Curve construction and Conditional Calibration across regimes and time-based slices for targeted improvements.
Daily calibration reporting and PnL attribution workflows to monitor model health in production.

Quick Start

Provide the latest prediction probabilities and outcomes to compute Brier decomposition, calibration curve, and conditional calibration, then generate and review the daily calibration report.

Frequently Asked Questions about Calibration Analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I compute Brier score decomposition for machine learning model evaluation?▼

Brier score decomposition splits model predictions into reliability, resolution, and uncertainty components to quantify calibration quality across prediction bins. You provide a prediction array and outcomes to compute these metrics and identify exactly where the model is wrong.

What is a calibration curve and how does it help with model reliability?▼

A calibration curve plots predicted probabilities against realized outcomes to visualize model reliability. By applying it across regimes and time-based slices, you can perform conditional calibration to identify targeted areas for improving model predictions in production ML environments.

How do I set up daily calibration reporting and PnL attribution workflows for production ML?▼

Daily calibration reporting and PnL attribution workflows monitor model health in production by systematically analyzing predictions versus realized outcomes. You compute Brier decomposition, information ratio, and conditional calibration to quantify calibration gaps and track model degradation over time.

Can I analyze model calibration across different market regimes and time slices?▼

Yes, conditional calibration analyzes model predictions across regimes and time-based slices. By applying Brier score decomposition and calibration curves to these segments, you identify specific conditions where the model underperforms and needs targeted improvements.

What data do I need to perform calibration analysis on my machine learning model?▼

Calibration analysis requires a prediction array of probabilities and corresponding realized outcomes, plus a binning strategy. These inputs enable Brier score decomposition, information ratio calculation, and calibration curve construction to evaluate model reliability and resolution.