probability-calibration

Probability calibration verifies model outputs by comparing predicted win probabilities against actual historical outcomes using reliability diagrams and Brier score analysis to detect overconfidence or underconfidence.

Platt scaling fits a logistic regression curve to calibrate probabilities, while isotonic regression uses a non-parametric approach. Calibration analysis guides which method best improves your model's probability quality.

Brier score and reliability diagrams are calculated by binning predicted probabilities into intervals and comparing observed win rates against predicted rates, requiring paired predictions and actual outcomes from historical games.

Drift monitoring detects when calibration degrades over time by comparing bin-level metrics across periods, recommending refitting correction methods on recent games when reliability drops below acceptable thresholds.

Probability calibration requires enough historical games with paired predicted probabilities and actual outcomes to compute bin-level metrics. Sufficient sample size ensures statistical significance for reliability diagrams and post-calibration comparisons.

This miscalibration indicates overconfident probabilities. Calibration diagnostics measure this bin-level error using reliability diagrams and Brier scores, then recommend Platt scaling or isotonic regression to correct the outputs.