evaluating-binary-classifiers

Evaluating a binary classifier on imbalanced data requires reframing the report around PR-AUC instead of standard accuracy. This Skill checks class balance first and applies precision-recall metrics to handle rare positive classes effectively.

Finding the optimal decision threshold involves sweeping multiple values instead of defaulting to 0.5. This Skill compares thresholds and selects an operating point using cost-aware or precision-floor rules to match your production requirements.

A complete binary classification scoring report needs ROC-AUC, PR-AUC, calibration, confusion matrices, and bootstrap confidence intervals. This Skill generates all these metrics from your held-out test sets to provide defensible performance analysis.

Yes, this binary classifier evaluation supports cost-aware decision handling. It selects operating points using cost-aware rules during threshold sweeps, making it suitable for fraud and medical screening workflows where false positives and negatives carry different weights.

Calculating bootstrap confidence intervals for ROC-AUC and PR-AUC involves resampling your test predictions. This Skill computes bootstrap confidence intervals for key metrics automatically, adding reliability analysis and Brier scores to your evaluation report.

Binary classifier accuracy often fails to reflect real performance because it masks poor minority class detection in imbalanced datasets. This Skill solves the problem by evaluating calibration, PR-AUC, and threshold-aware metrics to judge actual model behavior.