What problem does it solve?
You have an automatic judge or grader and human-labeled data, but no rigorous way to know whether the judge agrees with humans, where it is biased, or whether it can safely replace human review in production.
Core Features & Use Cases
- Calibration Statistics: Computes confusion matrix, TPR/TNR/F1 with bootstrap 95% confidence intervals, Cohen's kappa, Gwet's AC1, and directional bias from paired judge verdicts and human labels.
- Bias Detection: Runs five systematic bias checks covering position bias, verbosity bias, self-enhancement, progress illusion, and label drift, plus per-stratum breakdowns to expose hidden failures on boundary cases.
- Human-Reduction Roadmap: Produces a 4-phase transition plan (Advisory, Assisted, Auto-gate, Autonomous) gated by kappa thresholds, with disagreement clustering to diagnose root causes.
- Use Case: You have 200 human-labeled samples and a grader you want to deploy as a CI quality gate. Run the bundled calibration script to get a calibrated/not_calibrated verdict with the exact metrics blocking deployment.
Quick Start
Run the bundled calibration script on my paired judge verdicts and human labels JSONL files and tell me whether my grader is calibrated enough to reduce human review.
