calibration-bin-robustness

What problem does it solve?

This Skill addresses the issue of sparse bins in calibration binning modules, ensuring more reliable accuracy and confidence threshold recommendations.

Core Features & Use Cases

• Sparse Bin Mitigation: Protects against sparse bins by enforcing a minimum sample count.
• Bin Merging: Merges sparse bins into the nearest populated neighbor to improve accuracy.
• Threshold Recommendations: Adds a minimum sample requirement to threshold recommendations.
• Use Case: Ideal for calibration/evaluation modules that compute binned accuracy, ECE, MCE, or confidence thresholds.

Quick Start

Set a minimum samples-per-bin threshold and use the skill to compute binned accuracy with confidence threshold recommendations.