calibration-conformal

Calibrate temperature scaling by fitting the scaling parameter on a held-out dataset, separating the calibration logic from the training loop to ensure reliable decision making for open-world detectors.

Conformal prediction generates nonconformity scores to establish calibrated thresholds for abstention. Mondrian conformal methods provide distribution-free guarantees when data exhibits heterogeneity or shifts across different groups.

Serialize calibration artifacts by computing temperature scaling parameters and conformal thresholds on held-out data, then saving them as temperature.json and conformal.json files for reuse during evaluation and inference.

Abstention policies for detector inference include forced decision, threshold-based abstention, and conformal abstention. These policies determine when a detector should withhold predictions to maintain calibrated decision making.

Yes, the calibration workflow enforces strict separation of calibration logic from training loops. This separation ensures calibration is performed on held-out data, preventing data leakage and maintaining reliable abstention behavior.

The best way to compute conformal thresholds is using split conformal or Mondrian conformal methods on a held-out dataset, generating nonconformity scores that can be serialized into conformal.json artifacts for inference.