skills.rest
zkysar1
zkysar1COMMUNITY
Explore all skills by this author →
𝕏@ZacharyKysarGitHub

reflect-calibration

Bin resolved hypotheses by confidence and calculate accuracy per bin.

Updated Aug 23, 2026
Search Tags:#hypothesis testing#calibration#confidence#reflection#self-correction#ai accuracy
One-click install
npx skills add https://github.com/zkysar1/Claude-Skills-Continual-Learning-Base --skill reflect-calibration
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: reflect-calibration
Source: https://github.com/zkysar1/Claude-Skills-Continual-Learning-Base/tree/main/.claude/skills/reflect-calibration
Command: npx skills add https://github.com/zkysar1/Claude-Skills-Continual-Learning-Base --skill reflect-calibration

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill addresses the challenge of unreliable confidence scores in AI-generated hypotheses, ensuring more accurate self-assessment and improved decision-making.

Core Features & Use Cases

  • Confidence Binning: Groups hypotheses by confidence intervals (e.g., 70-79%).
  • Accuracy Calculation: Computes actual accuracy within each confidence bin.
  • Self-Consistency Check: Recommends methods for verifying hypothesis accuracy through multiple independent assessments.
  • Data Updates: Persists calibration findings to improve future performance.
  • Use Case: After generating 50 hypotheses, this skill analyzes how often hypotheses with 90%+ confidence were actually correct, identifying potential over or under-confidence in the AI's self-assessment.

Quick Start

Run the reflect calibration check to analyze hypothesis accuracy.

Frequently Asked Questions about reflect-calibration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calibrate AI confidence levels for generated hypotheses?▼

Calibrate AI confidence levels by binning resolved hypotheses into confidence intervals, calculating actual accuracy per bin, and updating meta-confidence calibration bias data to correct over or under-confidence in reasoning loops.

What is confidence binning in AI hypothesis testing?▼

Confidence binning groups AI hypotheses by confidence intervals like 70 to 79 percent, allowing you to calculate actual accuracy within each bin and identify calibration bias in self-assessment.

How can I check if my AI's self-assessment is overconfident?▼

Check for AI overconfidence by analyzing resolved hypotheses, calculating actual accuracy within confidence bins, and comparing results against original confidence levels to find calibration bias.

What is a self-consistency check for AI accuracy?▼

A self-consistency check verifies AI accuracy by recommending multiple independent assessments of hypotheses, ensuring reliable self-correction and improved decision-making in reasoning loops.

How many resolved hypotheses do I need for confidence calibration?▼

Confidence calibration requires a set of resolved hypotheses to bin by confidence intervals. An example use case analyzes 50 hypotheses to compute actual accuracy within bins and identify calibration bias.

Related Skills

Explore More Software Engineering Skills→
ruskibeatsruskibeats

calibration-bin-robustness

Enforce minimum sample counts and merge sparse bins in calibration binning modules.

Community
Intermediate
rfdiosuaorfdiosuao
30

predictor-hand-skill

Generates and scores probabilistic forecasts using superforecasting principles and reasoning chains.

Community
Advanced
moonlight-lupinmoonlight-lupin
15

fact-checker

Verify factual claims by cross-checking multiple independent sources.

Community
Advanced
brandonburrusbrandonburrus
1

statistical-analysis

Applies statistical methods to analyze data distributions, detect anomalies, and test hypotheses.

Community
Advanced