uncertainty-quantification

What problem does it solve?

Uncertainty-quantification helps an agent decide how much to trust LLM outputs and what to do when confidence is not reliable, preventing hallucinations from driving irreversible actions.

Core Features & Use Cases

• Three-tier sampling protocol: fast, standard, and thorough tiers using semantic entropy, SelfCheckGPT-style consistency checks, and conformal prediction/back-off for high-stakes factuality.
• Explicit abstain/escalate recommendations: outputs a policy decision (proceed / proceed_with_caveat / abstain / escalate) tied to measurable uncertainty signals.
• Compatibility with white-box and black-box settings: supports logprob-based methods when available and black-box estimators otherwise.
• Action gating for agentic systems: reduces risk before writes/deletes/sends/deploys and supports selective retrieval decisions in RAG.

Quick Start

Use uncertainty-quantification to score a draft response, then follow its recommendation to proceed, add caveats, abstain for more evidence, or escalate to a human when the action is irreversible.