Benchmarks AI agents and skills with synthetic problems to measure recall, precision, and calibration bias.
npx skills add https://github.com/Borda/.home --skill calibrate-borda
Please help me install this Agent Skill. Skill: calibrate Source: https://github.com/Borda/.home/tree/main/.claude/skills/calibrate Command: npx skills add https://github.com/Borda/.home --skill calibrate-borda
This Skill rigorously tests AI agents and skills against synthetic problems to measure their performance, identify systematic gaps, and ensure their self-reported confidence aligns with actual accuracy.
/calibrate sw-engineer full to test the software engineer agent on 10 synthetic coding problems, analyze its performance, and automatically generate updated instructions if needed.Use the calibrate skill to benchmark all agents and skills with full problem sets and apply any necessary changes.
High-intent search queries and answers about installing and using this skill.
Benchmark AI agent performance by generating synthetic problems with known outcomes to measure recall, precision, and calibration bias. This process identifies systematic gaps in agent outputs and proposes instruction updates for performance improvement.
Calibration analysis in AI agent testing detects over/under-confidence by comparing an agent's self-reported confidence levels against its actual recall on synthetic problems. This identifies systematic gaps where reported certainty does not align with real accuracy.
Measure recall and precision for AI skills by testing them against synthetic problems with known outcomes. This benchmarking calculates F1 scores and calibration bias to pinpoint recurring issues and anti-patterns in agent outputs.
You can automatically update agent instructions based on benchmark results. The calibration process analyzes performance gaps from synthetic problem testing and generates proposals to update instructions for immediate performance improvement.
Agent benchmarking supports A/B comparison against a general-purpose baseline. This feature allows you to evaluate specialized skill performance by comparing its recall, precision, and calibration metrics directly against baseline results.
yu13130122297Identify uncertainties in research framing, datasets, metrics, and baselines.
chetmanciniAnalyzes Agent Smith session metrics and generates tuning recommendations for Claude Code, Codex, OpenCode, and Pi.
48Nauts-OperatorImplement LLM evaluation strategies with automated metrics and human feedback.
bmsull560Evaluate agent performance and extraction accuracy in LangGraph pipelines.