What problem does it solve?
Evaluating LLM agents requires more than one-off scores because reliability varies across real inputs and failure modes, so this Skill helps you measure behavior, capability, and operational risk with repeatable eval methods.
Core Features & Use Cases
- Behavioral testing: build invariant, contract, snapshot, and regression suites that catch real-world regressions in tool use, safety, and output structure.
- Capability benchmarks: run established benchmarks like SWE-bench, AgentBench, and HumanEval, plus pass@k and calibration to understand strengths and limits.
- Reliability metrics: compute N-run consistency, worst-case percentiles, error budgets, and SLOs to quantify reliability variance.
- LLM-as-judge: design rubrics, mitigate judge bias, run dual-judge/ensemble scoring, and verify correlation with human judgment.
- Production monitoring: instrument tracing and do eval-in-production with drift detection and cost tracking for continuous assurance.
- Red teaming: create adversarial tests for prompt injection, jailbreaks, tool misuse, and data leakage to ensure robustness.
Quick Start
Use agent-evaluation to design an N-run eval plan for your agent that includes behavioral invariants, an LLM-judge rubric, and production monitoring checks.
