Evals

What problem does it solve?

The Evals Skill provides a structured framework to evaluate AI agent workflows—capturing transcripts, tool usage, and multi-turn interactions—so you can compare models, prompts, and configurations reliably.

Core Features & Use Cases

• Deterministic scorers for fast, reproducible checks (format, code, state, tools)
• Model-based rubrics and natural-language assertions for nuanced judgments
• Trial-based evaluation with pass@k and pass^k metrics
• Use-case driven workflows for creation, execution, comparison, and reporting
• Integrates with ALGORITHM workflows for automated validation and ISC updates

Quick Start

Run an evaluation by creating a use case, adding test cases, and running the evaluator through the CLI: bun run ~/.opencode/skills/Evals/EvalServer/cli-run.ts --use-case <name> --model <model>