What problem does it solve? Measuring whether a change to a skill or guideline text actually improves it requires repeatable, multi-provider evaluation, which is slow and error-prone to orchestrate by hand. ## Core Features & Use Cases - Scenario Execution: Runs a declared benchmark scenario (a skill, command, or question) via benchmark/run.py in a guideline repository checkout, with configurable judges, effort, repeats, and runtime (host, container, or vm). - Multi-Provider Judging: Scores the subject with frontier models from Anthropic, OpenAI, Gemini, and xAI, selected by bit flag or comma-joined names, and reports per-provider scores plus the overall mean. - Dry-Run Planning: Resolves the full run plan with --dry-run, writing run.json without calling any provider, so you can verify the subject command, judges, and fallbacks before spending money. - Use Case: After editing a review skill, run the explain-tenancy scenario with all four judges to see whether the change raised the rubric score before shipping it. ## Quick Start Run the explain-tenancy benchmark scenario with all four judges and one repeat, then summarize the report.
