skill-benchmark

Multi-run benchmarking executes tasks repeatedly to account for non-determinism, using isolated sandboxes per run to reduce interference. It applies layered grading with deterministic checks and LLM-as-judge scoring to produce robust performance metrics.

LLM-as-judge scoring evaluates skill performance after deterministic checks are applied, providing objective grades on response quality and behavior. This layered approach ensures robust scoring by combining automated validation with language model assessment.

Yes, negative controls use out-of-domain tasks to detect false positives and ensure reliability. By running benchmarks against tasks the skill should not solve, you can verify that performance improvements are specific to the intended domain.

No, a config.yml is optional. It allows you to customize models, tasks, and runs, but the benchmarking tool provides default fallbacks for task generation and configuration if no config file is supplied.

Reproducible skill grading generates detailed artifacts including response.json, transcript.json, meta.json, and behavior.json. These files capture outputs, conversation history, metadata, and behavioral analysis for deep evaluation and reporting.