eval-run

Skill evaluation automates testing by executing a skill against a dataset of test cases, applying judges to score outputs, and reporting results. The pipeline requires an eval.yaml file to configure datasets, models, tools, and outputs.

Benchmarking reports for AI skill outputs are generated by running the skill in batch or case mode, collecting artifacts, and producing a per-run summary. The pipeline outputs run_result.json, collection.json, and summary.yaml with costs.

Batch evaluation mode executes a skill against multiple test cases in a single run, collecting artifacts and applying judges to score outputs. The eval.yaml file configures the dataset, models, and runners to orchestrate the entire evaluation flow.

Yes, an eval.yaml configuration file is required to run skill evaluations. The evaluation pipeline requires eval.yaml to configure the dataset, models, tools, and outputs before executing test cases and producing summary reports.

The evaluation pipeline produces three output files: run_result.json, collection.json, and summary.yaml. These files contain run results, per-case artifacts, and a culminating analysis with costs for reporting.