Eval Running Skill

Run evaluation suites via a harness-driven pipeline to detect regressions and benchmark skill quality across your automation framework. It orchestrates end-to-end execution and baseline comparison for both CI and local validation runs.

Benchmark per-skill task quality in CI by executing a harness-driven evaluation pipeline. It ensures deterministic results and enables baseline management to quantify framework correctness across automated runs.

Baseline management for regression tests works by comparing current evaluation suite outputs against stored baselines. It enables updating reference baselines and validating results locally or within CI to detect regressions.

You can use structured outputs from evaluation suites for downstream analytics because the harness produces JSONL files. These structured logs support auditing and result comparison after running framework correctness or regression tests.

The evaluation harness supports both local runs and CI validation environments. It executes framework correctness, regression tests, and per-skill benchmarks deterministically across these scenarios for consistent baseline management.

Framework correctness suites validate the core orchestration pipeline, whereas per-skill benchmarks quantify individual task quality. Both are executed by the evaluation harness to detect regressions and produce structured JSONL outputs.