eval-run

To evaluate AI output quality, you run an automated evaluation against a structured evals.yaml file to generate a per-assertion score report. This process applies across any skill domain to deterministically measure output quality and detect regressions.

Detecting regressions after instruction changes is achieved by running an evals.yaml evaluation before and after modifications. This produces a structured before/after quality comparison report showing PASS/FAIL per assertion and total scores.

Yes, you can evaluate output using both deterministic code and llm-judge assertions. The evaluation verifies all assertions within the evals.yaml format and returns a machine-readable score report detailing any failures.

Generating a per-assertion score report for auditing skill performance involves supplying an output file and its corresponding evals.yaml. The evaluation produces a human-friendly report with PASS/FAIL details suitable for automated quality gates.

Yes, the evals.yaml format validation enforces strict syntax rules before running assertions. This ensures that the structured evaluation can reliably apply code and llm-judge assertions without processing errors.

You need an automated quality gate for skill optimization when you want to baseline scoring after instruction changes. Running an evals.yaml evaluation provides a structured score report to prevent quality degradation across releases.