output-dev-eval-testing

Offline evaluation tests for AI workflows validate quality without impacting production by using dataset YAMLs and verify() evaluators. You can build, run, and iterate these tests via the CLI to ensure repeatable quality checks and regression detection.

An eval workflow assembles dataset YAMLs and evaluators into automated quality gates. You construct it using evalWorkflow, define evaluators with verify() and Verdict helpers, and organize tests under tests/evals and tests/datasets for reproducible offline validation.

Evaluators using verify() and Verdict helpers define custom quality checks for AI workflows. You implement them with @outputai/evals to assess subjective quality or regression, returning structured verdicts that automate pass/fail decisions across multiple test scenarios.

Yes, offline evaluation testing supports subjective quality assessment with LLM judges. By defining evaluators with verify() and assembling them into eval workflows, you can automate nuanced quality scoring and reproducible validation across diverse datasets.

Dataset YAMLs for regression testing should be organized under tests/datasets. You define test scenarios in these YAML files, pair them with evaluators under tests/evals, and run the assembled eval workflows via the CLI to detect pipeline regressions.

Yes, @outputai/evals is required to build offline tests for Output SDK workflows. It provides the verify() function, Verdict helpers, and evalWorkflow constructor needed to define evaluators, assemble workflows, and run CLI-driven validation.