eval-harness

AI regression testing requires defining capability and regression evals with structured pass/fail criteria. You can define an eval with a feature name, run checks across model versions and prompts, and generate evaluation reports to detect regressions.

Eval-driven development is a formal framework to define, implement, and evaluate tests for AI sessions. It applies to building capability and regression evals by enforcing a structured workflow with phases: define, implement, evaluate, and report.

Yes, AI evals support both deterministic code-based graders and model-based graders. You can apply these grader types within the evaluate phase to assess outputs against defined pass/fail criteria and baselines.

Evaluating prompts across model versions involves storing baselines and running regression checks. The framework supports generating evaluation reports across model versions and prompts, applying deterministic or model-based graders to compare results against stored baselines.

No external dependencies are required to run eval-driven development tests. The framework operates independently to define, implement, evaluate, and report on AI capability and regression evals within your project eval store.