eval-harness

Pass@k metrics quantify the success probability of software features across multiple test attempts. Used within eval-driven testing workflows, these metrics validate whether API endpoints, UI components, and business logic maintain correctness and stability across iterative development cycles.

Define your capability and regression tests using structured YAML-formatted definitions that specify pass criteria. Apply these definitions to development workflows to validate features, generating integrated evaluation reports and trend metrics for CI/CD pipelines.

Yes, capability tests validate new features across API endpoints, UI components, and business logic. By applying eval-driven tests with defined pass criteria, you can prove features are correct across iterations, satisfying requirements for structured evals and integrated evaluation reports.

Generate integrated evaluation reports and trend metrics from YAML-formatted regression tests. Running eval-driven tests within your development workflow ensures existing functionality remains stable across iterations while providing structured trend metrics to track pass@k success probabilities over time.

No specific testing frameworks are required as dependencies. The eval-driven approach operates using YAML-formatted test definitions with defined pass criteria, allowing you to quantify software quality and generate evaluation reports directly within your existing development workflows without external framework constraints.