eval-harness

pass@k and pass^k are metrics for measuring AI model reliability. pass@k tracks the probability of at least one successful generation in k attempts, while pass^k measures the variance across multiple runs to track regressions over time.

This evaluation framework supports code-based, model-based, and human graders for comprehensive evaluation. These grader types allow you to evaluate AI model performance across different capabilities and regression scenarios during pre-implementation and post-implementation workflows.

The framework includes integration patterns for pre-implementation, during implementation, and post-implementation evaluation workflows. This ensures AI model performance is tracked reliably and regressions are caught across the entire development lifecycle.

A team developing a code generation AI can use this framework to define expected code quality, ensure new features don't break existing ones, and track the model's reliability over time using structured capability and regression evaluations.

The framework integrates into pre-implementation, during implementation, and post-implementation workflows. It enables teams to define expected AI behavior upfront and continuously evaluate model performance against those criteria throughout development.

Code-based, model-based, and human graders serve different evaluation needs. Code-based graders provide automated quantitative checks, model-based graders use AI to evaluate outputs, and human graders offer qualitative assessment for complex AI capabilities.