What problem does it solve?
The eval-harness provides a formal, repeatable framework for defining, running, and reporting evaluations for Gemini Code sessions so teams can quantify AI behavior, prevent regressions, and gate releases on objective metrics.
Core Features & Use Cases
- Eval-Driven Development: Define capability and regression evals before implementation to set clear success criteria.
- Multiple Grader Types: Support deterministic code graders, model-based graders for open-ended outputs, and human review flags for high-risk changes.
- Metrics & Reporting: Record pass@k and pass^k statistics, store eval definitions and run histories, and generate eval reports for release decisions.
- Integration Patterns: Create eval artifacts in .gemini/evals, run checks during implementation, and generate consolidated reports for CI and release notes.
- Use Case Example: Add an authentication feature with pre-defined capability checks, run graders to verify behavior, and use pass@k thresholds to decide readiness.
Quick Start
Define an eval for add-authentication, run the eval check to execute graders, and generate the eval report to review pass@k metrics.
