eval-harness

Automate eval workflows for Gemini CLI by defining capability and regression evals, running deterministic or model-based graders, and generating evaluation reports with pass@k metrics across code changes.

Eval-driven testing formalizes software quality by codifying capability expectations into eval definitions, running graders against changes, and comparing pass@k metrics against baselines to catch regressions in CLI behavior.

Yes, Gemini CLI regression testing supports code-based graders, model-based graders, and human reviews simultaneously to assess changes and generate comprehensive evaluation reports.

Eval definitions for Gemini CLI testing workflows should be stored under the .gemini/evals directory to maintain structure and ensure consistent evaluation report generation across code changes.

Yes, Gemini CLI testing outputs pass@k metrics and tracks baselines in evaluation reports, allowing you to measure reliability and detect regressions when evaluating code changes end-to-end.

Set up regression evals for Gemini CLI development by defining capability expectations in .gemini/evals, configuring deterministic graders, and running automated workflows to validate changes end-to-end before deployment.