agent-eval

To benchmark coding agents with reproducible trials, you run head-to-head comparisons on a defined task set using git worktrees for environment isolation. This yields a structured report with metrics like pass rate, cost, and time.

Comparing coding assistants on standardized tasks requires a YAML task format and automated evaluation. This approach generates objective comparison results using metrics like pass rate, cost, time, and consistency across multiple agents.

You run head-to-head coding agent comparisons by evaluating multiple agents on a defined set of tasks with standardized judges. The process produces a structured performance report highlighting pass rate, cost, time, and consistency.

Yes, agent-eval supports regression checks after model updates by running reproducible experiments. Using git worktrees to isolate runs prevents environment drift, ensuring consistent and reliable evaluation results for tool selection.

Yes, a YAML task format is required to define the tasks for evaluating multiple coding agents. This structured input enables standardized judges to accurately measure pass rate, cost, time, and consistency across reproducible runs.