agent-eval

You can evaluate coding agent performance by running head-to-head comparisons on custom tasks. The tool uses declarative task definitions to measure pass rate, cost, time, and consistency metrics, helping you determine which agent is best suited for your specific requirements.

Custom tasks for benchmarking coding agents are defined declaratively. You specify what the agent needs to do, which files to touch, and the criteria for success within a YAML file. This ensures consistent evaluation across multiple agents and runs.

To prevent interference when running multiple coding agent evaluations, each run is automatically isolated within a git worktree. This ensures that file modifications and environment changes made by one agent do not impact the results of another during comparative benchmarking.

Yes, you can compare Claude Code and Codex on the same code review task. By specifying multiple agents in the run command, the tool evaluates both on your custom task and provides comparative metrics like pass rate, cost, time, and consistency to aid your tool selection.

Evaluating coding agent performance uses pass rate, cost, time, and consistency as primary metrics. These quantitative measures provide a comprehensive view of agent effectiveness, efficiency, and reliability when completing custom coding tasks.

Yes, agent-eval works without external dependencies. The tool requires no additional packages or modules to function, allowing you to directly run comparative analysis on coding agents within your existing development environment.