agent-eval

Compare coding agents by running them against identical tasks within isolated git worktrees to measure pass rate, cost, and latency. This systematic benchmarking replaces subjective feelings with hard metrics for model selection.

Run head-to-head comparisons between coding agents like Claude Code and Aider by executing them against the same custom YAML task definitions over multiple iterations. This directly measures their success rates, costs, and execution time.

Coding agent benchmarking evaluates results using deterministic criteria like pytest test suites and grep commands, alongside model-based LLM judges. You define these validation methods within custom YAML task definitions to verify agent output.

Yes, you need Python and git to run coding agent benchmarks. Python manages task execution while git provides the isolated worktree environments required to test multiple agents safely without interfering with your main repository.

Isolated git worktrees provide separate environments for coding agent evaluation, preventing file conflicts when running multiple agents against the same repository. This ensures reproducible performance data for accurate comparative analysis.