agent-eval

To compare coding agents on custom programming tasks, you define reproducible YAML task files specifying prompts, files, and validation criteria, then run the agents in isolated Git worktrees to collect metrics like pass rate, cost, and execution time.

When evaluating coding agents on custom tasks, the metrics tracked include pass rate, cost, execution time, and consistency, providing a quantitative basis for comparing tools like Claude Code, Aider, and Codex.

Yes, you can isolate each coding agent run in its own Git worktree, which ensures clean and repeatable execution by preventing file conflicts and maintaining a consistent environment for every evaluation.

You define reproducible programming tasks for benchmarking AI coding tools using YAML task definitions, where you specify the prompts, target files, and validation criteria required for a consistent evaluation setup.

The best way to benchmark coding assistants for teams is using a systematic evaluation tool that runs reproducible YAML tasks in isolated Git worktrees, generating quantitative comparative reports on pass rate, cost, and execution time.