agent-eval

Benchmarking coding agents on reproducible tasks requires per-task YAML definitions and git worktrees to run repeatable multi-agent trials. The framework collects pass rate, cost, time, and consistency metrics to compile objective comparison reports.

Running head-to-head coding agent comparisons captures pass rate, cost, time, and consistency metrics across multiple trials. The framework compiles these metrics into reports to surface actionable insights for tool selection.

Yes, benchmarking coding agents for reproducible runs requires git worktrees and pinned task definitions to ensure repeatable experiments. This mechanism isolates agent environments to guarantee standardized task conditions across trials.

Yes, you can compare Claude, Codex, and Aider on real sprint tasks to select the best tool. The framework supports multi-agent runs with deterministic or model-based judges to measure regression performance and guide selection.

Configuring task definitions for coding agent evaluations involves creating YAML files under the tasks/ directory. These pinned definitions standardize evaluation criteria for deterministic or model-based judges across multi-agent runs.