agent-eval

Benchmark coding agents by defining tasks in a YAML directory and running the evaluation tool to generate a comparison report. It uses per-task git worktrees for isolated runs and deterministic judges to measure pass rate, cost, and time.

Yes, you can compare Claude Code, Aider, and Codex using built-in metrics like pass rate, cost, time, and consistency. The tool applies deterministic judges across reproducible tasks to ensure fair comparisons between different coding assistants.

Run reproducible tests for AI coding assistants by defining declarative task definitions that specify repo state, touched files, and judging criteria. The tool creates per-task git worktrees to ensure isolated, reproducible agent runs.

Evaluate coding agent performance before switching tools by running reproducible agent comparisons on your codebase. The tool extracts actionable insights from standardized tasks with cost-aware metrics to guide tool selection and adoption.

Define tasks for evaluating coding agents by creating a tasks directory with YAML task definitions. These declarative definitions specify the repo state, touched files, and judging criteria needed for deterministic evaluation.

Yes, benchmarking coding agents requires isolated git worktrees to ensure reproducible agent runs. The tool creates per-task worktrees so each agent operates in a clean, isolated environment for fair comparison.