agent-eval

Compare coding agents on reproducible tasks by defining YAML task definitions, evaluating them in isolated git worktrees, and reporting pass rate, cost, time, and consistency metrics. This approach ensures agents do not interfere with each other or the base repository during evaluation.

Benchmark LLM coding agents by running multiple trials against unit-test-backed tasks using deterministic judges like pytest. The evaluation collects structured metrics including pass rate, wall-clock time, and cost to produce a side-by-side comparison report.

Yes, agent evaluation supports deterministic judges such as pytest and command executions, pattern-based checks using grep, and optional LLM judging to verify task completion. These judges collect structured metrics for each run to ensure comprehensive evaluation.

Prevent interference during benchmark runs by creating per-run isolated git worktrees for each agent. Combined with pinned commits in YAML task definitions, this guarantees reproducible experiments without agents modifying the base repository.

Yes, reproducible experiments are achieved by defining tasks and pinned commits in YAML task definitions. By applying these definitions to isolated git worktrees across multiple runs per agent, the benchmark yields consistent and comparable metrics.