Compare coding agents on custom YAML-defined tasks in isolated git worktrees.
npx skills add https://github.com/zero3041/PREP --skill agent-eval-zero3041
Please help me install this Agent Skill. Skill: agent-eval Source: https://github.com/zero3041/PREP/tree/main/.claude/skills/skills/agent-eval Command: npx skills add https://github.com/zero3041/PREP --skill agent-eval-zero3041
The agent-eval Skill addresses the challenge of comparing coding agents' performance across various tasks, providing metrics on pass rate, cost, time, and consistency.
Run the task 'add-retry-logic' against the agents 'claude-code' and 'aider' with 3 trials using the command: agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3
High-intent search queries and answers about installing and using this skill.
You can compare coding agents by defining tasks in YAML, executing them in isolated git worktrees, and measuring performance metrics like pass rate, cost, time, and consistency for accurate head-to-head evaluation.
You define custom tasks for performance comparison using YAML files that specify the required actions, target files to touch, and the specific criteria for success to evaluate the coding agents accurately.
To ensure reproducibility when benchmarking coding agents, the evaluation system isolates each agent run in a fresh git worktree, preventing cross-contamination and keeping performance metrics accurate.
Yes, the system supports head-to-head comparison of coding agents like Claude Code and Aider, allowing you to run multiple trials against custom YAML tasks to measure their pass rate and cost.
When running coding agents against YAML tasks, the system tracks performance metrics including pass rate, execution cost, completion time, and consistency across multiple isolated runs.
You execute multiple trials by running the command with your YAML task file, specifying the target agents, and setting the desired number of runs to measure consistency and pass rate accurately.
visualityplCreate a Git worktree and launch an AI implementation session for a development plan.
hieuckAutomate head-to-head evaluation of coding agents on reproducible YAML-defined tasks.
Jurel89Coordinate N worker agents executing tasks in isolated git worktrees with back-pressure control.
muratcankoylanEvaluate agent performance across configurations and sessions with multi-dimensional rubrics.
