Compare coding agents across YAML-defined tasks using git worktree isolation.
npx skills add https://github.com/cescrafli/compyrasion --skill agent-eval-cescrafli
Please help me install this Agent Skill. Skill: agent-eval Source: https://github.com/cescrafli/compyrasion/tree/main/skills/agent-eval Command: npx skills add https://github.com/cescrafli/compyrasion --skill agent-eval-cescrafli
Head-to-head comparison of coding agents on reproducible tasks to generate data-backed insights on performance, cost, time, and reliability.
Run agent-eval with a defined tasks YAML and compare results across multiple agents.
High-intent search queries and answers about installing and using this skill.
Head-to-head comparison of coding agents on reproducible tasks requires YAML task definitions and deterministic judges. This process measures pass rate, cost, time, and consistency to reveal performance patterns across agents.
Benchmarking coding agents requires YAML task definitions that declaratively describe tasks and judging criteria. You also need per-task git worktree isolation to ensure reproducible agent runs and deterministic judges for fair comparisons.
Git worktree isolation ensures fair agent comparisons by creating separate working directories for each agent run. This per-task isolation prevents cross-contamination between tasks and guarantees that every agent operates under identical conditions.
Yes, you can use pattern-based judges alongside deterministic judges for multi-agent benchmarking. These judges evaluate task completion based on the criteria defined in your YAML task definitions to generate performance reports.
Coding agent comparison reports summarize pass rate, cost, time, and consistency across agents. These metrics provide data-backed insights into the performance and reliability of different coding agents on the evaluated tasks.
To define tasks for evaluating coding agents, you create YAML task definitions that declaratively describe the tasks and judging criteria. These definitions are then applied to reproducible tasks across codebases for multi-agent benchmarking.
EhraxCreates isolated git worktrees for feature work with verified ignore rules and project setup.
eduardoarantesManage git worktrees with automatic port isolation and per-worktree configuration.
blakesimsExecute implementation plan phases with tests and atomic commits.
yanochka11Benchmark large language models with lm-evaluation-harness across standardized metrics.
