Compares coding agents on YAML-defined tasks using git worktree isolation and pass-rate metrics.
npx skills add https://github.com/Femad-6/my-skills --skill agent-eval-femad-6
Please help me install this Agent Skill. Skill: agent-eval Source: https://github.com/Femad-6/my-skills/tree/main/.github/skills/agent-eval Command: npx skills add https://github.com/Femad-6/my-skills --skill agent-eval-femad-6
High-intent search queries and answers about installing and using this skill.
Define tasks in YAML files specifying the prompt, target files, and judge criteria, then run each agent multiple times with the agent-eval CLI. The report command produces a table comparing pass rate, cost, time, and consistency.
Create a YAML file with a name, description, repo path, target files, a prompt describing the change, and judge rules such as pytest commands or grep patterns. Pin a specific commit so results stay reproducible over time.
No, agent-eval uses git worktrees instead of Docker. Each agent run gets a fresh worktree created from the pinned commit, providing isolation without container overhead.
Run at least three trials per agent per task because coding agents are non-deterministic. Consistency is measured as pass rate across repeated runs, such as 3 out of 3 passes equaling 100 percent.
Three judge types are supported: deterministic code-based judges like pytest or build commands, pattern-based grep checks on files, and LLM-as-judge prompts. Include at least one deterministic judge per task since LLM judges add noise.
fallow-rsIteratively review and improve Fallow user-facing output against real-world benchmark projects.
Samuelca6399Generate operating budgets, rolling forecasts, and variance analysis for FP&A decisions.
aporto-techDetect web application performance regressions by comparing page load times, Core Web Vitals, and resource sizes.
hamelsmuDesign LLM-as-Judge evaluators for subjective criteria with binary pass/fail outcomes.
