skills.rest
reidemeister94
reidemeister94COMMUNITY
Explore all skills by this author →
𝕏@silviopavanettoGitHub

ai-agent-bench

Benchmark AI agents on coding tasks using git worktrees and metrics.

10|1|Updated Mar 19, 2026
Search Tags:#workflow#transcript#diff#benchmark#ai-agent#anomaly#coding-task
One-click install
npx skills add https://github.com/reidemeister94/development-skills --skill ai-agent-bench
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ai-agent-bench
Source: https://github.com/reidemeister94/development-skills/tree/main/skills/ai-agent-bench
Command: npx skills add https://github.com/reidemeister94/development-skills --skill ai-agent-bench

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables quantitative benchmarking of AI agents (Claude Code, Codex, OpenCode) on real coding tasks within the current repository, so you can compare performance, accuracy, and behavior.

Core Features & Use Cases

  • Orchestrated agent trials: isolated worktrees, baseline checks, agent execution, post-check, and artifact collection.
  • Rich telemetry: transcripts, diffs, timings, and resulting metrics for cross-agent comparison.
  • Real-world scenarios: refactoring tasks, performance experiments, and agent evaluation across diverse codebases.

Quick Start

Run the ai-agent-bench workflow on a repository with a task prompt to benchmark and compare agents.

Frequently Asked Questions about ai-agent-bench

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark AI coding agents on a real repository?▼

You can benchmark AI coding agents by running an orchestrated workflow that isolates runs using git worktrees, executes baseline checks, runs task prompts, and collects transcripts with diffs for comparison.

Can I compare Claude Code and Codex performance on the same coding task?▼

Yes, you can compare Claude Code, Codex, and OpenCode by executing the same structured task prompt across isolated worktrees and reviewing the resulting metrics, diffs, and timings.

What do I need to run an AI agent benchmark workflow?▼

You need a target repository, a structured task prompt, and a .agent-bench.toml configuration file to define and execute the benchmarking workflow successfully.

How are AI agent benchmarking results collected and reported?▼

Benchmark results are collected as transcripts, diffs, and timings under eval-results directories, alongside a metrics.json file and a human-readable report for cross-agent comparison.

Does benchmarking AI agents modify my current working directory?▼

No, the workflow isolates agent runs using separate git worktrees to ensure your current repository state remains untouched while executing and evaluating coding tasks.

What is the best way to evaluate AI agent efficiency on refactoring tasks?▼

The best way is to run an orchestrated trial that performs baseline outer checks, executes the refactoring prompt, and quantifies efficiency through collected timings and code diffs.

Related Skills

Explore More Software Engineering Skills→
EpiphyticEpiphytic
7

worktree

Create isolated git worktrees and structured prompts for parallel agent tasks.

Official
Intermediate
ClawBioClawBio
1.1k

profile-report

Generates unified genomic profile reports from uploaded JSON data.

Official
Intermediate
RamblurrRamblurr

coding-agents

Orchestrate external AI coding agents for task delegation and parallel processing.

Community
Intermediate
0xharryriddle0xharryriddle
3

evaluation

Assess agent performance with weighted rubrics and pass/fail thresholds.

Community
Advanced