skills.rest
greglas75
greglas75COMMUNITY
Explore all skills by this author →
GitHub

benchmark

Benchmarks AI coding providers in parallel and ranks them with a Claude meta-judge.

6|Updated Mar 27, 2026
Search Tags:#code generation#claude#benchmark#ai coding#leaderboard#meta-judge#provider comparison
One-click install
npx skills add https://github.com/greglas75/zuvo --skill benchmark-greglas75
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark
Source: https://github.com/greglas75/zuvo/tree/main/skills/benchmark
Command: npx skills add https://github.com/greglas75/zuvo --skill benchmark-greglas75

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Comparing AI coding assistants like Codex, Gemini, Claude, and Cursor-Agent on real tasks is slow and subjective. This Skill dispatches one task to all providers in parallel, scores responses with an opposite-model Claude meta-judge, and produces a ranked leaderboard with quality, cost, time, and self-eval bias metrics. ## Core Features & Use Cases - Parallel Multi-Provider Dispatch: Sends the same task to Codex, Gemini, Claude, and Cursor-Agent simultaneously and collects responses with cost and timing data. - Meta-Judge Scoring: Uses an opposite-model Claude judge to score completeness, accuracy, actionability, and hallucination-freedom, with shuffled presentation order to reduce bias. - Corpus Mode: Runs fixed OrderService and useSearchProducts tasks so results are comparable across runs over time, with optional test-writing and adversarial cross-review rounds. - Use Case: Run a benchmark in corpus mode each week to track whether a provider's code quality is improving, then compare two historical runs with the --compare flag to see quality, time, and cost deltas. ## Quick Start Ask the AI to run the benchmark skill in corpus mode with tests enabled to compare all available AI coding providers on the fixed tasks.

Frequently Asked Questions about benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark AI coding assistants against each other?▼

Dispatch the same coding task to Codex, Gemini, Claude, and Cursor-Agent in parallel, then score responses with a Claude meta-judge on completeness, accuracy, actionability, and hallucination-freedom. The skill outputs a ranked leaderboard with quality, time, and cost per provider.

How to compare AI code generation quality across runs over time?▼

Use corpus mode, which runs fixed OrderService and useSearchProducts tasks with identical prompts every run. Because the task never changes, leaderboard scores from different days are directly comparable, and the --compare flag shows deltas between two stored runs.

Which AI coding providers does the benchmark support?▼

The benchmark dispatches to Codex, Gemini, Claude, and Cursor-Agent, depending on which provider CLIs are installed. You can restrict a run to specific providers with the --provider flag using a comma-separated list.

Can the benchmark also evaluate generated tests and adversarial robustness?▼

Yes. The --with-tests flag runs a second round where providers write tests for their own code, and --with-adversarial runs cross-provider critiques of the code. Adversarial deltas and test composites are included in the corpus-mode quality score.

Why does the benchmark use a different model as the judge?▼

The opposite-model rule reduces self-serving bias: if the skill runs on Opus it judges with Sonnet, and vice versa. Responses are also shuffled and label-redacted before judging to reduce positional and identity bias.

Related Skills

Explore More Software Engineering Skills→
agent0aiagent0ai
19.1k

orchestrator

Delegates coding tasks to external terminal coding agents via shell execution.

Official
Advanced
shogo-labsshogo-labs
12

benchmark

Run browser-based performance benchmarks against a target URL and compare baselines.

Official
Advanced
Rx-K8Rx-K8

verification-loop-typescript

Validates builds, runs tsc/ESLint/tests with coverage, scans vulnerabilities, reviews diffs for TypeScript projects.

Community
Advanced
AlphaLab-USTCAlphaLab-USTC
131

bench

Run OhMyCode coding benchmarks and record pass-fail outcomes with token usage.

Community
Intermediate