skills.rest
tgmarinho
tgmarinhoCOMMUNITY
Explore all skills by this author →
Website𝕏@tgmarinhoGitHub

benchmark-models

Runs the same prompt through Claude, GPT, and Gemini to compare latency, cost, and quality.

Updated May 15, 2026
Search Tags:#llm evaluation#claude#model comparison#latency#benchmark#gemini#gpt
One-click install
npx skills add https://github.com/tgmarinho/canetaco --skill benchmark-models-tgmarinho
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-models
Source: https://github.com/tgmarinho/canetaco/tree/main/.claude/skills/benchmark-models
Command: npx skills add https://github.com/tgmarinho/canetaco --skill benchmark-models-tgmarinho

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing between AI models for a given task is usually guesswork. This Skill replaces vibes with data by running the same prompt through Claude, GPT (via Codex CLI), and Gemini side-by-side, then comparing latency, token usage, cost, and optionally output quality via an LLM judge. ## Core Features & Use Cases - Cross-model execution: Sends one prompt to multiple providers and reports per-model latency, tokens, and cost in a comparison table. - Optional quality judge: Uses Anthropic's Claude to score each model's output on a 0-10 scale for a small additional cost. - Auth preflight: A dry-run shows which providers are authenticated before any paid API calls happen, skipping unauthed providers cleanly. - Result persistence: Saves benchmark runs as dated JSON files so you can detect quality drift as providers update their models. - Use Case: You want to know which model handles your /review skill best. Run the benchmark against that skill's SKILL.md, enable the judge, and get a data-backed answer on speed, cost, and quality. ## Quick Start Ask the assistant to benchmark your prompt across Claude, GPT, and Gemini and show which model is fastest, cheapest, and highest quality.

Frequently Asked Questions about benchmark-models

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare Claude, GPT, and Gemini on the same prompt?▼

Run the benchmark with a prompt file or inline text and it executes the prompt through each provider's CLI, then prints a table comparing latency, tokens, and cost. Enable the judge flag to also score output quality on a 0-10 scale.

How do I benchmark an AI model against my own skill or prompt file?▼

Point the benchmark at a SKILL.md file or any prompt file on disk, or pass the prompt inline. The workflow lets you pick from discovered skills or supply your own path, then runs it across the providers you select.

What happens if a model provider is not authenticated?▼

A dry-run preflight shows each provider's auth status before any paid calls. Unauthenticated providers are skipped cleanly with a remediation hint, and the benchmark stops entirely if zero providers are authed.

How much does the LLM quality judge add to benchmark cost?▼

The judge adds roughly $0.05 per run and uses Anthropic's Claude to score each model's output on a 0-10 scale. It is opt-in only and never included automatically.

Can I track model performance changes over time?▼

Yes. Save benchmark results as dated JSON files under ~/.gstack/benchmarks/ and diff future runs against them. This catches quality regressions as providers update their underlying models.

Related Skills

Explore More Software Engineering Skills→
cheng80cheng80
1

benchmark-models

Benchmark AI models across Claude, GPT, and Gemini for latency, token usage, and cost.

Community
Advanced
ruvnetruvnet
70.1k

cost-trend

Analyzes persisted benchmark run JSON files to detect drift in win rate, latency, and cost.

Community
Basic
Exia-thdExia-thd

ai-engineer

Build and deploy production-grade AI/ML systems with MLOps pipelines.

Community
Advanced
Ar9avAr9av
635

paper-autoraters

Compute quality metrics and side-by-side judgments for research paper drafts.

Community
Advanced