skills.rest
ruvnet
ruvnetCOMMUNITY
Explore all skills by this author →
Website𝕏@ruvGitHub

cost-benchmark

Runs corpus benchmarks comparing booster against Gemini and Anthropic baselines and persists measured results.

70.1k|8.4k|Updated Jun 2, 2025
Search Tags:#verification#benchmark#cost tracking#anthropic#lLM#GEMINI#win rate
One-click install
npx skills add https://github.com/ruvnet/claude-flow --skill cost-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: cost-benchmark
Source: https://github.com/ruvnet/claude-flow/tree/main/plugins/ruflo-cost-tracker/skills/cost-benchmark
Command: npx skills add https://github.com/ruvnet/claude-flow --skill cost-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Claims about cost savings and routing accuracy are unverifiable without reproducible measurements. This Skill runs a benchmark suite against a structural and adversarial corpus, comparing the local booster against Gemini and Anthropic baselines, and persists a verifiable measured-vs-claimed record.

Core Features & Use Cases

  • Corpus Benchmarking: Executes scripts/bench.mjs against the booster corpus, reporting per-case and summary results with win rate and escalation rate metrics.
  • Optional LLM Baselines: Adds Gemini 2.0 Flash via BENCH_LLM_BASELINE=1 and Sonnet 4.6 / Opus 4.7 via BENCH_ANTHROPIC=1 for cost and quality comparisons.
  • Persisted Run History: Writes timestamped JSON results plus a latest.json pointer to docs/benchmarks/runs/ for downstream skills like cost-report.
  • Use Case: Before publishing a release, run the benchmark to confirm the booster win rate stays at or above 0.80 on Tier 1 cases and flip claimed metrics to verified.

Quick Start

Run the cost benchmark from the v3 directory with the booster only, then inspect the printed markdown summary and the persisted results in docs/benchmarks/runs/latest.json.

Frequently Asked Questions about cost-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run the cost benchmark for the booster?▼

Run `node ../plugins/ruflo-cost-tracker/scripts/bench.mjs` from the v3 directory. The booster-only run is free and takes about 85 ms, printing a markdown summary with win rate and escalation rate.

How do I benchmark against Gemini or Claude baselines?▼

Set BENCH_LLM_BASELINE=1 to add the Gemini 2.0 Flash baseline, and BENCH_ANTHROPIC=1 to add Sonnet 4.6 and Opus 4.7. API keys are pulled from gcloud secrets or overridden with BENCH_LLM_API_KEY and BENCH_ANTHROPIC_API_KEY.

Where are benchmark results stored?▼

Results are written to docs/benchmarks/runs/ as a timestamped JSON file per run, plus a latest.json pointer to the most recent run. Downstream skills like cost-report read latest.json for tier-spend numbers.

What win rate threshold does the benchmark enforce?▼

The smoke gate requires a win rate of at least 0.80 on Tier 1 cases, configurable in scripts/smoke.sh. The escalation rate on adversarial cases is reported but not gated.

Can I override the benchmark models or output path?▼

Yes. Use BENCH_LLM_MODEL and BENCH_LLM_BASE_URL for the OpenAI-compatible baseline, BENCH_ANTHROPIC_MODELS for Claude model IDs, and BENCH_OUT to change the output file path.

Related Skills

Explore More Software Engineering Skills→
DiologIRDiologIR

bench-corpus-review

Re-run whole-corpus quality reviews of diolog-swe-bench and diff against previous reviews.

Official
Advanced
mhdxbilalmhdxbilal
1

skill-cost-projections

Compute remaining workflow cost projections from per-phase averages.

Community
Advanced
omeraltnomeraltn

agency-autonomous-optimization-architect

Route multi-provider LLM and scraping API calls with cost limits and circuit breakers.

Community
Advanced
choice5346choice5346

nemo-evaluator-sdk

Evaluate LLMs across 100+ benchmarks using Docker, Slurm, or cloud backends.

Community
Advanced