skills.rest
cirne
cirneCOMMUNITY
Explore all skills by this author →
Website𝕏@sweetlewGitHub

model-eval-benchmark

Evaluate brain-app JSONL agent tasks across models to produce comparable reports.

Updated Aug 27, 2026
Search Tags:#llm#latency#benchmark#tokens#model#cost#evaluate
One-click install
npx skills add https://github.com/cirne/brain-app --skill model-eval-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-eval-benchmark
Source: https://github.com/cirne/brain-app/tree/main/.cursor/skills/model-eval-benchmark
Command: npx skills add https://github.com/cirne/brain-app --skill model-eval-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Runs brain-app JSONL agent evals (Enron v1, wiki v1) and compares model runs on pass rate, wall time, tokens, and estimated cost from report JSON. Use when benchmarking LLMs, comparing providers, price/performance analysis, or after changing supported-llm-models.json / LLM_PROVIDER + LLM_MODEL.

Core Features & Use Cases

  • Consistent eval harness across models with identical tasks and environments.
  • Detailed metrics: pass rate, wall time, tokens, and cost from aggregated usage.
  • Use Case: Benchmark new models or providers to decide on deployments or pricing.

Quick Start

Run the full evaluation with your chosen provider and model to generate a comparison report.

Frequently Asked Questions about model-eval-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM models to compare pass rate and cost?▼

You can benchmark LLMs by running brain-app JSONL agent tasks across multiple providers and model IDs to measure pass rate, wall time, tokens, and estimated cost, producing comparable JSON reports in data-eval/eval-runs.

What metrics are included when evaluating LLM performance across providers?▼

Evaluating LLM performance captures pass rate, wall time, token counts, and estimated cost from aggregated usage data, compiling these metrics into JSON reports stored in data-eval/eval-runs.

Do I need specific environment setup to run LLM benchmarks?▼

Running LLM benchmarks requires Node, npm scripts, and provider API keys configured, plus resolving supported-llm-models.json and LLM_PROVIDER/LLM_MODEL mappings before execution.

What tasks are used for consistent LLM evaluation harnessing?▼

LLM evaluation harnessing uses Enron v1 and wiki v1 brain-app JSONL agent tasks, running identical tasks across different models to ensure fair, comparable benchmark reporting.

When should I use a repeatable LLM evaluation harness?▼

Use a repeatable LLM evaluation harness when benchmarking new models, comparing providers, analyzing price versus performance, or after changing supported-llm-models.json and LLM_PROVIDER plus LLM_MODEL configurations.

Related Skills

Explore More Data & Analytics Skills→
midsphere-aimidsphere-ai
1

exo:models

Configure LLM providers and models for Exo agents via ModelConfig.

Official
Advanced
Michae2xlMichae2xl

agent-eval

Compare coding agents on custom tasks using pass rate, cost, time, and consistency metrics.

Community
Advanced
roanbrasilroanbrasil
5

llm-evals

Implement LLM evaluation patterns with LangSmith, RAGAS, and PromptFoo.

Community
Advanced
MicrockMicrock
322

Convex Agents Usage Tracking

Track LLM token usage and calculate costs for billing reports.

Community
Advanced