skills.rest
perminder-klair
perminder-klairCOMMUNITY
Explore all skills by this author →
Website𝕏@pinku1GitHub

subwave-llm-bench

Benchmarks LLM provider and model combinations against SUB/WAVE's on-air call kinds.

1.3k|113|Updated May 11, 2026
Search Tags:#benchmark#model evaluation#OpenRouter#ollama#lLM#Structured Output#ai radio
One-click install
npx skills add https://github.com/perminder-klair/subwave --skill subwave-llm-bench
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: subwave-llm-bench
Source: https://github.com/perminder-klair/subwave/tree/main/.claude/skills/subwave-llm-bench
Command: npx skills add https://github.com/perminder-klair/subwave --skill subwave-llm-bench

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Choosing an LLM for a live AI radio station is risky: a model that works through one provider can fail through another, and weak structured output or slow latency can break on-air calls. This Skill measures how candidate provider+model combinations handle the station's real LLM calls before trusting them on air.

Core Features & Use Cases

  • Matrix benchmarking: Runs every on-air call kind (track picks, segments, request matching, DJ scripts, banter, programme beats) in both candidate-pool and agent modes via the llm-bench harness, scoring reliability, rule violations, and p50 latency per model.
  • Legacy picker deep-dive: Uses picker-test.mjs and the bundled assess-models.sh script for high-iteration picker-only runs comparable with historical results, with automatic dev/prod detection and Ollama model discovery.
  • Recommendation reports: Produces a comparison table and a clear provider+model+mode recommendation, with a failure-mode glossary (hallucinated-id, no-object-generated, timeout) to explain disqualifiers.
  • Use Case: An operator asks "which Ollama model is good enough to run the DJ?" — the Skill runs the benchmark matrix, reads the JSON report, and recommends the best provider+model+mode without touching the live station configuration.

Quick Start

Ask the assistant to benchmark candidate models for the station, for example: run llm-bench comparing openrouter:google/gemma-4-31b-it and ollama:qwen3:8b with 3 iterations and recommend which one to deploy.

Frequently Asked Questions about subwave-llm-bench

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM models for the SUB/WAVE radio station?▼

Run npm run llm-bench from the controller directory with --models set to a comma list of provider:model pairs and --iterations for runs per scenario. The harness scores every on-air call kind and writes a JSON report with pass rates and latency.

Which LLM model should I use for the AI DJ picker?▼

Benchmark candidates through the exact provider routing you will deploy, then judge on thrown rate first, rule violations second, and p50 latency third. The same model can pass via OpenRouter and fail via a direct provider, so provider+model is the unit to evaluate.

Does the benchmark change the live station's configured model?▼

No. Both harnesses override provider and model only inside their own short-lived process, so the live controller's configured model is never touched. Changing the live model is a separate admin Settings action.

Why does a model fail with no-object-generated or hallucinated-id errors?▼

no-object-generated means the model cannot produce the required structured output schema, while hallucinated-id means it invents track ids. Both indicate weak structured output; check provider routing before rejecting the model entirely.

Can I benchmark Ollama models without naming each one?▼

Yes. The assess-models.sh script auto-discovers every model installed on the Ollama box when invoked with the ollama provider and no model list, then tests each in short and long modes and prints a comparison table.

When should I use picker-test.mjs instead of llm-bench?▼

Use picker-test.mjs for picker-only, high-iteration runs that need to be comparable with historical results. Use llm-bench for full coverage of all on-air call kinds including segments, scripts, banter, and programme beats.

Related Skills

Explore More Software Engineering Skills→
perminder-klairperminder-klair
1.2k

subwave-picker-benchmark

Benchmark LLM models for tool-calling reliability and latency in the SUB/WAVE DJ picker agent.

Community
Intermediate
armanisadeghiarmanisadeghi
2

admin-debug-context

Wire Next.js/React admin routes to a debug context panel with live console error capture.

Community
Intermediate
844196844196
8

analyze-session

Analyze Claude Code session logs to extract tool usage events and recovery patterns.

Community
Advanced
MesferAliMesferAli

evaluating-llms-harness

Evaluate LLMs across 60+ academic benchmarks using lm-evaluation-harness.

Community
Advanced