skills.rest
perminder-klair
perminder-klairCOMMUNITY
Explore all skills by this author →
Website𝕏@pinku1GitHub

subwave-picker-benchmark

Benchmark LLM models for tool-calling reliability and latency in the SUB/WAVE DJ picker agent.

1.2k|104|Updated May 11, 2026
Search Tags:#llm#benchmarking#tool calling#latency#radio dj#model reliability#event logs
One-click install
npx skills add https://github.com/perminder-klair/subwave --skill subwave-picker-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: subwave-picker-benchmark
Source: https://github.com/perminder-klair/subwave/tree/main/.claude/skills/subwave-picker-benchmark
Command: npx skills add https://github.com/perminder-klair/subwave --skill subwave-picker-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires python3, node, tsx, docker, curl, and includes scripts (resource) components.

What problem does it solve?

This Skill helps you determine which LLM model will reliably follow the music-picker tool-calling protocol used by the SUB/WAVE DJ picker agent, avoiding failures caused by models that respond with prose instead of tool calls.

Core Features & Use Cases

  • Runs a read-only picker benchmark harness: Executes the bundled controller harness that measures picker outcomes without changing the station’s configured live model.
  • Compares candidate models across modes: Evaluates models in both short and long session modes to catch regressions that only appear under realistic context length.
  • Produces actionable results: Outputs per-model success rate and latency (median/p95) and summarizes failure reasons from the controller’s event log so operators can choose a better-fit model.

Use case examples: assessing a new provider/model before putting it on air; diagnosing slow or failing djAgentPick behavior suspected to be caused by model choice; benchmarking multiple Ollama or cloud models to find the best tool-following reliability.

Quick Start

Run the benchmark harness for your chosen provider and one or more models inside the running controller container to generate a comparison table and failure-rate summary.

Frequently Asked Questions about subwave-picker-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM tool calling reliability for my DJ track picker?▼

You can benchmark LLM tool calling reliability by running a read-only picker harness that measures success rates and latency without altering your live station configuration. This evaluates whether candidate models follow the protocol or fail by returning prose.

What is the best way to compare LLM latency and model reliability for a music picker agent?▼

Comparing LLM latency and model reliability involves testing candidate models across short and long session modes to catch regressions under realistic context lengths. The benchmark outputs median and p95 latency alongside per-model success rates.

How do I diagnose why my djAgentPick tool loop is failing or returning prose?▼

Diagnosing djAgentPick failures requires running the bundled controller harness to execute the picker test and summarize failure reasons from the controller event log, helping you identify if the model choice is causing the tool-calling failures.

Do I need Docker to run the LLM picker benchmarking harness?▼

Yes, you need Docker to run the LLM picker benchmarking harness. The process requires executing the picker-test script inside the running sub-wave-controller container, alongside dependencies like Python3, Node, tsx, and curl.

Can I test multiple Ollama models without changing my live station's configured model?▼

Yes, you can test multiple Ollama or cloud models without changing your live station's configured model. The benchmark overrides the provider and model only within the test harness, leaving the live station configuration completely unchanged.

What limitations should I expect when benchmarking tool-calling protocols across different LLMs?▼

Limitations when benchmarking tool-calling protocols include models that respond with prose instead of tool calls, regressions that only appear in long session modes, and latency variations that require median and p95 measurements to accurately assess performance.

Related Skills

Explore More Data & Analytics Skills→
OpenHandsOpenHands
134

docker

Start a Docker daemon and run containers inside containerized environments.

Official
Advanced
rongarederongarede
4

collaborating-with-codex

Delegate coding tasks to Codex CLI via a Python bridge with JSON outputs.

Community
Intermediate
p-possp-poss
26

now-playing

Fetch current track details, Strudel code, and commentary from dj-claude MCP sessions.

Community
Basic
Dianshu-LiaoDianshu-Liao
2

llm-evaluation

Implement LLM evaluation with automated metrics and LLM-as-judge patterns.

Community
Advanced