subwave-picker-benchmark

What problem does it solve?

This Skill helps you determine which LLM model will reliably follow the music-picker tool-calling protocol used by the SUB/WAVE DJ picker agent, avoiding failures caused by models that respond with prose instead of tool calls.

Core Features & Use Cases

• Runs a read-only picker benchmark harness: Executes the bundled controller harness that measures picker outcomes without changing the station’s configured live model.
• Compares candidate models across modes: Evaluates models in both short and long session modes to catch regressions that only appear under realistic context length.
• Produces actionable results: Outputs per-model success rate and latency (median/p95) and summarizes failure reasons from the controller’s event log so operators can choose a better-fit model.

Use case examples: assessing a new provider/model before putting it on air; diagnosing slow or failing djAgentPick behavior suspected to be caused by model choice; benchmarking multiple Ollama or cloud models to find the best tool-following reliability.

Quick Start

Run the benchmark harness for your chosen provider and one or more models inside the running controller container to generate a comparison table and failure-rate summary.