router-benchmark-runner

To benchmark agentic tool-calls across multiple models, run a sequential evaluation script that orchestrates model swaps and warmup retries on the m5-router. This process handles output buffering and monitors progress to produce a final bench_summary.json.

Handling model warmup during an LLM benchmark requires a script that applies controlled swaps and retry logic. This ensures each model on the router is fully responsive before the tool-call evaluation begins, preventing cold-start errors.

To avoid output buffering during benchmark runs, redirect logs to a dedicated file like bench_run.log. This ensures real-time visibility into the evaluation progress and prevents the terminal from freezing during long sequential tests.

You can skip completed models on rerun by checking the results metadata. The benchmark script detects existing results and bypasses those models automatically, saving time and avoiding redundant evaluations across your router.

Validate model paths before running a router benchmark by pre-checking the router-preset.ini configuration file. This pre-run validation catches missing model files early, ensuring the sequential benchmarking process does not fail mid-execution.