router-benchmark-runner

What problem does it solve?

Run agentic tool-call benchmarks across all models on the m5-router sequentially. Handles model swap warmup, output buffering, and missing model files to ensure reliable evaluation.

Core Features & Use Cases

• Sequential benchmarking across models on the m5-router.
• Automatic model warmup with controlled swaps and retry logic to ensure responsiveness.
• Output buffering avoidance by redirecting logs to bench_run.log for real-time visibility.
• Pre-run validation of model paths using router-preset.ini to catch missing files.
• Skip completed models on rerun based on results metadata to save time.
• End-to-end workflow: restart router, clear stale results, run benches, monitor progress, and produce bench_summary.json.

Quick Start

Restart the router, clear stale results, run the bench script, monitor the log, and review bench_summary.json.