coding-benchmark-runner

What problem does it solve?

The Coding Benchmark Runner enables rapid evaluation of local model performance on coding tasks by running a standardized 15-problem Python benchmark against models served through the llama.cpp router.

Core Features & Use Cases

• Automates end-to-end benchmarking of coding tasks across local models.
• Provides per-problem timing, token usage, and scoring data for objective comparison.
• Use Case: A team wants to compare multiple local models to select the best candidate for code generation, then iterate on improvements.

Quick Start

Run the benchmark by executing the Python harness against your local model served via the llama.cpp router using the prompts.jsonl file