coding-benchmark-runner

Run a coding benchmark by executing the run_bench.py Python harness against local models served via the llama.cpp router endpoint. The harness uses prompts.jsonl to evaluate models and logs per-problem timing, token usage, and scoring data into a results JSON file.

You can benchmark multiple local models for Python code generation. The workflow supports running benchmarks against multiple models served via the llama.cpp router, recording per-problem timing, token usage, and scoring data for objective comparison into a results JSON file.

You need Python 3, the run_bench.py harness, prompts.jsonl, test_harness.py, and access to a llama.cpp router endpoint. These components execute a standardized 15-problem Python coding and algorithmic soundness benchmark against your local models.

The Python coding and algorithmic soundness benchmark includes 15 problems. It runs these standardized problems against local models served via the llama.cpp router, logging per-problem progress and recording timing and token usage into a results JSON.

When testing local coding models, the benchmark records per-problem timing, token usage, and scoring data. These metrics are logged into a results JSON file to provide objective comparison data for evaluating local model performance on Python coding tasks.