omniroute-cli-eval

You can run LLM evals from the command line by defining automated evaluation suites with JSONL samples and rubrics, then executing deterministic CLI commands to measure, compare, and generate scorecards for model quality.

To catch LLM regressions, build a reusable benchmark suite by defining inputs, expected outputs, and scoring rubrics like exact-match or llm-judge. Running these suites repeatedly validates output quality over time.

Available scoring rubrics for automated LLM evaluation include exact-match, contains, llm-judge, and regex. These options fit different quality criteria when measuring model outputs within your benchmark suites.

Yes, you can watch live progress during asynchronous model runs by using the --watch flag with your CLI execution command. This provides real-time visibility while the suite processes your samples.

You can export per-sample results and scorecards by fetching run details using specific run IDs. The CLI supports exporting outputs in JSON or scorecard formats to compare performance across different models.

The best way to automate regression testing for LLM outputs is creating reusable eval suites with deterministic CLI commands. This allows you to list, run, cancel, and export results repeatably to validate quality changes.