omniroute-cli-eval

Run OmniRoute eval suites from the CLI and export JSON or scorecard results.

44.0k|5.9k|Updated Feb 13, 2026

Search Tags:#regression testing #jsonl #scorecard #cli automation #llm evals #model benchmarking #routing quality

Ready to run

npx skills add https://github.com/diegosouzapw/OmniRoute --skill omniroute-cli-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: omniroute-cli-eval
Source: https://github.com/diegosouzapw/OmniRoute/tree/main/skills/omniroute-cli-eval
Command: npx skills add https://github.com/diegosouzapw/OmniRoute --skill omniroute-cli-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

OmniRoute CLI evals solve the problem of validating and comparing LLM output quality in a repeatable way, so you can catch regressions and make data-driven model choices.

Core Features & Use Cases

Evals as reusable suites: Define suites with inputs and expected outputs (or rubrics) stored in OmniRoute’s local database.
Multiple scoring options: Use exact-match, contains, llm-judge, or regex rubrics to fit different quality criteria.
Run management and scoring visibility: Create, run, watch live progress, fetch run details, and generate per-sample scorecards for comparison across models.

Quick Start

Tell your AI to run the eval suite with: omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --watch.

Frequently Asked Questions about omniroute-cli-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run LLM evals and compare model quality from the command line?▼

You can run LLM evals from the command line by defining automated evaluation suites with JSONL samples and rubrics, then executing deterministic CLI commands to measure, compare, and generate scorecards for model quality.

How do I build a benchmark suite to catch LLM regressions?▼

To catch LLM regressions, build a reusable benchmark suite by defining inputs, expected outputs, and scoring rubrics like exact-match or llm-judge. Running these suites repeatedly validates output quality over time.

What scoring rubrics can I use for automated LLM evaluation?▼

Available scoring rubrics for automated LLM evaluation include exact-match, contains, llm-judge, and regex. These options fit different quality criteria when measuring model outputs within your benchmark suites.

Can I watch live progress when executing asynchronous model runs?▼

Yes, you can watch live progress during asynchronous model runs by using the --watch flag with your CLI execution command. This provides real-time visibility while the suite processes your samples.

How do I export per-sample results and scorecards for model comparison?▼

You can export per-sample results and scorecards by fetching run details using specific run IDs. The CLI supports exporting outputs in JSON or scorecard formats to compare performance across different models.

What is the best way to automate regression testing for LLM outputs?▼

The best way to automate regression testing for LLM outputs is creating reusable eval suites with deterministic CLI commands. This allows you to list, run, cancel, and export results repeatably to validate quality changes.