omniroute-cli-eval
CommunityBenchmark models with live CLI scorecards.
Data & Analytics#regression testing#jsonl#scorecard#cli automation#llm evals#model benchmarking#routing quality
Authordiegosouzapw
Version1.0.0
Installs0
System Documentation
What problem does it solve?
OmniRoute CLI evals solve the problem of validating and comparing LLM output quality in a repeatable way, so you can catch regressions and make data-driven model choices.
Core Features & Use Cases
- Evals as reusable suites: Define suites with inputs and expected outputs (or rubrics) stored in OmniRouteās local database.
- Multiple scoring options: Use
exact-match,contains,llm-judge, orregexrubrics to fit different quality criteria. - Run management and scoring visibility: Create, run, watch live progress, fetch run details, and generate per-sample scorecards for comparison across models.
Quick Start
Tell your AI to run the eval suite with: omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --watch.
Dependency Matrix
Required Modules
None requiredComponents
Standard packageš» Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: omniroute-cli-eval Download link: https://github.com/diegosouzapw/OmniRoute/archive/main.zip#omniroute-cli-eval Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.