evalscope-cli

What problem does it solve?

Converts natural-language model evaluation and performance testing requests into the exact EvalScope CLI workflows, so you can benchmark accuracy and latency without manually assembling complex command flags.

Core Features & Use Cases

• Model accuracy evaluation (eval): Runs 156+ benchmarks (math, coding, Chinese, multimodal, agent, etc.) against local checkpoints or OpenAI-compatible / Anthropic API endpoints.
• Performance stress testing (perf): Measures TTFT, TPOT, throughput, and latency under configurable concurrency gradients and SLA auto-tuning.
• Benchmark discovery & result visualization: Lists/filter benchmarks by capability tag, retrieves rich metadata and examples, and launches a Web dashboard to compare outputs.

Typical use case: you want to evaluate a deployed model’s math + coding accuracy, then stress test its throughput and find the maximum concurrency that keeps latency under a target SLA.

Quick Start

Use the evalscope-cli skill to run an accuracy eval for a local model on gsm8k with a small sanity sample by telling it: evaluate Qwen2.5-0.5B on gsm8k with limit 10, then show me the commands to run and where to read the report.