llm-inference-benchmark
CommunityBenchmark and compare LLM inference servers efficiently.
Authorsoulmachine
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill enables users to benchmark and compare OpenAI-compatible LLM inference servers efficiently, providing insights into server performance, saturation knees, and prefill vs. decode-bound characteristics.
Core Features & Use Cases
- Benchmarking LLM Servers: Collects TTFT, TPOT, ITL, and input/output/total token throughput from OpenAI-compatible LLM inference servers.
- Insights on Saturation Curves: Determines the min and max knee of the saturation curve for optimal server performance.
- Prefill vs. Decode Analysis: Identifies whether the server is prefill or decode-bound, helping to optimize for either throughput or latency.
- Use Case: Use this Skill to compare different LLM servers like vLLM, SGLang, or anything serving /v1/completions, helping to make informed decisions about server configurations for production.
Quick Start
Run the llm-inference-benchmark skill with your desired concurrency and tokenizer model:
bash scripts/bench_sweep.sh TARGET_HOST=<server_ip> MODEL_NAME=kimi-k2.6 MODEL_REPO=nvidia/Kimi-K2.6-NVFP4
Dependency Matrix
Required Modules
sglang.bench_serving
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: llm-inference-benchmark Download link: https://github.com/soulmachine/skills/archive/main.zip#llm-inference-benchmark Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.