skills.rest
soulmachine
soulmachineCOMMUNITY
Explore all skills by this author →
𝕏@soulmachineGitHub

llm-inference-benchmark

Benchmark OpenAI-compatible LLM inference servers and analyze saturation curves.

3|1|Updated Mar 31, 2026
Search Tags:#performance#benchmarking#LLM#inference#server comparison
One-click install
npx skills add https://github.com/soulmachine/skills --skill llm-inference-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-inference-benchmark
Source: https://github.com/soulmachine/skills/tree/main/llm-inference-benchmark
Command: npx skills add https://github.com/soulmachine/skills --skill llm-inference-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sglang.bench_serving, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables users to benchmark and compare OpenAI-compatible LLM inference servers efficiently, providing insights into server performance, saturation knees, and prefill vs. decode-bound characteristics.

Core Features & Use Cases

  • Benchmarking LLM Servers: Collects TTFT, TPOT, ITL, and input/output/total token throughput from OpenAI-compatible LLM inference servers.
  • Insights on Saturation Curves: Determines the min and max knee of the saturation curve for optimal server performance.
  • Prefill vs. Decode Analysis: Identifies whether the server is prefill or decode-bound, helping to optimize for either throughput or latency.
  • Use Case: Use this Skill to compare different LLM servers like vLLM, SGLang, or anything serving /v1/completions, helping to make informed decisions about server configurations for production.

Quick Start

Run the llm-inference-benchmark skill with your desired concurrency and tokenizer model: bash scripts/bench_sweep.sh TARGET_HOST=<server_ip> MODEL_NAME=kimi-k2.6 MODEL_REPO=nvidia/Kimi-K2.6-NVFP4

Frequently Asked Questions about llm-inference-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark and compare LLM inference servers?▼

You can benchmark LLM inference servers by collecting TTFT, TPOT, ITL, and token throughput metrics using a load generator to analyze saturation curves and compare different server configurations.

What metrics are used for LLM inference server comparison?▼

LLM inference server comparison uses Time To First Token (TTFT), Time Per Output Token (TPOT), Inter-Token Latency (ITL), and input/output/total token throughput to evaluate server performance.

How do I find the saturation knee of an LLM inference server?▼

To find the saturation knee of an LLM inference server, run benchmark sweeps across different concurrency levels to determine the min and max knee of the saturation curve for optimal performance.

Can I use this to benchmark vLLM and SGLang servers?▼

Yes, you can benchmark vLLM, SGLang, or any server serving OpenAI-compatible /v1/completions endpoints to make informed decisions about server configurations for production.

Do I need sglang.bench_serving to benchmark LLM inference?▼

Yes, you need sglang.bench_serving installed as a load generator, and you must provide access to tokenizer model repositories for the server being tested to run the benchmarks.

How do I determine if my LLM server is prefill or decode-bound?▼

You can determine if an LLM server is prefill or decode-bound by analyzing benchmark metrics to identify performance bottlenecks, helping you optimize for either throughput or latency.

Related Skills

Explore More Software Engineering Skills→
tharunramagiritharunramagiri

add-atomic-chat-tool

Integrate local OpenAI-compatible model servers into containerized agents via MCP.

Community
Advanced
FutureMLS-LabFutureMLS-Lab
556

generate-profile

Generate end-to-end performance profiling traces for SGLang server runs.

Official
Intermediate
yanochka11yanochka11

serving-llms-vllm

Deploy and optimize vLLM inference servers with OpenAI-compatible endpoints.

Community
Advanced
Azeem-2Azeem-2

aiops

Automate IT operations decisions with AIOps patterns for monitoring and incident response.

Community
Advanced