skills.rest
BBuf
BBufCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llm-serving-capacity-planner

Parse SGLang and vLLM startup logs to decompose GPU memory and estimate request concurrency.

783|67|Updated Apr 1, 2026
Search Tags:#capacity planning#vllm#SGLang#kv cache#LLM serving#Log Analysis#GPU memory
One-click install
npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill llm-serving-capacity-planner
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-serving-capacity-planner
Source: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/tree/main/skills/llm-serving-capacity-planner
Command: npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill llm-serving-capacity-planner

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

When you launch an SGLang or vLLM serving instance, it is hard to tell where GPU HBM actually went or how many concurrent requests the deployment can sustain. This Skill parses startup logs to break memory into weights, KV pool, CUDA graph, framework overhead, and other categories, then estimates max concurrency for common token lengths.

Core Features & Use Cases

  • Memory Decomposition: Extracts weight load, KV pool, CUDA graph, and framework overhead from SGLang and vLLM startup log lines, with per-category GiB, percentage, and derivation notes.
  • Concurrency Estimation: Computes max concurrent requests from max_total_num_tokens and max_running_requests for token lengths like 4096, 6144, and 8192.
  • Cross-Validation: Compares per-rank nvidia-smi readings against log-derived values and supports theoretical KV byte calculation from the model's config.json, including MLA and SWA/CSA/HCA attention variants.
  • Use Case: After launching a DeepSeek model on 8xH200 with mem-fraction-static=0.88, run the analyzer on the startup log plus nvidia-smi output to see the KV pool budget, replication factor, and how many 8K-token requests can run concurrently before OOM.

Quick Start

Ask the agent to analyze your SGLang or vLLM startup log with the capacity analyzer, optionally providing nvidia-smi output and the model config.json, to get a memory breakdown and concurrency estimate.

Frequently Asked Questions about llm-serving-capacity-planner

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze SGLang startup logs for GPU memory usage?▼

Run the capacity_analyzer.py script with --log-file pointing to your SGLang startup log. It extracts weight loading, KV pool, CUDA graph, and framework overhead lines, then prints a memory breakdown table with percentages and derivation sources.

How to estimate max concurrent requests for an LLM serving deployment?▼

The analyzer computes max concurrency as min(max_total_num_tokens divided by tokens per request, max_running_requests) for token lengths like 4096, 6144, and 8192. It reads these limits from the final max_total_num_tokens log line or vLLM's concurrency report.

Does the capacity analyzer support vLLM logs as well as SGLang?▼

Yes, it parses current vLLM V1 startup lines such as Initial free memory, Model loading took, Available KV cache memory, and GPU KV cache size. Fields not present in vLLM logs are reported as unknown rather than mapped to SGLang-only concepts.

Why is my KV cache memory higher than expected with tensor parallelism?▼

When num_key_value_heads is smaller than tp_size, the KV cache is replicated across all TP ranks instead of being split. For example, kv_heads=1 with tp=8 means each of the 8 GPUs stores a full copy, multiplying per-card KV memory by 8.

What are the limitations of log-based memory analysis for SWA models?▼

For sliding-window attention models like DeepSeek-V4 with CSA/HCA, per-token KV bytes cannot be derived from config.json alone because framework-internal window parameters are needed. Use the bytes_per_full_token value reported directly in the log instead.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
3.2k

jetson-inference-mem-tune

Recommends inference runtimes and memory launch flags for LLM workloads on NVIDIA Jetson.

Official
Intermediate
sgl-projectsgl-project
33.0k

compute-mamba-ratio

Computes the optimal memory split ratio between state and KV pools for hybrid Mamba model serving.

Official
Advanced
lenadlmlenadlm

serving-llms-vllm

Serve LLMs via OpenAI-compatible endpoints using vLLM optimizations.

Community
Advanced
BagelHoleBagelHole
46

gpu-server-management

Configure NVIDIA GPU servers with drivers, CUDA, and MIG partitioning.

Community
Advanced