agentsop-vllm

What problem does it solve?

This Skill provides decision-grade guidance for serving LLMs with vLLM, helping you pick the right configuration and resolve common production issues like CUDA OOM, low throughput, and poor latency.

Core Features & Use Cases

• Inference-engine activation guidance: tells you when vLLM is the correct choice versus alternatives (TGI/SGLang/TensorRT-LLM/llama.cpp/Ollama) based on your workload and constraints.
• Practical serving SOP: covers the mental model (PagedAttention KV-cache as virtual memory, continuous batching, prefill vs decode), and then walks through precision, parallelism, memory/batch envelope sizing, prefix caching, and optional speculative decoding.
• Evidence-driven operation: includes targeted dilemma resolutions (batching vs latency, FP8 vs AWQ, TP=4 vs replicas, prefix caching ROI, speculative decoding tradeoffs) for tuning under real load.

Quick Start

Ask a coder-agent to use agentsop-vllm to diagnose your vLLM throughput/latency/OOM problem and produce a concrete configuration change plan for your exact model, GPU topology, and request length distribution.