What problem does it solve? Deploying large language models in production often means choosing between slow inference, excessive GPU memory consumption, and complex serving infrastructure. This Skill guides you through serving LLMs with vLLM, which uses PagedAttention and continuous batching to achieve up to 24x higher throughput than standard transformers while fitting larger models into limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for monitoring latency and throughput. - Offline Batch Inference: Process large prompt datasets efficiently with automatic internal batching, no manual chunking required. - Quantized Model Serving: Fit 70B models into 40GB of VRAM using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: You need to deploy Llama-3-8B as a production chatbot API handling 100+ requests per second. This Skill walks you through server configuration, load testing, monitoring setup, and Docker deployment with performance verification. ## Quick Start Ask the agent to help you deploy a Llama model as an OpenAI-compatible API server using vLLM with appropriate GPU memory settings.
