What problem does it solve? Serving large language models in production requires specialized infrastructure for continuous batching, KV-cache management, quantization, and GPU scaling, which is difficult to configure correctly without deep operational knowledge. ## Core Features & Use Cases - Multi-Framework Serving: Launch OpenAI-compatible inference endpoints with vLLM, Text Generation Inference, Ollama, or llama.cpp, with guidance on when each fits. - Quantization & Optimization: Apply AWQ, GPTQ, GGUF, and FP8 quantization plus PagedAttention and prefix caching to cut memory usage and raise throughput. - Production Deployment: Generate Docker Compose files and Kubernetes manifests with GPU scheduling, health probes, and autoscaling, plus SageMaker LMI and Bedrock guidance. - Use Case: You need to serve a fine-tuned 70B model to internal users. Use this Skill to pick AWQ quantization, generate a Kubernetes deployment with tensor parallelism across two A100s, and benchmark TTFT and tokens-per-second before going live. ## Quick Start Deploy Llama-3.1-8B-Instruct as an OpenAI-compatible API server using vLLM and then benchmark its latency and throughput.
