skills.rest
Shubh2310-developer
Shubh2310-developerCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs with high throughput using vLLM's PagedAttention and continuous batching.

2|Updated Jan 10, 2026
Search Tags:#quantization#inference#vllm#pagedattention#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Shubh2310-developer/ENGUNITYCORE --skill serving-llms-vllm-shubh2310-developer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Shubh2310-developer/ENGUNITYCORE/tree/main/.claude/skills/inference-serving-vllm
Command: npx skills add https://github.com/Shubh2310-developer/ENGUNITYCORE --skill serving-llms-vllm-shubh2310-developer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Deploying large language models in production often means choosing between slow inference, excessive GPU memory consumption, and complex serving infrastructure. This Skill guides you through serving LLMs with vLLM, which uses PagedAttention and continuous batching to achieve up to 24x higher throughput than standard transformers while fitting large models into limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with monitoring, prefix caching, and Docker/Kubernetes deployment patterns. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and tensor parallelism across multiple GPUs. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: You need to deploy Llama-3-8B as a chatbot backend handling 100+ requests per second. Use this Skill to configure the server, enable prefix caching, set up Prometheus metrics, and verify TTFT stays under 500ms. ## Quick Start Use the vLLM skill to help me deploy a Llama-3-8B model as an OpenAI-compatible API server on my GPU.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Llama-3-8B-Instruct, then query it with the OpenAI Python SDK pointed at http://localhost:8000/v1. Add flags like --enable-prefix-caching and --enable-metrics for production use.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

Use vLLM for production APIs needing high throughput and multi-platform support. TensorRT-LLM is roughly 12% faster but NVIDIA-only and harder to set up. llama.cpp suits CPU or edge single-user inference, not multi-user serving.

Can vLLM run a 70B model on a single 40GB GPU?▼

Yes, by serving an AWQ-quantized model such as TheBloke/Llama-2-70B-AWQ with --quantization awq and --gpu-memory-utilization 0.95. This reduces memory from about 140GB to roughly 35GB with under 1% accuracy loss.

Why does vLLM run out of memory during model loading?▼

OOM occurs when the model plus KV cache exceeds available VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I fix slow time to first token in vLLM?▼

Enable --enable-prefix-caching for repeated prompt prefixes and --enable-chunked-prefill for long prompts. If concurrency is high, reduce --max-num-seqs to prioritize latency over throughput.

Does vLLM support quantization methods like AWQ and GPTQ?▼

Yes, vLLM supports AWQ, GPTQ, and FP8 quantization via the --quantization flag. AWQ is recommended for 70B models, FP8 delivers the fastest inference on H100 GPUs, and GPTQ offers the widest model compatibility.

Related Skills

Explore More Software Engineering Skills→
Clay-HHKClay-HHK
1

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

Community
Intermediate
masermediagroup-stackmasermediagroup-stack
1

model-serving

Deploy LLM and ML models for production inference with vLLM, BentoML, and streaming APIs.

Community
Advanced
Cloud-Byte-ConsultingCloud-Byte-Consulting

inference-benchmark-runner

Benchmarks LLM inference latency, throughput, GPU utilization, and cost on Kubernetes.

Official
Advanced
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced