skills.rest
CHENHUI-X
CHENHUI-XCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploy and optimize high-throughput LLM inference servers using vLLM with OpenAI-compatible APIs.

Updated Jul 3, 2026
Search Tags:#quantization#inference#vllm#openai api#tensor parallelism#llm serving#gpu deployment
One-click install
npx skills add https://github.com/CHENHUI-X/toolbox --skill serving-llms-vllm-chenhui-x
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/CHENHUI-X/toolbox/tree/main/custom-skills/inference/serving-llms-vllm
Command: npx skills add https://github.com/CHENHUI-X/toolbox --skill serving-llms-vllm-chenhui-x

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow, memory-hungry, and hard to scale with naive transformers-based setups. This Skill guides you through deploying vLLM, which uses PagedAttention and continuous batching to achieve up to 24x higher throughput while fitting large models into limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for monitoring TTFT and throughput. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching via the Python LLM engine API. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: Deploy a Llama-3-70B chat API on 4x A100 GPUs with AWQ quantization, prefix caching, and load-tested throughput exceeding 100 requests per second. ## Quick Start Use the vLLM skill to help me deploy a Llama-3-8B model as an OpenAI-compatible API server on my GPU machine.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run 'vllm serve MODEL_NAME' to launch an OpenAI-compatible server on port 8000. Query it with the OpenAI Python SDK by setting base_url to http://localhost:8000/v1 and api_key to EMPTY.

How do I fit a 70B model on limited GPU memory?▼

Use AWQ or GPTQ quantization to reduce memory by roughly 4x, for example serving TheBloke/Llama-2-70B-AWQ with --quantization awq on a single 40GB GPU. Alternatively, split the model across multiple GPUs with --tensor-parallel-size.

vLLM vs TensorRT-LLM vs llama.cpp: which should I use?▼

Use vLLM for production multi-user APIs needing high throughput and broad hardware support. Choose TensorRT-LLM for maximum NVIDIA-only performance, llama.cpp for CPU or edge single-user inference, and HuggingFace transformers for research prototyping.

Why does vLLM run out of memory during model loading?▼

OOM occurs when the model plus KV cache exceeds available VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or add GPUs with tensor parallelism.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompt prefixes, and use --enable-chunked-prefill for long prompts. Reducing concurrent sequences with --max-num-seqs also prioritizes latency over throughput.

Does vLLM support monitoring and metrics in production?▼

Yes, vLLM exposes Prometheus metrics at the /metrics endpoint on the API port. Key metrics include vllm:time_to_first_token_seconds, vllm:num_requests_running, and vllm:gpu_cache_usage_perc, which integrate with Grafana dashboards.

Related Skills

Explore More Software Engineering Skills→
Clay-HHKClay-HHK
1

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

Community
Intermediate
handsomelong922handsomelong922

langchain

Build AI applications with agents, chains, and RAG using LangChain.

Community
Advanced
NVIDIANVIDIA
3.7k

evaluation

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher.

Official
Advanced
RedWoodOGRedWoodOG
174

llama-cpp

Run CPU-based LLM inference with llama.cpp using GGUF quantization.

Community
Intermediate