skills.rest
Chia1104
Chia1104COMMUNITY
Explore all skills by this author →
Website𝕏@chia7423GitHub

serving-llms-vllm

Deploy and optimize high-throughput LLM inference servers using vLLM with OpenAI-compatible APIs.

1|Updated Aug 11, 2026
Search Tags:#gpu#quantization#inference#vLLM#OpenAI API#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Chia1104/agent-air --skill serving-llms-vllm-chia1104
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Chia1104/agent-air/tree/main/skills/hermes/mlops/inference/serving-llms-vllm
Command: npx skills add https://github.com/Chia1104/agent-air --skill serving-llms-vllm-chia1104

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow, memory-hungry, and hard to scale with naive transformers-based setups. This Skill guides you through deploying vLLM to achieve high-throughput, low-latency inference with PagedAttention, continuous batching, and quantization, even on limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for monitoring. - Offline Batch Inference: Process large prompt datasets efficiently with automatic internal batching via the LLM engine API. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: You need to serve Llama-3-70B to hundreds of concurrent chatbot users. Use this Skill to configure tensor parallelism across 4 GPUs, enable AWQ quantization and prefix caching, then monitor TTFT and throughput via Prometheus metrics. ## Quick Start Ask the AI to help you launch a vLLM OpenAI-compatible server for your chosen model with appropriate GPU memory and quantization settings.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with vLLM in production?▼

Run vllm serve with your model name, set --gpu-memory-utilization to 0.9, and enable --enable-prefix-caching for repeated prompts. vLLM exposes an OpenAI-compatible API on port 8000 and Prometheus metrics at /metrics for monitoring TTFT and throughput.

vLLM vs TensorRT-LLM vs llama.cpp for inference serving?▼

vLLM offers high throughput with easy setup and multi-platform support including AMD and Intel GPUs. TensorRT-LLM is roughly 12% faster but NVIDIA-only with complex setup, while llama.cpp targets CPU and edge single-user inference.

How to fit a 70B model on limited GPU memory?▼

Use AWQ quantization to reduce a 70B model from 140GB to about 35GB VRAM, fitting on a single A100 40GB. Launch with --quantization awq using a pre-quantized model from HuggingFace, with under 1% accuracy loss.

Why does vLLM throw out of memory errors during model loading?▼

OOM occurs when model weights plus KV cache exceed available VRAM. Reduce --gpu-memory-utilization to 0.7, lower --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

Does vLLM support multi-GPU and multi-node serving?▼

Yes, vLLM supports tensor parallelism with --tensor-parallel-size using a power-of-2 GPU count, and pipeline parallelism across nodes with --pipeline-parallel-size. Multi-node setups require matching MASTER_ADDR, MASTER_PORT, and WORLD_SIZE environment variables.

How do I reduce time to first token in vLLM?▼

Enable --enable-prefix-caching to reuse KV cache for repeated prompt prefixes and --enable-chunked-prefill for long prompts. Prefix caching can reduce TTFT by over 80% when system prompts repeat across requests.

Related Skills

Explore More Software Engineering Skills→
choice5346choice5346

awq-quantization

Quantizes LLMs to 4-bit using activation-aware weight optimization.

Community
Advanced
allthingslinuxallthingslinux
1

prometheus-grafana

Deploy Prometheus and Grafana to monitor systems and services.

Official
Advanced
cosmixcosmix
53

performance-testing

Run load tests with k6, Locust, or Artillery and report latency, throughput, and error rate.

Community
Advanced
RedWoodOGRedWoodOG
174

llama-cpp

Run CPU-based LLM inference with llama.cpp using GGUF quantization.

Community
Intermediate