skills.rest
AvaTar-ArTs
AvaTar-ArTsCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and optimize LLM inference servers using vLLM with OpenAI-compatible APIs.

Updated Jul 10, 2026
Search Tags:#quantization#inference#vllm#openai api#tensor parallelism#llm serving#gpu deployment
One-click install
npx skills add https://github.com/AvaTar-ArTs/.Agent-skills --skill serving-llms-vllm-avatar-arts
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/AvaTar-ArTs/.Agent-skills/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/AvaTar-ArTs/.Agent-skills --skill serving-llms-vllm-avatar-arts

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow, memory-hungry, and hard to scale. This Skill guides you through deploying high-throughput LLM inference with vLLM, covering PagedAttention memory management, continuous batching, quantization, and production monitoring. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for multi-user applications. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and structured result export. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: Deploy Llama-3-8B as an OpenAI-compatible API handling 100+ requests per second, monitor TTFT and GPU cache usage via Prometheus, and troubleshoot OOM or latency issues using the included reference guides. ## Quick Start Use the vLLM skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM with vLLM as an OpenAI-compatible API?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Llama-3-8B-Instruct --port 8000. The server exposes an OpenAI-compatible endpoint at /v1 that works directly with the OpenAI Python SDK using base_url http://localhost:8000/v1.

How do I fit a 70B model on a single GPU with vLLM?▼

Use AWQ quantization with a pre-quantized model such as TheBloke/Llama-2-70B-AWQ and launch with --quantization awq --gpu-memory-utilization 0.95. This reduces memory from 140GB to roughly 35GB, fitting on one 40GB A100.

vLLM vs TensorRT-LLM vs llama.cpp: which should I use?▼

Use vLLM for production APIs needing high throughput and multi-platform support. Choose TensorRT-LLM only for NVIDIA-only deployments needing maximum performance, and llama.cpp for CPU or edge single-user inference.

Why does vLLM throw out of memory errors during model loading?▼

OOM occurs when model weights plus KV cache exceed VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompt prefixes, and use --enable-chunked-prefill for long prompts. Target TTFT under 500ms and monitor vllm:time_to_first_token_seconds via Prometheus metrics.

Does vLLM support multi-GPU and multi-node serving?▼

Yes, vLLM supports tensor parallelism across GPUs with --tensor-parallel-size using a power of 2, and pipeline parallelism across nodes with --pipeline-parallel-size using MASTER_ADDR, RANK, and WORLD_SIZE environment variables.

Related Skills

Explore More Software Engineering Skills→
Clay-HHKClay-HHK
1

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

Community
Intermediate
devMoezdevMoez
1

serving-llms-vllm

Deploy vLLM to serve LLMs through an OpenAI-compatible API.

Community
Intermediate
shakfushakfu

benchmark-model

Benchmark MAX model servers to measure throughput, latency, and GPU utilization.

Community
Intermediate
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced