skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploy and optimize LLM inference servers using vLLM with OpenAI-compatible APIs.

Updated Sep 10, 2026
Search Tags:#quantization#inference#vLLM#OpenAI API#tensor parallelism#gpu optimization#llm serving
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill serving-llms-vllm-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/inference/serving-llms-vllm
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill serving-llms-vllm-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production requires high throughput, low latency, and efficient GPU memory usage, which standard transformers inference cannot deliver. This Skill guides you through deploying vLLM servers with PagedAttention, continuous batching, and quantization to fit large models on limited hardware. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for monitoring. - Offline Batch Inference: Process large prompt datasets efficiently with automatic internal batching and configurable sampling parameters. - Quantized Model Serving: Serve 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: Deploy a Llama 3 70B model behind an OpenAI-compatible API on 4 A100 GPUs, enable prefix caching for repeated system prompts, and monitor TTFT and throughput via Prometheus metrics. ## Quick Start Use the vLLM skill to deploy a Llama 3 8B model as an OpenAI-compatible API server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM with vLLM as an OpenAI-compatible API?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Meta-Llama-3-8B-Instruct, then query it with the OpenAI SDK pointed at http://localhost:8000/v1. Add flags like --gpu-memory-utilization 0.9 and --enable-prefix-caching for production settings.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

Use vLLM for production APIs needing high throughput and multi-platform support. TensorRT-LLM is roughly 12% faster but NVIDIA-only with complex setup, while llama.cpp suits CPU or edge single-user inference.

How do I serve a 70B model on limited GPU memory?▼

Use a pre-quantized AWQ model such as TheBloke/Llama-2-70B-AWQ and launch with --quantization awq, which fits a 70B model in about 35-40GB VRAM. Alternatively, split the model across GPUs with --tensor-parallel-size.

Why does vLLM throw out of memory errors during model loading?▼

OOM occurs when the model plus KV cache exceeds available VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or use tensor parallelism across multiple GPUs.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompts and chunked prefill with --enable-chunked-prefill for long prompts. Reducing --max-num-seqs also prioritizes latency over throughput.

Does vLLM support monitoring and metrics in production?▼

Yes, vLLM exposes Prometheus metrics at the /metrics endpoint on the API port. Key metrics include vllm:time_to_first_token_seconds, vllm:num_requests_running, and vllm:gpu_cache_usage_perc, which integrate with Grafana dashboards.

Related Skills

Explore More Software Engineering Skills→
Clay-HHKClay-HHK
1

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

Community
Intermediate
ArogyaReddyArogyaReddy

prometheus-configuration

Configure Prometheus for metric collection, alerting, and monitoring in Kubernetes or Docker.

Community
Advanced
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
jacardljacardl

llama-cpp

Provide CPU-based LLM inference for non-NVIDIA hardware using llama.cpp.

Community
Advanced