skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and optimize LLM inference servers using vLLM with OpenAI-compatible APIs.

Updated Jun 5, 2026
Search Tags:#quantization#inference#vllm#openai api#tensor parallelism#llm serving#gpu deployment
One-click install
npx skills add https://github.com/xu1713/openhorse --skill serving-llms-vllm-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/skills/mlops/inference/vllm
Command: npx skills add https://github.com/xu1713/openhorse --skill serving-llms-vllm-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow and memory-hungry with standard transformers, causing high latency, low throughput, and GPU out-of-memory errors. This Skill guides you through deploying high-throughput LLM APIs with vLLM's PagedAttention and continuous batching. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics monitoring. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and configurable sampling parameters. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization. - Use Case: Deploy Llama-3-8B as an OpenAI-compatible chat API handling 100+ requests per second on a single A100, with metrics dashboards tracking TTFT and GPU cache usage. ## Quick Start Use the vLLM skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible API server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Llama-3-8B-Instruct --port 8000. Then query it using the OpenAI Python SDK with base_url set to http://localhost:8000/v1 and any placeholder API key.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

vLLM offers high throughput with easy setup and multi-platform support. TensorRT-LLM is slightly faster but NVIDIA-only with complex setup. llama.cpp targets CPU and edge single-user inference, not production multi-user serving.

How do I fix vLLM out of memory errors when loading a model?▼

Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, or enable quantization with --quantization awq. For large models, split across GPUs using --tensor-parallel-size with a power-of-2 GPU count.

Does vLLM support quantized models like AWQ and GPTQ?▼

Yes, vLLM supports AWQ, GPTQ, and FP8 quantization. Pass --quantization awq or gptq when serving pre-quantized models from HuggingFace, enabling a 70B model to fit in roughly 35-40GB of VRAM.

Why is my vLLM server throughput below 50 requests per second?▼

Low throughput usually means insufficient concurrency. Increase --max-num-seqs to 256 or 512, enable --enable-prefix-caching, and verify GPU utilization exceeds 80% with nvidia-smi.

How do I reduce time to first token in vLLM?▼

Enable --enable-prefix-caching for repeated prompt prefixes and --enable-chunked-prefill for long prompts. If TTFT remains high, reduce concurrent sequences or parallelize prefill across multiple GPUs.

Related Skills

Explore More Software Engineering Skills→
Clay-HHKClay-HHK
1

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

Community
Intermediate
allthingslinuxallthingslinux
1

prometheus-grafana

Deploy Prometheus and Grafana to monitor systems and services.

Official
Advanced
keqingli1129keqingli1129

load-testing

Run Locust-based streaming load tests on Databricks Apps to measure QPS, TTFT, and latency.

Community
Advanced
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced