skills.rest
luckybbjason1
luckybbjason1COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and optimize LLM inference servers with vLLM using OpenAI-compatible APIs and quantization.

Updated Sep 9, 2026
Search Tags:#quantization#inference#vllm#openai api#tensor parallelism#llm serving#gpu deployment
One-click install
npx skills add https://github.com/luckybbjason1/trading --skill serving-llms-vllm-luckybbjason1
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/luckybbjason1/trading/tree/main/.hermes/skills/mlops/inference/vllm
Command: npx skills add https://github.com/luckybbjason1/trading --skill serving-llms-vllm-luckybbjason1

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow and memory-hungry with standard transformers. This Skill guides you through deploying high-throughput LLM APIs with vLLM, achieving up to 24x higher throughput via PagedAttention and continuous batching while fitting large models into limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for monitoring TTFT and throughput. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and multi-GPU tensor parallelism. - Quantized Model Serving: Serve 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: Deploy a Llama-3-8B chatbot backend handling 100+ requests per second on one A100, with Docker deployment, Nginx load balancing, and Grafana dashboards tracking latency. ## Quick Start Use the vLLM skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible API server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Llama-3-8B-Instruct --port 8000. Then query it using the OpenAI Python SDK with base_url set to http://localhost:8000/v1 and any placeholder API key.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

Use vLLM for production APIs needing high throughput and multi-platform support. TensorRT-LLM is about 12% faster but NVIDIA-only with complex setup. llama.cpp suits CPU or edge single-user inference, not multi-user serving.

How do I serve a 70B model on limited GPU memory?▼

Use a pre-quantized AWQ model like TheBloke/Llama-2-70B-AWQ and launch with --quantization awq. This reduces memory from 140GB to about 35GB, fitting on a single 40GB A100 with under 1% accuracy loss.

Why does vLLM throw out of memory errors during model loading?▼

The model plus KV cache exceeds available VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len to 4096, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I fix slow time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompts, and use --enable-chunked-prefill for long prompts. Also reduce --max-num-seqs if too many concurrent requests are queuing ahead of new ones.

Does vLLM support multiple GPUs and distributed serving?▼

Yes, use --tensor-parallel-size with a power-of-2 GPU count to shard models across GPUs. For multi-node setups, combine tensor parallelism with --pipeline-parallel-size and configure MASTER_ADDR, RANK, and WORLD_SIZE environment variables.

Related Skills

Explore More Software Engineering Skills→
huggingfacehuggingface
10.9k

hf-mem

Estimate inference memory for Safetensors and GGUF models from Hugging Face Hub.

Official
Basic
api7api7
1

a6-plugin-prometheus

Expose APISIX metrics in Prometheus format for monitoring and dashboards.

Official
Intermediate
project-purupuruproject-purupuru

eval

Run Loa framework evaluation suites to detect regressions and benchmark skill quality.

Official
Intermediate
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced