skills.rest
eddielueng
eddieluengCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs via vLLM with PagedAttention and continuous batching.

1|Updated Apr 14, 2026
Search Tags:#quantization#vllm#pagedattention#tensor parallelism#llm serving#continuous batching#openai-compatible api
One-click install
npx skills add https://github.com/eddielueng/hermes-agent-zh --skill serving-llms-vllm-eddielueng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill serving-llms-vllm-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill solves the challenge of deploying LLMs in a production setting where you need high throughput and low latency without over-provisioning GPU memory.

Core Features & Use Cases

  • High-throughput serving with vLLM: Uses PagedAttention and continuous batching to reduce idle GPU time and improve tokens/sec for real traffic.
  • Production-ready OpenAI-compatible API: Exposes an OpenAI-compatible server interface so applications can call it with standard chat/completions tooling.
  • GPU-memory optimizations: Supports quantization (GPTQ/AWQ/FP8) and tensor parallelism to fit larger models and improve cost efficiency.
  • Common use case: Deploy an OpenAI-compatible endpoint for a multi-user chatbot and tune latency/throughput to hit targets like TTFT and req/sec.

Quick Start

Run the vLLM OpenAI-compatible server for your model by issuing: vllm serve meta-llama/Llama-3-8B-Instruct --port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with an OpenAI-compatible API for multiple users?▼

Serve an LLM with an OpenAI-compatible API for multiple users by running vLLM serve to expose standard chat and completions endpoints. It uses continuous batching to handle concurrent requests efficiently without over-provisioning GPU memory.

How does PagedAttention reduce GPU memory bottlenecks during LLM serving?▼

PagedAttention reduces GPU memory bottlenecks during LLM serving by managing attention keys and values in paged blocks, minimizing fragmentation. This allows continuous batching to process more requests concurrently, significantly increasing throughput and lowering latency.

Can I use tensor parallelism and quantization to fit larger models in limited GPU environments?▼

Yes, you can use tensor parallelism and quantization to fit larger models in limited GPU environments. Tensor parallelism splits the model across GPUs, while GPTQ, AWQ, or FP8 quantization reduces memory footprint to improve cost efficiency.

What is the best way to improve LLM serving throughput and reduce latency for production traffic?▼

The best way to improve LLM serving throughput and reduce latency for production traffic is using vLLM. It applies PagedAttention and continuous batching to reduce idle GPU time, and allows enabling prefix caching and chunked prefill for further optimization.

How do I deploy a vLLM server for batch inference workloads?▼

Deploy a vLLM server for batch inference workloads by running the vllm serve command with your target model, such as meta-llama/Llama-3-8B-Instruct. You can optionally enable metrics and tune features like chunked prefill to handle heavy inference loads.

Does vLLM support GPTQ and AWQ quantization modes for serving large language models?▼

Yes, vLLM supports GPTQ and AWQ quantization modes for serving large language models, alongside FP8. These modes compress model weights to lower memory requirements, enabling deployment in memory-constrained GPU environments while maintaining fast inference speeds.

Related Skills

Explore More Software Engineering Skills→
hsliuustc0106hsliuustc0106
84

vllm-omni-setup

Install and configure vLLM-Omni with Python environments and smoke tests.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
huidgehuidge
2

serving-llms-vllm

Deploy LLMs with vLLM for OpenAI-compatible serving and tensor parallelism.

Community
Advanced
aws-samplesaws-samples
18

vllm-serving-setup

Automate vLLM inference serving deployment on EKS Kubernetes clusters.

Official
Advanced