skills.rest
uv-xiao
uv-xiaoCOMMUNITY
Explore all skills by this author →
Website𝕏@youwei_xiaoGitHub

uv-serving-llms-vllm

Serve LLMs with vLLM using OpenAI-compatible endpoints and quantization.

1|Updated Feb 6, 2026
Search Tags:#production deployment#inference#vllm#pagedattention#llm serving#high throughput#continuous batching
One-click install
npx skills add https://github.com/uv-xiao/pkbllm --skill uv-serving-llms-vllm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: uv-serving-llms-vllm
Source: https://github.com/uv-xiao/pkbllm/tree/main/knowledge/ML/serving/vllm
Command: npx skills add https://github.com/uv-xiao/pkbllm --skill uv-serving-llms-vllm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of efficiently deploying and serving Large Language Models (LLMs) at scale, optimizing for high throughput and low latency in production environments.

Core Features & Use Cases

  • High-Throughput Serving: Leverages vLLM's PagedAttention and continuous batching to maximize inference speed and concurrency.
  • Production Deployment: Ideal for deploying LLM APIs, supporting OpenAI-compatible endpoints, and handling demanding workloads.
  • Memory Optimization: Supports quantization (GPTQ, AWQ, FP8) and tensor parallelism to serve large models with limited GPU memory.
  • Use Case: Deploying a chatbot service that needs to handle thousands of concurrent user requests with minimal response time, or performing large-scale batch inference on a dataset.

Quick Start

Serve the Llama-3-8B-Instruct model using vLLM with an OpenAI-compatible endpoint.

Frequently Asked Questions about uv-serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs with high throughput and low latency in production?▼

To serve LLMs with high throughput, you can use vLLM's PagedAttention and continuous batching to maximize inference speed and concurrency for production API workloads.

Can I deploy an OpenAI-compatible endpoint for large language models using vLLM?▼

Yes, you can deploy an OpenAI-compatible endpoint using vLLM, which allows you to handle demanding chatbot workloads and large-scale batch inference efficiently.

How does PagedAttention and continuous batching optimize LLM inference?▼

PagedAttention and continuous batching optimize LLM inference by managing memory more efficiently and processing multiple concurrent requests simultaneously, thereby maximizing throughput.

How do I serve large language models on hardware with limited GPU memory?▼

You can serve large models on limited GPU memory by applying quantization methods like GPTQ, AWQ, or FP8, and utilizing tensor parallelism within the vLLM serving framework.

Does vLLM support GPTQ, AWQ, and FP8 quantization for inference?▼

Yes, vLLM supports GPTQ, AWQ, and FP8 quantization methods, enabling efficient inference and high-throughput serving on resource-constrained hardware setups.

Related Skills

Explore More Software Engineering Skills→
yangwhaleyangwhale
9

vllm-installer

Install, configure, and run vLLM on NVIDIA GPUs with CUDA.

Community
Advanced
ax-llmax-llm
2.9k

ax-rust-ai

Generate Rust AxAI provider client code for axllm workflows.

Official
Intermediate
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
zechenzhangAGIzechenzhangAGI
11.5k

knowledge-distillation

Compress large language models by transferring teacher knowledge to smaller student models.

Community
Intermediate