skills.rest
zerwiz
zerwizCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM and OpenAI-compatible endpoints.

1|Updated Apr 10, 2026
Search Tags:#gpu#quantization#inference#vllm#pagedattention#llm serving#continuous batching
One-click install
npx skills add https://github.com/zerwiz/wayofpi --skill serving-llms-vllm-zerwiz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/zerwiz/wayofpi/tree/main/.hermes/hermes-agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/zerwiz/wayofpi --skill serving-llms-vllm-zerwiz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It removes the complexity of running large language models efficiently by turning GPU-heavy inference into a reliable, high-throughput API service.

Core Features & Use Cases

  • Production LLM Serving: Launch OpenAI-compatible endpoints for chat, completion, and model access workflows.
  • Performance Optimization: Improve latency and throughput with PagedAttention, continuous batching, prefix caching, and speculative decoding.
  • Deployment Flexibility: Support single-GPU, multi-GPU, Docker, Kubernetes, and distributed serving setups for production environments.
  • Quantized Model Serving: Run AWQ, GPTQ, and FP8 models to fit larger models into limited VRAM.
  • Troubleshooting and Monitoring: Diagnose OOM, networking, and performance issues with health checks, metrics, and tuning guidance.

Quick Start

Ask the skill to help you launch a vLLM server for your target model and optimize it for your hardware and traffic pattern.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM inference server with vLLM for production API hosting?▼

To deploy an LLM inference server with vLLM, launch an OpenAI-compatible endpoint configured for your target model and optimize vLLM settings for your specific GPU hardware and traffic pattern to ensure reliable serving.

Can I run quantized models with vLLM to fit larger LLMs into limited VRAM?▼

Yes, vLLM supports quantized model serving using AWQ, GPTQ, and FP8 formats, allowing you to fit larger LLMs into limited VRAM while maintaining high-throughput inference for production environments.

How does PagedAttention and continuous batching improve LLM serving throughput?▼

PagedAttention and continuous batching improve LLM serving throughput by optimizing memory management and dynamically batching incoming requests, which significantly reduces latency and increases overall API performance.

What is the best way to scale vLLM for multi-GPU distributed serving?▼

The best way to scale vLLM for multi-GPU distributed serving involves applying tensor parallelism across your available GPUs and deploying within Kubernetes or Docker to manage distributed workloads effectively.

Why does my vLLM server run out of memory or drop connections under heavy traffic?▼

vLLM server out-of-memory or dropped connections often stem from suboptimal batch sizes or hardware limits, requiring troubleshooting via health checks, metrics monitoring, and performance tuning to resolve.

Related Skills

Explore More Software Engineering Skills→
tangzheng202202tangzheng202202
1

serving-llms-vllm

Configure vLLM servers with OpenAI-compatible endpoints, quantization options, and tensor parallelism for scalable LLM deployments on GPU infrastructure.

Community
Advanced
ax-llmax-llm
2.9k

ax-rust-ai

Generate Rust AxAI provider client code for axllm workflows.

Official
Intermediate
peteromalletpeteromallet
97

serving-llms-vllm

Deploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.

Community
Intermediate
Z43LZ43L

tensorrt-llm

Deploys TensorRT-LLM to optimize LLM inference on NVIDIA GPUs.

Community
Advanced