skills.rest
ricable
ricableCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve large language models with vLLM high-throughput inference and OpenAI-compatible endpoints.

Updated Feb 15, 2026
Search Tags:#quantization#inference#vllm#openai api#gpu optimization#llm serving#continuous batching
One-click install
npx skills add https://github.com/ricable/mcai --skill serving-llms-vllm-ricable
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/ricable/mcai/tree/main/.agents/skills/serving-llms-vllm
Command: npx skills add https://github.com/ricable/mcai --skill serving-llms-vllm-ricable

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill simplifies production LLM serving by turning complex inference setup, scaling, and optimization decisions into a guided workflow for reliable deployment.

Core Features & Use Cases

  • High-throughput model serving with vLLM for API endpoints and internal tools.
  • Quantized deployment for large models using AWQ, GPTQ, or FP8 to fit tighter GPU memory budgets.
  • Performance and operations guidance for batching, prefix caching, monitoring, troubleshooting, and distributed serving.
  • Use case: deploy a chat or completion API for a large model, then tune it for latency, throughput, and memory efficiency across one or more GPUs.

Quick Start

Ask me to configure a vLLM serving setup for your model and workload, and I will recommend the right deployment, quantization, and performance settings.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up high-throughput LLM serving with an OpenAI-compatible API?▼

High-throughput LLM serving with an OpenAI-compatible API is configured using vLLM, which applies PagedAttention and continuous batching to optimize model inference. You can deploy a chat or completion API endpoint and tune it for latency, throughput, and memory efficiency.

What is the best way to fit a large language model into a tight GPU memory budget?▼

To fit a large language model into a tight GPU memory budget, apply quantized deployment using AWQ, GPTQ, or FP8 formats. This reduces the memory footprint of the model, allowing it to run on fewer or less expensive GPUs during inference.

Does vLLM support multi-GPU and multi-node deployment for production inference?▼

vLLM supports multi-GPU and multi-node deployment for production API serving and offline batch generation. It provides performance and operations guidance for distributed serving to scale large language models across multiple hardware resources.

How does continuous batching and prefix caching improve inference throughput?▼

Continuous batching and prefix caching improve inference throughput by dynamically managing incoming requests and reusing previously computed key-value pairs. This reduces latency and increases the number of requests the model can process concurrently.

Can I use speculative decoding to optimize LLM serving performance?▼

Speculative decoding is supported to optimize LLM serving performance by reducing the time required for token generation. Combined with continuous batching and PagedAttention, it lowers latency and increases overall API throughput.

Related Skills

Explore More Software Engineering Skills→
DarkArty07DarkArty07
3

llama-cpp

Deploy local LLM inference with GGUF quantized weights via llama.cpp.

Community
Advanced
twjarviszyra-webtwjarviszyra-web
3

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible endpoints and tensor parallelism.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced