skills.rest
1thirteeng3
1thirteeng3COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy vLLM inference APIs with PagedAttention and continuous batching.

Updated Mar 24, 2026
Search Tags:#deployment#inference#openai-api#vllm#llm-serving#tensor-parallelism
One-click install
npx skills add https://github.com/1thirteeng3/greenmoire --skill serving-llms-vllm-1thirteeng3
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/1thirteeng3/greenmoire/tree/main/integrations/hermes-agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/1thirteeng3/greenmoire --skill serving-llms-vllm-1thirteeng3

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Production-grade LLM inference services require high throughput, low latency, and memory efficiency to support OpenAI-compatible endpoints and real-time workloads.

Core Features & Use Cases

  • High-throughput serving using vLLM's PagedAttention and continuous batching to maximize GPU utilization.
  • OpenAI-compatible endpoints with support for quantization methods (AWQ, GPTQ, FP8) and tensor parallelism for memory-constrained deployments.
  • Production deployment guidance, including Docker/Kubernetes patterns and monitoring recommendations for multi-user AI applications.

Quick Start

Launch a vLLM-based LLM service and expose it via an OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible LLM API with vLLM?▼

Deploy an OpenAI-compatible LLM API by launching vLLM to expose endpoints, utilizing PagedAttention and continuous batching to maximize GPU throughput for real-time workloads. The skill guides configuring the service to ensure high performance and memory efficiency.

What is the best way to scale LLM inference across multiple GPUs?▼

The best way to scale LLM inference across multiple GPUs is using vLLM's tensor parallelism. This approach splits model layers across available hardware, enabling memory-constrained deployments to handle large models without exceeding GPU memory limits.

Does vLLM support quantization methods like AWQ, GPTQ, and FP8?▼

vLLM supports quantization methods including AWQ, GPTQ, and FP8. These techniques reduce memory footprint and increase throughput, allowing production-grade LLM inference services to run efficiently on memory-constrained hardware setups.

Can I run vLLM inference services in Docker and Kubernetes?▼

You can run vLLM inference services in Docker and Kubernetes. The skill provides production deployment patterns for multi-node setups, including monitoring recommendations and scaling configurations for multi-user AI applications.

Why use PagedAttention and continuous batching for LLM serving?▼

Use PagedAttention and continuous batching for LLM serving to achieve high throughput and low latency. These vLLM features maximize GPU utilization by dynamically managing memory and processing multiple requests simultaneously in production environments.

Do I need torch and transformers to serve models with vLLM?▼

You need torch and transformers dependencies to serve models with vLLM. These frameworks provide the underlying neural network operations and model loading capabilities required to execute production-grade LLM inference workloads.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

quantizing-models-bitsandbytes

Quantize LLMs to 8-bit or 4-bit precision using BitsAndBytes.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
tianhao909tianhao909
1

serving-llms-vllm

Serve LLMs with vLLM, PagedAttention, and continuous batching.

Community
Advanced
ihatesea69ihatesea69
3

serving-llms-vllm

Serve LLMs with vLLM PagedAttention and continuous batching.

Community
Advanced