skills.rest
KappTech88
KappTech88COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy vLLM servers with quantization and tensor parallelism on GPU clusters.

Updated Mar 30, 2026
Search Tags:#benchmarking#quantization#openai-compatible#vllm#llm-serving#tensor-parallelism#production-api
One-click install
npx skills add https://github.com/KappTech88/AI-RESEARCH-SKILLS-MCP --skill serving-llms-vllm-kapptech88
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/KappTech88/AI-RESEARCH-SKILLS-MCP/tree/main/skills/vllm
Command: npx skills add https://github.com/KappTech88/AI-RESEARCH-SKILLS-MCP --skill serving-llms-vllm-kapptech88

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill provides production-grade, high-throughput LLM serving with vLLM to run OpenAI-compatible APIs and scale inference on constrained hardware.

Core Features & Use Cases

  • High-throughput production serving: OpenAI-compatible endpoints with continuous batching and PagedAttention for low latency.
  • Memory-efficient deployment: Supports tensor parallelism and quantization (AWQ, GPTQ, FP8) to fit large models on limited GPUs.
  • Deployment patterns & monitoring: Provides guidance for Docker/Kubernetes deployments, health checks, and Prometheus metrics for production reliability.

Quick Start

Launch a vLLM server with your model and verify it responds to OpenAI-compatible requests.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve large language models with an OpenAI-compatible API in production?▼

You can serve large language models using an OpenAI-compatible API by deploying vLLM, which provides high-throughput endpoints with continuous batching and PagedAttention for low-latency production inference.

What's the best way to fit large LLMs on constrained GPUs during inference?▼

To fit large LLMs on constrained GPUs, apply memory-efficient deployment techniques like tensor parallelism and quantization methods such as AWQ, GPTQ, or FP8 to reduce the memory footprint during serving.

How does PagedAttention and continuous batching improve LLM serving throughput?▼

PagedAttention optimizes memory allocation for KV cache while continuous batching dynamically processes incoming requests, together maximizing GPU utilization and significantly improving LLM serving throughput.

Can I deploy vLLM on Kubernetes with production monitoring?▼

Yes, you can deploy vLLM on Docker and Kubernetes, utilizing provided deployment patterns, health checks, and Prometheus metrics to ensure production-grade reliability and monitoring.

Does vLLM support prefix caching to speed up repeated inference prompts?▼

Yes, vLLM implements prefix caching alongside PagedAttention and continuous batching to accelerate inference for repeated prompts and optimize overall serving latency.

Related Skills

Explore More Software Engineering Skills→
2nth-ai2nth-ai
1

tech/ollama/deployment

Deploy Ollama AI service on Docker, Kubernetes, Linux, and Apple Silicon with GPU support.

Official
Advanced
TitoPrauseeTitoPrausee

serving-llms-vllm

Deploy vLLM for high-throughput LLM serving with OpenAI-compatible APIs.

Community
Advanced
peteromalletpeteromallet
97

serving-llms-vllm

Deploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.

Community
Intermediate
zeronx798zeronx798

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible endpoints and PagedAttention.

Community
Advanced