Verified:GitHub

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Updated May 2, 2026

Search Tags:#quantization #vllm #pagedattention #tensor parallelism #llm serving #continuous batching #openai compatible api

Ready to run

npx skills add https://github.com/qcmuu/AI-Research-Skills --skill serving-llms-vllm-qcmuu

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/12-inference-serving/vllm
Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill serving-llms-vllm-qcmuu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve?

This Skill helps you deploy large language models behind a fast, reliable inference server so you can achieve high throughput and low latency in real applications.

Core Features & Use Cases

High-throughput serving: Uses vLLM’s PagedAttention and continuous batching to increase tokens/sec and requests/sec for multi-user workloads.
Production-ready API deployment: Runs an OpenAI-compatible server for chat/completions style integrations.
Memory-efficient model scaling: Supports tensor parallelism and quantization (AWQ/GPTQ/FP8) to fit larger models in limited VRAM.

Quick Start

Run a vLLM OpenAI-compatible server for your model on port 8000 with GPU memory utilization set to 0.9 so clients can immediately start sending requests.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I serve LLMs with high throughput and low latency for production traffic?▼

Serve LLMs with high throughput by deploying an inference endpoint using vLLM, which utilizes PagedAttention and continuous batching to increase tokens/sec for multi-user workloads. This configuration handles high concurrency reliably in real applications.

Can I fit larger LLMs in limited VRAM using quantization and tensor parallelism?▼

Yes, you can fit larger LLMs in limited VRAM by applying memory-efficient model scaling techniques. vLLM supports tensor parallelism alongside quantization methods like AWQ, GPTQ, and FP8 to reduce memory footprint during deployment.

How do I deploy an OpenAI-compatible API for chat completions using vLLM?▼

Deploy an OpenAI-compatible API by running a vLLM server configured on port 8000 with GPU memory utilization set to 0.9. This allows clients to immediately start sending requests for chat and completions style integrations.

What metrics should I validate to optimize vLLM server settings for latency?▼

Validate vLLM server performance by measuring Time To First Token (TTFT) and requests per second (req/sec) metrics. These metrics help tune throughput and latency settings for your specific production inference workload.

Does vLLM support offline batch inference in addition to real-time API serving?▼

Yes, vLLM supports offline batch inference in addition to real-time API serving. The inference endpoint applies to both API serving and offline batch processing, allowing flexible deployment across different traffic patterns.

Can I enable Prometheus monitoring for my vLLM inference endpoint?▼

Yes, you can optionally enable Prometheus monitoring for your vLLM inference endpoint. This allows you to track production traffic metrics and validate ongoing performance alongside the OpenAI-compatible server configuration.