serving-llms-vllm

Serve large language models via vLLM with OpenAI-compatible endpoints and quantization.

2|Updated Apr 25, 2026

Search Tags:#LLM #throughput #quantization #OpenAI-compatible #inference-serving #vllm #tensor-parallelism

Ready to run

npx skills add https://github.com/AlexiosBluffMara/mercury --skill serving-llms-vllm-alexiosbluffmara

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/AlexiosBluffMara/mercury/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/AlexiosBluffMara/mercury --skill serving-llms-vllm-alexiosbluffmara

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers.

What problem does it solve?

This skill provides production-grade serving of large language models using vLLM, enabling high throughput with OpenAI-compatible endpoints and memory-efficient deployment via PagedAttention and continuous batching.

Core Features & Use Cases

High-throughput OpenAI-compatible API serving for LLMs
Memory-efficient deployment with quantization (GPTQ/AWQ/FP8) and tensor parallelism
Suitable for production APIs, batch inference, and multi-model deployments

Quick Start

Launch the vLLM server with your model to start serving OpenAI-compatible requests immediately.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I serve an LLM with an OpenAI-compatible API using vLLM?▼

To serve an LLM with an OpenAI-compatible API using vLLM, launch the vLLM server with your selected model. This immediately provides high-throughput OpenAI-compatible endpoints for production requests.

What is the best way to run batch inference on memory-constrained GPUs?▼

The best way to run batch inference on memory-constrained GPUs is using vLLM with quantization formats like GPTQ, AWQ, or FP8. This reduces memory consumption while maintaining production-grade throughput.

How does PagedAttention and continuous batching improve LLM throughput?▼

PagedAttention and continuous batching improve LLM throughput by dynamically managing memory pages and grouping incoming requests. This mechanism significantly maximizes GPU utilization during inference serving.

Can I use tensor parallelism in vLLM for multi-GPU model deployment?▼

Yes, you can use tensor parallelism in vLLM for multi-GPU model deployment. It splits model tensors across multiple GPUs, enabling low-latency chat services and high-throughput inference for large models.

Does vLLM support GPTQ and AWQ quantization for inference serving?▼

Yes, vLLM supports GPTQ, AWQ, and FP8 quantization for inference serving. These formats enable memory-efficient deployment of large language models on hardware with limited VRAM.

What are the limitations of using vLLM for production LLM serving?▼

Limitations of using vLLM for production LLM serving include strict dependencies on torch and transformers. Users must ensure their GPU hardware supports the required tensor parallelism and quantization features.