skills.rest
Dq666dinger
Dq666dingerCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs via an OpenAI-compatible API using vLLM with quantization and tensor parallelism.

1|Updated May 10, 2026
Search Tags:#quantization#openai-compatible#vllm#llm-serving#multi-gpu#production-inference
One-click install
npx skills add https://github.com/Dq666dinger/my-hermes --skill serving-llms-vllm-dq666dinger
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Dq666dinger/my-hermes/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/Dq666dinger/my-hermes --skill serving-llms-vllm-dq666dinger

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill enables production-grade LLM serving by running models with vLLM and exposing an OpenAI-compatible API, delivering high throughput and efficient memory usage.

Core Features & Use Cases

  • High-throughput LLM serving using vLLM's PagedAttention and continuous batching.
  • Quantization options (AWQ, GPTQ, FP8) to fit large models on smaller GPUs.
  • OpenAI-compatible endpoints and multi-GPU tensor parallelism for scalable inference.

Quick Start

Launch a vLLM server for your model with your chosen quantization and tensor-parallel settings to begin serving an OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an OpenAI-compatible API for production LLM inference?▼

You can deploy a production LLM serving endpoint using vLLM to expose an OpenAI-compatible API, leveraging high throughput and efficient memory management for scalable inference.

What is the best way to fit large LLMs on smaller GPUs for production serving?▼

Quantization is the best way to fit large LLMs on smaller GPUs. Applying formats like AWQ, GPTQ, or FP8 reduces memory requirements for production serving on budget-constrained hardware.

How does vLLM achieve high throughput for LLM serving?▼

vLLM achieves high throughput for LLM serving by using PagedAttention for efficient memory allocation and continuous batching to process incoming requests dynamically, maximizing GPU utilization.

Can I use multi-GPU tensor parallelism for scalable LLM serving?▼

Yes, you can use multi-GPU tensor parallelism for scalable LLM serving. vLLM supports distributing model computation across multiple GPUs to increase inference capacity for large models.

Do I need quantization to deploy an OpenAI-compatible API with vLLM?▼

No, quantization is optional. You can deploy an OpenAI-compatible API with vLLM using standard model weights, but applying quantization helps fit larger models on smaller GPUs.

When should I not use continuous batching for production LLM inference?▼

You should avoid continuous batching if your workload consists of single, isolated requests requiring minimal latency, as the batching mechanism optimizes for high throughput over individual request speed.

Related Skills

Explore More Software Engineering Skills→
DaddyElonMusk69DaddyElonMusk69

gguf-quantization

Convert models to GGUF format with 2-8 bit quantization for llama.cpp.

Community
Intermediate
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
manutejmanutej
1

api-gateway-patterns

Implement Kong API gateway patterns for routing, authentication, and rate limiting.

Community
Advanced
ihatesea69ihatesea69
3

serving-llms-vllm

Serve LLMs with vLLM PagedAttention and continuous batching.

Community
Advanced