serving-llms-vllm

Deploys high-throughput LLM serving with PagedAttention and continuous batching for OpenAI-compatible endpoints.

1|Updated Apr 30, 2025

Search Tags:#quantization #inference #vllm #llm-serving #tensor-parallelism #production-api

Ready to run

npx skills add https://github.com/lucasfth/config --skill serving-llms-vllm-lucasfth

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/lucasfth/config/tree/main/.hermes/skills/mlops/inference/vllm
Command: npx skills add https://github.com/lucasfth/config --skill serving-llms-vllm-lucasfth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

LLM serving at production scale can be latency-sensitive and memory intensive; this Skill provides high-throughput hosting using vLLM's PagedAttention and continuous batching to optimize latency and throughput for OpenAI-compatible endpoints and model quantization workflows.

Core Features & Use Cases

High-throughput LLM serving with PagedAttention and continuous batching for production APIs.
OpenAI-compatible endpoints, quantization options (AWQ, GPTQ, FP8), and tensor parallelism to fit large models on multi-GPU setups.
Use Case: Deploy a production chat API for a 70B-model with low latency and high concurrency.

Quick Start

Start a vLLM server and expose an OpenAI-compatible endpoint for production traffic.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deploy an OpenAI-compatible API for high-throughput LLM serving?▼

Deploy an OpenAI-compatible API for high-throughput LLM serving using vLLM to leverage PagedAttention and continuous batching, optimizing latency and concurrency for production traffic.

Can I run a 70B LLM on multiple GPUs with limited GPU memory?▼

Run large 70B LLMs on multi-GPU setups with limited memory by applying tensor parallelism and quantization formats like AWQ, GPTQ, or FP8 to distribute the workload efficiently.

What is continuous batching and how does it optimize production inference workloads?▼

Continuous batching optimizes production inference workloads by dynamically scheduling incoming requests, maximizing GPU utilization and reducing latency for high-concurrency API endpoints.

What is the best way to reduce latency for a production chat API using a large language model?▼

Reduce latency for a production chat API by deploying the model with vLLM, which uses PagedAttention and continuous batching to handle high-concurrency requests efficiently.

Does vLLM support quantization options like AWQ and GPTQ for production inference?▼

vLLM supports quantization options including AWQ, GPTQ, and FP8 for production inference, allowing large models to fit within limited GPU memory while maintaining throughput.

When should I use tensor parallelism for LLM serving?▼

Use tensor parallelism for LLM serving when deploying large models across multiple GPUs to overcome limited GPU memory constraints and achieve production-grade throughput.