skills.rest
lucasfth
lucasfthCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploys high-throughput LLM serving with PagedAttention and continuous batching for OpenAI-compatible endpoints.

1|Updated Apr 30, 2025
Search Tags:#quantization#inference#vllm#llm-serving#tensor-parallelism#production-api
One-click install
npx skills add https://github.com/lucasfth/config --skill serving-llms-vllm-lucasfth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/lucasfth/config/tree/main/.hermes/skills/mlops/inference/vllm
Command: npx skills add https://github.com/lucasfth/config --skill serving-llms-vllm-lucasfth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

LLM serving at production scale can be latency-sensitive and memory intensive; this Skill provides high-throughput hosting using vLLM's PagedAttention and continuous batching to optimize latency and throughput for OpenAI-compatible endpoints and model quantization workflows.

Core Features & Use Cases

  • High-throughput LLM serving with PagedAttention and continuous batching for production APIs.
  • OpenAI-compatible endpoints, quantization options (AWQ, GPTQ, FP8), and tensor parallelism to fit large models on multi-GPU setups.
  • Use Case: Deploy a production chat API for a 70B-model with low latency and high concurrency.

Quick Start

Start a vLLM server and expose an OpenAI-compatible endpoint for production traffic.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible API for high-throughput LLM serving?▼

Deploy an OpenAI-compatible API for high-throughput LLM serving using vLLM to leverage PagedAttention and continuous batching, optimizing latency and concurrency for production traffic.

Can I run a 70B LLM on multiple GPUs with limited GPU memory?▼

Run large 70B LLMs on multi-GPU setups with limited memory by applying tensor parallelism and quantization formats like AWQ, GPTQ, or FP8 to distribute the workload efficiently.

What is continuous batching and how does it optimize production inference workloads?▼

Continuous batching optimizes production inference workloads by dynamically scheduling incoming requests, maximizing GPU utilization and reducing latency for high-concurrency API endpoints.

What is the best way to reduce latency for a production chat API using a large language model?▼

Reduce latency for a production chat API by deploying the model with vLLM, which uses PagedAttention and continuous batching to handle high-concurrency requests efficiently.

Does vLLM support quantization options like AWQ and GPTQ for production inference?▼

vLLM supports quantization options including AWQ, GPTQ, and FP8 for production inference, allowing large models to fit within limited GPU memory while maintaining throughput.

When should I use tensor parallelism for LLM serving?▼

Use tensor parallelism for LLM serving when deploying large models across multiple GPUs to overcome limited GPU memory constraints and achieve production-grade throughput.

Related Skills

Explore More Software Engineering Skills→
zengbaochengzengbaocheng

serving-llms-vllm

Deploy OpenAI-compatible LLM endpoints with vLLM and tensor parallelism.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
projectedanxprojectedanx
1

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and continuous batching.

Community
Advanced
LamseyahEliasLamseyahElias

serving-llms-vllm

Deploy vLLM servers exposing OpenAI-compatible endpoints for LLM inference.

Community
Advanced