skills.rest
CUexter
CUexterCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy vLLM servers with OpenAI-compatible endpoints and tensor parallelism.

2|Updated Mar 26, 2026
Search Tags:#deployment#llm#inference#openai-compatible#serving#multi-gpu
One-click install
npx skills add https://github.com/CUexter/hermes-agent --skill serving-llms-vllm-cuexter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/CUexter/hermes-agent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/CUexter/hermes-agent --skill serving-llms-vllm-cuexter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploy production-grade LLM serving with vLLM to achieve high throughput and low latency for OpenAI-compatible endpoints, efficiently utilizing GPU resources.

Core Features & Use Cases

  • OpenAI-compatible endpoints with a scalable API
  • PagedAttention memory-efficient KV cache and continuous batching
  • Tensor parallelism across multiple GPUs and optional quantization (AWQ/GPTQ/FP8)
  • Use cases include hosting chatbots, assistants, and enterprise APIs with high request volume

Quick Start

Launch the vLLM server with your chosen model to expose an OpenAI-compatible API endpoint.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible LLM serving endpoint?▼

Deploy an OpenAI-compatible LLM serving endpoint using vLLM to expose a scalable API for high request volumes, enabling hosting for chatbots and enterprise assistants.

What's the best way to improve LLM inference throughput and reduce latency?▼

Improve LLM inference throughput and reduce latency by deploying vLLM, which utilizes PagedAttention for memory-efficient KV cache management alongside continuous batching.

Can I use tensor parallelism to serve LLMs across multiple GPUs?▼

Yes, you can serve LLMs across multiple GPUs using vLLM, which supports tensor parallelism to efficiently distribute workloads and manage high request volumes.

Does vLLM support model quantization for production deployment?▼

vLLM supports optional model quantization including AWQ, GPTQ, and FP8 formats, enabling memory-efficient operation and optimized runtime configuration for production LLM serving.

When do I need continuous batching for LLM inference?▼

You need continuous batching for LLM inference when managing high request volumes in production, allowing vLLM to maximize GPU utilization and maintain low latency.

Related Skills

Explore More Software Engineering Skills→
tangzheng202202tangzheng202202
1

serving-llms-vllm

Configure vLLM servers with OpenAI-compatible endpoints, quantization options, and tensor parallelism for scalable LLM deployments on GPU infrastructure.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
CODE-SAURABHCODE-SAURABH
2

system-design

Design scalable software architectures with explicit scale targets and failure tolerance.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced