skills.rest
t2ance
t2anceCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and manage high-throughput LLM serving with vLLM-powered OpenAI-compatible endpoints.

Updated Aug 23, 2026
Search Tags:#production#llm#quantization#inference#vllm#serving#tensor-parallelism
One-click install
npx skills add https://github.com/t2ance/dr-claw-plugin --skill serving-llms-vllm-t2ance
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/t2ance/dr-claw-plugin/tree/main/plugins/ml-training-stack/skills/inference-serving/vllm
Command: npx skills add https://github.com/t2ance/dr-claw-plugin --skill serving-llms-vllm-t2ance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill enables scalable LLM inference by leveraging vLLM's paged attention and continuous batching to maximize throughput and minimize latency on GPUs.

Core Features & Use Cases

  • High-throughput LLM serving with OpenAI-compatible endpoints.
  • Quantization options (GPTQ, AWQ, FP8) to reduce memory usage.
  • Tensor parallelism across multiple GPUs for large models.
  • Production deployment guidance and monitoring via the vLLM CLI and documentation.

Quick Start

Launch a vLLM server with your model and connect your client to the OpenAI-compatible API endpoint.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy high-throughput LLM serving with an OpenAI-compatible API?▼

High-throughput LLM serving is deployed using vLLM to launch a server that provides an OpenAI-compatible API endpoint. This leverages paged attention and continuous batching to maximize throughput and minimize latency on GPUs.

Can I use tensor parallelism across multiple GPUs for large language models?▼

Yes, tensor parallelism across multiple GPUs is supported for serving large language models. You can configure multi-GPU setups via the vLLM CLI to distribute model inference workloads and maximize throughput.

What quantization options are available to reduce memory usage during LLM inference?▼

Quantization options available to reduce memory usage during LLM inference include GPTQ, AWQ, and FP8. These formats are configurable when launching your vLLM server to optimize production deployments.

How do I configure and tune a vLLM server using the CLI for production?▼

You configure and tune a vLLM server for production using the vLLM CLI to set parameters for quantization, tensor parallelism, and OpenAI-compatible endpoints, satisfying configuration, tuning, monitoring, and reliability requirements.

What's the best way to maximize LLM inference throughput and minimize latency on GPUs?▼

The best way to maximize LLM inference throughput and minimize latency on GPUs is using vLLM. It achieves this through paged attention and continuous batching mechanisms optimized for high-volume production serving.

Does this approach to LLM serving support continuous batching and paged attention?▼

Yes, this approach to LLM serving supports continuous batching and paged attention. These mechanisms are core to vLLM, enabling scalable inference and optimized memory management for high-throughput endpoints.

Related Skills

Explore More Software Engineering Skills→
dawsonblockdawsonblock

serving-llms-vllm

Serve OpenAI-compatible LLM endpoints with vLLM and PagedAttention.

Community
Advanced
heysuhasheysuhas
2

serving-llms-vllm

Deploy LLM inference servers with the vLLM engine and OpenAI-compatible APIs.

Community
Advanced
davpatel605-beepdavpatel605-beep

serving-llms-vllm

Deploy and optimize vLLM inference servers with OpenAI-compatible APIs.

Community
Advanced
afel6afel6

serving-llms-vllm

Deploy LLM serving with vLLM and OpenAI-compatible endpoints.

Community
Advanced