skills.rest
attentiondotnet
attentiondotnetCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploy vLLM servers with OpenAI-compatible endpoints and tensor parallelism.

Updated Mar 30, 2026
Search Tags:#llm#quantization#inference#openai-api#tensor-parallel#vllm#serving
One-click install
npx skills add https://github.com/attentiondotnet/hermes-agent --skill serving-llms-vllm-attentiondotnet
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/attentiondotnet/hermes-agent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/attentiondotnet/hermes-agent --skill serving-llms-vllm-attentiondotnet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve?

Deploy production-grade LLM servers that meet latency, throughput, and memory constraints for real-world applications.

Core Features & Use Cases

  • PagedAttention with continuous batching enables high throughput and lower latency for multi-user workloads.
  • OpenAI-compatible endpoints for easy integration with existing clients and tooling.
  • Quantization options (GPTQ, AWQ, FP8) to fit large models on limited GPU memory.
  • Tensor parallelism and multi-GPU deployments for scaling to larger models.
  • Production-ready workflows including monitoring, health checks, and safe deployment patterns.
  • Use Case: Deploy a chat assistant/API in production across teams with consistent latency and reliability.

Quick Start

Install vLLM and start a minimal server with your chosen model to expose an OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM server with high throughput and an OpenAI-compatible API?▼

Deploy high-throughput LLM servers using vLLM to expose OpenAI-compatible REST endpoints. PagedAttention with continuous batching enables lower latency and higher throughput for multi-user production workloads.

Can I fit large quantized LLMs on limited GPU memory for inference?▼

Yes, you can serve quantized LLMs on limited GPU memory using vLLM. It supports GPTQ, AWQ, and FP8 quantization options to reduce memory footprint while maintaining inference capabilities.

How does tensor parallelism work for multi-GPU LLM serving?▼

Tensor parallelism in vLLM enables multi-GPU LLM serving by splitting model layers across GPUs. This allows you to scale to larger models that exceed a single GPU's memory capacity during inference.

What is the best way to configure gpu-memory-utilization for production LLM inference?▼

Configuring gpu-memory-utilization in vLLM manages memory allocation for production LLM inference. This setting optimizes memory efficiency alongside PagedAttention to prevent out-of-memory errors under concurrent requests.

Does vLLM support continuous batching for concurrent chat API requests?▼

Yes, vLLM uses continuous batching with PagedAttention to process concurrent chat API requests efficiently. This mechanism dynamically schedules inference operations to maintain consistent latency across multi-user workloads.

What are the limitations when serving quantized models with tensor parallelism?▼

Serving quantized models with tensor parallelism requires careful configuration of gpu-memory-utilization and deployment patterns. You must ensure compatibility between your quantization format and multi-GPU hardware setup for deterministic serving.

Related Skills

Explore More Software Engineering Skills→
yangwhaleyangwhale
9

vllm-installer

Install, configure, and run vLLM on NVIDIA GPUs with CUDA.

Community
Advanced
frankxaifrankxai
2

nvidia-nim

Configure and run NVIDIA NIM inference via OpenAI-compatible APIs.

Community
Advanced
NITISH-gitbitNITISH-gitbit

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization via vLLM.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced