skills.rest
john-data-chen
john-data-chenCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Configure vLLM servers for OpenAI-compatible endpoints with Docker or Kubernetes.

2|Updated Apr 25, 2026
Search Tags:#production#optimization#quantization#inference#vllm#serving
One-click install
npx skills add https://github.com/john-data-chen/hermes-agent-backup --skill serving-llms-vllm-john-data-chen
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/john-data-chen/hermes-agent-backup/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/john-data-chen/hermes-agent-backup --skill serving-llms-vllm-john-data-chen

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides production-grade LLM serving using vLLM to maximize throughput and minimize latency, enabling scalable inference for OpenAI-compatible endpoints.

Core Features & Use Cases

  • OpenAI-compatible endpoints with vLLM serving
  • Quantization and tensor-parallel deployment for memory efficiency
  • Production-grade inference for batch and streaming workloads across multi-GPU setups

Quick Start

Start the vLLM server with your model to begin serving requests immediately.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs in production with vLLM to maximize throughput and minimize latency?▼

vLLM enables high-throughput LLM serving by utilizing PagedAttention for memory efficiency, OpenAI-compatible endpoints for API integration, and multi-GPU tensor parallelism to maximize throughput and minimize latency.

How do I deploy vLLM using Docker or Kubernetes for scalable inference?▼

Deploy vLLM in production using Docker or Kubernetes to manage multi-GPU configurations, enabling scalable batch and streaming workloads with OpenAI-compatible endpoints for distributed inference.

Can I use tensor parallelism and quantization in vLLM for multi-GPU memory efficiency?▼

Yes, vLLM supports tensor parallelism for multi-GPU setups and quantization techniques to optimize memory usage, allowing efficient deployment of large models across distributed hardware configurations.

What is PagedAttention and how does it optimize vLLM inference batching?▼

PagedAttention is a memory management mechanism in vLLM that optimizes inference batching by reducing memory fragmentation, enabling higher throughput and more efficient continuous batching for large language models.

Does vLLM support OpenAI-compatible endpoints for streaming workloads?▼

Yes, vLLM provides OpenAI-compatible endpoints that support both batch and streaming workloads, allowing seamless integration with existing applications built for the OpenAI API standard.

What are the limitations of serving quantized models with vLLM in production?▼

Serving quantized models with vLLM requires careful tuning of memory, batching, and monitoring, and may face limitations with specific hardware configurations or reduced model accuracy in multi-GPU deployments.

Related Skills

Explore More Software Engineering Skills→
x-TheFoxx-TheFox

serving-llms-vllm

Serve LLMs via an OpenAI-compatible API using vLLM.

Community
Advanced
CyapstayeCyapstaye
9

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and Prometheus metrics.

Community
Advanced
GitnappGitnapp

serving-llms-vllm

Deploy large language model inference services with vLLM and OpenAI-compatible endpoints.

Community
Advanced
jordanhubbardjordanhubbard
9

serving-llms-vllm

Serve large language models with high-throughput inference and OpenAI-compatible endpoints.

Community
Advanced