skills.rest
tangzheng202202
tangzheng202202COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Configure vLLM servers with OpenAI-compatible endpoints, quantization options, and tensor parallelism for scalable LLM deployments on GPU infrastructure.

1|Updated Apr 13, 2026
Search Tags:#deployment#production#gpu#quantization#openai-api#vllm#llm-serving
One-click install
npx skills add https://github.com/tangzheng202202/hermes-skills --skill serving-llms-vllm-tangzheng202202
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/tangzheng202202/hermes-skills/tree/main/03-mlops/mlops/inference/vllm
Command: npx skills add https://github.com/tangzheng202202/hermes-skills --skill serving-llms-vllm-tangzheng202202

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploying scalable LLM services for production environments is complex, requiring efficient inference, memory management, and robust API endpoints.

Core Features & Use Cases

  • OpenAI-compatible endpoints for seamless client integration.
  • Quantization options (AWQ, GPTQ, FP8) to fit large models into available VRAM.
  • Tensor parallelism and multi-GPU scaling for large models and higher throughput.
  • Production deployment guidance including Docker and Kubernetes, plus monitoring and health checks.
  • Use Case: Build a production-grade chat API capable of handling thousands of concurrent users with low latency.

Quick Start

Start the vLLM server to expose a model as an OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API in production?▼

To deploy an LLM as an OpenAI-compatible API in production, you can use vLLM to serve models with high throughput. It provides endpoints that seamlessly integrate with existing OpenAI API clients for scalable deployment.

What's the best way to fit large LLMs into limited GPU VRAM during serving?▼

The best way to fit large LLMs into limited GPU VRAM during serving is applying quantization. vLLM supports AWQ, GPTQ, and FP8 quantization options to reduce memory footprint while maintaining inference performance.

How does tensor parallelism work for multi-GPU LLM serving?▼

Tensor parallelism for multi-GPU LLM serving works by partitioning model tensors across multiple GPUs. vLLM leverages this technique to scale large models and achieve higher throughput in production environments.

Can I use Docker and Kubernetes for scalable LLM serving deployment?▼

Yes, you can use Docker and Kubernetes for scalable LLM serving deployment. vLLM provides production deployment guidance that includes containerization, orchestration, monitoring, and health checks for robust infrastructure.

Do I need GPU-enabled infrastructure to run a vLLM server?▼

Yes, you need GPU-enabled infrastructure to run a vLLM server. High-throughput LLM serving requires a vLLM runtime, a Python environment, and GPU hardware to efficiently handle thousands of concurrent users with low latency.

Related Skills

Explore More Software Engineering Skills→
samlaggzsamlaggz

serving-llms-vllm

Serve large language models via vLLM for OpenAI-compatible endpoints.

Community
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
Ced3-hanCed3-han

serving-llms-vllm

Serve LLMs with vLLM using PagedAttention and continuous batching.

Community
Advanced