Verified:GitHub

serving-llms-vllm

Name: serving-llms-vllm
Availability: InStock
Author: tangzheng202202

Configure vLLM servers with OpenAI-compatible endpoints, quantization options, and tensor parallelism for scalable LLM deployments on GPU infrastructure.

1|Updated Apr 13, 2026

Search Tags:#deployment #production #gpu #quantization #openai-api #vllm #llm-serving

Ready to run

npx skills add https://github.com/tangzheng202202/hermes-skills --skill serving-llms-vllm-tangzheng202202

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/tangzheng202202/hermes-skills/tree/main/03-mlops/mlops/inference/vllm
Command: npx skills add https://github.com/tangzheng202202/hermes-skills --skill serving-llms-vllm-tangzheng202202

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploying scalable LLM services for production environments is complex, requiring efficient inference, memory management, and robust API endpoints.

Core Features & Use Cases

OpenAI-compatible endpoints for seamless client integration.
Quantization options (AWQ, GPTQ, FP8) to fit large models into available VRAM.
Tensor parallelism and multi-GPU scaling for large models and higher throughput.
Production deployment guidance including Docker and Kubernetes, plus monitoring and health checks.
Use Case: Build a production-grade chat API capable of handling thousands of concurrent users with low latency.

Quick Start

Start the vLLM server to expose a model as an OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deploy an LLM as an OpenAI-compatible API in production?▼

To deploy an LLM as an OpenAI-compatible API in production, you can use vLLM to serve models with high throughput. It provides endpoints that seamlessly integrate with existing OpenAI API clients for scalable deployment.

What's the best way to fit large LLMs into limited GPU VRAM during serving?▼

The best way to fit large LLMs into limited GPU VRAM during serving is applying quantization. vLLM supports AWQ, GPTQ, and FP8 quantization options to reduce memory footprint while maintaining inference performance.

How does tensor parallelism work for multi-GPU LLM serving?▼

Tensor parallelism for multi-GPU LLM serving works by partitioning model tensors across multiple GPUs. vLLM leverages this technique to scale large models and achieve higher throughput in production environments.

Can I use Docker and Kubernetes for scalable LLM serving deployment?▼

Yes, you can use Docker and Kubernetes for scalable LLM serving deployment. vLLM provides production deployment guidance that includes containerization, orchestration, monitoring, and health checks for robust infrastructure.

Do I need GPU-enabled infrastructure to run a vLLM server?▼

Yes, you need GPU-enabled infrastructure to run a vLLM server. High-throughput LLM serving requires a vLLM runtime, a Python environment, and GPU hardware to efficiently handle thousands of concurrent users with low latency.