skills.rest
zengbaocheng
zengbaochengCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy OpenAI-compatible LLM endpoints with vLLM and tensor parallelism.

Updated May 23, 2026
Search Tags:#production#api#llm#quantization#inference#vllm#serving
One-click install
npx skills add https://github.com/zengbaocheng/hermes-tech-hub --skill serving-llms-vllm-zengbaocheng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/zengbaocheng/hermes-tech-hub/tree/main/mlops/inference/vllm
Command: npx skills add https://github.com/zengbaocheng/hermes-tech-hub --skill serving-llms-vllm-zengbaocheng

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides scalable, production-grade LLM serving via a vLLM-based API that is compatible with OpenAI endpoints.

Core Features & Use Cases

  • High-throughput, low-latency LLM serving for OpenAI-compatible APIs.
  • Supports quantization for memory efficiency and tensor-parallel deployment across GPUs.
  • Use Case: Deploy a production chat API with multi-GPU scaling and monitoring.

Quick Start

Start a vLLM server configured to serve OpenAI-compatible endpoints for production-grade inference.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible API for LLM serving in production?▼

You can deploy an OpenAI-compatible API for LLM serving by starting a vLLM server configured with scalable endpoints, enabling high-throughput and low-latency inference directly in production environments.

What quantization methods does vLLM support for memory-efficient LLM inference?▼

vLLM supports AWQ, GPTQ, and FP8 quantization strategies for LLM inference, reducing memory consumption while maintaining scalable performance across production-grade deployments.

Can I use tensor parallelism to scale LLM serving across multiple GPUs?▼

Yes, you can use multi-GPU tensor parallelism in vLLM to scale LLM serving, distributing model execution across multiple GPUs to achieve high-throughput production inference.

How do I add monitoring to a production-grade vLLM API endpoint?▼

You can add integrated monitoring to your production-grade vLLM API endpoint by configuring the vLLM server, ensuring scalable LLM serving with continuous performance tracking across your deployment.

What is the best way to achieve high-throughput LLM inference without sacrificing OpenAI API compatibility?▼

Using vLLM is an effective way to achieve high-throughput LLM inference while maintaining OpenAI API compatibility, providing a production-grade serving environment with low latency and multi-GPU scaling.

Related Skills

Explore More Software Engineering Skills→
yangwhaleyangwhale
9

vllm-installer

Install, configure, and run vLLM on NVIDIA GPUs with CUDA.

Community
Advanced
koderoverkoderover
7

apisix

Manage Apache APISIX routes, upstreams, services, and plugins via the Admin API.

Official
Intermediate
ashiqcodeleafashiqcodeleaf

serving-llms-vllm

Deploy and serve LLM APIs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced