skills.rest
nelohenriq
nelohenriqCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve large language models with vLLM via OpenAI-compatible endpoints.

1|Updated Mar 22, 2026
Search Tags:#production#quantization#inference#openai-api#vllm#llm-serving#tensor-parallelism
One-click install
npx skills add https://github.com/nelohenriq/hermes-agent-plus --skill serving-llms-vllm-nelohenriq
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/nelohenriq/hermes-agent-plus/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/nelohenriq/hermes-agent-plus --skill serving-llms-vllm-nelohenriq

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provide production-grade serving for large language models with high throughput using vLLM, enabling scalable, low-latency inference in multi-user environments.

Core Features & Use Cases

  • High-throughput LLM serving with PagedAttention and continuous batching
  • OpenAI-compatible endpoints and quantization support (AWQ, GPTQ, FP8)
  • Tensor parallelism across GPUs for large model deployment
  • Use cases include API-backed LLM services, batch inference pipelines, and scalable chat systems.

Quick Start

Launch a vLLM server configured for your model and monitoring.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy a high-throughput LLM serving endpoint?▼

To deploy high-throughput LLM serving, you can use vLLM to expose an OpenAI-compatible API, applying PagedAttention and continuous batching for scalable, low-latency inference in multi-user environments.

What is the best way to serve large language models across multiple GPUs?▼

Serving large language models across multiple GPUs is best handled using vLLM's tensor parallelism, which splits model execution to support large deployments while maximizing throughput.

How do I set up an OpenAI-compatible API for inference?▼

You can set up an OpenAI-compatible API for inference by launching a vLLM server, which natively exposes a REST API endpoint compatible with OpenAI client libraries for your large language models.

Can I use quantization methods like AWQ or FP8 for LLM serving?▼

Yes, you can use quantization methods like AWQ, GPTQ, and FP8 for LLM serving with vLLM to reduce memory usage and increase throughput during deployment.

Do I need torch and transformers to run a vLLM server?▼

Yes, you need dependencies such as torch and transformers installed in your environment to run a vLLM server and execute production-grade inference workflows.

Related Skills

Explore More Software Engineering Skills→
handsomelong922handsomelong922

llamaguard

Classify and filter unsafe content in prompts and model responses.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM with tensor parallelism and quantization.

Community
Intermediate
dawsonblockdawsonblock

serving-llms-vllm

Serve OpenAI-compatible LLM endpoints with vLLM and PagedAttention.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced