skills.rest
Harries
HarriesCOMMUNITY
Explore all skills by this author →
Website𝕏@HarriesSteeleGitHub

serving-llms-vllm

Serve LLMs with vLLM for OpenAI-compatible endpoints using tensor parallelism and quantization.

Updated Apr 24, 2026
Search Tags:#quantization#inference#openai-api#vllm#llm-serving#tensor-parallelism
One-click install
npx skills add https://github.com/Harries/hermes-agent --skill serving-llms-vllm-harries
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Harries/hermes-agent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/Harries/hermes-agent --skill serving-llms-vllm-harries

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers.

What problem does it solve?

Efficiently deploy production-grade LLM serving by combining vLLM's paging attention with continuous batching to maximize throughput and minimize memory usage, while exposing OpenAI-compatible endpoints.

Core Features & Use Cases

  • High-throughput LLM serving with PagedAttention and continuous batching.
  • OpenAI-compatible API endpoints, multi-model support, and streaming responses.
  • Quantization options (GPTQ, AWQ, FP8) to fit large models on limited GPUs.
  • Tensor parallelism across GPUs for scaling large models.

Quick Start

Launch a vLLM server with your chosen model to expose a production-ready OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs with high throughput and low memory usage?▼

Serve LLMs using vLLM to combine PagedAttention with continuous batching, maximizing inference throughput and minimizing memory usage while exposing OpenAI-compatible endpoints.

How do I deploy an OpenAI-compatible API for large language models?▼

Deploy an OpenAI-compatible API by launching a vLLM server with your chosen model, providing production-ready endpoints with streaming responses and multi-model support.

Can I fit large quantized models on limited GPUs for inference?▼

You can fit large models on limited GPUs using vLLM quantization options including GPTQ, AWQ, and FP8, reducing memory requirements for efficient inference.

What's the best way to scale LLM serving across multiple GPUs?▼

Scale LLM serving across multiple GPUs using vLLM tensor parallelism, distributing large model inference workloads to achieve low latency and high throughput.

Do I need torch and transformers to run vLLM for inference?▼

Yes, you need torch and transformers as dependencies alongside vllm to run production-grade LLM inference, enabling high-throughput serving with continuous batching.

Related Skills

Explore More Software Engineering Skills→
RedWoodOGRedWoodOG
174

serving-llms-vllm

Deploy scalable LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
Clay-HHKClay-HHK
2

serving-llms-vllm

Serve LLMs via vLLM with OpenAI-compatible endpoints and quantization options.

Community
Advanced
ihatesea69ihatesea69
3

serving-llms-vllm

Serve LLMs with vLLM PagedAttention and continuous batching.

Community
Advanced