serving-llms-vllm

Name: serving-llms-vllm
Availability: InStock
Author: Harries

Serve LLMs with vLLM for OpenAI-compatible endpoints using tensor parallelism and quantization.

Updated Apr 24, 2026

Search Tags:#quantization #inference #openai-api #vllm #llm-serving #tensor-parallelism

Ready to run

npx skills add https://github.com/Harries/hermes-agent --skill serving-llms-vllm-harries

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Harries/hermes-agent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/Harries/hermes-agent --skill serving-llms-vllm-harries

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers.

What problem does it solve?

Efficiently deploy production-grade LLM serving by combining vLLM's paging attention with continuous batching to maximize throughput and minimize memory usage, while exposing OpenAI-compatible endpoints.

Core Features & Use Cases

High-throughput LLM serving with PagedAttention and continuous batching.
OpenAI-compatible API endpoints, multi-model support, and streaming responses.
Quantization options (GPTQ, AWQ, FP8) to fit large models on limited GPUs.
Tensor parallelism across GPUs for scaling large models.

Quick Start

Launch a vLLM server with your chosen model to expose a production-ready OpenAI-compatible API.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I serve LLMs with high throughput and low memory usage?▼

Serve LLMs using vLLM to combine PagedAttention with continuous batching, maximizing inference throughput and minimizing memory usage while exposing OpenAI-compatible endpoints.

How do I deploy an OpenAI-compatible API for large language models?▼

Deploy an OpenAI-compatible API by launching a vLLM server with your chosen model, providing production-ready endpoints with streaming responses and multi-model support.

Can I fit large quantized models on limited GPUs for inference?▼

You can fit large models on limited GPUs using vLLM quantization options including GPTQ, AWQ, and FP8, reducing memory requirements for efficient inference.

What's the best way to scale LLM serving across multiple GPUs?▼

Scale LLM serving across multiple GPUs using vLLM tensor parallelism, distributing large model inference workloads to achieve low latency and high throughput.

Do I need torch and transformers to run vLLM for inference?▼

Yes, you need torch and transformers as dependencies alongside vllm to run production-grade LLM inference, enabling high-throughput serving with continuous batching.