skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

serving-llms-vllm

Deploy vLLM serving infrastructure with OpenAI-compatible endpoints and quantization.

228|35|Updated Mar 9, 2026
Search Tags:#openai-api#inference-serving#vllm#high-throughput#tensor-parallelism#paged-attention#continuous-batching
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha --skill serving-llms-vllm-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/kaminocorp/hermes-alpha/tree/main/hermes-agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/kaminocorp/hermes-alpha --skill serving-llms-vllm-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploy production-grade LLM serving infrastructure using vLLM to achieve high throughput and low latency.

Core Features & Use Cases

  • OpenAI-compatible endpoints: Expose scalable, OpenAI-compatible APIs for apps and experiments.
  • Memory-efficient serving: Leverage PagedAttention and continuous batching to maximize throughput on GPU clusters.
  • Quantization & tensor parallelism: Support AWQ, GPTQ, FP8 quantization and tensor parallelism to fit large models in constrained hardware.

Quick Start

Launch a vLLM server with your chosen model to expose an OpenAI-compatible API endpoint.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy a high-throughput LLM serving infrastructure?▼

Deploy high-throughput LLM serving infrastructure using vLLM to achieve low latency and maximum GPU utilization. It leverages PagedAttention and continuous batching to handle production workloads efficiently.

Can I expose an OpenAI-compatible API endpoint for my local models?▼

Yes, you can expose an OpenAI-compatible API endpoint for your local models. Launching a vLLM server with your chosen model directly provides a scalable, OpenAI-compatible API for apps and experiments.

How does PagedAttention and continuous batching improve LLM inference serving?▼

PagedAttention and continuous batching improve LLM inference serving by maximizing throughput and memory efficiency on GPU clusters. They dynamically manage memory allocation to process incoming requests concurrently without padding overhead.

Does vLLM support quantized model serving with tensor parallelism?▼

Yes, vLLM supports quantized model serving with tensor parallelism. It applies AWQ, GPTQ, and FP8 quantization methods alongside tensor parallelism to fit large models within memory-constrained hardware deployments.

What is the best way to serve large LLMs on memory-constrained GPUs?▼

The best way to serve large LLMs on memory-constrained GPUs is applying vLLM with AWQ, GPTQ, or FP8 quantization. This reduces the memory footprint while maintaining high-throughput inference and low latency.

Related Skills

Explore More Software Engineering Skills→
DaddyElonMusk69DaddyElonMusk69

serving-llms-vllm

Serve large language models with high throughput using vLLM.

Community
Advanced
ChimeraFoundationaChimeraFoundationa

serving-llms-vllm

Serve LLMs with vLLM using pagedAttention and continuous batching.

Community
Advanced
bailynlovebailynlove
1

serving-llms-vllm

Serve OpenAI-compatible LLM APIs with PagedAttention and continuous batching.

Community
Advanced
ihatesea69ihatesea69
3

serving-llms-vllm

Serve LLMs with vLLM PagedAttention and continuous batching.

Community
Advanced