skills.rest
ovachiever
ovachieverCOMMUNITY
Explore all skills by this author →
Website𝕏@erikjfritschGitHub

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

52|6|Updated Nov 24, 2025
Search Tags:#production#quantization#OpenAI API#inference-serving#vllm#pagedattention
One-click install
npx skills add https://github.com/ovachiever/droid-tings --skill serving-llms-vllm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/ovachiever/droid-tings/tree/main/skills/vllm
Command: npx skills add https://github.com/ovachiever/droid-tings --skill serving-llms-vllm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill introduces vLLM-based serving for LLMs, delivering scalable, low-latency inference with features like PagedAttention, continuous batching, and OpenAI-compatible endpoints, including quantization and tensor parallelism.

Core Features & Use Cases

  • Production-grade serving: Deploy OpenAI-compatible LLM APIs with high throughput.
  • Memory-efficient inference: Leverage PagedAttention and quantization to fit large models on available GPUs.
  • Operational excellence: Tune latency, throughput, and monitoring in production, including metrics and health checks.

Quick Start

Start a local server for a 7B model with 1 GPU and test a completion request.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM with low latency and high throughput in production?▼

Deploy LLMs with vLLM using PagedAttention and continuous batching to achieve low-latency, high-throughput inference. vLLM optimizes GPU memory efficiency and serves OpenAI-compatible APIs for production workloads at scale.

Can I fit large language models on a single GPU using quantization?▼

Yes, vLLM supports AWQ, GPTQ, and FP8 quantization to reduce model memory footprint, allowing larger models to run on limited GPU resources while maintaining inference quality.

What's the best way to serve multiple LLM requests concurrently?▼

vLLM's continuous batching groups multiple inference requests into single batches, maximizing GPU utilization and throughput. This approach significantly increases concurrent request capacity compared to sequential serving.

Does vLLM support multi-GPU tensor parallelism for larger models?▼

Yes, vLLM enables tensor parallelism across multiple GPUs, distributing model layers to serve models larger than individual GPU memory while maintaining low latency.

How do I monitor and tune latency and throughput metrics in production?▼

vLLM provides built-in monitoring, health checks, and metrics collection to track latency and throughput. Operational tuning options let you balance latency-throughput trade-offs for your production SLA.

Can I use vLLM with existing OpenAI API client libraries?▼

Yes, vLLM exposes OpenAI-compatible endpoints, allowing drop-in replacement with standard OpenAI client libraries and existing integrations without code changes.

Related Skills

Explore More Software Engineering Skills→
Z43LZ43L

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference API with vLLM.

Community
Intermediate
koralliskorallis
90

monitoring-observability

Implement monitoring, logging, metrics, and tracing with Grafana and Sentry.

Community
Intermediate
informatico-madridinformatico-madrid
1

serving-llms-vllm

Serve LLMs with vLLM via OpenAI-compatible endpoints and quantization.

Community
Advanced
RheasilviaRheasilvia

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers using vLLM with PagedAttention and continuous batching.

Community
Advanced