Verified:GitHub

serving-llms-vllm

Serve large language models with high-throughput inference for OpenAI-compatible endpoints.

1|Updated Apr 12, 2026

Search Tags:#quantization #inference #vllm #serving #tensor-parallelism #paged-attention #continuous-batching

One-click install

npx skills add https://github.com/cryptopafi/nexusos-skills --skill serving-llms-vllm-cryptopafi

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/cryptopafi/nexusos-skills/tree/main/mlops/inference/vllm
Command: npx skills add https://github.com/cryptopafi/nexusos-skills --skill serving-llms-vllm-cryptopafi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

vLLM enables serving large language models with high throughput and low latency while reducing GPU memory overhead, solving the challenge of scaling interactive and batch LLM workloads in production environments.

Core Features & Use Cases

High throughput serving: Continuous batching and PagedAttention deliver orders-of-magnitude higher request throughput and tokens/sec.
Production-ready API: OpenAI-compatible endpoints, Prometheus metrics, health checks, and deployment patterns for Docker and Kubernetes.
Memory- and cost-optimized deployments: Built-in support for AWQ/GPTQ/FP8 quantization, tensor parallelism, prefix caching, and speculative decoding to fit large models into limited GPU memory.
Use Case: Host an OpenAI-compatible LLM inference service that handles 100+ req/sec with TTFT under target, or run offline batch inference on large datasets with automatic internal batching.

Quick Start

Launch vLLM to serve meta-llama/Llama-3-8B-Instruct on localhost:8000 with gpu-memory-utilization=0.9 and metrics enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How does PagedAttention improve LLM inference throughput for production APIs?▼

Continuous batching processes new inference requests dynamically without waiting for ongoing batches to finish, maximizing GPU utilization and delivering high throughput for production LLM serving.

Can I deploy quantized models with AWQ or GPTQ for memory-optimized inference?▼

Tensor parallelism splits large language model layers across multiple GPUs, enabling you to serve models that exceed single-GPU memory limits while maintaining high-throughput inference.

How do I serve an OpenAI-compatible endpoint for high-throughput LLM inference?▼

Built-in Prometheus metrics, health checks, and Docker or Kubernetes deployment patterns provide the necessary monitoring and orchestration for production-ready LLM serving.

What is the best way to run offline batch inference on large datasets?▼

Prefix caching and speculative decoding optimize prompt processing and token generation speed, reducing latency further without compromising the OpenAI-compatible API output quality.