skills.rest
perasyudha
perasyudhaCOMMUNITY
Explore all skills by this author →
Website𝕏@Nyxora_AIGitHub

serving-llms-vllm

Deploy and optimize high-throughput LLM inference servers using vLLM with OpenAI-compatible APIs.

5|2|Updated May 26, 2026
Search Tags:#quantization#inference#vllm#openai api#tensor parallelism#gpu optimization#llm serving
One-click install
npx skills add https://github.com/perasyudha/Nyxora --skill serving-llms-vllm-perasyudha
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/perasyudha/Nyxora/tree/main/packages/core/playbooks/mlops/inference/vllm
Command: npx skills add https://github.com/perasyudha/Nyxora --skill serving-llms-vllm-perasyudha

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow, memory-hungry, and hard to scale. This Skill guides you through deploying vLLM, which uses PagedAttention and continuous batching to achieve up to 24x higher throughput than standard transformers while fitting large models into limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with monitoring, prefix caching, and Docker/Kubernetes deployment patterns. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and tensor parallelism across multiple GPUs. - Quantized Model Serving: Fit 70B models into 40GB VRAM using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: You need to serve Llama-3-8B to hundreds of concurrent chatbot users. Use this Skill to configure a vLLM server with prefix caching and metrics, verify TTFT under 500ms, and deploy via Docker. ## Quick Start Use the vllm skill to deploy a Llama-3-8B-Instruct server with an OpenAI-compatible API on port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM with vLLM as an OpenAI-compatible API?▼

Run vllm serve with your model name, such as vllm serve meta-llama/Llama-3-8B-Instruct, which starts an OpenAI-compatible server on port 8000. Query it using the OpenAI Python SDK with base_url set to http://localhost:8000/v1.

How do I serve a 70B model on limited GPU memory?▼

Use a pre-quantized AWQ model like TheBloke/Llama-2-70B-AWQ and launch with the --quantization awq flag. This reduces memory from 140GB to roughly 35GB, fitting on a single 40GB A100 GPU.

vLLM vs TensorRT-LLM vs llama.cpp: which should I use?▼

Use vLLM for production APIs needing high throughput and multi-platform support. Choose TensorRT-LLM for maximum NVIDIA-only performance, and llama.cpp for CPU or edge single-user inference.

Why does vLLM run out of memory during model loading?▼

OOM occurs when the model plus KV cache exceeds VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I fix slow time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompts, and use --enable-chunked-prefill for long prompts. Reducing concurrent sequences with --max-num-seqs also lowers latency.

Does vLLM support multiple GPUs and distributed serving?▼

Yes, vLLM supports tensor parallelism with --tensor-parallel-size using a power-of-2 GPU count, and pipeline parallelism across nodes for multi-node serving. It runs on NVIDIA, AMD ROCm, Intel GPUs, and TPUs.

Related Skills

Explore More Software Engineering Skills→
DaddyElonMusk69DaddyElonMusk69

obliteratus

Remove built-in refusal guardrails from HuggingFace model checkpoints.

Community
Advanced
api7api7
1

a6-plugin-prometheus

Expose APISIX metrics in Prometheus format for monitoring and dashboards.

Official
Intermediate
Supporter09Supporter09

evaluating-llms-harness

Benchmarks LLMs on-demand with plug-and-play pre-trained AI models.

Community
Advanced
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced