skills.rest
JamesFincher
JamesFincherCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM.

Updated May 4, 2026
Search Tags:#quantization#vllm#pagedattention#tensor parallelism#llm serving#openai-compatible api#throughput optimization
One-click install
npx skills add https://github.com/JamesFincher/gengar --skill serving-llms-vllm-jamesfincher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/JamesFincher/gengar/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/JamesFincher/gengar --skill serving-llms-vllm-jamesfincher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill solves the problem of deploying high-throughput, low-latency LLM inference services without rewriting your application when you need production-grade performance.

Core Features & Use Cases

  • High-throughput serving via vLLM’s PagedAttention and continuous batching to raise tokens/sec and requests/sec.
  • OpenAI-compatible API endpoint so clients can use the standard /v1 chat/completions workflow.
  • Memory-efficient deployment using quantization (AWQ/GPTQ/FP8) and tensor parallelism to fit larger models on available GPUs.

Use case example: You need an OpenAI-compatible endpoint for a multi-user chatbot backed by a 70B-class model, and you must keep TTFT low while maximizing throughput on limited GPU VRAM.

Quick Start

Run a local OpenAI-compatible vLLM server for your model by starting vLLM with vllm serve meta-llama/Llama-3-8B-Instruct --port 8000 --host 0.0.0.0.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with an OpenAI-compatible API using vLLM?▼

Serve LLMs with an OpenAI-compatible API by starting the `vllm serve` command with your model, port, and host. This deploys a high-throughput inference server supporting standard `/v1` chat/completions workflows for direct client requests.

How does vLLM achieve high throughput for LLM inference?▼

vLLM achieves high LLM inference throughput using PagedAttention and continuous batching. These mechanisms maximize tokens/sec and requests/sec by efficiently managing GPU memory utilization and processing concurrent multi-user generation requests.

Can I fit a 70B-class model on limited GPUs for LLM serving?▼

Fit 70B-class models on limited GPUs for LLM serving by applying quantization methods like AWQ, GPTQ, or FP8 alongside tensor parallelism. This memory-efficient deployment strategy maximizes throughput while keeping latency low on constrained VRAM.

What is the best way to reduce latency for a multi-user LLM chatbot?▼

Reduce latency for a multi-user LLM chatbot by deploying an OpenAI-compatible vLLM inference server. It utilizes continuous batching, prefix caching, and configurable GPU memory utilization to keep Time-To-First-Token low while maximizing overall throughput.

Does vLLM support offline batch inference pipelines?▼

vLLM supports offline batch inference pipelines alongside online serving. It leverages high-throughput generation features like continuous batching and PagedAttention to efficiently process large volumes of offline generation requests.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

llama-cpp

Run LLM inference on CPU and non-NVIDIA hardware with GGUF quantization.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
jayll1303jayll1303
17

model-quantization

Quantize large language models with GGUF, GPTQ, AWQ, and bitsandbytes.

Community
Advanced
lxh755818-botlxh755818-bot

serving-llms-vllm

Serve LLMs with OpenAI API, quantization, and tensor parallelism.

Community
Advanced