skills.rest
MarbleSodas
MarbleSodasCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploy high-throughput LLM inference with vLLM's OpenAI-compatible server interface.

Updated Apr 9, 2026
Search Tags:#llm#quantization#vllm#pagedattention#openai api#tensor parallelism#inference serving
One-click install
npx skills add https://github.com/MarbleSodas/Mavis --skill serving-llms-vllm-marblesodas
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/MarbleSodas/Mavis/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/MarbleSodas/Mavis --skill serving-llms-vllm-marblesodas

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill solves the challenge of deploying large language models behind an API with low latency and high throughput while staying within practical GPU memory limits.

Core Features & Use Cases

  • High-performance LLM serving: Uses vLLM’s PagedAttention and continuous batching to improve tokens/sec and reduce time to first token (TTFT).
  • OpenAI-compatible endpoints: Exposes an OpenAI-style server interface for easy client integration.
  • Quantized and parallel serving: Supports AWQ/GPTQ/FP8 quantization and tensor parallelism to fit bigger models or improve performance on supported hardware.

Quick Start

Run a local OpenAI-compatible server for an instruction model by starting vLLM with vllm serve meta-llama/Llama-3-8B-Instruct --port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with an OpenAI-compatible API using vLLM?▼

To serve LLMs with an OpenAI-compatible API, run vLLM's server with a model argument like `vllm serve meta-llama/Llama-3-8B-Instruct --port 8000` to expose high-throughput inference endpoints.

What is the best way to improve LLM inference throughput and reduce latency?▼

Using vLLM for LLM inference improves throughput and reduces time to first token (TTFT) by implementing PagedAttention and continuous batching for efficient memory management during generation.

Can I use quantization and tensor parallelism to fit larger LLMs on limited GPU memory?▼

Yes, vLLM supports AWQ, GPTQ, and FP8 quantization alongside tensor parallelism to fit larger models within GPU memory limits and improve serving performance on supported hardware.

Does vLLM support offline batch generation as well as interactive chat endpoints?▼

vLLM supports both interactive chat assistant endpoints and offline batch generation tasks through its OpenAI-compatible server interface, allowing flexible high-throughput LLM deployment.

What context length and GPU utilization settings do I need to configure for vLLM inference serving?▼

Deploying vLLM requires configuring the served context length, GPU utilization, and optional metrics or caching settings within the vllm runtime to optimize your specific LLM serving environment.

Why does my LLM serving deployment run out of memory, and when should I use quantization?▼

Memory constraints during LLM serving occur when models exceed GPU limits; applying AWQ, GPTQ, or FP8 quantization reduces the memory footprint, allowing bigger models to deploy successfully.

Related Skills

Explore More Software Engineering Skills→
guccangguccang
3

serving-llms-vllm

Deploy vLLM inference endpoints with OpenAI-compatible APIs and GPU optimization.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM with tensor parallelism and quantization.

Community
Intermediate
hung-phanhung-phan
1

vllm

Optimize LLM inference with PagedAttention, continuous batching, and quantization.

Community
Advanced