skills.rest
peteromallet
peteromalletCOMMUNITY
Explore all skills by this author →
𝕏@peteromalletGitHub

serving-llms-vllm

Deploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.

97|8|Updated Mar 20, 2026
Search Tags:#quantization#inference#openai-api#vllm#serving#tensor-parallelism
One-click install
npx skills add https://github.com/peteromallet/megaplan --skill serving-llms-vllm-peteromallet
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/peteromallet/megaplan/tree/main/megaplan/agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/peteromallet/megaplan --skill serving-llms-vllm-peteromallet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploying scalable LLM inference pipelines with limited GPU memory can be challenging; this Skill provides high-throughput serving with vLLM's PagedAttention and continuous batching, supporting OpenAI-compatible endpoints and quantization.

Core Features & Use Cases

  • High-throughput LLM serving using PagedAttention and continuous batching.
  • OpenAI-compatible endpoints with simple integration.
  • Quantization options (GPTQ/AWQ/FP8) and tensor parallelism for memory-constrained hardware.
  • Suitable for production deployments requiring low latency and scalable throughput.

Quick Start

Start by launching vLLM with your model to expose an OpenAI-compatible API endpoint and begin serving with optional quantization and tensor parallelism.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy high-throughput LLM inference with an OpenAI-compatible API endpoint?▼

Deploy high-throughput LLM inference using vLLM to expose an OpenAI-compatible API endpoint. It leverages PagedAttention and continuous batching to achieve low latency and scalable throughput for production-grade model serving.

How does vLLM handle serving large models on memory-constrained GPU hardware?▼

Serving large models on memory-constrained hardware is handled through vLLM's support for quantization options like AWQ, GPTQ, and FP8. These techniques reduce the memory footprint required for inference without severely degrading performance.

Can I use tensor parallelism in vLLM for multi-GPU cluster deployments?▼

Yes, you can use tensor parallelism in vLLM for multi-GPU deployments. Tensor parallelism splits the model across multiple GPUs in your cluster, enabling you to serve large language models that exceed the memory capacity of a single GPU.

What is the best way to achieve low latency and scalable throughput for production LLM serving?▼

The best way to achieve low latency and scalable throughput for production LLM serving is using vLLM. It optimizes inference pipelines using PagedAttention and continuous batching while supporting OpenAI-compatible endpoints and tensor parallelism.

Does vLLM support quantization formats like GPTQ and AWQ for model deployment?▼

Yes, vLLM supports quantization formats including GPTQ, AWQ, and FP8 for model deployment. Applying these quantization options allows you to deploy production-grade inference pipelines on memory-constrained hardware efficiently.

Related Skills

Explore More Software Engineering Skills→
vasilyu1983vasilyu1983
73

ai-llm-inference

Optimize LLM inference performance and cost with vLLM and quantization.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
Quill-AgentQuill-Agent
3

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization using vLLM.

Community
Advanced