skills.rest
founderphantom
founderphantomCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLM inference with vLLM for OpenAI-compatible endpoints and multi-GPU deployment.

Updated Apr 1, 2026
Search Tags:#production#llm#inference#openai-api#vllm#serving#tensor-parallelism
One-click install
npx skills add https://github.com/founderphantom/zola-agent --skill serving-llms-vllm-founderphantom
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/founderphantom/zola-agent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/founderphantom/zola-agent --skill serving-llms-vllm-founderphantom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve?

Scalable, low-latency LLM inference for production-ready APIs by providing a robust serving solution that can handle OpenAI-compatible requests, memory optimization, and multi-GPU deployment.

Core Features & Use Cases

  • High-throughput OpenAI-compatible endpoints using vLLM for scalable inference.
  • Quantization support (GPTQ, AWQ, FP8) to fit larger models into limited GPU memory.
  • Tensor parallelism and multi-GPU deployment for production workloads such as chatbots, assistants, and enterprise deployments.
  • Use Case: Deploy a production API that serves real-time queries with low latency and high throughput while controlling memory usage.

Quick Start

Start a vLLM server configured for OpenAI-compatible endpoints and production workloads.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLM inference with an OpenAI-compatible API in production?▼

Serve LLM inference with an OpenAI-compatible API using vLLM to provide high-throughput endpoints for production workloads. It deploys scalable, low-latency APIs for chatbots and enterprise deployments.

How do I fit large LLMs into limited GPU memory for inference?▼

Fit large LLMs into limited GPU memory by applying quantization techniques during inference. vLLM supports GPTQ, AWQ, and FP8 quantization to control memory usage while serving production APIs.

What is the best way to deploy multi-GPU tensor parallelism for LLM serving?▼

Deploy multi-GPU tensor parallelism for LLM serving using vLLM to distribute inference workloads across GPUs. This provides memory-optimized, high-throughput deployments for production APIs.

Does vLLM require PyTorch and Transformers for production LLM serving?▼

vLLM does require PyTorch and Transformers for production LLM serving. These dependencies provide the foundation for memory-optimized inference and high-throughput endpoint deployment.

Can I use vLLM to optimize both latency and throughput for real-time LLM queries?▼

You can use vLLM to optimize both latency and throughput for real-time LLM queries. It provides production-grade inference designed to handle concurrent requests while maintaining low response times.

When should I use tensor parallelism instead of quantization for LLM inference?▼

Use tensor parallelism when multi-GPU deployment is available to maximize throughput, and use GPTQ or AWQ quantization to compress models into limited GPU memory for efficient inference serving.

Related Skills

Explore More Software Engineering Skills→
hsliuustc0106hsliuustc0106
84

vllm-omni-setup

Install and configure vLLM-Omni with Python environments and smoke tests.

Community
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
huidgehuidge
2

serving-llms-vllm

Deploy LLMs with vLLM for OpenAI-compatible serving and tensor parallelism.

Community
Advanced
peteromalletpeteromallet
97

serving-llms-vllm

Deploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.

Community
Intermediate