skills.rest
humanerd-drew
humanerd-drewOFFICIAL
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM quantization and tensor parallelism.

2|7|Updated Jun 19, 2026
Search Tags:#quantization#OpenAI API#tensor parallelism#inference latency#LLM serving
One-click install
npx skills add https://github.com/humanerd-drew/opencode-drewgent --skill serving-llms-vllm-humanerd-drew
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/humanerd-drew/opencode-drewgent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/humanerd-drew/opencode-drewgent --skill serving-llms-vllm-humanerd-drew

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of deploying production-grade Large Language Model (LLM) APIs with optimized inference latency and throughput, suitable for serving models with limited GPU memory and supporting OpenAI-compatible endpoints.

Core Features & Use Cases

  • High Throughput: Achieves 24x higher throughput than standard transformers through PagedAttention and continuous batching.
  • OpenAI Compatibility: Supports OpenAI-compatible endpoints for seamless integration with existing systems.
  • Quantization: Offers quantization options like GPTQ, AWQ, and FP8 to fit large models in limited GPU memory.
  • Use Case: Deploying a chatbot that handles 100+ requests per second while maintaining high performance on a 40GB GPU.

Quick Start

Install vLLM and run:

vllm serve meta-llama/Llama-3-8B-Instruct

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM serving API with high throughput and low inference latency?▼

To deploy a high-throughput LLM serving API, use vllm with PagedAttention and continuous batching to achieve 24x higher throughput than standard transformers while maintaining optimized inference latency.

Can I serve large language models on limited GPU memory?▼

Yes, you can serve large language models on limited GPU memory by applying quantization options like GPTQ, AWQ, and FP8 to fit the model within your available VRAM constraints.

Does vllm support OpenAI-compatible API endpoints for production deployments?▼

Yes, vllm supports OpenAI-compatible endpoints, allowing seamless integration of your production-grade large language model APIs with existing systems and tools built for the OpenAI API standard.

What is the best way to scale LLM serving performance across multiple GPUs?▼

The best way to scale LLM serving performance is using tensor parallelism, which distributes the model across multiple GPUs to enhance performance and handle hundreds of requests per second.

Do I need torch and transformers to run an LLM serving deployment with vllm?▼

Yes, you need torch and transformers libraries installed as dependencies alongside vllm to successfully deploy and run production-grade large language model serving APIs.

Related Skills

Explore More Software Engineering Skills→
juliuss1907juliuss1907

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization for limited GPU memory.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
peteromalletpeteromallet
97

serving-llms-vllm

Deploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.

Community
Intermediate
carterwayneskhizeinecarterwayneskhizeine
19

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM with tensor parallelism and quantization.

Community
Intermediate