skills.rest
whichguy
whichguyCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs with vLLM, OpenAI API, quantization, and tensor parallelism.

1|Updated Jun 23, 2026
Search Tags:#quantization#OpenAI API#tensor parallelism#high throughput#LLM serving
One-click install
npx skills add https://github.com/whichguy/hermes-skills-marketplace --skill serving-llms-vllm-whichguy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/whichguy/hermes-skills-marketplace/tree/main/skills/serving-llms-vllm
Command: npx skills add https://github.com/whichguy/hermes-skills-marketplace --skill serving-llms-vllm-whichguy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines production-level LLM API deployment and inference by optimizing for high throughput, low latency, and efficient GPU usage.

Core Features & Use Cases

  • High Throughput: Achieves 24x higher throughput through PagedAttention and continuous batching.
  • OpenAI API: Supports OpenAI-compatible endpoints for seamless integration.
  • Quantization: Offers quantization methods like GPTQ/AWQ/FP8 for efficient GPU usage.
  • Use Case: Deploy vLLM in production to serve large LLM models with high throughput and low latency, even on limited GPU resources.

Quick Start

Use the vLLM skill to serve the meta-llama/Llama-3-8B-Instruct model with PagedAttention and quantization:

vllm serve meta-llama/Llama-3-8B-Instruct --quantization awq --tensor-parallel-size 2

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs with high throughput and low latency in production?▼

High-throughput LLM serving is achieved using PagedAttention and continuous batching, delivering up to 24x higher throughput. This approach optimizes inference latency and GPU memory usage for production-grade deployments.

Can I use an OpenAI-compatible API for my self-hosted LLM deployment?▼

Yes, you can use an OpenAI-compatible API for self-hosted LLM serving. This provides seamless integration with existing OpenAI API clients while running models on your own GPU infrastructure.

What's the best way to reduce GPU memory usage when serving large LLM models?▼

Quantization is the best way to reduce GPU memory usage when serving large LLMs. Methods like GPTQ, AWQ, and FP8 enable efficient GPU utilization, allowing deployment on limited hardware resources.

How do I deploy a large LLM across multiple GPUs using tensor parallelism?▼

To deploy a large LLM across multiple GPUs, use the tensor parallelism feature by setting the tensor-parallel-size parameter. This splits the model layers across available GPUs to accelerate inference.

Do I need torch and transformers to run a vLLM serving endpoint?▼

Yes, you need torch and transformers Python libraries to run a vLLM serving endpoint. These dependencies are required alongside vllm to execute production-grade LLM inference workloads.

Related Skills

Explore More Software Engineering Skills→
lxh755818-botlxh755818-bot

serving-llms-vllm

Serve LLMs with OpenAI API, quantization, and tensor parallelism.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
NITISH-gitbitNITISH-gitbit

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization via vLLM.

Community
Advanced
luokai25luokai25

serving-llms-vllm

Deploy high-throughput LLM APIs with vLLM's PagedAttention and continuous batching.

Community
Advanced