skills.rest
Ced3-han
Ced3-hanCOMMUNITY
Explore all skills by this author →
Website𝕏@JiahanCedGitHub

serving-llms-vllm

Serve LLMs with vLLM using PagedAttention and continuous batching.

Updated Apr 30, 2026
Search Tags:#OpenAI#LLM#quantization#vLLM#high throughput#inference serving
One-click install
npx skills add https://github.com/Ced3-han/Harness-Settings --skill serving-llms-vllm-ced3-han
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Ced3-han/Harness-Settings/tree/main/skills/vllm
Command: npx skills add https://github.com/Ced3-han/Harness-Settings --skill serving-llms-vllm-ced3-han

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a solution for high-throughput, production-grade LLM serving, addressing issues related to inference latency and GPU memory constraints, while supporting OpenAI-compatible endpoints and model quantization.

Core Features & Use Cases

  • High Throughput Serving: Offers significantly improved latency and throughput for deploying LLM APIs, optimized for inference.
  • Supports OpenAI APIs: Ensures seamless integration with existing OpenAI-compatible infrastructure.
  • Model Quantization: Implements support for GPTQ/AWQ/FP8 to optimize inference with limited GPU memory.
  • Use Case: Ideal for applications like chatbots, virtual assistants, or multi-user applications that require low latency with high throughput, serving models with limited GPU memory.

Quick Start

Run the vllm serve command for your desired model. Example: vllm serve meta-llama/Llama-3-8B-Instruct --gpu-memory-utilization 0.9 --port 8000

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs with high throughput and low latency for production?▼

Serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. This approach optimizes inference latency and maximizes throughput for production-grade deployment, ideal for multi-user chatbots and virtual assistants.

Can I deploy an OpenAI-compatible API endpoint for my self-hosted LLM?▼

Yes, you can deploy an OpenAI-compatible API endpoint. This serving solution supports OpenAI endpoints, ensuring seamless integration with existing OpenAI-compatible infrastructure for your self-hosted models.

How do I run LLM inference on systems with limited GPU memory?▼

Run LLM inference on limited GPU memory by applying model quantization. This Skill supports GPTQ, AWQ, and FP8 quantization formats to optimize memory usage and maintain performance on constrained hardware.

What is the best way to start serving a model like Llama-3 using vLLM?▼

The best way to start is using the vllm serve command with your desired model. For example, run `vllm serve meta-llama/Llama-3-8B-Instruct --gpu-memory-utilization 0.9 --port 8000` to quickly deploy your API.

Does vLLM support tensor parallelism for multi-GPU inference serving?▼

Yes, vLLM supports tensor parallelism for inference serving. This feature allows you to distribute model computations across multiple GPUs, optimizing inference performance for large language models.

Why does continuous batching improve LLM serving latency compared to traditional methods?▼

Continuous batching improves LLM serving latency by dynamically processing incoming requests. Combined with PagedAttention, it manages memory efficiently and maximizes throughput, overcoming the memory fragmentation issues of traditional batching.

Related Skills

Explore More Software Engineering Skills→
DonZzzillaDonZzzilla

llama-cpp

Run local GGUF inference for llama.cpp models with Hugging Face Hub discovery.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
granietgraniet
247

vllm

Launches vLLM-based OpenAI-compatible endpoints with PagedAttention and continuous batching.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced