skills.rest
iani-kuli
iani-kuliCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploy vLLM inference servers with quantization, tensor parallelism, and OpenAI-compatible APIs.

Updated May 5, 2026
Search Tags:#deployment#llm#throughput#quantization#inference#vllm
One-click install
npx skills add https://github.com/iani-kuli/harness_bro --skill serving-llms-vllm-iani-kuli
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/iani-kuli/harness_bro/tree/main/.claude/skills/ported/serving-llms-vllm
Command: npx skills add https://github.com/iani-kuli/harness_bro --skill serving-llms-vllm-iani-kuli

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, locust, transformers, autoawq, autogptq, and includes references (resource) components.

What problem does it solve?

This skill addresses the challenges of deploying large language models in production by optimizing memory usage and maximizing request throughput.

Core Features & Use Cases

  • High-Performance Serving: Utilizes PagedAttention and continuous batching to significantly increase request-per-second capacity.
  • Quantization Support: Enables deployment of large models on limited GPU hardware using AWQ, GPTQ, or FP8 methods.
  • Production Readiness: Provides standardized workflows for API deployment, load testing, and monitoring via Prometheus metrics.

Quick Start

Use the serving-llms-vllm skill to launch an OpenAI-compatible API server for the Llama-3-8B-Instruct model on port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy a high-throughput LLM inference server?▼

Deploy a high-throughput LLM inference server using the vLLM engine to optimize GPU memory and request throughput, utilizing PagedAttention and continuous batching to significantly increase request-per-second capacity.

How does vLLM optimize GPU memory for LLM serving?▼

vLLM optimizes GPU memory for LLM serving by utilizing PagedAttention and continuous batching, which enables efficient KV cache management and maximizes request throughput during high-concurrency inference.

Can I deploy large models on limited GPU hardware using vLLM?▼

You can deploy large models on limited GPU hardware using vLLM by applying model quantization methods such as AWQ, GPTQ, or FP8, which reduce memory consumption while maintaining inference performance.

How do I host an OpenAI-compatible API for LLM inference?▼

Host an OpenAI-compatible API for LLM inference by deploying the vLLM engine, which provides standardized workflows for production-ready API hosting, load testing, and monitoring via Prometheus metrics.

Does vLLM support tensor parallelism for production inference?▼

vLLM supports tensor parallelism for production inference, allowing you to distribute large language models across multiple GPUs to optimize memory usage and maximize request throughput.

What is the best way to load test LLM inference throughput?▼

Load test LLM inference throughput by deploying the vLLM engine with standardized workflows that integrate locust for load testing and Prometheus metrics for monitoring high-concurrency batching performance.

Related Skills

Explore More Software Engineering Skills→
dawsonblockdawsonblock

serving-llms-vllm

Serve OpenAI-compatible LLM endpoints with vLLM and PagedAttention.

Community
Advanced
cosmixcosmix
53

performance-testing

Run load tests with k6, Locust, or Artillery and report latency, throughput, and error rate.

Community
Advanced
heysuhasheysuhas
2

serving-llms-vllm

Deploy LLM inference servers with the vLLM engine and OpenAI-compatible APIs.

Community
Advanced
tianhao909tianhao909
1

serving-llms-vllm

Serve LLMs with vLLM, PagedAttention, and continuous batching.

Community
Advanced