skills.rest
jordanhubbard
jordanhubbardCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Serve large language models with high-throughput inference and OpenAI-compatible endpoints.

9|3|Updated May 18, 2026
Search Tags:#quantization#inference optimization#tensor parallelism#LLM serving#GPU memory management
One-click install
npx skills add https://github.com/jordanhubbard/mac --skill serving-llms-vllm-jordanhubbard
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/jordanhubbard/mac/tree/main/src/mac/_hermes/skills/mlops/inference/vllm
Command: npx skills add https://github.com/jordanhubbard/mac --skill serving-llms-vllm-jordanhubbard

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill solves the challenge of deploying production LLM APIs with high throughput and low latency, especially for large models that require GPU memory optimization.

Core Features & Use Cases

  • High Throughput Inference: Achieves 24x higher throughput than standard transformers.
  • GPU Memory Optimization: Fits large models within limited GPU memory with quantization and tensor parallelism.
  • OpenAI API Compatibility: Supports OpenAI-compatible endpoints.
  • Use Cases: Ideal for chatbots, assistants, and multi-user applications that require high performance and efficiency.

Quick Start

Install the vLLM skill:

pip install vllm

To perform basic offline inference, run:

from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM serving throughput for production APIs?▼

Optimize LLM serving throughput for production APIs by utilizing PagedAttention, continuous batching, and tensor parallelism to achieve 24x higher throughput than standard transformers while maintaining low latency.

What's the best way to fit large language models in limited GPU memory?▼

Fit large language models in limited GPU memory by applying quantization and tensor parallelism techniques. These methods partition model weights across multiple GPUs and reduce precision to minimize memory footprint during inference.

How does continuous batching improve inference latency for multi-user applications?▼

Continuous batching improves inference latency for multi-user applications by dynamically processing incoming requests without waiting for a full batch. This maximizes GPU utilization and ensures efficient scaling for chatbots and assistants.

Can I use OpenAI-compatible endpoints with vllm and torch?▼

Yes, you can use OpenAI-compatible endpoints with vllm and torch. The serving framework supports OpenAI API compatibility, allowing seamless integration with existing applications requiring high performance and efficiency.

Do I need tensor parallelism to deploy high-throughput LLM APIs?▼

You need tensor parallelism to deploy high-throughput LLM APIs when working with large models that exceed single GPU memory limits. It partitions the model across multiple GPUs to enable efficient scaling and memory management.

Related Skills

Explore More Software Engineering Skills→
jacardljacardl

stable-diffusion-image-generation

Generate images from text prompts using Stable Diffusion via HuggingFace Diffusers.

Community
Advanced
webdevtodayjasonwebdevtodayjason

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints, quantization, and tensor parallelism.

Community
Advanced
overviewlabsoverviewlabs

serving-llms-vllm

Deploy and scale LLM inference with vLLM for low-latency production serving.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced