skills.rest
markwang2658
markwang2658COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy OpenAI-compatible LLM serving endpoints with vLLM on Linux and macOS.

31|3|Updated May 7, 2026
Search Tags:#quantization#openai-api#inference-serving#vllm#pagedattention#high-throughput#continuous-batching
One-click install
npx skills add https://github.com/markwang2658/hermes-windows-native --skill serving-llms-vllm-markwang2658
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/markwang2658/hermes-windows-native/tree/main/hermes-agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/markwang2658/hermes-windows-native --skill serving-llms-vllm-markwang2658

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve?

vLLM provides high-performance LLM serving, enabling OpenAI-compatible endpoints with efficient memory usage and quantization options for large models.

Core Features & Use Cases

  • High-throughput LLM serving with vLLM
  • OpenAI API compatibility, quantization options (AWQ/GPTQ/FP8)
  • Tensor parallelism and continuous batching for production workloads

Quick Start

Launch a vLLM server with a model of your choice and send an OpenAI-compatible completion request to begin serving.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible LLM serving endpoint with vLLM?▼

To deploy an OpenAI-compatible LLM serving endpoint with vLLM, launch a vLLM server with your chosen model and send OpenAI-compatible completion requests. It supports Linux and macOS for real-time API inference.

What quantization methods does vLLM support for high-throughput inference?▼

vLLM supports AWQ, GPTQ, and FP8 quantization methods for high-throughput inference. These options help meet specific throughput and memory targets when serving large language models in production.

How does vLLM achieve efficient memory usage and high throughput for LLM serving?▼

vLLM achieves efficient memory usage and high throughput for LLM serving using PagedAttention and continuous batching. These mechanisms enable scalable inference and optimized memory allocation for large models.

Can I use tensor parallelism for multi-GPU configurations in vLLM?▼

Yes, you can use tensor parallelism for multi-GPU configurations in vLLM. It supports scalable inference across multiple GPUs to handle production workloads and meet high throughput targets.

What dependencies do I need to run a vLLM inference server?▼

You need vllm, torch, and transformers dependencies to run a vLLM inference server. These packages provide the foundational framework for loading models and executing high-throughput inference.

Related Skills

Explore More Software Engineering Skills→
SriRamkunamsettySriRamkunamsetty

huggingface-accelerate

Launch distributed PyTorch training with HuggingFace Accelerate across CPU, GPU, and TPU environments.

Community
Advanced
overviewlabsoverviewlabs

serving-llms-vllm

Deploy and scale LLM inference with vLLM for low-latency production serving.

Community
Advanced
MesferAliMesferAli

serving-llms-vllm

Deploy LLMs with vLLM for OpenAI-compatible endpoints and quantization.

Community
Advanced
BermudaLocalsBermudaLocals
1

serving-llms-vllm

Deploy vLLM inference servers with PagedAttention, continuous batching, and quantization.

Community
Advanced