skills.rest
zangjeicy
zangjeicyCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy high-performance LLM inference endpoints using vLLM with quantization and tensor parallelism.

Updated Jul 13, 2026
Search Tags:#deployment#llm#quantization#inference#vllm#serving
One-click install
npx skills add https://github.com/zangjeicy/Hermes --skill serving-llms-vllm-zangjeicy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/zangjeicy/Hermes/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/zangjeicy/Hermes --skill serving-llms-vllm-zangjeicy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve?

This skill addresses the challenges of deploying Large Language Models in production environments by providing high-throughput, low-latency inference capabilities that optimize GPU memory usage and request handling.

Core Features & Use Cases

  • High-Performance Serving: Utilizes PagedAttention and continuous batching to maximize GPU utilization and throughput.
  • Production Flexibility: Supports OpenAI-compatible APIs, various quantization methods like AWQ and GPTQ, and multi-node distributed serving.
  • Use Case: Deploy a 70B parameter model on limited hardware using quantization or scale a 8B model to handle hundreds of concurrent requests per second with minimal latency.

Quick Start

Use the serving-llms-vllm skill to launch an OpenAI-compatible inference server for the meta-llama/Llama-3-8B-Instruct model on port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible LLM inference server for high-throughput serving?▼

To deploy an OpenAI-compatible LLM inference server for high-throughput serving, use vLLM to launch the model and expose an API endpoint on your desired port. This enables low-latency API delivery and scalable model hosting on NVIDIA hardware.

How does vLLM optimize GPU memory usage for large language models?▼

vLLM optimizes GPU memory usage for large language models by utilizing PagedAttention and continuous batching. These mechanisms maximize GPU utilization, significantly improving inference throughput and overall memory efficiency for production serving.

Can I deploy a 70B parameter model on limited hardware using quantization?▼

Yes, you can deploy a 70B parameter model on limited hardware using quantization methods like AWQ and GPTQ. This reduces the memory footprint, allowing large model architectures to run on constrained NVIDIA GPU resources.

What is the best way to scale an 8B model to handle hundreds of concurrent requests per second?▼

The best way to scale an 8B model to handle hundreds of concurrent requests per second is using vLLM with continuous batching and tensor parallelism. This configuration ensures high-throughput, low-latency inference for concurrent API delivery.

Does vLLM support multi-node distributed serving for production LLM inference?▼

Yes, vLLM supports multi-node distributed serving for production LLM inference. This allows you to scale model hosting across multiple nodes using tensor parallelism to handle increased workloads and large model architectures.

When should I use prefix caching during LLM inference?▼

You should use prefix caching during LLM inference when processing requests with shared prompt prefixes. It optimizes memory efficiency and reduces latency by reusing computed key and value tensors for repeated context structures.

Related Skills

Explore More Software Engineering Skills→
juliuss1907juliuss1907

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization for limited GPU memory.

Community
Advanced
CyapstayeCyapstaye
9

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and Prometheus metrics.

Community
Advanced
GregsGreyCodeGregsGreyCode
6

serving-llms-vllm

Deploy vLLM servers with PagedAttention, continuous batching, and OpenAI-compatible endpoints.

Community
Advanced
KappTech88KappTech88

serving-llms-vllm

Deploy vLLM servers with quantization and tensor parallelism on GPU clusters.

Community
Advanced