skills.rest
hungthinh04
hungthinh04COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and optimized inference.

Updated May 12, 2026
Search Tags:#quantization#vllm#pagedattention#inference optimization#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/hungthinh04/Hermes_AI_Agent --skill serving-llms-vllm-hungthinh04
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/hungthinh04/Hermes_AI_Agent/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/hungthinh04/Hermes_AI_Agent --skill serving-llms-vllm-hungthinh04

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you deploy and operate large language models with high throughput, low latency, and production-ready serving patterns, reducing the complexity of running LLM inference at scale.

Core Features & Use Cases

  • High-Performance Serving: Launch OpenAI-compatible LLM endpoints with continuous batching, prefix caching, and tensor parallelism.
  • Quantized Deployment: Fit larger models into limited GPU memory using AWQ, GPTQ, or FP8 configurations.
  • Operations and Reliability: Use built-in guidance for monitoring, Docker, Kubernetes, load balancing, troubleshooting, and performance tuning.
  • Use Case: Run a chatbot API for an internal product team, benchmark its latency and throughput, then tune it for better GPU utilization and lower time to first token.

Quick Start

Ask for a vLLM deployment plan for meta-llama/Llama-3-8B-Instruct that includes installation, server launch, and a basic OpenAI-compatible test call.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with an OpenAI-compatible API?▼

Serve an LLM with an OpenAI-compatible API by launching a vLLM server that supports continuous batching, prefix caching, and PagedAttention to optimize inference throughput and latency.

What's the best way to fit a large language model into limited GPU memory?▼

Fit large language models into limited GPU memory by deploying quantized configurations using AWQ, GPTQ, or FP8 formats to reduce the memory footprint during inference.

Does vLLM support multi-GPU tensor parallelism for inference?▼

vLLM supports multi-GPU tensor parallelism, allowing you to execute large language models across multiple GPUs or multi-node environments to maximize inference performance.

How do I monitor LLM serving latency and throughput in production?▼

Monitor LLM serving latency and throughput in production using built-in Prometheus metrics, alongside operational guidance for Docker, Kubernetes, load balancing, and performance tuning.

Can I use vLLM for offline batch generation?▼

vLLM supports offline batch generation, enabling efficient inference for large datasets while maintaining the same PagedAttention and continuous batching optimizations.

When do I need PagedAttention for LLM inference?▼

PagedAttention is needed for LLM inference when managing memory allocation efficiently during continuous batching, significantly reducing latency and improving overall throughput.

Related Skills

Explore More Software Engineering Skills→
ricablericable

serving-llms-vllm

Serve large language models with vLLM high-throughput inference and OpenAI-compatible endpoints.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
gztcodegztcode

serving-llms-vllm

Optimizes LLM inference with PagedAttention and continuous batching for production API serving.

Community
Advanced