skills.rest
rickyananda
rickyanandaCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs with vLLM using OpenAI-compatible APIs and quantization.

1|Updated May 18, 2026
Search Tags:#quantization#inference#openai-api#vllm#pagedattention#llm-serving#tensor-parallelism
One-click install
npx skills add https://github.com/rickyananda/hermes-skills --skill serving-llms-vllm-rickyananda
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/rickyananda/hermes-skills/tree/main/mlops/inference/vllm
Command: npx skills add https://github.com/rickyananda/hermes-skills --skill serving-llms-vllm-rickyananda

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you deploy large language models with high throughput and low latency so they can serve real users reliably instead of struggling under heavy inference load.

Core Features & Use Cases

  • Production LLM Serving: Run model APIs with OpenAI-compatible endpoints for chat, completions, and tool-based applications.
  • Performance Optimization: Improve throughput and latency with PagedAttention, continuous batching, prefix caching, and chunked prefill.
  • Memory-Efficient Deployment: Fit larger models into limited GPU memory using AWQ, GPTQ, or FP8 quantization.
  • Real-World Use Case: Use it to host a multi-user chatbot, batch-generate outputs for a dataset, or scale a model across multiple GPUs or nodes.

Quick Start

Ask the Skill to set up a vLLM deployment plan for your target model, hardware, and serving requirements.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve large language models with an OpenAI-compatible API?▼

You can serve large language models with an OpenAI-compatible API by deploying a vLLM inference server. This provides high-throughput, low-latency endpoints for chat, completions, and tool-based applications to handle real user workloads reliably.

What is the best way to fit large models into limited GPU memory for inference?▼

The best way to fit large models into limited GPU memory during inference is applying AWQ, GPTQ, or FP8 quantization. These quantization techniques reduce the memory footprint of large language models, allowing deployment on hardware with constrained VRAM capacity.

How does PagedAttention improve LLM serving performance?▼

PagedAttention improves LLM serving performance by optimizing memory management during inference. Combined with continuous batching and prefix caching, it significantly increases throughput and reduces latency for concurrent multi-user requests on large language models.

Can I run distributed LLM inference across multiple GPUs or nodes?▼

Yes, you can run distributed LLM inference across multiple GPUs or nodes using tensor parallelism. This approach splits the model workload across available hardware, enabling high-throughput serving for models that exceed single-GPU memory limits.

How do I batch generate outputs for a dataset using a large language model?▼

You can batch generate outputs for a dataset using offline batch generation features in vLLM. This allows high-throughput processing of large datasets without needing to run a persistent live API server, maximizing inference efficiency.

Do I need PyTorch and Transformers to deploy models with vLLM?▼

Yes, you need PyTorch and Transformers support to deploy models with vLLM. These frameworks provide the foundational model loading and execution environment required to run the inference server and support features like PagedAttention.

Related Skills

Explore More Software Engineering Skills→
overviewlabsoverviewlabs

serving-llms-vllm

Deploy and scale LLM inference with vLLM for low-latency production serving.

Community
Advanced
CyapstayeCyapstaye
9

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and Prometheus metrics.

Community
Advanced
stephanjstephanj
9

rag-architect

Architect RAG systems integrating vector databases, embedding models, and retrieval pipelines.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced