skills.rest
jacardl
jacardlCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs via OpenAI-compatible endpoints using vLLM with PagedAttention and continuous batching.

Updated Apr 16, 2026
Search Tags:#latency#throughput#quantization#openai-api#vllm#llm-serving#tensor-parallelism
One-click install
npx skills add https://github.com/jacardl/New-Radar --skill serving-llms-vllm-jacardl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/jacardl/New-Radar/tree/main/backend/frameworks/hermes-agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/jacardl/New-Radar --skill serving-llms-vllm-jacardl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve?

Serves LLMs at production scale by leveraging vLLM to maximize throughput and minimize latency, enabling robust OpenAI-compatible endpoints.

Core Features & Use Cases

  • High-throughput LLM serving with PagedAttention and continuous batching
  • OpenAI-compatible endpoints for chat and completions
  • Quantization support (GPTQ, AWQ, FP8) and tensor parallelism for memory-constrained deployments
  • Production-grade deployment workflows with multi-GPU setups

Quick Start

Launch vLLM with your model and expose the OpenAI-compatible /v1/chat/completions endpoint to start serving requests.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve large language models with low latency and high throughput?▼

Serve large language models with low latency and high throughput by using vLLM. It implements PagedAttention and continuous batching to maximize request processing speed for production-grade APIs.

How do I deploy an OpenAI-compatible API endpoint for my LLM?▼

Deploy an OpenAI-compatible API endpoint by launching vLLM with your model. This exposes the /v1/chat/completions endpoint, enabling chat and completion services to integrate seamlessly.

Can I run quantized LLMs in memory-constrained environments using tensor parallelism?▼

Run quantized LLMs in memory-constrained environments using vLLM with support for AWQ, GPTQ, and FP8 quantization. Tensor parallelism distributes the workload across multi-GPU setups to reduce memory pressure.

What is the best way to scale LLM serving for production traffic?▼

Scale LLM serving for production traffic by deploying vLLM with continuous batching and multi-GPU tensor parallelism. This approach maximizes throughput and handles concurrent requests efficiently.

Do I need PyTorch and transformers to serve models with vLLM?▼

You need PyTorch and transformers to serve models with vLLM. These dependencies are required to support model inference and integration within the serving framework.

Related Skills

Explore More Software Engineering Skills→
lxh755818-botlxh755818-bot

serving-llms-vllm

Serve LLMs with OpenAI API, quantization, and tensor parallelism.

Community
Advanced
polymerdaopolymerdao
1

polymer-pay-llm

Route OpenAI-compatible chat completions through Polymer Pay's category-based provider routing.

Official
Advanced
HarriesHarries

serving-llms-vllm

Serve LLMs with vLLM for OpenAI-compatible endpoints using tensor parallelism and quantization.

Community
Advanced
ihatesea69ihatesea69
3

serving-llms-vllm

Serve LLMs with vLLM PagedAttention and continuous batching.

Community
Advanced