skills.rest
DarkArty07
DarkArty07COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy high-throughput LLM inference servers with OpenAI-compatible API endpoints.

3|Updated Apr 21, 2026
Search Tags:#production deployment#quantization#vllm#inference optimization#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/DarkArty07/Aether-Agents --skill serving-llms-vllm-darkarty07
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/DarkArty07/Aether-Agents/tree/main/home/skills/mlops/inference/vllm
Command: npx skills add https://github.com/DarkArty07/Aether-Agents --skill serving-llms-vllm-darkarty07

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates the complexity of building slow, memory-heavy LLM inference infrastructure that fails under production traffic, reducing deployment time from days to minutes.

Core Features & Use Cases

  • High-Throughput Production Serving: Leverages PagedAttention and continuous batching to handle 100+ requests per second with low latency.
  • Memory-Optimized Quantization: Supports AWQ, GPTQ, and FP8 quantization to fit 70B+ models on limited GPU hardware.
  • OpenAI-Compatible Endpoints: Provides drop-in API compatibility for existing LLM applications without code changes.
  • Use Case: If you need to serve a 70B parameter Llama model to a team of 50+ concurrent users with sub-500ms time-to-first-token, this Skill handles all configuration, optimization, and deployment steps.

Quick Start

Use the serving-llms-vllm skill to deploy a production-ready OpenAI-compatible LLM API endpoint for your chosen model with optimized throughput and memory settings.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM for production serving with high throughput and low latency?▼

Production serving for LLMs requires continuous batching and PagedAttention to handle 100+ requests per second. This Skill deploys an optimized inference server that reduces deployment time from days to minutes while maintaining sub-500ms time-to-first-token.

Does OpenAI-compatible API serving work with existing LLM applications without code changes?▼

OpenAI-compatible API serving provides drop-in endpoint compatibility for existing LLM applications. This Skill deploys an inference server that matches the OpenAI API format, allowing your applications to switch models without requiring code modifications.

Can I fit a 70B parameter Llama model on memory-constrained GPUs for inference?▼

Fitting 70B parameter Llama models on limited GPU hardware requires memory-optimized quantization. This Skill supports AWQ, GPTQ, and FP8 quantization techniques to reduce memory consumption and enable serving large models on constrained hardware.

What is the best way to handle concurrent requests for large language model inference?▼

Handling concurrent LLM inference requests requires continuous batching and PagedAttention-based memory management. This Skill deploys a server optimized for production workloads, enabling 50+ concurrent users to receive fast response times without performance degradation.

How do I use tensor parallelism to optimize LLM serving across multiple GPUs?▼

Tensor parallelism optimizes LLM serving by distributing model weights across multiple GPUs. This Skill configures tensor parallelism alongside continuous batching and production monitoring integration to ensure high-throughput inference for large language models.

Why does LLM inference fail under production traffic and how can I fix it?▼

LLM inference fails under production traffic due to memory-heavy infrastructure lacking continuous batching and PagedAttention. This Skill eliminates deployment complexity by building a memory-optimized inference server that handles high request volumes efficiently.

Related Skills

Explore More Software Engineering Skills→
QingHuan-6QingHuan-6

sglang-diffusion-performance

Identify fastest SGLang Diffusion flags for a model, GPU, and VRAM budget.

Community
Advanced
handsomelong922handsomelong922

serving-llms-vllm

Deploy scalable LLM serving with vLLM for OpenAI-compatible endpoints.

Community
Advanced
granietgraniet
247

vllm

Launches vLLM-based OpenAI-compatible endpoints with PagedAttention and continuous batching.

Community
Advanced
overviewlabsoverviewlabs

serving-llms-vllm

Deploy and scale LLM inference with vLLM for low-latency production serving.

Community
Advanced