skills.rest
danmartinez78
danmartinez78COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy vLLM servers with OpenAI-compatible endpoints and quantization.

Updated Apr 25, 2026
Search Tags:#quantization#inference#openai-api#vllm#llm-serving#tensor-parallelism
One-click install
npx skills add https://github.com/danmartinez78/echo-ghost-backup --skill serving-llms-vllm-danmartinez78
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/danmartinez78/echo-ghost-backup/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/danmartinez78/echo-ghost-backup --skill serving-llms-vllm-danmartinez78

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploy production-grade LLM serving infrastructure that can host OpenAI-compatible endpoints with efficient memory usage and configurable quantization options.

Core Features & Use Cases

  • High-throughput LLM serving with vLLM, leveraging PagedAttention and continuous batching.
  • OpenAI-compatible API endpoints with production-grade deployment and easy scalability.
  • Support for quantization (GPTQ/AWQ/FP8) and tensor parallelism to run large models on memory-constrained GPUs.
  • Use Case: Expose a robust LLM inference API for a 70B model with high concurrency and controlled latency.

Quick Start

Install vllm and launch a server configured for your model; then point clients to the OpenAI-compatible endpoint.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve a 70B LLM with high throughput on memory-constrained GPUs?▼

Serve a 70B LLM with high throughput on memory-constrained GPUs by applying quantization and tensor parallelism. This approach uses PagedAttention and continuous batching to maintain controlled latency and high concurrency under strict memory limits.

What's the best way to expose an OpenAI-compatible API endpoint for a custom LLM?▼

Expose an OpenAI-compatible API endpoint for a custom LLM by deploying a vLLM server configured for your model. Clients can then point directly to the endpoint to leverage production-grade inference without changing existing API integrations.

Does vLLM support quantization formats like AWQ, GPTQ, and FP8 for inference?▼

vLLM supports quantization formats including AWQ, GPTQ, and FP8 for inference. Applying these quantization methods allows large models to run efficiently on memory-constrained GPUs while maintaining high-throughput serving.

How do I configure tensor parallelism and gpu-memory-utilization for LLM serving?▼

Configure tensor parallelism and gpu-memory-utilization for LLM serving through deployment controls provided by vLLM. Adjusting parameters like max-num-seqs alongside tensor parallelism optimizes memory usage and throughput across multiple GPUs.

When do I need continuous batching and PagedAttention for LLM inference?▼

You need continuous batching and PagedAttention for LLM inference when running production workloads requiring high throughput for large models. These mechanisms manage memory efficiently and handle high concurrency with controlled latency.

Can I scale LLM serving horizontally to handle high concurrency requests?▼

You can scale LLM serving horizontally to handle high concurrency requests by leveraging tensor parallelism across GPUs. This configuration provides production-grade deployment controls and maintains robust inference latency under heavy load.

Related Skills

Explore More Software Engineering Skills→
DarkArty07DarkArty07
3

llama-cpp

Deploy local LLM inference with GGUF quantized weights via llama.cpp.

Community
Advanced
copilot-ldcopilot-ld
1

libllm

Expose an HTTP LLM API client for OpenAI-compatible chat completions and embeddings.

Official
Advanced
PriyanshKuniyalPriyanshKuniyal

istio-traffic-management

Configure Istio traffic routing, canary deployments, circuit breakers, and load balancing.

Community
Advanced
PiaoxuemoliPiaoxuemoli

serving-llms-vllm

Serve LLMs with OpenAI-compatible APIs using vLLM and quantization.

Community
Advanced