skills.rest
Signmanal
SignmanalCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy OpenAI-compatible LLM serving endpoints with vLLM and quantization.

1|Updated Jun 25, 2026
Search Tags:#production deployment#quantization#vllm#inference optimization#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Signmanal/VIGIL --skill serving-llms-vllm-signmanal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Signmanal/VIGIL/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/Signmanal/VIGIL --skill serving-llms-vllm-signmanal

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploying production-grade LLM serving endpoints with high throughput, low latency, and efficient GPU memory usage is challenging, especially when working with large models or limited hardware resources.

Core Features & Use Cases

  • Production API Deployment: Set up OpenAI-compatible serving endpoints for 7B to 70B+ models with configurable GPU memory utilization, tensor parallelism, and built-in Prometheus metrics for monitoring.
  • Offline Batch Inference: Process large datasets of prompts efficiently without server overhead, with automatic continuous batching to maximize throughput.
  • Quantized Model Serving: Run large quantized models (AWQ, GPTQ, FP8) on limited GPU memory with minimal accuracy loss, enabling cost-effective deployment of 70B parameter models on single GPUs. A common use case is a machine learning engineering team using this skill to deploy a local Llama 3 70B model on a single A100 40GB GPU via AWQ quantization for internal analyst querying, with Docker for consistent production deployment.

Quick Start

Use the vLLM skill to deploy a production-ready OpenAI-compatible API endpoint for the Llama 3 8B Instruct model on your local GPU, with metrics enabled for performance monitoring.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible LLM API for production workloads?▼

To deploy an OpenAI-compatible LLM API for production workloads, use vLLM to set up serving endpoints with configurable GPU memory utilization, tensor parallelism, and built-in Prometheus metrics for monitoring.

Can I run a 70B parameter LLM on a single GPU?▼

You can run a 70B parameter LLM on a single GPU by applying quantized model serving with formats like AWQ, GPTQ, or FP8, which minimize accuracy loss while fitting large models into limited GPU memory.

What is the best way to process large datasets of prompts for offline batch inference?▼

The best way to process large datasets of prompts for offline batch inference is using vLLM, which bypasses server overhead and applies automatic continuous batching to maximize throughput.

Does vLLM support tensor parallelism across different GPU platforms?▼

vLLM supports tensor parallelism and flexible deployment across NVIDIA, AMD, and Intel GPU platforms, utilizing PagedAttention memory optimization and continuous batching for high-throughput inference.

How does PagedAttention memory optimization improve LLM serving performance?▼

PagedAttention memory optimization improves LLM serving performance by reducing GPU memory fragmentation during continuous batching, enabling high-throughput and low-latency inference for large language models.

What quantization formats can I use to reduce GPU memory usage during LLM deployment?▼

You can use AWQ, GPTQ, and FP8 quantization formats during LLM deployment to run large models on limited GPU memory with minimal accuracy loss, enabling cost-effective production serving.

Related Skills

Explore More Software Engineering Skills→
tangzheng202202tangzheng202202
1

serving-llms-vllm

Configure vLLM servers with OpenAI-compatible endpoints, quantization options, and tensor parallelism for scalable LLM deployments on GPU infrastructure.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
icyzhicyzh
1

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints.

Community
Advanced
hung-phanhung-phan
1

vllm

Optimize LLM inference with PagedAttention, continuous batching, and quantization.

Community
Advanced