skills.rest
Junkfooooood
JunkfoooooodCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and optimize LLM inference servers with vLLM using OpenAI-compatible APIs and quantization.

Updated May 4, 2026
Search Tags:#deployment#quantization#inference#vllm#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Junkfooooood/hermes-governance --skill serving-llms-vllm-junkfooooood
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Junkfooooood/hermes-governance/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/Junkfooooood/hermes-governance --skill serving-llms-vllm-junkfooooood

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow, memory-hungry, and hard to scale. This Skill guides you through deploying high-throughput LLM APIs with vLLM, achieving up to 24x higher throughput than standard transformers through PagedAttention and continuous batching. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with Docker, Kubernetes, load balancing, and Prometheus monitoring. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and tensor parallelism. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization. - Use Case: Deploy Llama-3-8B as an OpenAI-compatible chat API handling 100+ requests per second on a single A100, with prefix caching and metrics dashboards. ## Quick Start Use the vLLM skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible API server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Llama-3-8B-Instruct --port 8000. The server exposes /v1/chat/completions endpoints compatible with the OpenAI Python SDK using base_url http://localhost:8000/v1.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

vLLM offers high throughput with easy setup and multi-platform support. TensorRT-LLM is about 12% faster but NVIDIA-only with complex setup. llama.cpp targets CPU and edge single-user inference, not production multi-user serving.

How do I serve a 70B model on limited GPU memory?▼

Use a pre-quantized AWQ model such as TheBloke/Llama-2-70B-AWQ and launch with --quantization awq. This reduces memory from 140GB to about 35GB, fitting on a single 40GB A100 with under 1% accuracy loss.

Why does vLLM throw out of memory errors during model loading?▼

The model plus KV cache exceeds available VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompt prefixes, and use --enable-chunked-prefill for long prompts. Also verify tensor parallelism uses a power-of-2 GPU count and GPU utilization stays above 80%.

Does vLLM support monitoring and production metrics?▼

Yes, launch with --enable-metrics --metrics-port 9090 to expose Prometheus metrics including time to first token, requests running, and GPU cache usage. These integrate with Grafana dashboards and Kubernetes health probes.

Related Skills

Explore More Software Engineering Skills→
Clay-HHKClay-HHK
1

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

Community
Intermediate
mrsknetworkmrsknetwork

ai-integration

Integrates LLMs and RAG capabilities into production applications with streaming, function calling, and embeddings across OpenAI, Anthropic, Gemini.

Community
Advanced
Cloud-Byte-ConsultingCloud-Byte-Consulting

inference-benchmark-runner

Benchmarks LLM inference latency, throughput, GPU utilization, and cost on Kubernetes.

Official
Advanced
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced