skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and optimize LLM inference servers using vLLM with OpenAI-compatible APIs.

Updated Jun 7, 2026
Search Tags:#quantization#inference#vLLM#OpenAI API#pagedattention#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill serving-llms-vllm-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill serving-llms-vllm-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow and memory-hungry with standard transformers, causing high latency, low throughput, and GPU out-of-memory errors when handling concurrent requests. ## Core Features & Use Cases - High-Throughput Serving: Uses PagedAttention and continuous batching to achieve up to 24x higher throughput than standard transformers. - OpenAI-Compatible API: Launch a production server with vllm serve and query it using the standard OpenAI SDK. - Quantization Support: Serve AWQ, GPTQ, or FP8 quantized models to fit 70B models into a single 40GB GPU. - Use Case: Deploy a Llama-3-8B chatbot backend handling 100+ requests per second on one A100, with Prometheus metrics monitoring TTFT and GPU cache usage. ## Quick Start Use the vLLM skill to launch an OpenAI-compatible server for meta-llama/Llama-3-8B-Instruct on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM with an OpenAI-compatible API using vLLM?▼

Run `vllm serve meta-llama/Llama-3-8B-Instruct` to start a server on port 8000. Then query it with the OpenAI Python SDK by setting base_url to http://localhost:8000/v1 and api_key to EMPTY.

vLLM vs TensorRT-LLM for LLM serving, which should I choose?▼

TensorRT-LLM is roughly 12% faster but NVIDIA-only with high setup complexity. vLLM is much easier to set up, supports multiple platforms including AMD and Intel GPUs, and offers more quantization options like AWQ and GPTQ.

How do I fit a 70B model on a single 40GB GPU?▼

Use AWQ quantization with a pre-quantized model like TheBloke/Llama-2-70B-AWQ and launch with `--quantization awq --gpu-memory-utilization 0.95`. This reduces memory from 140GB to about 35GB with under 1% accuracy loss.

Why does vLLM throw out of memory errors during model loading?▼

The model plus KV cache exceeds available VRAM. Reduce `--gpu-memory-utilization` to 0.7, lower `--max-model-len` to 4096, enable AWQ quantization, or split the model across GPUs with `--tensor-parallel-size 2`.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with `--enable-prefix-caching` for repeated prompts and chunked prefill with `--enable-chunked-prefill` for long prompts. Also verify tensor parallelism uses a power-of-2 GPU count.

When should I use llama.cpp instead of vLLM?▼

Use llama.cpp for CPU or edge inference and single-user scenarios. vLLM is designed for production multi-user deployments needing high throughput above 100 requests per second on GPU infrastructure.

Related Skills

Explore More Software Engineering Skills→
HughYauHughYau
2.5k

borzoi

Predict genome-wide functional tracks from DNA sequence using the Borzoi deep learning model.

Community
Advanced
BjzyLabsBjzyLabs
1

grafana-mcp

Manage Grafana dashboards, data sources, users, and alerts via the MCP API.

Official
Advanced
MLT-OSSMLT-OSS
14

serving-llms-vllm

Serve LLMs with high throughput using vLLM's PagedAttention and continuous batching.

Official
Advanced
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced