skills.rest
jayll1303
jayll1303COMMUNITY
Explore all skills by this author →
GitHub

vllm-tgi-inference

Deploy local vLLM or HuggingFace TGI inference servers with OpenAI-compatible APIs.

17|3|Updated Apr 3, 2026
Search Tags:#quantization#vllm#kv-cache#llm-serving#tensor-parallelism#tgi
One-click install
npx skills add https://github.com/jayll1303/AIEKit --skill vllm-tgi-inference
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vllm-tgi-inference
Source: https://github.com/jayll1303/AIEKit/tree/main/.kiro/skills/vllm-tgi-inference
Command: npx skills add https://github.com/jayll1303/AIEKit --skill vllm-tgi-inference

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploying and operating local LLM inference servers is complex and error-prone due to GPU memory limits, tensor-parallel sharding, quantization formats, batching and KV cache sizing, and differing engine flags. This Skill consolidates server launch patterns, engine decision guidance, quantized model serving, OpenAI-compatible API usage, and a diagnostic checklist to reduce downtime and OOM failures.

Core Features & Use Cases

  • Engine Selection: Guidance to choose between vLLM (pip, strong TP and quantization support) and TGI (Docker-native, grammar & watermarking).
  • Server Launch & Sharding: Commands and validation checks for single-GPU, multi-GPU tensor parallelism, and Docker-based TGI sharding.
  • Quantized Model Serving: Instructions for serving AWQ, GPTQ, GGUF and bitsandbytes formats and their engine-specific flags.
  • Performance Tuning & Diagnostics: VRAM estimation, KV cache tuning, batch scheduling, metrics, and OOM troubleshooting steps.
  • Client Patterns: OpenAI-compatible Python and curl examples for chat, completion, and streaming clients.
  • Use Case: Launch a multi-GPU vLLM server to serve a quantized 70B model with tensor parallelism and monitor Prometheus metrics to tune throughput.

Quick Start

Start a local vLLM server to serve meta-llama/Llama-3.1-8B-Instruct on port 8000 with an OpenAI-compatible API and validate the /v1/models endpoint.

Frequently Asked Questions about vllm-tgi-inference

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve a local LLM with an OpenAI-compatible API using vLLM?▼

Serve local LLMs with an OpenAI-compatible API by launching a vLLM server with the model name and port, then validate the endpoint. This Skill provides launch commands and client patterns for chat, completion, and streaming workloads.

What's the best way to deploy a quantized 70B model across multiple GPUs?▼

Deploy quantized 70B models across multiple GPUs using tensor-parallel sharding. This Skill provides multi-GPU server launch flags and validation checks for serving AWQ, GPTQ, GGUF, and bitsandbytes formats.

Why does my LLM inference server throw an OOM error when loading models?▼

LLM inference servers throw OOM errors due to GPU memory limits, improper KV cache sizing, or batch scheduling issues. This Skill provides VRAM estimation, tuning parameters, and a diagnostic checklist to troubleshoot model loading failures.

Can I use HuggingFace TGI for single-GPU model serving?▼

HuggingFace TGI supports single-GPU model serving alongside multi-GPU Docker-based sharding. This Skill provides server launch commands, quantization options, and validation checks for both single- and multi-GPU deployments.

How do I tune KV cache and batch scheduling for high-throughput LLM serving?▼

Tune KV cache and batch scheduling for high-throughput LLM serving by adjusting VRAM allocation and monitoring Prometheus metrics. This Skill provides performance tuning guidance and diagnostic steps to optimize throughput.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
TehminanazTehminanaz
1

openai-agents-sdk-groq-integration

Integrate the OpenAI Agents SDK with Groq's endpoint in FastAPI applications.

Community
Advanced
TitoPrauseeTitoPrausee

serving-llms-vllm

Deploy vLLM for high-throughput LLM serving with OpenAI-compatible APIs.

Community
Advanced
yanochka11yanochka11

serving-llms-vllm

Deploy and optimize vLLM inference servers with OpenAI-compatible endpoints.

Community
Advanced