skills.rest
air-gapped
air-gappedOFFICIAL
Explore all skills by this author →
GitHub

vllm-quantization

Quantize LLM checkpoints for vLLM serving with FP8, NVFP4, and MXFP4 formats.

5|1|Updated Apr 19, 2026
Search Tags:#quantization#vllm#fp8#nvfp4#modelopt#llm-compressor
One-click install
npx skills add https://github.com/air-gapped/skills --skill vllm-quantization
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vllm-quantization
Source: https://github.com/air-gapped/skills/tree/main/.claude/skills/vllm-quantization
Command: npx skills add https://github.com/air-gapped/skills --skill vllm-quantization

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill enables efficient quantization workflows for vLLM deployments across Hopper, Blackwell, and ROCm fleets, reducing memory and compute with validated formats like FP8, NVFP4, and MXFP4, plus integration with llm-compressor and NVIDIA ModelOpt.

Core Features & Use Cases

  • Supports multiple production quantization paths (FP8, NVFP4, MXFP4, and online quantization) and guidance for model export pipelines.
  • Provides end-to-end coverage from PTQ to deployment, including kv cache options and MoE support across vendors.
  • Use cases include preparing 70B-class models for datacenter inference, upgrading existing Qwen3/Qi models, and enabling offline model sharing with vendor formats.

Quick Start

Quantize a 70B Hopper model using FP8_DYNAMIC with llm-compressor and load it into vLLM.

Frequently Asked Questions about vllm-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a large language model for vLLM inference?▼

Quantize large language models for vLLM by applying post-training quantization workflows using llm-compressor or NVIDIA ModelOpt, enabling efficient deployment on datacenter GPUs.

What quantization formats are supported for vLLM serving?▼

Supported vLLM quantization formats include FP8, NVFP4, and MXFP4, along with online quantization paths tailored for Hopper, Blackwell, and ROCm hardware architectures.

Does vLLM quantization work with Qwen3 and Llama model families?▼

vLLM quantization supports model families including Qwen3, Llama, and GLM, allowing you to prepare 70B-class checkpoints for datacenter inference across various vendor platforms.

How do I apply FP8_DYNAMIC quantization to a 70B model using llm-compressor?▼

Use llm-compressor to apply FP8_DYNAMIC quantization to 70B-class models on Hopper GPUs, then load the resulting checkpoint directly into vLLM for efficient serving.

Can I use online quantization instead of pre-quantized checkpoints in vLLM?▼

vLLM supports online quantization as an alternative to pre-quantized checkpoints, applying formats like FP8 dynamically during deployment without requiring offline model export.

What are the limitations of using NVFP4 and MXFP4 quantization with vLLM?▼

NVFP4 and MXFP4 quantization in vLLM requires specific hardware like Blackwell GPUs and involves version gating for vendor PTQ workflows, limiting compatibility with older systems.

Related Skills

Explore More Software Engineering Skills→
slowlyCslowlyC
154

triton-skill

Write, debug, and optimize GPU kernels using Triton and Gluon DSLs.

Community
Advanced
jayll1303jayll1303
17

vllm-tgi-inference

Deploy local vLLM or HuggingFace TGI inference servers with OpenAI-compatible APIs.

Community
Advanced
annealing-inversionannealing-inversion

sglang-diffusion-modelopt-quant

Quantize diffusion DiT models with NVIDIA ModelOpt for SGLang Diffusion.

Community
Advanced
baleen37baleen37
4

competitive-agents

Dispatch two parallel subagents to solve a task and synthesize the best solution.

Community
Advanced