skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

awq-quantization

Quantize large language models to 4-bit with activation-aware weight quantization.

2|Updated Apr 12, 2026
Search Tags:#llm#quantization#inference#4-bit#vllm#awq#marlin
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill awq-quantization-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: awq-quantization
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/10-optimization/awq
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill awq-quantization-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

AWQ solves the challenge of deploying large language models when GPU memory is limited and inference speed and accuracy must be preserved by providing activation-aware 4-bit weight quantization that reduces memory footprint and speeds up runtime with minimal accuracy degradation.

Core Features & Use Cases

  • Activation-aware 4-bit quantization: Protects salient weights identified from activation patterns to minimize accuracy loss (<5% typical degradation).
  • Multiple kernel backends: GEMM for batch throughput, GEMV for single-token streaming, Marlin for Ampere+ GPUs, and ExLlama for AMD compatibility.
  • Production deployment: Works with transformers, vLLM, fused modules, multi-GPU device mapping, and supports models across Llama, Mistral, Qwen, Falcon and multimodal families.
  • Use case: Quantize a 7B–70B instruction-tuned model to serve with vLLM on limited GPU memory while maintaining high throughput and low perplexity increase.

Quick Start

Quantize my Mistral-7B model to AWQ 4-bit with q_group_size 128 and zero-point enabled, save the quantized model, and load it for fused inference.

Frequently Asked Questions about awq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce GPU memory usage for large LLM inference without losing accuracy?▼

4-bit activation-aware weight quantization (AWQ) reduces GPU memory and speeds up large LLM inference by protecting salient weights identified from activation patterns, minimizing accuracy degradation to under 5%. It allows deploying 7B-70B models efficiently in memory-constrained environments.

How much calibration data is needed to quantize a model to 4-bit using AWQ?▼

AWQ 4-bit quantization requires minimal calibration data, typically between 128 and 1024 tokens. This small sample size is sufficient to identify activation patterns and protect salient weights, ensuring minimal accuracy degradation during the quantization process.

Can I serve AWQ quantized models using vLLM on Ampere GPUs?▼

Yes, AWQ quantized models are fully compatible with vLLM serving. For Ampere+ GPUs, you can optionally use Marlin kernels to maximize batch and single-token generation throughput, while ExLlama kernels provide AMD compatibility.

Does AWQ 4-bit quantization support multimodal and instruction-tuned models?▼

Yes, AWQ supports deploying 7B-70B instruction-tuned and multimodal models across Llama, Mistral, Qwen, and Falcon families. It works seamlessly with transformers, autoawq, and vLLM for production serving.

What is the best way to speed up single-token streaming for 4-bit quantized models?▼

To speed up single-token streaming for 4-bit quantized models, use the GEMV kernel backend. For batch throughput, use the GEMM backend, and for Ampere+ GPUs, enable the Marlin kernel to maximize runtime speed.

When should I not use AWQ for model quantization?▼

You should avoid AWQ if your deployment environment lacks compatible GPU hardware like Ampere+ or AMD architectures, or if your model architecture falls outside supported families like Llama, Mistral, Qwen, and Falcon.

Related Skills

Explore More Software Engineering Skills→
hhhi21ghhhi21g

awq-quantization

Compress large language models to 4-bit precision using activation-aware weight quantization.

Community
Advanced
GitnappGitnapp

serving-llms-vllm

Deploy large language model inference services with vLLM and OpenAI-compatible endpoints.

Community
Advanced
handsomelong922handsomelong922

awq-quantization

Quantize large language models to 4-bit weights using AutoAWQ.

Community
Advanced
ovachieverovachiever
52

gptq

Compress large language models to 4-bit precision with GPTQ.

Community
Advanced