skills.rest
handsomelong922
handsomelong922COMMUNITY
Explore all skills by this author →
GitHub

awq-quantization

Quantize large language models to 4-bit weights using AutoAWQ.

Updated Apr 3, 2026
Search Tags:#llm#calibration#gpu#quantization#inference#backends#awq
One-click install
npx skills add https://github.com/handsomelong922/my-codex-skills --skill awq-quantization-handsomelong922
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: awq-quantization
Source: https://github.com/handsomelong922/my-codex-skills/tree/main/skills/awq
Command: npx skills add https://github.com/handsomelong922/my-codex-skills --skill awq-quantization-handsomelong922

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

AWQ (Activation-aware Weight Quantization) provides 4-bit quantization that preserves salient weights based on activation patterns, reducing memory usage and enabling faster inference for large language models with minimal accuracy loss.

Core Features & Use Cases

  • Activation-aware 4-bit quantization that protects salient weights to minimize accuracy loss.
  • 3x speedup with memory reduction across backends (GEMM, GEMV, Marlin, ExLlama, ExLLama) and supports multi-GPU deployment and calibration data.
  • Flexible calibration and fusion options, including automatic layer fusion and selective module handling for large-scale models.

Quick Start

Quantize a target model with 4-bit weight quantization using AutoAWQ workflow.

Frequently Asked Questions about awq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How does activation-aware 4-bit quantization reduce memory usage for large language models?▼

Activation-aware 4-bit quantization reduces memory usage by identifying and preserving salient weights based on activation patterns, minimizing accuracy loss while accelerating LLM inference.

What backends are supported for 4-bit weight quantization and multi-GPU deployment?▼

4-bit weight quantization supports GEMM, GEMV, Marlin, ExLlama, and ExLlamaV2 backends, and includes multi-GPU deployment capabilities to accelerate inference across distributed hardware.

How do I quantize a model using AutoAWQ and custom calibration data?▼

You can quantize a target model using the AutoAWQ workflow with custom calibration data, applying automatic layer fusion and selective module handling to optimize large-scale model performance.

Do I need Python tooling like torch and transformers to apply AWQ quantization?▼

Yes, applying AWQ quantization requires Python tooling such as autoawq, transformers, and torch to execute model-specific calibration and fusion options during the 4-bit weight quantization process.

Can I achieve 3x inference speedup with minimal accuracy loss using 4-bit quantization?▼

Yes, activation-aware 4-bit quantization can achieve a 3x inference speedup with memory reduction by protecting salient weights, ensuring minimal accuracy loss for large language models.

When should I use activation-aware weight quantization instead of standard 4-bit quantization?▼

Use activation-aware weight quantization when standard 4-bit quantization causes excessive accuracy loss, as it protects salient weights based on activation patterns to preserve model quality.

Related Skills

Explore More Software Engineering Skills→
Orchestra-ResearchOrchestra-Research
11.5k

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight quantization.

Official
Advanced
informatico-madridinformatico-madrid
1

awq-quantization

Quantize large language models to 4-bit using the AWQ algorithm.

Community
Intermediate
melissa-pereira-deelmelissa-pereira-deel

On-Device ML Optimization

Optimize on-device ML models for speed, size, and energy efficiency.

Community
Advanced
ovachieverovachiever
52

gptq

Compress large language models to 4-bit precision with GPTQ.

Community
Advanced