skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

hqq-quantization

Quantize transformer-based causal LM weights to low-bit precision without calibration data.

2|Updated Apr 12, 2026
Search Tags:#llm#quantization#inference#huggingface#vllm#model-compression#hqq
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill hqq-quantization-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/10-optimization/hqq
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill hqq-quantization-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

HQQ enables rapid, calibration-free weight quantization of large language models so you can compress models for memory-efficient inference and deployment without collecting calibration datasets.

Core Features & Use Cases

  • Calibration-free quantization: Quantize weights to 8/4/3/2/1-bit precision instantly without sample data.
  • Multiple optimized backends: Choose from PyTorch, ATEN, TorchAO int4, Marlin, BitBlas and other backends to optimize inference throughput.
  • Framework integration & fine-tuning: Native support for HuggingFace and vLLM loading, with PEFT/LoRA compatibility for continuing training on quantized models.
  • Use cases: Compress transformer-based causal LMs for GPU inference, deploy low-bit models with vLLM for high-throughput serving, or perform QLoRA-style fine-tuning on quantized checkpoints.

Quick Start

Quantize meta-llama/Llama-3.1-8B to 4-bit HQQ with group_size 64 and prepare the model for vLLM deployment.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize LLM weights without collecting calibration data?▼

Calibration-free LLM quantization compresses model weights to low-bit representations instantly without sample data. HQQ applies this to transformer-based causal LMs, enabling rapid compression and memory-efficient inference without datasets.

Can I deploy 4-bit quantized models with vLLM for high-throughput serving?▼

Yes, quantized models can be deployed with vLLM for high-throughput serving. The Skill supports multiple optimized backends including Marlin, TorchAO int4, BitBlas, and ATEN to maximize inference throughput during vLLM deployment.

Does PEFT or LoRA fine-tuning work on quantized LLM checkpoints?▼

Yes, PEFT and LoRA fine-tuning are natively supported on quantized checkpoints. You can perform QLoRA-style continuing training on low-bit models within HuggingFace workflows while keeping the base weights quantized.

What backends are available for low-bit inference optimization?▼

Available optimized backends include PyTorch, ATEN, TorchAO int4, Marlin, and BitBlas. These backends integrate with hqq and torch runtimes to optimize inference throughput for low-bit quantized models.

How do I configure group_size and nbits for mixed-precision tuning?▼

Mixed-precision tuning is supported through configurable nbits, group_size, and axis parameters. You can set these values, such as using 4-bit precision with group_size 64, to balance compression ratio and model accuracy.

Why use calibration-free quantization instead of standard quantization methods?▼

Calibration-free quantization eliminates the need to collect and process calibration datasets, enabling instant model compression. This significantly reduces deployment preparation time while still supporting mixed-precision tuning and optimized inference backends.

Related Skills

Explore More Software Engineering Skills→
qcmuuqcmuu

peft-fine-tuning

Train LoRA or QLoRA adapters for parameter-efficient fine-tuning of transformer models.

Community
Intermediate
GitnappGitnapp

peft-fine-tuning

Configure LoRA and QLoRA adapters for memory-efficient LLM fine-tuning.

Community
Advanced
MesferAliMesferAli

hqq-quantization

Quantize Large Language Model weights to 4/3/2-bit precision without calibration data.

Community
Intermediate
ovachieverovachiever
52

gptq

Compress large language models to 4-bit precision with GPTQ.

Community
Advanced