skills.rest
OpenLAIR
OpenLAIROFFICIAL
Explore all skills by this author →
GitHub

hqq-quantization

Quantize large language models without calibration data across multiple backends and bit-widths.

1.0k|117|Updated Feb 26, 2026
Search Tags:#llm#quantization#inference#backends#peft#calibration-free
One-click install
npx skills add https://github.com/OpenLAIR/dr-claw --skill hqq-quantization-openlair
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/OpenLAIR/dr-claw/tree/main/skills/optimization/hqq
Command: npx skills add https://github.com/OpenLAIR/dr-claw --skill hqq-quantization-openlair

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

HQQ enables fast, calibration-free weight quantization for large language models, dramatically reducing memory and compute needs without requiring calibration data.

Core Features & Use Cases

  • No calibration data required: quantize any model instantly for inference.
  • Multiple backends and bit-widths (8/4/3/2/1-bit): flexible deployment across GPUs and runtimes.
  • PEFT/LORA friendly: fine-tune quantized models with common PEFT methods.

Quick Start

Install hqq and quantize a pretrained model to 4-bit precision for fast, calibration-free inference.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a large language model without calibration data?▼

Calibration-free quantization allows you to quantize large language models instantly, reducing memory footprint and accelerating inference without requiring a calibration dataset to pre-process model weights.

What bit-widths and backends are supported for LLM quantization and inference?▼

LLM quantization supports 8, 4, 3, 2, and 1-bit precision across multiple backends including pytorch, pytorch_compile, aten, marlin, bitblas, and torchao_int4, enabling flexible deployment across various GPUs and runtimes.

Can I fine-tune a quantized model using PEFT or LoRA?▼

Yes, you can fine-tune quantized models using common PEFT methods. The quantization process is PEFT and LoRA friendly, allowing you to fine-tune models after applying low-bit weight quantization.

Does calibration-free quantization work with HuggingFace and vLLM backends?▼

Yes, calibration-free quantization integrates with HuggingFace and vLLM, allowing you to load and run quantized models directly within these frameworks for accelerated inference.

How do I apply per-layer quantization configurations to a large language model?▼

You can apply per-layer configurations to a large language model by specifying individual quantization settings during the calibration-free process, optimizing memory and compute distribution across different model layers.

Related Skills

Explore More Software Engineering Skills→
qcmuuqcmuu

peft-fine-tuning

Train LoRA or QLoRA adapters for parameter-efficient fine-tuning of transformer models.

Community
Intermediate
ovachieverovachiever
52

quantizing-models-bitsandbytes

Quantize LLMs to 8-bit or 4-bit precision using BitsAndBytes.

Community
Advanced
ihatesea69ihatesea69
3

hqq-quantization

Quantize LLMs to 4/3/2-bit precision without calibration data.

Community
Advanced
OpenLAIROpenLAIR
1.0k

gptq

Quantize large language models to 4-bit GPTQ with 128-group configuration.

Official
Advanced