hqq-quantization

Quantize LLM weights without calibration data across 8/4/3/2/1-bit configurations.

Updated Mar 16, 2026

Search Tags:#llm #pytorch #quantization #inference #backends #memory-optimization #calibration-free

Ready to run

npx skills add https://github.com/arsity/scholar-tools --skill hqq-quantization-arsity

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/10-optimization/hqq
Command: npx skills add https://github.com/arsity/scholar-tools --skill hqq-quantization-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Half-Quadratic Quantization enables weight quantization for large language models without requiring calibration data, reducing memory footprint and speeding up inference.

Core Features & Use Cases

Calibration-free quantization across 8/4/3/2/1-bit configurations with configurable group sizes.
Multiple backends (PyTorch, TorchAO, BitBlas, Marlin) for flexible deployment.
Mixed-precision per-layer configurations and easy integration with HuggingFace Transformers.
Production-ready workflows with vLLM for efficient serving and deployment.
Clear paths for model compression and deployment across diverse hardware.

Quick Start

Install hqq and quantize a model with a 4-bit config using the default backend, then test with a sample prompt.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I quantize LLM weights without calibration data?▼

You can quantize LLM weights without calibration data using Half-Quadratic Quantization, which directly processes model weights to reduce memory footprint and accelerate inference without needing a calibration dataset.

Can I use mixed-precision quantization per layer for PyTorch models?▼

Yes, mixed-precision quantization supports per-layer configurations for PyTorch models, allowing you to assign different bit-widths across layers and integrate seamlessly with HuggingFace Transformers.

Does hqq quantization support vLLM for production inference?▼

Yes, hqq quantization supports production-ready workflows with vLLM for efficient model serving and deployment across diverse hardware configurations.

What bit-width configurations are available for calibration-free LLM quantization?▼

Calibration-free LLM quantization supports 8, 4, 3, 2, and 1-bit configurations with configurable group sizes to optimize memory usage and inference speed.

Which inference backends are compatible with Half-Quadratic Quantization?▼

Half-Quadratic Quantization is compatible with multiple backends including PyTorch, TorchAO, BitBlas, and Marlin, enabling flexible deployment across various hardware environments.

What do I need to install to start quantizing models with hqq?▼

You need Python, PyTorch, and the hqq library installed, along with optional dependencies for specific backends and HuggingFace Transformers integration to begin quantizing models.