skills.rest
tianhao909
tianhao909COMMUNITY
Explore all skills by this author →
GitHub

hqq-quantization

Quantize LLMs to 1-8 bit precision without calibration data.

1|Updated Feb 21, 2026
Search Tags:#quantization#model optimization#inference acceleration#llm compression#4-bit quantization#hqq
One-click install
npx skills add https://github.com/tianhao909/AI-Research-SKILLs-cn --skill hqq-quantization-tianhao909
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/tianhao909/AI-Research-SKILLs-cn/tree/main/10-optimization/hqq
Command: npx skills add https://github.com/tianhao909/AI-Research-SKILLs-cn --skill hqq-quantization-tianhao909

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires hqq, torch, transformers, vllm, peft, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill enables efficient compression of Large Language Models (LLMs) to extremely low bit precision (e.g., 4-bit, 3-bit, 2-bit) without requiring a calibration dataset, significantly reducing model size and memory footprint for faster inference.

Core Features & Use Cases

  • Calibration-Free Quantization: Quantize models instantly without needing representative data.
  • Extreme Precision: Supports 8/4/3/2/1-bit quantization for maximum compression.
  • Optimized Backends: Integrates with high-performance inference engines like vLLM, HuggingFace Transformers, Marlin, and BitBlas.
  • Use Case: Deploy a large LLM on resource-constrained hardware by quantizing it to 4-bit, reducing its VRAM requirements by 4x while maintaining acceptable performance for real-time applications.

Quick Start

Use the hqq-quantization skill to quantize the 'meta-llama/Llama-3.1-8B' model to 4-bit precision with a group size of 64.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize an LLM to 4-bit without a calibration dataset?▼

You can perform calibration-free weight quantization on LLMs to reduce them to 4-bit precision instantly. This approach compresses models without needing representative data, significantly reducing memory footprint and accelerating inference.

What levels of bit precision can I use for LLM compression?▼

LLM compression supports ultra-low bit precision ranging from 1 to 8 bits. You can select 8, 4, 3, 2, or 1-bit quantization to achieve your desired balance between maximum model compression and acceptable inference performance.

Can I fine-tune a quantized model using PEFT or LoRA?▼

Yes, you can fine-tune quantized models directly using PEFT and LoRA. This allows you to adapt extremely compressed models to specific tasks while maintaining the reduced memory footprint achieved through low-bit quantization.

Does HQQ quantization work with vLLM and HuggingFace Transformers for inference acceleration?▼

Yes, calibration-free quantization integrates with optimized backends like vLLM, HuggingFace Transformers, Marlin, and BitBlas. This compatibility ensures faster inference speeds and efficient deployment of compressed models on resource-constrained hardware.

Why does 4-bit quantization reduce VRAM requirements for large language models?▼

4-bit quantization reduces VRAM requirements by compressing model weights to a lower bit precision. This decreases the overall model size by approximately 4x, allowing deployment of large LLMs on hardware with limited memory for real-time applications.

When should I avoid using calibration-free quantization for model optimization?▼

You should avoid calibration-free quantization if maintaining the absolute maximum accuracy of the original model is critical. While it drastically reduces memory usage, extreme compression to 1 or 2-bit precision may degrade performance beyond acceptable limits.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
ihatesea69ihatesea69
3

hqq-quantization

Quantize LLMs to 4/3/2-bit precision without calibration data.

Community
Advanced
Clay-HHKClay-HHK
2

peft-fine-tuning

Fine-tune 7B-70B language models with LoRA and QLoRA adapters.

Community
Advanced
arsityarsity

awq-quantization

Quantize large language models to 4-bit precision using activation-aware weight quantization.

Community
Advanced