skills.rest
Supporter09
Supporter09COMMUNITY
Explore all skills by this author →
GitHub

hqq-quantization

Quantize Hugging Face models to low-bit weights without calibration data.

Updated May 4, 2026
Search Tags:#llm#compression#quantization#inference#transformers#vllm#hqq
One-click install
npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill hqq-quantization-supporter09
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/Supporter09/Face_Anti_Spoofing_Biometric/tree/main/.claude/skills/hqq
Command: npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill hqq-quantization-supporter09

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the delay and complexity of compressing large language models by enabling fast, calibration-free quantization for smaller, cheaper, and faster inference-ready models.

Core Features & Use Cases

  • Calibration-Free Quantization: Reduce model precision to 8/4/3/2/1-bit without needing a calibration dataset.
  • Deployment Backends: Prepare models for optimized inference with PyTorch, TorchAO, Marlin, BitBlas, or vLLM.
  • Mixed-Precision Tuning: Apply different bit-widths to attention and MLP layers for better quality-speed tradeoffs.
  • Use Case: A team can quantize a Hugging Face causal language model, validate output quality, and deploy it with a faster backend while keeping memory usage low.

Quick Start

Use the hqq-quantization skill to compress a Hugging Face model to 4-bit without calibration data and prepare it for faster inference.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a large language model without needing calibration data?▼

Calibration-free quantization reduces model precision to 8/4/3/2-bit weights directly without needing a calibration dataset. This enables fast compression workflows for Hugging Face Transformers or vLLM while keeping memory usage low.

What is mixed-precision tuning for LLM quantization?▼

Mixed-precision tuning applies different bit-widths to attention and MLP layers during quantization. This balances the tradeoff between output quality and inference speed for large language models.

Can I use vLLM for backend-specific deployment of quantized models?▼

Yes, you can prepare quantized models for optimized inference with vLLM. The skill also supports PyTorch, TorchAO, Marlin, and BitBlas backends for backend-specific deployment.

Does HQQ quantization work with LoRA configurations?▼

Yes, HQQ quantization supports LoRA compatibility. You can combine low-bit weight quantization with LoRA adapters while tuning bit-width, group-size, and axis selection.

What are the limitations of calibration-free quantization?▼

Limitations of calibration-free quantization include potential memory, quality, and hardware constraints. You must troubleshoot bit-width, group-size, and axis selection to balance performance and prevent inference issues.

What bit-widths can I use to compress a Hugging Face causal language model?▼

You can compress a Hugging Face causal language model to 8, 4, 3, 2, or 1-bit weights. This allows you to reduce model precision for faster and cheaper inference without needing calibration data.

Related Skills

Explore More Software Engineering Skills→
zaiyemeiyou404zaiyemeiyou404

llama-cpp

Run local GGUF inference with llama.cpp and discover models from Hugging Face Hub.

Community
Advanced
informatico-madridinformatico-madrid
1

hqq-quantization

Quantize large language models to 1-8 bit precision without calibration.

Community
Intermediate
itsmostafaitsmostafa
24

qlora

Fine-tune 7B+ models on consumer GPUs with 4-bit quantization and LoRA adapters.

Community
Advanced
ihatesea69ihatesea69
3

hqq-quantization

Quantize LLMs to 4/3/2-bit precision without calibration data.

Community
Advanced