Quantize LLM weights to 8/4/3/2/1-bit using calibration-free HQQ.
npx skills add https://github.com/qcmuu/AI-Research-Skills --skill hqq-quantization-qcmuu
Please help me install this Agent Skill. Skill: hqq-quantization Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/10-optimization/hqq Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill hqq-quantization-qcmuu
This Skill solves the slow, dataset-dependent quantization problem by enabling fast calibration-free weight quantization for LLMs, so you can reduce memory and improve inference efficiency without running expensive calibration passes.
Use case example: When you need to deploy a large LLM on limited GPU memory, quantize a Llama model to 4-bit using HQQ, choose an optimized backend for your hardware, and serve it with vLLM for faster, lower-cost generation.
Use the hqq-quantization skill to quantize a model to 4-bit with a single instruction: pip install hqq, then load the model with an HqqConfig(nbits=4, group_size=64) and generate text to verify quality.
High-intent search queries and answers about installing and using this skill.
Calibration-free LLM quantization uses half-quadratic quantization to directly reduce model weights to 8/4/3/2/1-bit precision without needing a calibration dataset. This allows immediate memory reduction for inference without running expensive calibration passes.
Yes, you can deploy 4-bit quantized models with vLLM by configuring HqqConfig with nbits and group_size. Selecting optimized runtime backends like Marlin or TorchAO ensures efficient matrix multiplication for faster, lower-cost inference generation.
Using half-quadratic quantization to compress model weights into 8/4/3/2/1-bit representations effectively reduces GPU memory usage. Applying per-layer mixed precision and serving the quantized model via vLLM maximizes inference efficiency on limited hardware.
Yes, HQQ quantization supports optional LoRA fine-tuning on quantized models. This allows you to apply parameter-efficient fine-tuning to your 8/4/3/2/1-bit quantized LLMs within your existing HuggingFace Transformers and PEFT workflows.
You configure per-layer mixed precision by adjusting the BaseQuantizeConfig or HqqConfig settings. This allows you to specify symmetric or asymmetric options, group size, axis, and varying bit depths across different layers for optimized inference performance.
You need to install HQQ, alongside Torch, Transformers, PEFT, and vLLM. These dependencies enable you to load the model with HqqConfig, apply half-quadratic quantization, and execute efficient generation using available runtime backends.
vasilyu1983Optimize LLM inference performance and cost with vLLM and quantization.
GitnappConfigure LoRA and QLoRA adapters for memory-efficient LLM fine-tuning.
tianhao909Quantize LLMs to 1-8 bit precision without calibration data.
gagan114662Quantize Large Language Models to 4/3/2-bit precision using HQQ.
