hqq-quantization

You can quantize an LLM without calibration data using Half-Quadratic Quantization. It enables fast 4/3/2/1-bit weight quantization directly, allowing immediate deployment with vLLM or HuggingFace Transformers without requiring a calibration dataset.

The best way to deploy a quantized model with vLLM is using calibration-free weight quantization. Applying Half-Quadratic Quantization allows you to quantize model weights to lower bits and seamlessly integrate the result into vLLM inference pipelines.

Yes, weight quantization works with HuggingFace Transformers. The process applies quantization to model weights and provides framework integration, allowing you to load the optimized model directly for inference within the HuggingFace ecosystem.

Yes, you can apply mixed-precision settings during LLM quantization. This approach supports flexible quantization configurations, including per-layer dynamic settings, allowing different bits of precision across various layers to balance memory efficiency and performance.

Yes, it is possible to fine-tune a quantized model with PEFT or LoRA. The quantization process supports integration with PEFT and LoRA fine-tuning workflows, enabling memory-efficient adaptation of large language models after weight compression.