quantizing-models-bitsandbytes

Load large Hugging Face models in 4-bit or 8-bit by applying bitsandbytes quantization configurations like NF4 or INT8. This reduces GPU memory usage during inference while preserving model accuracy for memory-constrained deployments.

You can fine-tune a 13B model on a 24 GB GPU using QLoRA. This approach trains adapters on a quantized base model, combining 4-bit loading with paged optimizers to prevent out-of-memory errors and lower infrastructure costs.

NF4 and INT8 quantization reduce transformer model size by representing weights in 4-bit or 8-bit formats. NF4 is optimized for QLoRA fine-tuning, while INT8 is commonly used for standard Hugging Face inference to lower memory consumption.

Bitsandbytes quantization works with paged optimizers and CPU offloading. Combining these memory efficiency workflows with gradient checkpointing allows you to run significantly larger models on limited GPU memory without causing system crashes.

You need accelerate, torch, transformers, and bitsandbytes to configure quantization. These dependencies are required to set up INT8, NF4, FP4, and double quantization workflows for loading large language models on your GPU.

Use double quantization for large language models when you need extreme memory optimization. It quantizes the quantization constants themselves, further reducing GPU memory usage while maintaining accuracy for memory-constrained deployment scenarios.