qlora

What problem does it solve?

Memory-efficient fine-tuning for large models on consumer GPUs by combining 4-bit quantization with LoRA adapters, enabling training of large models with limited VRAM.

Core Features & Use Cases

• Memory-efficient fine-tuning with 4-bit Quantization: NF4-based weight quantization combined with full-precision LoRA adapters to minimize GPU memory usage during training.
• Double quantization and paged optimizers: advanced techniques to further reduce memory footprint and handle memory spikes during training.
• Workflow support for large-scale models: designed for 7B+ models on consumer GPUs and scalable to larger sizes with careful resource management.
• Inference and merging workflows: options to merge adapters into full precision for deployment when needed.

Quick Start

Configure a memory-efficient QLoRA fine-tuning run for a 7B+ model using 4-bit NF4, double quantization, and LoRA adapters.