skills.rest
Tgoldi
TgoldiCOMMUNITY
Explore all skills by this author →
GitHub

quantizing-models-bitsandbytes

Quantizes LLMs to 8-bit or 4-bit formats for reduced GPU memory usage with HuggingFace Transformers.

1|Updated May 10, 2026
Search Tags:#llm#memory optimization#quantization#huggingface#bitsandbytes#4-bit#qlora
One-click install
npx skills add https://github.com/Tgoldi/claude-skills --skill quantizing-models-bitsandbytes-tgoldi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quantizing-models-bitsandbytes
Source: https://github.com/Tgoldi/claude-skills/tree/main/quantizing-models-bitsandbytes
Command: npx skills add https://github.com/Tgoldi/claude-skills --skill quantizing-models-bitsandbytes-tgoldi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bitsandbytes, transformers, accelerate, torch, and includes references (resource) components.

What problem does it solve? Large language models often exceed available GPU memory, blocking inference and fine-tuning on consumer hardware. This Skill reduces model memory by 50-75% through 8-bit and 4-bit quantization with minimal accuracy loss. ## Core Features & Use Cases - 8-bit and 4-bit Quantization: Load models with INT8, NF4, or FP4 formats via BitsAndBytesConfig, cutting a 14GB Llama 2 7B down to 3.5GB. - QLoRA Fine-tuning: Train LoRA adapters on 4-bit base models, enabling fine-tuning of 70B models on a single GPU. - 8-bit Optimizers: Replace standard AdamW with paged 8-bit optimizers to cut optimizer memory by 75%. - Use Case: You have an RTX 4090 with 24GB VRAM and need to fine-tune Llama 2 13B. Load it in 4-bit with NF4 and double quantization, attach LoRA adapters, and train within roughly 18GB of memory. ## Quick Start Quantize the meta-llama/Llama-2-7b-hf model to 4-bit using bitsandbytes and run a test inference to verify memory usage.

Frequently Asked Questions about quantizing-models-bitsandbytes

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a HuggingFace model to 4-bit?▼

Create a BitsAndBytesConfig with load_in_4bit=True, bnb_4bit_quant_type="nf4", and bnb_4bit_use_double_quant=True, then pass it to AutoModelForCausalLM.from_pretrained with device_map="auto". This reduces memory by about 75% with under 1% accuracy loss.

What is QLoRA and how do I fine-tune with it?▼

QLoRA fine-tunes a 4-bit quantized base model by training small LoRA adapters instead of full weights. Load the model in 4-bit, call prepare_model_for_kbit_training, attach adapters with peft's LoraConfig, and train with a standard Trainer using paged_adamw_8bit.

Should I use NF4 or FP4 quantization for LLMs?▼

NF4 is recommended for transformer models because its quantization bins match normally distributed weights, giving better accuracy than FP4. FP4 suits uniform distributions and should only be used if NF4 fails.

bitsandbytes vs GPTQ vs GGUF: which should I use?▼

Use bitsandbytes for QLoRA training and quick HuggingFace integration. Choose GPTQ or AWQ for faster production inference serving, GGUF for CPU inference with llama.cpp, and FP8 on H100 hardware.

Why do I get a CUDA error when loading a quantized model?▼

CUDA errors usually mean a version mismatch between bitsandbytes and your CUDA toolkit. Check your version with nvcc --version and reinstall bitsandbytes with pip install bitsandbytes --no-cache-dir. CUDA 11.1+ and compute capability 7.0+ are required.

What GPU memory do I need for a 4-bit quantized model?▼

Estimate memory as parameters multiplied by 0.5 bytes: Llama 2 7B needs about 4GB, 13B about 8GB, and 70B about 24GB in 4-bit. If the model still does not fit, enable CPU offloading with max_memory and device_map="auto".

Related Skills

Explore More Software Engineering Skills→
JKhyroJKhyro

axolotl

Configure and validate Axolotl YAML files for model fine-tuning workflows.

Community
Advanced
tylertitsworthtylertitsworth

huggingface-transformers

Manage HuggingFace models and datasets in Python workflows.

Community
Advanced
dongzhuoyaodongzhuoyao
10

gpu-training-acceleration

Optimize PyTorch training speed and memory usage on CUDA GPUs.

Community
Advanced
VYRE-StudiosVYRE-Studios

serving-llms-vllm

Serve LLMs with vLLM using PagedAttention and continuous batching.

Official
Advanced