skills.rest
Jensen-Yao
Jensen-YaoCOMMUNITY
Explore all skills by this author →
GitHub

quantizing-models-bitsandbytes

Quantizes LLMs to 8-bit or 4-bit formats for reduced GPU memory usage.

2|Updated Aug 15, 2026
Search Tags:#llm#fine-tuning#quantization#gpu memory#bitsandbytes#4-bit#QLoRA
One-click install
npx skills add https://github.com/Jensen-Yao/agents-skills --skill quantizing-models-bitsandbytes-jensen-yao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quantizing-models-bitsandbytes
Source: https://github.com/Jensen-Yao/agents-skills/tree/main/skills/bitsandbytes
Command: npx skills add https://github.com/Jensen-Yao/agents-skills --skill quantizing-models-bitsandbytes-jensen-yao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bitsandbytes, transformers, accelerate, torch, and includes references (resource) components.

What problem does it solve? Large language models often exceed available GPU memory, blocking inference and fine-tuning on consumer hardware. This Skill reduces model memory by 50-75% through 8-bit and 4-bit quantization with minimal accuracy loss. ## Core Features & Use Cases - 8-bit and 4-bit Quantization: Load models with INT8, NF4, or FP4 formats via HuggingFace Transformers BitsAndBytesConfig, cutting a 14GB Llama 2 7B down to 3.5GB. - QLoRA Fine-tuning: Train LoRA adapters on 4-bit base models to fine-tune 70B models on a single GPU, saving only ~20MB adapter weights. - 8-bit Optimizers: Replace standard AdamW with paged 8-bit optimizers to reduce optimizer state memory by 75% during training. - Use Case: You have an RTX 4090 with 24GB VRAM and need to fine-tune Llama 2 13B. Load it in 4-bit NF4 with double quantization, attach LoRA adapters, and train with paged_adamw_8bit within roughly 18GB of memory. ## Quick Start Quantize the Llama 2 7B model to 4-bit NF4 format and load it for inference on my GPU.

Frequently Asked Questions about quantizing-models-bitsandbytes

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a HuggingFace model to 4-bit?▼

Create a BitsAndBytesConfig with load_in_4bit=True, bnb_4bit_quant_type="nf4", and bnb_4bit_use_double_quant=True, then pass it as quantization_config to AutoModelForCausalLM.from_pretrained. This reduces memory by about 75% with under 1% accuracy loss.

What is the difference between NF4 and FP4 quantization?▼

NF4 (NormalFloat4) uses bins optimized for normally distributed weights, making it more accurate for transformer models. FP4 uses symmetric bins suited to uniform distributions. For LLMs, NF4 consistently outperforms FP4 on benchmarks like MMLU.

How to fine-tune a 70B model on a single GPU with QLoRA?▼

Load the base model in 4-bit with NF4 and double quantization, prepare it with prepare_model_for_kbit_training, attach LoRA adapters via PEFT, and train with the paged_adamw_8bit optimizer. A 70B model fits in roughly 35GB, enabling training on one A100 80GB.

Does bitsandbytes work with FSDP multi-GPU training?▼

Yes, but you must set bnb_4bit_quant_storage=torch.bfloat16 in the BitsAndBytesConfig so 4-bit layers are wrapped identically to regular layers for FSDP sharding. The model torch_dtype must match the quant_storage dtype.

When should I use GPTQ or AWQ instead of bitsandbytes?▼

Use GPTQ or AWQ for production serving where inference speed matters most, as they run faster than bitsandbytes. Use GGUF for CPU inference with llama.cpp, and FP8 on H100 GPUs with hardware FP8 support. bitsandbytes suits QLoRA training and flexible HuggingFace workflows.

Why do I get CUDA errors when loading a quantized model?▼

CUDA errors usually come from a version mismatch between bitsandbytes and your installed CUDA toolkit. Check your CUDA version with nvcc --version and reinstall bitsandbytes with pip install bitsandbytes --no-cache-dir to get a matching build.

Related Skills

Explore More Software Engineering Skills→
ricablericable

llama-factory

Guide LLaMA-Factory fine-tuning and deployment workflows for large language models.

Community
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
dongzhuoyaodongzhuoyao
10

gpu-training-acceleration

Optimize PyTorch training speed and memory usage on CUDA GPUs.

Community
Advanced
HermesagentsHermesagents

gguf-quantization

Quantize and deploy GGUF models for llama.cpp CPU/GPU inference.

Community
Advanced