Compress large language models to 4-bit precision with GPTQ.
npx skills add https://github.com/ovachiever/droid-tings --skill gptq
Please help me install this Agent Skill. Skill: gptq Source: https://github.com/ovachiever/droid-tings/tree/main/skills/gptq Command: npx skills add https://github.com/ovachiever/droid-tings --skill gptq
This Skill explains GPTQ post-training 4-bit quantization, enabling deployment of very large models (70B+) on consumer GPUs with reduced memory and competitive accuracy, plus integration with PEFT for QLoRA fine-tuning.
Load a quantized model with AutoGPTQ, perform a forward pass, and compare performance against FP16.
High-intent search queries and answers about installing and using this skill.
4-bit quantization compresses models to one-quarter their original size, enabling 70B+ parameter models to run on consumer GPUs with ~4× memory savings and 3–4× inference speedups. GPTQ achieves this through post-training group-wise quantization with minimal perplexity loss.
Yes. AutoGPTQ integrates with Transformers to load quantized models and works with PEFT for QLoRA efficient fine-tuning, letting you adapt 4-bit compressed models without full retraining.
GPTQ uses group-wise 4-bit quantization with per-group scales and zero-points, designed specifically for post-training compression of very large models with calibration workflows that preserve accuracy better than simpler uniform quantization approaches.
GPTQ quantization achieves approximately 4× memory reduction compared to full precision while delivering 3–4× inference speedups in many setups, with competitive accuracy and minimal perplexity degradation on large models.
Yes. GPTQ requires calibration data workflows to compute per-group scales and zero-points during post-training quantization, ensuring the 4-bit representation accurately captures model behavior.
GPTQ is optimized for very large models (70B+ parameters) where memory and inference speed are bottlenecks on consumer GPUs; smaller models may not justify the quantization overhead.
Andrew-GirgisTrain and fine-tune vision models with Hugging Face Transformers and Jobs.
ovachieverCompress large language models to 4-bit precision with GPTQ.
ricableCompress large language models into GPTQ 4-bit checkpoints.
ovachieverOptimize LLM inference on NVIDIA GPUs with TensorRT-LLM.
