Quantize large language models to 4-bit with GPTQ calibration.
npx skills add https://github.com/qcmuu/AI-Research-Skills --skill gptq-qcmuu
Please help me install this Agent Skill. Skill: gptq Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/10-optimization/gptq Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill gptq-qcmuu
GPTQ solves the problem of deploying large LLMs with limited GPU memory and high inference cost by converting FP16/FP32 model weights into 4-bit weights while keeping accuracy degradation low.
Use it when you need to run a large model on consumer or single-GPU setups (e.g., RTX 4090/3090) with roughly 4× memory reduction and under ~2% perplexity degradation, or when you want significantly faster generation than FP16.
Run GPTQ quantization for your model by telling your AI to quantize meta-llama/Llama-2-7b-chat-hf to 4-bit using group_size 128 and calibrate on C4 for 128 samples, then save the resulting artifacts locally.
High-intent search queries and answers about installing and using this skill.
4-bit post-training quantization reduces VRAM usage by converting FP16 or FP32 model weights to 4-bit, achieving roughly 4× memory reduction with under ~2% perplexity degradation. This allows large models to run on consumer setups like RTX 4090 or 3090 GPUs.
Group-wise GPTQ quantization maintains accuracy by using a small calibration dataset to minimize quantization error via Hessian-aware procedures. Configurable parameters like group_size, desc_act, and damp_percent allow fine-tuning the balance between compression and quality loss.
Yes, GPTQ supports multiple execution backends including ExLlamaV2, Marlin, and Triton. These backend-focused inference accelerators improve tokens per second generation speed, providing significantly faster inference compared to FP16 models.
Yes, the GPTQ workflow supports integration paths for both inference and QLoRA-style fine-tuning. It covers end-to-end quantization, saving and pushing models, and deployment optimization for research and production environments.
Calibration data is required to minimize quantization error during the conversion process. By running a small set of samples through the model, the Hessian-aware procedure adjusts group-wise quantization parameters to keep accuracy degradation low.
GPTQ is limited to post-training quantization and requires group-wise calibration data. While it achieves under ~2% perplexity degradation, some quality loss is inevitable compared to FP16, and users must configure parameters like bits, group_size, and damp_percent carefully.
tianhao909Quantize LLM weights to 4-bit precision for reduced memory usage.
zhuangbiaoweiQuantize large language models to 4-bit with the GPTQ algorithm.
afel6Fine-tune large language models with Unsloth using LoRA and QLoRA.
ihatesea69Quantize large language models to 4-bit weights using AWQ.
