Quantize large language models to 4-bit GPTQ with 128-group configuration.
npx skills add https://github.com/OpenLAIR/dr-claw --skill gptq-openlair
Please help me install this Agent Skill. Skill: gptq Source: https://github.com/OpenLAIR/dr-claw/tree/main/skills/optimization/gptq Command: npx skills add https://github.com/OpenLAIR/dr-claw --skill gptq-openlair
Post-training 4-bit quantization of LLMs to reduce memory usage and speed up inference.
Quantize a pretrained LLM to 4-bit GPTQ using a 128-group configuration and calibrate with a representative dataset before saving the quantized model.
High-intent search queries and answers about installing and using this skill.
You can deploy 70B+ models on consumer GPUs by applying post-training 4-bit GPTQ quantization with a 128-group configuration and calibrating with a representative dataset to reduce memory usage.
Yes, 4-bit GPTQ quantization integrates with transformers and PEFT, enabling QLoRA fine-tuning workflows directly on the quantized models to optimize memory efficiency during training.
GPTQ quantized models support multiple inference backends including ExLlama, Marlin, and Triton, along with multi-GPU setups to accelerate inference across different hardware environments.
Yes, calibration data is required to perform 4-bit GPTQ quantization. You must use a representative dataset to calibrate the model during the quantization process to maintain accuracy.
The best way to run 70B+ models in memory-constrained environments is post-training 4-bit GPTQ quantization, which significantly reduces memory usage and speeds up inference on consumer GPUs.
ricableCompress large language models into GPTQ 4-bit checkpoints.
tianhao909Quantize LLMs to 1-8 bit precision without calibration data.
ihatesea69Quantize LLMs to 4/3/2-bit precision without calibration data.
qcmuuQuantize large language models to 4-bit with GPTQ calibration.
