gptq

Quantize large language models to 4-bit precision for memory reduction and faster inference.

Updated Aug 27, 2026

Search Tags:#optimization #llm #calibration #quantization #transformers #peft #gptq

Ready to run

npx skills add https://github.com/t2ance/dr-claw-plugin --skill gptq-t2ance

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: gptq
Source: https://github.com/t2ance/dr-claw-plugin/tree/main/plugins/ml-training-stack/skills/optimization/gptq
Command: npx skills add https://github.com/t2ance/dr-claw-plugin --skill gptq-t2ance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Post-training 4-bit quantization enables deploying large language models on consumer hardware by dramatically reducing memory usage while preserving accuracy and enabling faster inference.

Core Features & Use Cases

Group-wise 4-bit quantization with minimal perplexity degradation
Seamless integration with transformers and PEFT for LoRA/QLoRA workflows
Supports multiple backends (ExLlama, Marlin, Triton) and multi-GPU/offload setups

Quick Start

Install AutoGPTQ and follow calibration-guided quantization steps to produce a quantized model ready for deployment.

Frequently Asked Questions about gptq

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I quantize large language models to 4-bit precision?▼

Quantize large language models to 4-bit precision by applying group-wise quantization with configurable bits and group_size across supported backends using calibration data. This reduces memory usage while preserving model accuracy for faster inference.

What is group-wise 4-bit quantization for LLMs?▼

Group-wise 4-bit quantization is a post-training technique reducing model memory footprint with minimal perplexity degradation. It configures group_size parameters during calibration to maintain accuracy while enabling deployment on consumer hardware.

Does 4-bit LLM quantization work with PEFT and transformers workflows?▼

Yes, 4-bit LLM quantization integrates seamlessly with transformers and PEFT for LoRA or QLoRA workflows. This ensures compatibility with multi-GPU and CPU-offload deployment setups while maintaining efficient model fine-tuning capabilities.

Can I deploy quantized LLMs on consumer hardware without losing accuracy?▼

You can deploy quantized LLMs on consumer hardware with minimal loss by applying post-training 4-bit quantization. Calibration-guided steps ensure the quantized model retains high accuracy while dramatically reducing memory requirements for faster local execution.

Which backends are supported for 4-bit LLM quantization and multi-GPU offload?▼

Supported backends for 4-bit LLM quantization include ExLlama, Marlin, and Triton. These backends facilitate multi-GPU offloading and CPU-offload setups, ensuring flexible deployment across various hardware configurations for quantized models.

Do I need calibration data to perform 4-bit quantization on transformers?▼

Yes, you need calibration data to perform 4-bit quantization on transformers. Calibration-guided quantization steps are required to produce a quantized model ready for deployment, ensuring minimal perplexity degradation during the group-wise quantization process.