skills.rest
box755
box755COMMUNITY
Explore all skills by this author →
GitHub

gptq

Apply GPTQ post-training quantization to compress LLM weights into 4-bit representations.

Updated Aug 27, 2026
Search Tags:#model deployment#inference acceleration#4-bit#gptq#qlora#llm quantization#group-wise quantization
One-click install
npx skills add https://github.com/box755/simlens-research --skill gptq-box755
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gptq
Source: https://github.com/box755/simlens-research/tree/main/skill-packs/AI-Research-SKILLs/10-optimization/gptq
Command: npx skills add https://github.com/box755/simlens-research --skill gptq-box755

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires auto-gptq, transformers, optimum, peft, and includes references (resource) components.

What problem does it solve?

GPTQ reduces the memory footprint of large language models while keeping quality close to FP16, so you can run much bigger models on limited consumer or single-server GPU resources.

Core Features & Use Cases

  • Post-training 4-bit quantization: Compresses pretrained LLM weights to 4-bit using group-wise quantization with minimal accuracy loss.
  • Calibration-driven accuracy control: Uses representative calibration data to minimize quantization error and preserve perplexity.
  • Optimized inference backends: Supports multiple kernel backends (e.g., ExLlama, Marlin, Triton) for faster generation.

Use cases: deploy 70B–405B-class models on consumer GPUs, reduce VRAM needs for serving, and enable faster inference when FP16 is too large or too slow.

Quick Start

Use the gptq skill to quantize Llama-2-7B-Chat from HuggingFace by running the quantization with 4-bit group size 128 on a small calibration sample and saving the resulting quantized model directory.

Frequently Asked Questions about gptq

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a large language model to 4-bit for inference acceleration?▼

To quantize a large language model to 4-bit, apply GPTQ post-training quantization with group-wise quantization using representative calibration data to compress weights while minimizing accuracy loss.

Can I deploy a 70B model on consumer GPUs using 4-bit quantization?▼

Yes, 4-bit quantization reduces VRAM requirements enough to deploy 70B-class models on consumer GPUs by compressing pretrained weights while keeping quality close to FP16.

Does GPTQ quantization work with HuggingFace transformers workflows?▼

GPTQ integrates directly with HuggingFace and transformers workflows, requiring group-wise quantization configuration and calibration data processing to save compressed model directories.

What is the best way to preserve model accuracy during LLM quantization?▼

The best way to preserve accuracy during LLM quantization is using calibration-driven control, which processes representative data to minimize quantization error and maintain perplexity.

Can I use QLoRA and PEFT fine-tuning with a GPTQ quantized model?▼

Yes, GPTQ provides optional support for PEFT and QLoRA fine-tuning on top of the quantized weights, allowing parameter-efficient training on the 4-bit compressed base model.

Which inference backends are supported for GPTQ 4-bit models?▼

GPTQ supports multiple optimized inference kernel backends including ExLlama, Marlin, and Triton to accelerate text generation on the 4-bit compressed models.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

ptq

Quantize pretrained models to nvfp4, fp8, or int4_awq checkpoints using ModelOpt PTQ.

Official
Advanced
ihatesea69ihatesea69
3

hqq-quantization

Quantize LLMs to 4/3/2-bit precision without calibration data.

Community
Advanced
ScientiaCapitalScientiaCapital
1

training-optimization

Configure learning rates, LoRA, batch sizes, and gradient strategies for LLM fine-tuning.

Community
Advanced
jonnabiojonnabio
6

gptq

Quantize LLMs to 4-bit precision using group-wise quantization.

Community
Advanced