skills.rest
ricable
ricableCOMMUNITY
Explore all skills by this author →
GitHub

gptq

Compress large language models into GPTQ 4-bit checkpoints.

Updated Feb 15, 2026
Search Tags:#calibration#quantization#transformers#vllm#gptq#llm deployment#autogptq
One-click install
npx skills add https://github.com/ricable/mcai --skill gptq-ricable
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gptq
Source: https://github.com/ricable/mcai/tree/main/.agents/skills/gptq
Command: npx skills add https://github.com/ricable/mcai --skill gptq-ricable

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you compress large language models into 4-bit GPTQ formats so they fit on smaller GPUs, run faster, and retain strong output quality.

Core Features & Use Cases

  • Post-training quantization: Turn FP16 Hugging Face models into compact GPTQ checkpoints with group-wise quantization.
  • Deployment optimization: Load quantized models with AutoGPTQ, transformers, vLLM, or TGI for efficient inference on consumer or server GPUs.
  • Fine-tuning and validation: Calibrate models with representative data, tune quantization settings, and verify quality with perplexity or benchmark checks.

Quick Start

Ask the GPTQ skill to help you quantize a target model, choose calibration data, and prepare the result for deployment or evaluation.

Frequently Asked Questions about gptq

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compress a Hugging Face model into a GPTQ 4-bit checkpoint?▼

You compress a Hugging Face model into a GPTQ 4-bit checkpoint by applying post-training quantization with AutoGPTQ, using representative calibration data to maintain inference quality and reduce GPU memory usage.

What is the best way to deploy GPTQ quantized models for faster inference?▼

The best way to deploy GPTQ quantized models is by loading the 4-bit checkpoints with vLLM, TGI, AutoGPTQ, or transformers to achieve efficient inference on consumer or server GPUs.

Can I fine-tune and validate models after applying 4-bit quantization?▼

Yes, you can fine-tune and validate models after 4-bit quantization by integrating with PEFT, calibrating with representative data, and verifying output quality using perplexity or benchmark checks.

Does GPTQ quantization work with vLLM for multi-GPU server deployment?▼

Yes, GPTQ quantization works with vLLM for multi-GPU server deployment, allowing you to load 4-bit checkpoints efficiently and serve inference across consumer or multi-GPU servers.

How much accuracy is lost when shrinking LLMs to 4-bit with GPTQ?▼

Shrinking LLMs to 4-bit with GPTQ results in little accuracy loss, preserving strong output quality through group-wise quantization and calibration tuning during the compression process.

What calibration data do I need for AutoGPTQ post-training quantization?▼

For AutoGPTQ post-training quantization, you need representative calibration data that matches your target use case to tune group-wise quantization settings and verify output quality without significant accuracy loss.

Related Skills

Explore More Software Engineering Skills→
zaiyemeiyou404zaiyemeiyou404

llama-cpp

Run local GGUF inference with llama.cpp and discover models from Hugging Face Hub.

Community
Advanced
HarriesHarries

serving-llms-vllm

Serve LLMs with vLLM for OpenAI-compatible endpoints using tensor parallelism and quantization.

Community
Advanced
unclehowellunclehowell

peft-fine-tuning

Fine-tune large language models using LoRA or QLoRA with 4-bit quantization.

Community
Advanced
jonnabiojonnabio
6

gptq

Quantize LLMs to 4-bit precision using group-wise quantization.

Community
Advanced