skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

quantization

Quantize pre-trained Large Language Models using PTQ, QAT, GPTQ, AWQ, bitsandbytes, FP8, and GGUF.

1|Updated Jun 4, 2026
Search Tags:#LLM#quantization#model compression#large language models#inference optimization
One-click install
npx skills add https://github.com/hung-phan/ml-skills --skill quantization-hung-phan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quantization
Source: https://github.com/hung-phan/ml-skills/tree/main/skills/ml-review/references/ml-architectures/quantization
Command: npx skills add https://github.com/hung-phan/ml-skills --skill quantization-hung-phan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides solutions for reducing the memory footprint and accelerating inference of Large Language Models (LLMs) through quantization methods.

Core Features & Use Cases

  • Model Quantization: Offers various quantization methods like PTQ, QAT, GPTQ, AWQ, bitsandbytes, FP8, and GGUF.
  • Memory and Speed: Enables running large models on smaller GPUs, reducing inference cost, and improving inference speed.
  • Use Case: When deploying a 70B parameter model on a single 48GB GPU, quantization allows for a significant reduction in VRAM usage and a 2-4x faster inference speed.

Quick Start

Run the quantization skill to apply PTQ to a model with a calibration dataset of 128-512 samples.

Frequently Asked Questions about quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce LLM memory footprint to run a 70B model on a single 48GB GPU?▼

Model quantization reduces memory footprint for large language models using methods like PTQ, GPTQ, and bitsandbytes. This allows deploying a 70B parameter model on a single 48GB GPU by significantly decreasing VRAM usage.

What is the difference between PTQ, GPTQ, AWQ, and bitsandbytes for large language model inference optimization?▼

Quantization methods for large language models include PTQ, QAT, GPTQ, AWQ, bitsandbytes, FP8, and GGUF. These inference optimization techniques reduce memory footprint and accelerate inference speed across various LLM deployment scenarios.

How to quantize a pre-trained large language model step by step?▼

To quantize a pre-trained large language model, run the quantization process using Post-Training Quantization (PTQ) and provide a calibration dataset of 128-512 samples to optimize model inference and reduce memory footprint.

Do I need a calibration dataset to apply GPTQ or AWQ quantization?▼

Yes, applying GPTQ or AWQ quantization requires a calibration dataset. Providing 128-512 samples during the quantization process ensures accurate model compression and effective memory footprint reduction for large language models.

When should I use FP8 versus GGUF formats for LLM model compression?▼

Use FP8 or GGUF formats for LLM model compression based on your specific deployment scenario and hardware. Both quantization methods reduce memory footprint and improve inference speed, enabling large models to run on smaller GPUs.

Does quantization affect inference speed on smaller GPUs?▼

Quantization improves inference speed on smaller GPUs by reducing the memory footprint of large language models. Applying methods like PTQ or bitsandbytes can accelerate inference by 2-4x while significantly decreasing VRAM usage.

Related Skills

Explore More Software Engineering Skills→
OpenLAIROpenLAIR
1.0k

gptq

Quantize large language models to 4-bit GPTQ with 128-group configuration.

Official
Advanced
ihatesea69ihatesea69
3

llama-cpp

Run GGUF quantized LLM inference on CPU, Apple Silicon, and consumer GPUs.

Community
Advanced
MesferAliMesferAli

quantizing-models-bitsandbytes

Quantize LLMs to 8-bit or 4-bit to reduce GPU memory usage.

Community
Advanced
hhhi21ghhhi21g

model-pruning

Prune large language models with Wanda and SparseGPT techniques.

Community
Advanced