skills.rest
hhhi21g
hhhi21gCOMMUNITY
Explore all skills by this author →
GitHub

hqq-quantization

Quantize machine learning models without calibration data for vLLM or HuggingFace Transformers.

Updated Apr 11, 2026
Search Tags:#machine learning#model compression#model optimization#inference speed#model quantization
One-click install
npx skills add https://github.com/hhhi21g/HealthCenter --skill hqq-quantization-hhhi21g
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/hhhi21g/HealthCenter/tree/main/.codex/skills/hqq
Command: npx skills add https://github.com/hhhi21g/HealthCenter --skill hqq-quantization-hhhi21g

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires hqq>=0.2.0, torch>=2.0.0, torchao, marlin, bitblas, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of quantizing models for model compression and inference speed without the need for calibration data, making it ideal for fast quantization workflows, deployment with vLLM or HuggingFace Transformers, and fine-tuning with LoRA/PEFT.

Core Features & Use Cases

  • Calibration-Free Quantization: Quantize models instantly without sample data.
  • Multiple Backends: Supports PyTorch, ATEN, TorchAO, Marlin, and BitBlas for optimized inference.
  • Flexible Precision: Offers 8/4/3/2/1-bit precision with configurable group sizes.
  • Framework Integration: Integrates with HuggingFace and vLLM.
  • PEFT Compatible: Fine-tune quantized models with LoRA.

Quick Start

Use the hqq-quantization skill to quantize the 'llama-3.1-8b' model to 4-bit precision.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a machine learning model without calibration data?▼

Model quantization without calibration data is achieved by applying the hqq-quantization Skill, which compresses models instantly for reduced size and increased inference speed without requiring sample datasets.

Can I fine-tune a quantized model using LoRA or PEFT?▼

Yes, you can fine-tune quantized models using LoRA or PEFT. The Skill produces models explicitly compatible with parameter-efficient fine-tuning workflows, maintaining flexibility despite reduced precision.

Does model quantization work with vLLM and HuggingFace Transformers?▼

Model quantization integrates directly with vLLM and HuggingFace Transformers for deployment. It supports multiple backends like PyTorch, ATEN, TorchAO, Marlin, and BitBlas for optimized inference.

What bit precisions are supported for model compression?▼

Model compression supports 8, 4, 3, 2, and 1-bit precision levels. You can configure group sizes alongside these flexible precision options to balance memory savings and inference accuracy.

Do I need specific libraries to run quantized models for inference?▼

Yes, running quantized models requires specific backend libraries like torchao or marlin for optimized inference. You must install hqq, torch, and these dependencies to ensure proper execution.

Related Skills

Explore More Software Engineering Skills→
JoaquinCampoJoaquinCampo
3

kvpress

Compress HuggingFace transformer model KV-cache during generation with press strategies.

Community
Advanced
tianhao909tianhao909
1

hqq-quantization

Quantize LLMs to 1-8 bit precision without calibration data.

Community
Advanced
zeronx798zeronx798

peft-fine-tuning

Performs parameter-efficient fine-tuning of LLMs with LoRA and QLora via transformers, peft, bitsandbytes libraries.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

efficient-ai

Compress PyTorch models using quantization, pruning, and knowledge distillation.

Community
Advanced