skills.rest
amd
amdOFFICIAL
Explore all skills by this author →
WebsiteGitHub

quark-torch-quant-plan

Build a Quark Torch post-training quantization plan from model analysis and deployment intent.

154|27|Updated Jun 19, 2024
Search Tags:#llm#quantization#torch#fp8#ptq#int4#quark
One-click install
npx skills add https://github.com/amd/Quark --skill quark-torch-quant-plan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quark-torch-quant-plan
Source: https://github.com/amd/Quark/tree/main/.claude/skills-impl/l1-atomic/torch/quark-torch-quant-plan
Command: npx skills add https://github.com/amd/Quark --skill quark-torch-quant-plan

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you decide how to quantize a Torch language model in Quark by turning model analysis and deployment goals into a concrete PTQ plan. It removes guesswork around scheme selection, algorithm choice, exclusions, and KV cache decisions.

Core Features & Use Cases

  • Quantization scheme selection: Recommends FP8, INT4 weight-only, INT8, MXFP, or AMD-specific formats based on accuracy, size, and hardware targets.
  • Plan assembly: Produces a structured quantization plan with global scheme, layer overrides, calibration settings, and evaluation intent.
  • Decision support: Useful when you need to compare options such as AWQ, GPTQ, SmoothQuant, or mixed-precision overrides for specific modules like attention or lm_head.
  • Use Case: A user provides model analysis for a large LLM and asks for the best inference-ready configuration on a specific accelerator, and this Skill drafts the quantization plan to confirm before execution.

Quick Start

Ask for a Quark Torch quantization plan for my model analysis and deployment target, and recommend the best scheme, algorithm, exclusions, and KV cache settings.

Frequently Asked Questions about quark-torch-quant-plan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the right quantization scheme for my LLM in Torch?▼

Choosing the right quantization scheme requires analyzing your model and deployment target. Quark recommends FP8, INT4 weight-only, INT8, or MXFP formats by evaluating accuracy, size, and hardware constraints to build an accelerator-aware PTQ plan.

What's the best way to plan post-training quantization for large language models?▼

The best way to plan post-training quantization is to generate a structured PTQ plan before execution. Quark assembles a configuration with global schemes, layer-specific overrides, and calibration settings based on your deployment intent to remove execution guesswork.

Can I use mixed-precision overrides for specific modules like attention or lm_head?▼

Yes, you can use mixed-precision overrides for specific modules like attention or lm_head. Quark supports layer-specific overrides, allowing you to exclude sensitive layers or apply different algorithms to maintain model accuracy during PTQ.

How do AWQ, GPTQ, and SmoothQuant compare for Torch quantization?▼

AWQ, GPTQ, and SmoothQuant are distinct PTQ algorithms for optimizing LLMs. Quark provides decision support to compare these options, recommending the best algorithm based on your specific model analysis and hardware accelerator target.

When should I exclude layers or adjust KV cache settings during PTQ?▼

You should exclude layers or adjust KV cache settings during PTQ when sensitive modules degrade accuracy. Quark identifies these layers and recommends specific KV cache configurations and exclusions to maintain model performance during inference.

Related Skills

Explore More Software Engineering Skills→
amdamd
154

quark-onnx-quant-plan

Build hardware-aware Quark ONNX quantization plans from model analysis and deployment intent.

Official
Advanced
air-gappedair-gapped
5

vllm-quantization

Quantize LLM checkpoints for vLLM serving with FP8, NVFP4, and MXFP4 formats.

Official
Advanced
SignmanalSignmanal
1

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

Community
Intermediate
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced