skills.rest
amd
amdOFFICIAL
Explore all skills by this author →
WebsiteGitHub

quark-torch-llm-ptq-eval

Orchestrate post-training quantization workflows for Torch LLMs with validation and accuracy evaluation.

154|27|Updated Jun 19, 2024
Search Tags:#validation#llm#evaluation#pytorch#quantization#huggingface#rocm
One-click install
npx skills add https://github.com/amd/Quark --skill quark-torch-llm-ptq-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quark-torch-llm-ptq-eval
Source: https://github.com/amd/Quark/tree/main/.claude/skills-impl/l3-recipes/torch/quark-torch-llm-ptq-eval
Command: npx skills add https://github.com/amd/Quark --skill quark-torch-llm-ptq-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill runs a complete post-training quantization workflow for Torch-based large language models, then verifies that the resulting model is structurally valid and optionally measures accuracy impact.

Core Features & Use Cases

  • End-to-End PTQ Orchestration: Chains model intake, quantization planning, manifest generation, and execution into one guided workflow.
  • Mandatory Validation: Checks the quantized model against the source model using layout, config, auxiliary files, and MD5-based integrity verification.
  • Optional Accuracy Evaluation: Runs quick perplexity checks, lm_eval benchmarks, or accelerated ROCm-based evaluation for deeper quality assessment.
  • Use Case: Use this Skill when you want to quantize a HuggingFace PyTorch LLM such as Llama, Qwen, or Mistral, confirm the output is correct, and decide whether accuracy remains acceptable.

Quick Start

Ask the assistant to quantize your Torch LLM, validate the output, and optionally evaluate accuracy for the model and target precision you want.

Frequently Asked Questions about quark-torch-llm-ptq-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a HuggingFace PyTorch LLM and validate the output?▼

To quantize a HuggingFace PyTorch LLM, you orchestrate end-to-end post-training quantization that includes manifest generation, structural validation, and optional accuracy evaluation. This workflow ensures the quantized model is structurally valid and checks accuracy impact.

Can I evaluate model accuracy after post-training quantization using lm_eval or vLLM?▼

Yes, you can evaluate model accuracy after post-training quantization by running quick perplexity checks, lm_eval benchmarks, or accelerated ROCm-based evaluation with vLLM. This optional step measures the accuracy impact of applying FP8 or INT4 settings.

Does this quantization workflow support ROCm for LLM evaluation?▼

Yes, the quantization workflow supports optional ROCm-backed evaluation for deeper quality assessment of LLMs. You can use ROCm acceleration when running lm_eval benchmarks or vLLM to measure accuracy after applying quantization.

What is the best way to verify a quantized Torch LLM against its source model?▼

The best way to verify a quantized Torch LLM is through mandatory structural validation that checks layout, config, and auxiliary files with MD5-based integrity verification. This ensures the quantized output correctly matches the original HuggingFace PyTorch transformer model.

What precision settings are available for quantizing Torch transformer models?▼

Available precision settings for quantizing Torch transformer models include FP8 and INT4, along with related configurations. These quantize, validate, and measure workflows apply to HuggingFace PyTorch models like Llama, Qwen, or Mistral.

Why does post-training quantization require strict artifact handling and checkpointed execution?▼

Post-training quantization requires strict artifact handling and checkpointed execution to maintain model integrity throughout the quantize, validate, and measure workflow. This process ensures validation reports are accurately generated and structural consistency is preserved.

Related Skills

Explore More Software Engineering Skills→
amdamd
154

quark-torch-ptq

Quantize PyTorch transformer models into deployment artifacts using AMD Quark PTQ workflows.

Official
Advanced
JamesFincherJamesFincher

tensorrt-llm

Optimizes LLM inference serving on NVIDIA GPUs using TensorRT-LLM with quantization and parallelism options.

Community
Advanced
amdamd
154

quark-torch-llm-eval

Evaluate LLM accuracy on AMD ROCm with vLLM, SGLang, or ATOM serving.

Official
Advanced
jonnabiojonnabio
6

gptq

Quantize LLMs to 4-bit precision using group-wise quantization.

Community
Advanced