skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

quantized-export

Export promoted fine-tuned checkpoints as FP8, AWQ, GGUF, or merged safetensors with smoke-test validation.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#fine-tuning#fp8#gguf#llama.cpp#awq#model export#Quantization
One-click install
npx skills add https://github.com/wshobson/agents --skill quantized-export
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quantized-export
Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/quantized-export
Command: npx skills add https://github.com/wshobson/agents --skill quantized-export

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires unsloth, autoawq, transformers, sentencepiece, vllm, and includes references (resource) components.

What problem does it solve?

A checkpoint that passed promotion still isn't deployable until it's exported in the right format for its target runtime, and export bugs are silent — a malformed export still produces a loadable artifact. This Skill picks the correct quantization format for the target hardware and workload, then gates every export behind a mandatory pre/post smoke test.

Core Features & Use Cases

  • Format Selection Map: Chooses FP8 for Hopper-class GPUs, AWQ INT4 for older GPUs, GGUF Q4_K_M with imatrix for edge/llama.cpp, and explicitly avoids NVFP4 on GB10 hardware.
  • Workload Overrides: Blocks INT4 quantization for long-context, code, and math workloads where quantization error compounds, requiring FP8 or W8A8 instead.
  • Mandatory Smoke Test: Loads the exported artifact in its actual target runtime, runs 3–5 golden prompts with deterministic decoding, and gates on byte match for lossless exports or task-grader verdict agreement for lossy ones.
  • Use Case: After a LoRA fine-tune receives a PROMOTE verdict, export it as GGUF Q4_K_M with an imatrix for laptop serving via llama.cpp, then verify the quantized model still passes the same task graders as the pre-export checkpoint.

Quick Start

Export my promoted checkpoint as GGUF Q4_K_M with an imatrix for llama.cpp serving and run the smoke test against my goldens file.

Frequently Asked Questions about quantized-export

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I export a fine-tuned model to GGUF for llama.cpp?▼

Convert the merged safetensors to f16 GGUF with convert_hf_to_gguf.py, generate an importance matrix from a domain-representative calibration corpus using llama-imatrix, then quantize to Q4_K_M with llama-quantize. Skipping the imatrix step works but leaves accuracy on the table.

What quantization format should I use for my GPU?▼

FP8 is the default on Hopper-class GPUs and newer, AWQ INT4 targets older GPU generations, and GGUF Q4_K_M is for edge or llama.cpp serving. On GB10 hardware, avoid NVFP4 entirely — it runs about 32% slower than FP8 there.

Can I use INT4 quantization for code or math models?▼

No. Long-context, code, and math workloads break at INT4 because quantization error compounds across long sequences and precise reasoning. Stay on FP8 or W8A8 for these workload classes regardless of target hardware, and validate with actual task evals rather than broad benchmarks like MMLU.

Should I export a merged model or LoRA-only adapter?▼

Merged export folds the adapter into base weights for a self-contained artifact with no base-model dependency at serve time. LoRA-only produces a much smaller artifact but requires the serving stack to load the exact same base model revision alongside it.

Why does my exported model produce garbled or run-on output?▼

Garbled output typically indicates a chat template mismatch between the export and the checkpoint's training template. Fluent but semantically wrong output points to quantization incorrectly applied to the lm_head. Both are caught by the mandatory pre/post smoke test.

Why does my smoke test fail byte match on a quantized export?▼

Byte match is expected to fail for any lossy quantized export — quantization legitimately perturbs logits. The correct gate for lossy exports is task-grader verdict agreement between pre- and post-export outputs; byte match applies only to lossless fp16/bf16 merges.

Related Skills

Explore More Software Engineering Skills→
EstomEstom

fine-tuning-expert

Fine-tune large language models with LoRA, QLoRA, and PEFT workflows.

Community
Advanced
GitnappGitnapp

serving-llms-vllm

Deploy large language model inference services with vLLM and OpenAI-compatible endpoints.

Community
Advanced
Veloxia-agencyVeloxia-agency

prompt-governance

Version, evaluate, and promote production prompt assets with deployment gates.

Community
Advanced
yo-stevenyo-steven

deployment

Deploy HF LLM checkpoints as OpenAI-compatible endpoints via vLLM, SGLang, or TRT-LLM.

Community
Advanced