skills.rest
Mesh-LLM
Mesh-LLMOFFICIAL
Explore all skills by this author →
WebsiteGitHub

hf-gguf-quant-jobs

Quantize split BF16/FP16 GGUF model repos into custom quant GGUF repos with skippy-quantize.

3.4k|405|Updated Feb 11, 2026
Search Tags:#model compression#lLM#gguf#hugging face#hf jobs#Quantization#skippy-quantize
One-click install
npx skills add https://github.com/Mesh-LLM/mesh-llm --skill hf-gguf-quant-jobs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hf-gguf-quant-jobs
Source: https://github.com/Mesh-LLM/mesh-llm/tree/main/.agents/skills/hf-gguf-quant-jobs
Command: npx skills add https://github.com/Mesh-LLM/mesh-llm --skill hf-gguf-quant-jobs

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Quantizing large language models normally requires holding the entire model in memory or on local disk, which is infeasible for very large models. This Skill runs low-memory, resumable quantization jobs that process split BF16/FP16 GGUF repos one shard window at a time, either locally or as Hugging Face Jobs.

Core Features & Use Cases

  • Low-memory windowed quantization: Uses skippy-quantize to stage, quantize, publish, and clean up one split window at a time, keeping memory bounded via --max-memory.
  • Resumable HF Jobs: Launches Hugging Face Jobs with mounted Hub repos, HF_TOKEN secrets, and manifests so cancelled jobs resume from the first missing target shard.
  • Monitoring and validation: Polls a compact JSON status file, inspects job logs for healthy or concerning markers, and validates the target repo by counting shards and checking the quant plan and tensor-type file.
  • Use Case: You have a 200GB split BF16 GGUF model on the Hugging Face Hub and need a Q4_K_M quant repo. Launch an HF Job with the source and target repos mounted, let skippy-quantize publish shards incrementally, and validate the completed target repo.

Quick Start

Ask the agent to create and launch a low-memory skippy-quantize job that quantizes your split BF16 GGUF source repo into a target quant repo on Hugging Face with a specified quant type and tensor-type recipe.

Frequently Asked Questions about hf-gguf-quant-jobs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a large GGUF model without loading it fully into memory?▼

Use skippy-quantize with a split BF16/FP16 GGUF source repo and a small window size, typically --window-size 1. It stages one shard at a time, quantizes the window, publishes the output shard, and deletes staged files, keeping memory bounded by --max-memory.

How do I run a GGUF quantization job on Hugging Face Jobs?▼

Launch with hf jobs uv run, mounting the source and target model repos as volumes and passing HF_TOKEN as a secret. The job script builds skippy-quantize, prepares the manifest, runs run-quant, verifies the job, and uploads sidecars.

Should I use llama-quantize or skippy-quantize for split GGUF repos?▼

Use skippy-quantize for this workflow; the Skill explicitly forbids llama-quantize, llama-quantise, or wrapper scripts around them. skippy-quantize supports resumable split-window quantization with manifests and preflight validation.

What happens if a quantization job is cancelled or fails midway?▼

Cancel the stalled job before changing code or hardware, then relaunch the same manifest. The next run skips already published shards and resumes at the first missing output shard in the target repo.

How do I verify a quantized GGUF repo after the job completes?▼

Check the target repo via the Hub API for commit SHA, file count, GGUF shard count, first and last shard names, and presence of quant-plan.json and the tensor-type file. Locally, run verify-job with --llama-load and validate-splits against the target.

Related Skills

Explore More Software Engineering Skills→
amdamd
154

quark-torch-file2file-quantization

Quantize large safetensors LLM checkpoints file-to-file with low memory usage.

Official
Advanced
yanochka11yanochka11

huggingface-local-models

Discover and deploy local Hugging Face GGUF models with llama.cpp workflows.

Community
Advanced
castform-aicastform-ai
49

view-progress

Monitor castform training runs and inspect status, reward scalars, and logs.

Official
Intermediate
LynxLabVNLynxLabVN

llama-cpp

Load, quantize, and run llama.cpp models for local inference.

Official
Advanced