skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

gguf-quantization

Convert and quantize HuggingFace models to GGUF format for llama.cpp inference.

14|5|Updated Apr 9, 2026
Search Tags:#apple silicon#quantization#gguf#llama.cpp#llm deployment#cpu inference#model conversion
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill gguf-quantization-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gguf-quantization
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/skills/mlops/inference/gguf
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill gguf-quantization-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models on consumer hardware, CPUs, or Apple Silicon requires compressing models into efficient formats, and this Skill guides the full GGUF conversion and quantization workflow for llama.cpp. ## Core Features & Use Cases - Model Conversion: Convert HuggingFace models to GGUF format using convert_hf_to_gguf.py with FP16 or other output types. - Flexible Quantization: Apply K-quant methods from Q2_K to Q8_0, with importance matrix (imatrix) support for better low-bit quality. - Multi-Hardware Inference: Run quantized models on CPU, NVIDIA CUDA, or Apple Silicon Metal via llama-cli, llama-server, or llama-cpp-python. - Use Case: Download a Llama 3.1 8B model from HuggingFace, convert it to GGUF, quantize to Q4_K_M with an importance matrix, and serve it through an OpenAI-compatible local server. ## Quick Start Convert my HuggingFace model to GGUF format and quantize it to Q4_K_M so I can run it locally with llama.cpp.

Frequently Asked Questions about gguf-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a HuggingFace model to GGUF format?▼

Use the convert_hf_to_gguf.py script from the llama.cpp repository: run python convert_hf_to_gguf.py ./model --outfile model-f16.gguf --outtype f16. Then quantize the FP16 output with llama-quantize to your target bit level.

What GGUF quantization level should I use for a 7B model?▼

Q4_K_M is the recommended default, offering about 4.1 GB size with high quality for a 7B model. Use Q5_K_M or Q6_K for higher quality, or Q3_K_M when memory is severely constrained.

Does GGUF work on Apple Silicon Macs?▼

Yes, llama.cpp supports Metal acceleration on Apple Silicon. Build with make GGML_METAL=1 and set n_gpu_layers to 99 to offload all layers to the GPU via Metal.

GGUF vs AWQ vs GPTQ: which quantization should I choose?▼

GGUF targets CPU, Apple Silicon, and flexible local inference with llama.cpp. AWQ and GPTQ are better for maximum accuracy on NVIDIA GPUs with calibration, while TensorRT-LLM suits production NVIDIA deployments.

Why does my quantized model produce poor quality output?▼

Low-bit quantization without calibration degrades quality. Generate an importance matrix with llama-imatrix using diverse calibration text, then pass it to llama-quantize via the --imatrix flag for Q4 and below.

How do I fix CUDA out of memory errors with GGUF models?▼

Reduce the number of offloaded GPU layers with the -ngl flag, switch to a smaller quantization like Q3_K_M, lower the context length, or quantize the KV cache using type_k and type_v settings.

Related Skills

Explore More Software Engineering Skills→
jinzhezenggroupjinzhezenggroup
124

deepmd-finetune-dpa3

Fine-tunes a pre-trained DPA3 model on a downstream dataset using DeePMD-kit.

Official
Advanced
AVOI-CEOAVOI-CEO

llama-cpp

Run local LLM inference with llama.cpp across CPU, Apple Silicon, and GPUs.

Community
Advanced
CxxxxDxxxFCxxxxDxxxF

llmfit-advisor

Analyze local hardware to recommend LLMs with quantization and speed estimates.

Community
Intermediate
zhouboyu-xrealzhouboyu-xreal

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Community
Advanced