gguf-quantization

Convert and quantize HuggingFace models to GGUF format for llama.cpp inference.

2|Updated Jan 10, 2026

Search Tags:#apple silicon #quantization #model compression #gguf #llama.cpp #cpu inference #llama-cpp-python

Ready to run

npx skills add https://github.com/Shubh2310-developer/ENGUNITYCORE --skill gguf-quantization-shubh2310-developer

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: gguf-quantization
Source: https://github.com/Shubh2310-developer/ENGUNITYCORE/tree/main/.claude/skills/optimization-gguf
Command: npx skills add https://github.com/Shubh2310-developer/ENGUNITYCORE --skill gguf-quantization-shubh2310-developer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models on consumer hardware, CPUs, or Apple Silicon requires compressing models into an efficient format, and doing this manually involves complex conversion scripts, quantization trade-offs, and hardware-specific configuration. ## Core Features & Use Cases - Model Conversion: Convert HuggingFace models to GGUF format using llama.cpp's convert_hf_to_gguf.py script with FP16 or other output types. - Flexible Quantization: Apply K-quant methods from Q2_K to Q8_0, with importance matrix (imatrix) support for better low-bit quality. - Multi-Hardware Inference: Run quantized models on CPU, NVIDIA CUDA, or Apple Silicon Metal, via CLI, Python bindings, or an OpenAI-compatible server. - Use Case: You want to run Llama-3.1-8B on a MacBook. Convert it to GGUF, quantize to Q4_K_M with an imatrix, and serve it locally through llama-server or Ollama. ## Quick Start Convert my HuggingFace model to GGUF format and quantize it to Q4_K_M so I can run it locally on my laptop.

Frequently Asked Questions about gguf-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I convert a HuggingFace model to GGUF format?▼

Use the convert_hf_to_gguf.py script from llama.cpp: run python convert_hf_to_gguf.py ./model --outfile model-f16.gguf --outtype f16. Then quantize the FP16 output with llama-quantize to your target bit level such as Q4_K_M.

What GGUF quantization level should I use for a 7B model?▼

Q4_K_M is the recommended default, offering about 4.1 GB size with high quality for a 7B model. Use Q5_K_M or Q6_K for higher quality, or Q3_K_M when memory is constrained. Always use an importance matrix for Q4 and below.

Does GGUF work on Apple Silicon Macs?▼

Yes, GGUF supports Apple Silicon through Metal acceleration. Build llama.cpp with make GGML_METAL=1 and set n_gpu_layers to 99 in llama-cpp-python to offload all layers to the GPU via Metal.

GGUF vs AWQ vs GPTQ: which quantization should I choose?▼

GGUF targets CPU, Apple Silicon, and flexible local inference with 2-8 bit K-quants. AWQ and GPTQ are better for maximum accuracy on NVIDIA GPUs with calibration. Choose GGUF for consumer hardware and tools like Ollama or LM Studio.

Why does my quantized GGUF model produce poor quality output?▼

Low-bit quantization without an importance matrix often degrades quality. Generate an imatrix with llama-imatrix using diverse calibration text, then quantize with the --imatrix flag, or move to a higher precision type like Q5_K_M.

How do I fix CUDA out of memory errors when running GGUF models?▼

Reduce n_gpu_layers to offload fewer layers to the GPU, use a smaller quantization like Q3_K_M, lower the context length, or quantize the KV cache with type_k and type_v settings in llama-cpp-python.