skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Runs GGUF-quantized LLM inference on CPUs, Apple Silicon, and non-NVIDIA GPUs.

14|5|Updated Apr 9, 2026
Search Tags:#apple silicon#quantization#edge deployment#gguf#llama.cpp#cpu inference#llm inference
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill llama-cpp-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill llama-cpp-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally without NVIDIA GPUs or datacenter infrastructure is difficult; this Skill enables LLM inference on CPUs, Apple Silicon, AMD/Intel GPUs, and edge devices using llama.cpp with GGUF quantization. ## Core Features & Use Cases - Cross-hardware inference: Run models on CPU-only machines, Apple Silicon via Metal, AMD GPUs via ROCm, and NVIDIA GPUs via CUDA with layer offloading. - GGUF quantization: Convert HuggingFace models to GGUF and quantize from 1.5 to 8 bits, reducing memory footprint and achieving 4-10x speedup over PyTorch on CPU. - OpenAI-compatible server: Deploy llama-server with chat completions, streaming, continuous batching, and metrics endpoints. - Use Case: Deploy a Llama 2-7B chatbot on an M3 MacBook by downloading a Q4_K_M GGUF model and serving it through a local OpenAI-compatible API at 50 tokens per second. ## Quick Start Download a Q4_K_M GGUF model from HuggingFace and run llama-cli or llama-server with the model path to start local inference.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run Llama models on a CPU without a GPU?▼

Use llama.cpp with a GGUF-quantized model such as Q4_K_M, which runs a 7B model in about 7 GB of RAM. Build from source or install via brew, then run llama-cli with the model path and thread count set to your physical core count.

What GGUF quantization format should I use?▼

Q4_K_M is the recommended default, offering about 1.7% perplexity increase over FP16 while cutting a 7B model to 4.1 GB. Use Q6_K or Q8_0 for maximum quality, and Q3_K_M or lower only for very large models or memory-constrained edge devices.

llama.cpp vs vLLM vs TensorRT-LLM: which should I choose?▼

Choose llama.cpp for CPU, Apple Silicon, AMD/Intel GPUs, or edge deployment without CUDA. Choose vLLM or TensorRT-LLM when you have NVIDIA GPUs and need maximum datacenter throughput with features like PagedAttention.

Does llama.cpp support Apple Silicon GPU acceleration?▼

Yes, build with LLAMA_METAL=1 to enable Metal acceleration on M1/M2/M3/M4 chips. Offload layers with -ngl 999; an M3 Max achieves roughly 50 tokens per second on Llama 2-7B Q4_K_M.

Why is my llama.cpp inference slow or running out of memory?▼

Slow inference often results from hyperthreading or high-bit quantization; set threads to physical cores and use Q4_K_M. Out-of-memory errors are fixed by lowering quantization, reducing context size with -c, or offloading fewer GPU layers with -ngl.

How do I convert a HuggingFace model to GGUF format?▼

Download the HuggingFace model, run convert_hf_to_gguf.py to produce an FP16 GGUF file, then run llama-quantize with your target format such as Q4_K_M. Alternatively, download pre-quantized GGUF models directly from HuggingFace.

Related Skills

Explore More Software Engineering Skills→
lenadlmlenadlm

gguf-quantization

Convert HuggingFace models to GGUF format and quantize them for CPU inference.

Community
Advanced
diskd-aidiskd-ai

cerebras-api

Integrate Cerebras Chat Completions API into Python and TypeScript applications.

Official
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced