skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run GGUF model inference on CPUs and non-NVIDIA GPUs with llama.cpp.

2|Updated Apr 12, 2026
Search Tags:#quantization#llama-cpp#gguf#model-serving#apple-silicon#edge-deployment#cpu-inference
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill llama-cpp-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/12-inference-serving/llama-cpp
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill llama-cpp-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Enables efficient local LLM inference on CPU-only machines, Apple Silicon, and consumer GPUs without relying on NVIDIA CUDA, allowing models to run on edge devices and non-datacenter hardware.

Core Features & Use Cases

  • CPU and non‑NVIDIA GPU inference: Build and run llama.cpp on macOS (Metal), Linux (ROCm/CUDA where available), and pure-CPU environments.
  • GGUF quantization support: Convert and run GGUF-quantized models (Q2–Q8 variants) to reduce memory and improve speed on limited hardware.
  • Model serving and deployment: Start an OpenAI-compatible server, enable hybrid CPU/GPU offloading, batch processing, and context tuning for chatbots, batch generation, and embedded assistants.
  • Use Case: Deploy a Llama 2 7B Q4_K_M model on an M3 Mac or an AMD GPU to serve low-latency chat completions for a local assistant without needing CUDA-enabled infrastructure.

Quick Start

Start a local OpenAI-compatible llama.cpp server using a GGUF model and serve requests on port 8080.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on CPU or Apple Silicon without NVIDIA CUDA?▼

You can run LLM inference on CPU and Apple Silicon without CUDA by using llama.cpp with platform-specific builds like Metal for Mac, enabling efficient local model serving on edge devices and consumer hardware.

What is the best way to serve GGUF quantized models locally on an M3 Mac?▼

The best way to serve GGUF quantized models on an M3 Mac is using an OpenAI-compatible llama.cpp server, which supports low-latency chat completions by leveraging Metal for hardware acceleration.

Can I convert HuggingFace models to GGUF format for edge deployment on Raspberry Pi?▼

Yes, you can convert HuggingFace models to GGUF format for edge deployment on Raspberry Pi, applying quantization variants like Q4_K_M to reduce memory requirements and improve inference speed.

Does llama.cpp support hybrid CPU and GPU layer offloading for batch processing?▼

Yes, llama.cpp supports hybrid CPU and GPU layer offloading along with context sizing, allowing you to optimize batch processing and chat generation across available hardware resources.

What are the limitations of running local LLMs on non-NVIDIA GPUs?▼

Running local LLMs on non-NVIDIA GPUs requires platform-specific builds like ROCm for AMD or Metal for Apple Silicon, and you may face memory constraints that require applying GGUF quantization to fit available hardware.

Related Skills

Explore More Software Engineering Skills→
ToqsickToqsick

llama-cpp

Run local GGUF inference for llama.cpp models with quantization selection and Hugging Face Hub discovery.

Community
Advanced
cascade-protocolcascade-protocol
14

agentbox-inference

Generate chat completions via an OpenAI-compatible API with USDC micropayments.

Official
Intermediate
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-local-models

Search Hugging Face Hub and launch GGUF models via llama.cpp.

Community
Intermediate
helix4uhelix4u

llama-cpp

Run LLM inference on CPU, Apple Silicon, and non-NVIDIA GPUs with GGUF models.

Community
Intermediate