llama-cpp

Run local LLM inference on CPU and GPU with GGUF quantization.

Updated Apr 19, 2026

Search Tags:#server #quantization #inference #cpu #gguf #llama.cpp #edge-deployment

Ready to run

npx skills add https://github.com/gqf2008/hermez-ai --skill llama-cpp-gqf2008

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/gqf2008/hermez-ai/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/gqf2008/hermez-ai --skill llama-cpp-gqf2008

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Running large language models locally can be resource-intensive and platform-specific. llama-cpp provides a portable, efficient path to perform LLM inference on CPU, Apple Silicon, or paired GPU setups with GGUF-quantized weights, plus an OpenAI-compatible server and optional bindings.

Core Features & Use Cases

Local inference across CPU and GPU hardware with broad platform support (CPU, Apple Silicon, NVIDIA/AMD GPUs).
GGUF quantization and model conversion workflows to enable 2–8 bit precision for edge deployment.
OpenAI-compatible server and ecosystem integrations (Ollama, LM Studio, text-generation-webui) for flexible serving options.
Python bindings and a range of deployment scenarios from edge devices to small servers.

Quick Start

Install llama.cpp and run a sample inference on your local machine.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run LLM inference locally on CPU without CUDA?▼

Local LLM inference without CUDA uses llama.cpp to run models directly on CPU or Apple Silicon hardware. It provides a portable, efficient path for executing large language models on desktop machines or edge deployments where GPU acceleration is unavailable.

What is GGUF quantization and when do I need it for edge deployment?▼

GGUF quantization compresses model weights to 2–8 bit precision, significantly reducing memory requirements. It is needed for edge deployment to fit large language models onto resource-constrained local hardware while maintaining acceptable inference speeds.

How do I convert and serve a quantized model with an OpenAI-compatible API?▼

Convert and serve quantized models by transforming weights to the GGUF format, then launching the built-in OpenAI-compatible server. This workflow provides a local API endpoint, allowing ecosystem integrations like Ollama or LM Studio to interact with the model.

Does llama.cpp work with Apple Silicon and AMD GPUs for local inference?▼

llama.cpp supports local inference across Apple Silicon and AMD GPUs, alongside standard CPU and NVIDIA setups. This broad platform compatibility allows flexible hybrid configurations for running large language models without being locked into a single hardware vendor.

Can I use Python bindings to run llama.cpp inference in my application?▼

Python bindings are available to integrate llama.cpp inference directly into custom applications. This allows developers to execute local LLM inference and serve model predictions programmatically within their existing Python-based software workflows.

What are the limitations of running LLM inference purely on CPU?▼

Running LLM inference purely on CPU limits processing speed compared to GPU-accelerated setups, especially for larger models. However, using GGUF quantization mitigates this by lowering memory bandwidth requirements and enabling functional local execution on machines lacking dedicated GPUs.