llama-cpp

Execute local GGUF inference for llama.cpp models across multiple hardware platforms.

Updated May 11, 2026

Search Tags:#GGUF #Hugging Face Hub #llama.cpp #local execution #model inference

Ready to run

npx skills add https://github.com/richardnguyen0715/keep-it-real --skill llama-cpp-richardnguyen0715

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/richardnguyen0715/keep-it-real/tree/main/refer-projects/hermes-agent/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/richardnguyen0715/keep-it-real --skill llama-cpp-richardnguyen0715

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill allows for local inference with llama.cpp, providing efficient and versatile access to quantized models for a variety of hardware and use cases.

Core Features & Use Cases

Local GGUF Inference: Run llama.cpp models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
Model Discovery: Find and select the appropriate Hugging Face Hub models for llama.cpp.
Command-Line Tools: Access llama-server and llama-cli for model inference and management.
Use Case: Suppose you are developing a text generation tool and need to quickly find and test a llama.cpp model from the Hugging Face Hub for your application.

Quick Start

Run the llama.cpp model from the Hugging Face Hub with the command: llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run local GGUF inference using llama.cpp models from Hugging Face Hub?▼

Run local GGUF inference by using the llama-server command with a Hugging Face Hub model repository and quantization tag, such as: llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

Can I execute llama.cpp inference on Apple Silicon and CUDA hardware?▼

Yes, local GGUF inference supports multiple hardware configurations including CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs, allowing efficient execution of quantized models across diverse hardware setups.

What is the best way to discover and select Hugging Face Hub models for llama.cpp?▼

Model discovery is handled through a comprehensive interface that helps find and select appropriate Hugging Face Hub models for llama.cpp, suitable for AI research, development, and production environments.

Do I need llama-cpp-python installed to run local GGUF models?▼

Yes, executing local GGUF inference requires the llama-cpp-python bindings version 0.2.0 or higher, along with the actual GGUF model files to perform text generation and inference tasks.

Are there limitations when running quantized models locally with llama.cpp?▼

Local GGUF inference requires specific hardware configurations and GGUF formatted model files. You must ensure your system matches supported platforms like CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs for optimal quantized model performance.