skills.rest
ldzhhxx
ldzhhxxCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run Hugging Face Hub GGUF models locally via llama.cpp on CPU or GPU.

1|1|Updated May 9, 2026
Search Tags:#model selection#gguf#llama.cpp#local inference#hugging face#ai inference
One-click install
npx skills add https://github.com/ldzhhxx/Hermes_offline_v2 --skill llama-cpp-ldzhhxx
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/ldzhhxx/Hermes_offline_v2/tree/main/hermes-agent/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/ldzhhxx/Hermes_offline_v2 --skill llama-cpp-ldzhhxx

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, llama-cpp, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the need for fast and flexible local inference, providing tools to process models from Hugging Face Hub on CPU or GPU with a variety of quantizations and options for both development and deployment.

Core Features & Use Cases

  • Local Inference: Utilizes the llama.cpp framework to run Hugging Face Hub models directly on your device with customizable quantizations and hardware options.
  • Model Discovery: Navigate and extract the correct inference model for your application based on availableGGUF files, with specific emphasis on Hugging Face Hub models that are llama.cpp-compatible.
  • Quantization: Supports Q4/K-M to Q8/0 quantization, enabling both fast and compact model usage while preserving accuracy.
  • Deployment Options: From direct use on local terminals or within web applications using provided servers to custom configurations that optimize memory usage.
  • Use Case: When building an AI-based application that requires fast inference, this Skill allows you to use the latest models with optimal performance for your specific hardware setup.

Quick Start

Install llama.cpp:

brew install llama.cpp  # macOS/Linux

Run a model directly from the Hub:

llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run Hugging Face models locally using llama.cpp?▼

You can run Hugging Face models locally using llama.cpp by downloading compatible GGUF files and executing them via the llama-server command, specifying the repository and quantization level for direct CPU or GPU inference on your device.

What GGUF quantization levels are supported for local inference?▼

Local inference supports quantization levels ranging from Q4/K-M to Q8/0, allowing you to balance model size and processing speed while preserving accuracy for optimized deployment across different hardware configurations.

Can I use llama.cpp for AI inference on both CPU and GPU hardware?▼

Yes, llama.cpp provides high-performance local inference optimized for both CPU and GPU hardware, enabling flexible deployment of Hugging Face Hub models with customizable options for memory usage and processing speed.

Do I need Python libraries to process GGUF files with llama.cpp?▼

Yes, processing GGUF files requires the llama.cpp framework and Python libraries like llama-cpp-python to handle model loading, quantization, and inference execution within your local development or deployment environment.

What is the best way to find llama.cpp-compatible models on Hugging Face Hub?▼

To find compatible models, navigate Hugging Face Hub repositories specifically offering GGUF files, which are formatted for llama.cpp processing and support direct deployment through the llama-server command line interface.

How do I deploy a local AI model as a web application using llama.cpp?▼

You can deploy local AI models as web applications by utilizing the provided server components within llama.cpp, which allow direct integration into web interfaces or terminal applications with optimized memory configurations.

Related Skills

Explore More Software Engineering Skills→
lxt070409lxt070409
1

llama-cpp

Run local inference with llama.cpp models and discover GGUF models from the Hugging Face Hub.

Community
Advanced
guccangguccang
3

serving-llms-vllm

Deploy vLLM inference endpoints with OpenAI-compatible APIs and GPU optimization.

Community
Advanced
eliottbusinesseliottbusiness

llama-cpp

Run llama.cpp GGUF model inference locally with Hugging Face Hub discovery.

Community
Advanced
hochoa13hochoa13

llama-cpp

Run GGUF quantized LLM inference on CPUs, Apple Silicon, and consumer GPUs.

Community
Intermediate