skills.rest
aayushsoam
aayushsoamCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run llama.cpp GGUF model inference locally across CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs.

1|1|Updated May 25, 2026
Search Tags:#GGUF#quantization#Hugging Face Hub#llama.cpp#model inference#local model running
One-click install
npx skills add https://github.com/aayushsoam/clawbot-agent --skill llama-cpp-aayushsoam
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/aayushsoam/clawbot-agent/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/aayushsoam/clawbot-agent --skill llama-cpp-aayushsoam

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill allows for local GGUF inference and discovery of Hugging Face Hub models for llama.cpp, enabling users to run local models on various hardware and find the right model for their needs.

Core Features & Use Cases

  • Local GGUF Inference: Run llama.cpp models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
  • Model Discovery: Find the right GGUF for a specific Hugging Face repo, build a llama-server or llama-cli command, and search the Hub for compatible models.
  • Use Case: Suppose you have a model you want to run locally but need to find the correct GGUF and quantization for it. Use this Skill to discover the right model and quantization level, then run it locally with the provided commands.

Quick Start

Use the llama-cpp skill to find the GGUF for the model 'bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0' and run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF inference using a Hugging Face model?▼

To run local GGUF inference, search the Hugging Face Hub for compatible models, select the correct quantization, and execute the model locally using llama-cli or llama-server commands.

How do I find the right GGUF quantization for a specific model on Hugging Face Hub?▼

Finding the right GGUF quantization involves searching the Hugging Face Hub for the specific model repository and evaluating available quantization levels to match your local hardware capabilities.

Can I run llama.cpp models on Apple Silicon and CUDA GPUs?▼

Yes, you can run llama.cpp models locally across multiple hardware targets including Apple Silicon, CUDA, ROCm, Intel GPUs, and standard CPU-based systems for flexible inference.

How do I build a llama-server command for local model inference?▼

Build a llama-server command by selecting a GGUF model from Hugging Face Hub, determining the appropriate quantization, and configuring the execution parameters to run inference locally.

What is the best way to discover compatible GGUF models for local execution?▼

The best way to discover compatible GGUF models is to search the Hugging Face Hub directly, filtering for available quantization options that suit your specific hardware configuration.

Related Skills

Explore More Software Engineering Skills→
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-local-models

Search Hugging Face Hub and launch GGUF models via llama.cpp.

Community
Intermediate
JuanMS20JuanMS20

llama-cpp

Run local GGUF inference and discover models from Hugging Face Hub.

Community
Advanced
webdevtodayjasonwebdevtodayjason

llama-cpp

Automate local GGUF inference and model discovery for llama.cpp.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced