skills.rest
NITISH-gitbit
NITISH-gitbitCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run GGUF models locally and discover compatible Hugging Face Hub models.

Updated Jun 26, 2026
Search Tags:#GGUF#inference#model discovery#Hugging Face Hub#local deployment
One-click install
npx skills add https://github.com/NITISH-gitbit/hermes-custom --skill llama-cpp-nitish-gitbit
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/NITISH-gitbit/hermes-custom/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/NITISH-gitbit/hermes-custom --skill llama-cpp-nitish-gitbit

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies local GGUF inference with Hugging Face Hub model discovery, allowing for efficient model selection and deployment.

Core Features & Use Cases

  • Local GGUF Inference: Run quantized models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
  • Model Discovery: Discover and select Hugging Face Hub models compatible with llama.cpp.
  • Use Case: If you need to quickly deploy a model for inference and you have specific size constraints, this Skill can help you find and deploy the right model from the Hugging Face Hub.

Quick Start

To get started, install llama.cpp and then use the skill to run a specific model from the Hub. For example, to run the Q8 model from the 'bartowski/Llama-3.2-3B-Instruct-GGUF' repository, use the following command:

llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF inference using models from the Hugging Face Hub?▼

Local GGUF inference is executed by installing llama.cpp and using the llama-server command with the -hf flag to pull and run quantized models directly from the Hugging Face Hub repositories.

Does llama.cpp support local deployment on Apple Silicon and CUDA GPUs?▼

Yes, local deployment supports multiple hardware configurations including CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs, allowing efficient model execution across various hardware setups.

How do I find and select a compatible GGUF model for my specific hardware constraints?▼

Model discovery is facilitated by searching the Hugging Face Hub for GGUF repositories, enabling you to filter and select models that match your specific size and hardware constraints for deployment.

What is the best way to deploy a quantized Llama-3.2-3B model locally?▼

To deploy a quantized model locally, use the llama-server command targeting a specific Hugging Face Hub repository, such as running the Q8_0 variant of the bartowski/Llama-3.2-3B-Instruct-GGUF model.

Do I need llama-cpp-python bindings to execute GGUF models locally?▼

Yes, the llama-cpp-python dependency is required for execution, providing the necessary Python bindings to interface with the underlying llama.cpp framework for running local inference.

What are the limitations when running local GGUF inference with large model sizes?▼

Limitations depend on your hardware capabilities, as running large quantized models requires sufficient memory and processing power across CPU, Apple Silicon, or GPU configurations for efficient execution.

Related Skills

Explore More Software Engineering Skills→
ToqsickToqsick

llama-cpp

Run local GGUF inference for llama.cpp models with quantization selection and Hugging Face Hub discovery.

Community
Advanced
datathingsdatathings
10

llamacpp

Reference the llama.cpp C API for model loading and inference workflows.

Official
Advanced
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
DaddyElonMusk69DaddyElonMusk69

gguf-quantization

Convert models to GGUF format with 2-8 bit quantization for llama.cpp.

Community
Intermediate