skills.rest
Tommy-yw
Tommy-ywCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local inference and discover llama.cpp models from Hugging Face Hub.

539|39|Updated May 1, 2026
Search Tags:#quantization#inference#Hugging Face Hub#llama.cpp#model conversion
One-click install
npx skills add https://github.com/Tommy-yw/RunbookHermes --skill llama-cpp-tommy-yw
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Tommy-yw/RunbookHermes/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/Tommy-yw/RunbookHermes --skill llama-cpp-tommy-yw

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill allows for efficient local inference of llama.cpp models, with support for quantization, Hugging Face Hub integration, and custom builds.

Core Features & Use Cases

  • Local Inference: Perform inference directly on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
  • Quantization: Convert models for reduced memory footprint and improved performance.
  • Hugging Face Hub Discovery: Find and download llama.cpp compatible models directly from the Hugging Face Hub.
  • Custom Model Conversion: Convert Hugging Face models to GGUF format for use with llama.cpp.
  • Use Case: Imagine you want to perform fast text generation with a llama.cpp model without internet access. This Skill enables you to download the model and use it locally, converting it to a quantized version if needed.

Quick Start

Install llama.cpp, run the command to use the model from the Hugging Face Hub: llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local inference on a llama.cpp model without an internet connection?▼

Local inference allows you to download a model from Hugging Face Hub and run it directly on your hardware. You can execute text generation offline using CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs after the initial download.

How do I convert a Hugging Face model to GGUF format for llama.cpp?▼

Model conversion transforms Hugging Face models into the GGUF format required by llama.cpp. This custom conversion process ensures compatibility so you can execute local inference using the llama.cpp framework.

Does llama.cpp support quantization to reduce memory footprint?▼

Quantization is supported to convert models for a reduced memory footprint and improved performance. This optimization allows efficient local inference across various hardware platforms without requiring excessive system resources.

Can I discover and download GGUF models directly from the Hugging Face Hub?▼

Hugging Face Hub discovery enables you to find and download llama.cpp compatible models directly. You can quickly access repositories like bartowski/Llama-3.2-3B-Instruct-GGUF and specify quantization levels such as Q8_0 for immediate use.

What dependencies do I need to run llama.cpp inference locally?▼

Executing local inference requires installing llama.cpp and the llama-cpp-python dependency. These components provide the necessary environment to perform model discovery, quantization, and text generation on your chosen hardware platform.

Related Skills

Explore More Software Engineering Skills→
kngender5kngender5

llama-cpp

Discover Hugging Face Hub models and build llama.cpp inference command lines.

Community
Advanced
GitnappGitnapp

llama-cpp

Discover GGUF models on Hugging Face and launch llama.cpp inference commands.

Community
Advanced
eliottbusinesseliottbusiness

llama-cpp

Run llama.cpp GGUF model inference locally with Hugging Face Hub discovery.

Community
Advanced
jonnabiojonnabio
6

awq-quantization

Quantize large language models to 4-bit using activation-aware weight quantization.

Community
Advanced