skills.rest
markwang2658
markwang2658COMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Performs GGUF inference and discovers Hugging Face Hub models for llama.cpp across CPU and GPU backends.

31|3|Updated May 7, 2026
Search Tags:#gpu#inference#cpu#llama-cpp#model-discovery#gguf#hugging-face
One-click install
npx skills add https://github.com/markwang2658/hermes-windows-native --skill llama-cpp-markwang2658
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/markwang2658/hermes-windows-native/tree/main/hermes-agent/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/markwang2658/hermes-windows-native --skill llama-cpp-markwang2658

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve?

Enables local GGUF inference with llama.cpp and simplifies Hugging Face Hub-based model discovery, removing the need for Docker or WSL2.

Core Features & Use Cases

  • Local GGUF inference on CPU, Apple Silicon, or GPU backends via llama-cpp-python.
  • Quantization guidance and model discovery directly from Hugging Face Hub.
  • HF local-app and tree API workflows enable quick selection of compatible GGUF files.

Quick Start

Run the llama.cpp Quick Start to perform local GGUF inference and model discovery.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF inference without using Docker or WSL2?▼

You can run local GGUF inference directly on Linux, macOS, or Windows using llama-cpp-python, which removes the need for Docker or WSL2 by selecting native CPU or GPU backends.

How do I find compatible GGUF models on Hugging Face Hub?▼

Find compatible GGUF models on Hugging Face Hub using model discovery workflows that query HF local-app and tree APIs, enabling quick selection of files that match your hardware backend.

Can I use Apple Silicon GPU acceleration for local llama.cpp inference?▼

Yes, local llama.cpp inference supports Apple Silicon GPU acceleration alongside standard CPU and GPU backends across Linux, macOS, and Windows environments.

Do I need a specific version of llama-cpp-python to run GGUF models locally?▼

Yes, you need llama-cpp-python version 0.2.0 or higher because the skill enforces this dependency compatibility to ensure proper quantization selection and local server execution.

What is the best way to get quantization guidance for local GGUF models?▼

The best way to get quantization guidance for local GGUF models is using the skill's built-in discovery features, which match Hugging Face GGUF files to your specific CPU or GPU backend.

Related Skills

Explore More Software Engineering Skills→
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
matthew-johnsonmatthew-johnson

llama-cpp

Run local GGUF inference with llama.cpp and discover Hugging Face Hub models.

Community
Intermediate
SriRamkunamsettySriRamkunamsetty

llama-cpp

Run local GGUF inference and discover Hugging Face models for llama.cpp.

Community
Intermediate
harperaaharperaa
30

setup-windows

Automate BastionClaw setup on Windows with WSL2 and Docker/Podman.

Community
Advanced