skills.rest
matthew-johnson
matthew-johnsonCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local GGUF inference with llama.cpp and discover Hugging Face Hub models.

Updated Apr 20, 2026
Search Tags:#server#quantization#inference#local-models#gguf#huggingface-hub#llama.cpp
One-click install
npx skills add https://github.com/matthew-johnson/hermes-agent --skill llama-cpp-matthew-johnson
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/matthew-johnson/hermes-agent/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/matthew-johnson/hermes-agent --skill llama-cpp-matthew-johnson

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Enable local GGUF inference and HF Hub model discovery.

Core Features & Use Cases

  • Local GGUF inference via llama.cpp supporting CPU, Apple Silicon, CUDA, and ROCm.
  • Model discovery on Hugging Face Hub with hardware-compatibility guidance.
  • Simple, deployable server workflows using llama-server or llama-cli for OpenAI-compatible endpoints.

Quick Start

Install llama.cpp, then run the server against a local GGUF file and explore Hugging Face Hub models.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF inference with hardware acceleration?▼

Local GGUF inference runs via llama.cpp, supporting CPU, Apple Silicon, CUDA, and ROCm environments. You execute local server or CLI commands against a downloaded GGUF file to generate text efficiently on your available hardware.

How do I find and download GGUF models from Hugging Face Hub?▼

You discover GGUF models on Hugging Face Hub using guided discovery features that provide hardware-compatibility guidance. This helps you identify and download GGUF files that match your local CPU or GPU environment for inference.

Can I create an OpenAI-compatible endpoint using llama.cpp?▼

Yes, you can deploy an OpenAI-compatible endpoint using the llama-server workflow. This allows you to run a local server that accepts API requests, providing a standardized interface for local GGUF inference.

Does this GGUF inference workflow support both CPU and GPU environments?▼

GGUF inference supports CPU, Apple Silicon, CUDA, and ROCm environments. The workflow validates your hardware setup and guides you to compatible Hugging Face Hub models to ensure smooth local execution.

What is the best way to start a local server for GGUF models?▼

The best way to start a local server is installing llama.cpp and running llama-server against a local GGUF file. This workflow enforces frontmatter validation and repository inventory before executing server or CLI commands.

Related Skills

Explore More Software Engineering Skills→
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-local-models

Search Hugging Face Hub and launch GGUF models via llama.cpp.

Community
Intermediate
EndUser123EndUser123

ai-groq

Streams Llama, Mixtral, Gemma, and DeepSeek inference via Groq's OpenAI-compatible endpoint.

Community
Advanced
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
guccangguccang
3

serving-llms-vllm

Deploy vLLM inference endpoints with OpenAI-compatible APIs and GPU optimization.

Community
Advanced