skills.rest
InverterNetwork
InverterNetworkOFFICIAL
Explore all skills by this author →
Website𝕏@inverternetworkGitHub

llama-cpp

Discovers GGUF models from Hugging Face Hub and configures llama.cpp inference servers.

Updated May 4, 2026
Search Tags:#quantization#inference#gguf#llama.cpp#local llm#hugging face
One-click install
npx skills add https://github.com/InverterNetwork/hermes-agent --skill llama-cpp-inverternetwork
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/InverterNetwork/hermes-agent/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/InverterNetwork/hermes-agent --skill llama-cpp-inverternetwork

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve?

This skill solves the complexity of finding, quantizing, and running local LLMs by providing a streamlined workflow for GGUF model discovery and inference on consumer hardware.

Core Features & Use Cases

  • Model Discovery: Search and identify the optimal GGUF model for your hardware directly from the Hugging Face Hub.
  • Inference Optimization: Configure GPU offloading, batching, and quantization settings to maximize performance on your specific machine.
  • Use Case: If you need to run a private, high-performance chat assistant on your local machine without relying on cloud APIs, this skill helps you select the right model size and quant format to fit your available VRAM.

Quick Start

Use the llama-cpp skill to launch a local server for the model bartowski/Llama-3.2-3B-Instruct-GGUF using the Q8_0 quantization.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF model inference on consumer hardware?▼

Run local GGUF model inference by discovering models on Hugging Face Hub and configuring hardware-optimized deployment across CPU, Apple Silicon, and GPU architectures to fit your available VRAM.

What's the best way to select a quantization format for a local LLM?▼

Select a quantization format by matching the model size and quant settings, such as Q8_0, to your available VRAM, enabling a private, high-performance chat assistant without relying on cloud APIs.

Can I use llama.cpp to create an OpenAI-compatible API endpoint?▼

Yes, you can use llama.cpp to launch a local server and configure it to provide OpenAI-compatible API endpoints for your selected GGUF models.

Does llama-cpp-python support GPU offloading and batching?▼

Yes, llama-cpp-python supports inference optimization by configuring GPU offloading and batching settings to maximize performance across CPU, Apple Silicon, and GPU architectures.

How do I find optimal GGUF models on Hugging Face Hub?▼

Find optimal GGUF models on Hugging Face Hub by searching and identifying the right model size and quant format to fit your available VRAM and specific hardware configuration.

Related Skills

Explore More Software Engineering Skills→
sheawinklersheawinkler
78

llama-cpp

Run local GGUF inference and discover Hugging Face Hub models with llama.cpp.

Community
Advanced
VYRE-StudiosVYRE-Studios

llama-cpp

Run GGUF model inference on CPU and non-NVIDIA hardware.

Official
Advanced
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
ovachieverovachiever
52

llama-cpp

Run LLM inference on CPU and non-NVIDIA hardware with GGUF quantization.

Community
Advanced