skills.rest
SriRamkunamsetty
SriRamkunamsettyCOMMUNITY
Explore all skills by this author →
𝕏@SriRam41258GitHub

llama-cpp

Run local GGUF inference and discover Hugging Face models for llama.cpp.

Updated May 20, 2026
Search Tags:#server#model-discovery#gguf#llama.cpp#hugging-face#local-inference
One-click install
npx skills add https://github.com/SriRamkunamsetty/SITA2.0-HermesAgent --skill llama-cpp-sriramkunamsetty
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/SriRamkunamsetty/SITA2.0-HermesAgent/tree/main/hermes-agent/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/SriRamkunamsetty/SITA2.0-HermesAgent --skill llama-cpp-sriramkunamsetty

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

llama.cpp's local GGUF inference and Hugging Face model discovery workflow enables developers to run GGUF models locally while easily locating suitable models from the HF repository.

Core Features & Use Cases

  • Local GGUF inference on CPU or GPU, with guidance for selecting quantizations like Q4_K_M, Q5_K_M, or Q8_0.
  • Hugging Face repository discovery to locate GGUF models compatible with llama.cpp and to compare file sizes and hardware requirements.
  • Build a llama-server or llama-cli command from the Hub data and enumerate available GGUF files and sizes across repositories.
  • Use case: quickly prototype local AI assistants with offline capabilities and reproducible environments.

Quick Start

Install llama.cpp, choose a GGUF model from Hugging Face, and start a local inference server with llama-server.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF inference using llama.cpp?▼

Run local GGUF inference by installing llama.cpp, selecting a GGUF model from Hugging Face, and starting a local inference server with llama-server. It supports both CPU and GPU workflows for offline execution.

How do I find suitable GGUF models on Hugging Face for local inference?▼

Discover suitable GGUF models on Hugging Face by using repository-wide enumeration. This guides you from Hub local-app pages to exact GGUF filenames via the tree API, allowing comparison of file sizes and hardware requirements.

Which quantization should I choose when running a GGUF model locally?▼

Choose quantizations like Q4_K_M, Q5_K_M, or Q8_0 when running a GGUF model locally. Your selection depends on target hardware capabilities and the required balance between model precision and memory footprint.

Does llama.cpp support both CPU and GPU workflows for GGUF models?▼

Yes, llama.cpp supports both CPU and GPU workflows for running GGUF models. You can build llama-server or llama-cli commands from Hugging Face data to execute local inference across different hardware configurations.

Can I build a llama-server command directly from Hugging Face model data?▼

Yes, you can build a llama-server or llama-cli command directly from Hugging Face Hub data. The workflow enumerates available GGUF files and sizes across repositories to construct the exact server commands needed.

Related Skills

Explore More Software Engineering Skills→
zhu637882-stackzhu637882-stack

env-setup

Detect, install, and configure Homebrew, Node.js, pnpm, PostgreSQL, Redis, Git, then output a summary.

Community
Advanced
diskrotrepodiskrotrepo
9

yulan-mini

Generate song lyrics, audio prompts, and chat completions via a llama.cpp server.

Official
Intermediate
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
cryptopaficryptopafi

serving-llms-vllm

Serve large language models with high-throughput inference for OpenAI-compatible endpoints.

Community
Advanced