skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

llama-cpp

Run C/C++ LLM inference on CPU and non-NVIDIA hardware with GGUF quantization.

228|35|Updated Mar 9, 2026
Search Tags:#openai-compatible#apple-silicon#llama.cpp#edge-deployment#cpu-inference#gguf-quantization#non-cuda
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha --skill llama-cpp-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/kaminocorp/hermes-alpha/tree/main/hermes-agent/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/kaminocorp/hermes-alpha --skill llama-cpp-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill provides fast and portable LLM inference on CPU-only and non-NVIDIA hardware by leveraging llama.cpp and GGUF quantization to reduce memory usage and increase throughput on edge devices and consumer hardware.

Core Features & Use Cases

  • Pure C/C++ inference engine optimized for CPU and integrated GPUs, including Apple Silicon.
  • GGUF quantization support enabling 4- to 8-bit models to run on limited memory.
  • OpenAI-compatible API via llama-server and CLI tools for local deployment and testing.
  • Use cases include running small-to-mid sized LLMs on desktops, laptops, Raspberry Pi-like devices, or servers without CUDA.

Quick Start

Install a GGUF-quantized model and run the provided CLI to start CPU-based LLM inference locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on CPU-only hardware without CUDA?▼

To run LLM inference on CPU-only hardware without CUDA, use a pure C/C++ engine with GGUF quantization to enable fast, local model execution on edge devices and consumer hardware.

Can I use an OpenAI-compatible API for local CPU inference on Apple Silicon?▼

Yes, you can expose an OpenAI-compatible API for local CPU inference on Apple Silicon by running the provided server tooling, allowing local applications to connect to the quantized models.

What is GGUF quantization and when do I need it for edge deployment?▼

GGUF quantization reduces memory usage by compressing models to 4- or 8-bit formats, which is needed for edge deployment when running LLMs on limited-memory devices like laptops or embedded systems.

How do I start running a local model with pure C/C++ inference?▼

To start running a local model with pure C/C++ inference, install a GGUF-quantized model and execute the provided command-line interface tool to initiate CPU-based LLM inference locally.

Does CPU-based LLM inference work on embedded systems like Raspberry Pi?▼

Yes, CPU-based LLM inference works on embedded systems like Raspberry Pi by leveraging GGUF quantization and an optimized C/C++ engine to run small-to-mid sized models on limited hardware.

What are the limitations of running non-CUDA LLM inference on edge devices?▼

Limitations of running non-CUDA LLM inference on edge devices include restricted memory capacity requiring 4- to 8-bit quantization, and reduced throughput compared to dedicated GPU hardware when processing large models.

Related Skills

Explore More Software Engineering Skills→
SignmanalSignmanal
1

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

Community
Intermediate
kwcantrellkwcantrell

llama-server

Serve local GGUF LLMs with an OpenAI-compatible HTTP API.

Community
Advanced
huggingfacehuggingface
10.9k

hf-mem

Estimate inference memory for Safetensors and GGUF models from Hugging Face Hub.

Official
Basic
founderphantomfounderphantom

llama-cpp

Perform CPU-based LLM inference with GGUF quantization using llama.cpp.

Community
Intermediate