skills.rest
moltis-org
moltis-orgOFFICIAL
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Run local LLM inference on CPU and GPU with llama.cpp and GGUF models.

2.8k|332|Updated Jan 29, 2026
Search Tags:#quantization#gguf#apple-silicon#llama.cpp#local-inference#server-api#python-bindings
One-click install
npx skills add https://github.com/moltis-org/moltis --skill llama-cpp-moltis-org
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/moltis-org/moltis/tree/main/crates/skills/src/assets/mlops/inference/llama-cpp
Command: npx skills add https://github.com/moltis-org/moltis --skill llama-cpp-moltis-org

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Enables fast, local LLM inference by running llama.cpp on CPU, Apple Silicon, and GPUs without relying on cloud services, while providing robust quantization workflows.

Core Features & Use Cases

  • Deterministic inference on CPU and GPU, including GGUF quantization formats (Q4_K_M, Q5_K_M, Q6_K, Q8_0) and optional IMATRIX calibration.
  • Multiple interfaces: CLI for quick experiments, Python bindings for integration, and an OpenAI-compatible server for local API access.
  • Edge and offline deployments: Suitable for edge devices (Apple Silicon/M-series) and air-gapped environments, with Ollama/LM Studio compatibility.

Quick Start

Install the llama-cpp toolchain, prepare a GGUF model, and start a local server to verify setup.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local LLM inference on Apple Silicon without an internet connection?▼

Local LLM inference on Apple Silicon works offline by running llama.cpp with GGUF quantized models, enabling deterministic execution on edge devices without relying on cloud services or continuous internet access.

What is GGUF quantization and when do I need it for local inference?▼

GGUF quantization compresses large language models into formats like Q4_K_M and Q8_0, reducing memory requirements so they can run efficiently on CPU-only or CUDA-less environments during local inference.

Can I use an OpenAI-compatible API for local inference on a CPU-only server?▼

Yes, you can expose local inference through an OpenAI-compatible server API, allowing applications designed for OpenAI to interact seamlessly with local GGUF models running on CPU-only or CUDA-less servers.

What's the best way to integrate local LLM inference into a Python application?▼

The best way to integrate local inference is by using Python bindings provided by the llama.cpp toolchain, allowing direct model loading and execution within Python scripts for custom workflows.

Does local inference with llama.cpp support IMATRIX calibration for quantized models?▼

Yes, local inference supports optional IMATRIX calibration alongside standard GGUF quantization formats, allowing users to refine model weights for improved accuracy during edge deployments and offline execution.

Related Skills

Explore More Software Engineering Skills→
vikrant-projectvikrant-project
2

llama-cpp

Execute GGUF inference locally with llama.cpp and Python bindings.

Community
Intermediate
jayll1303jayll1303
17

llama-cpp-inference

Run local GGUF model inference and OpenAI-compatible serving with llama.cpp.

Community
Advanced
agentic-inagentic-in
577

llama.cpp

Run local LLM inference with llama.cpp across CPU, Apple Silicon, and GPUs.

Official
Advanced
hochoa13hochoa13

llama-cpp

Run GGUF quantized LLM inference on CPUs, Apple Silicon, and consumer GPUs.

Community
Intermediate