skills.rest
thisismynewfmail-ui
thisismynewfmail-uiCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run GGUF model inference on CPU and non-NVIDIA GPU platforms.

Updated Apr 12, 2026
Search Tags:#quantization#inference#gguf#model-serving#apple-silicon#llama.cpp#edge-deployment
One-click install
npx skills add https://github.com/thisismynewfmail-ui/Monika-agent --skill llama-cpp-thisismynewfmail-ui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/thisismynewfmail-ui/Monika-agent/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/thisismynewfmail-ui/Monika-agent --skill llama-cpp-thisismynewfmail-ui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Enables reliable LLM inference on machines without NVIDIA CUDA, including CPU-only servers, Apple Silicon Macs, AMD/Intel GPUs, and low-power edge devices by using GGUF models and lightweight C/C++ runtimes.

Core Features & Use Cases

  • Platform portability: Build and run llama.cpp on macOS (Metal), Linux (CPU), ROCm (AMD), and CUDA-enabled systems where available.
  • Quantization & memory efficiency: Support for GGUF quantization formats (Q2–Q8) to reduce model size and RAM requirements for edge deployment.
  • Serving and tooling: OpenAI-compatible server mode, Docker deployment recipes, monitoring endpoints, and guidance for layer offloading and batching.
  • Use cases: Interactive chat on an M1/M2 laptop, local API serving on a Raspberry Pi or VPS, batch prompt processing for offline pipelines.

Quick Start

Start an OpenAI-compatible server by running the llama-server binary with your GGUF model on port 8080 to serve chat completions.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on CPU-only servers or Apple Silicon Macs?▼

Run LLM inference on CPU-only servers or Apple Silicon Macs by using GGUF models with lightweight C/C++ runtimes. This approach enables reliable execution on machines without NVIDIA CUDA, utilizing Metal for macOS or native CPU processing.

What is the best way to deploy an OpenAI-compatible server for local LLMs?▼

Deploy an OpenAI-compatible server for local LLMs by running the llama-server binary with your GGUF model. It provides chat completions on a specified port, supported by Docker deployment recipes and monitoring endpoints.

Can I use GGUF quantization to reduce model size for edge deployment?▼

Yes, you can use GGUF quantization to reduce model size for edge deployment. It supports formats ranging from Q2 to Q8, significantly lowering RAM requirements on low-power edge devices and standard hardware.

Does this inference approach work with AMD or Intel GPUs?▼

Yes, this inference approach works with AMD and Intel GPUs. It supports platform portability across ROCm for AMD GPUs and native CPU processing for Intel, alongside Metal for Apple Silicon and CUDA where available.

How do I handle layer offloading and context sizing during batch prompt processing?▼

Handle layer offloading and context sizing during batch prompt processing by applying the runtime configuration guidance. This optimizes memory usage and processing efficiency for offline pipelines and interactive chat environments.

Related Skills

Explore More Software Engineering Skills→
maystudiosmaystudios
15

llama-cpp

Build, configure, and deploy llama.cpp for local LLM inference.

Community
Advanced
Syh1906Syh1906
25

openai-compatible-imagegen

Automates image generation, editing, and batch creation via OpenAI-compatible API with Python.

Community
Advanced
pmcdowallpmcdowall

serving-llms-vllm

Configure vLLM deployments with OpenAI-compatible APIs and performance tuning.

Community
Advanced
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced