skills.rest
KarlinskyS
KarlinskySCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run transformer language models on CPU and non-NVIDIA GPUs with GGUF quantization.

Updated Apr 10, 2026
Search Tags:#quantization#inference#gguf#apple-silicon#llama.cpp#edge-deployment#gpu-offloading
One-click install
npx skills add https://github.com/KarlinskyS/hermesSkills --skill llama-cpp-karlinskys
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/KarlinskyS/hermesSkills/tree/main/mlops/inference/llama-cpp
Command: npx skills add https://github.com/KarlinskyS/hermesSkills --skill llama-cpp-karlinskys

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Allows running transformer language models without NVIDIA CUDA by providing optimized CPU and non‑NVIDIA GPU inference paths, enabling local and edge deployments where CUDA is unavailable or impractical.

Core Features & Use Cases

  • CPU-first inference optimized for Apple Silicon and x86 CPUs for local development and edge devices.
  • Non‑NVIDIA GPU support and offloading via Metal and ROCm for AMD and Apple GPUs, plus hybrid CPU/GPU layering.
  • GGUF quantization workflows to reduce memory and improve throughput for models from 7B to 70B parameters.
  • Server mode and OpenAI-compatible API for exposing local models as a service for chatbots, batch processing, or integration testing.
  • Use Case: Deploy a quantized Llama 2 7B model on an M2/M3 Mac or a Raspberry Pi to serve chat completions locally without CUDA.

Quick Start

Run llama.cpp with a GGUF model to start a local OpenAI-compatible server on port 8080 using a quantized model.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on Apple Silicon or AMD GPUs without CUDA?▼

You can run LLM inference on Apple Silicon or AMD GPUs without CUDA by using optimized CPU paths alongside Metal and ROCm builds, enabling local execution on non-NVIDIA hardware. It supports hybrid CPU/GPU layering to maximize throughput.

What is GGUF quantization and when do I need it for local LLM deployment?▼

GGUF quantization compresses transformer models to 2–8 bit precision to reduce memory usage and improve inference throughput. You need it for local LLM deployment on edge devices or desktops with constrained memory capacity.

Can I expose a local LLM as an OpenAI-compatible API for chatbot integration?▼

You can expose a local LLM as an OpenAI-compatible API by running the server mode on a specified port. This creates a local endpoint for chatbot integration, batch processing, or integration testing using quantized models.

Does llama.cpp work on edge devices like a Raspberry Pi for local inference?▼

llama.cpp works on edge devices like a Raspberry Pi by utilizing its CPU-first inference optimization. By applying GGUF quantization, you can reduce the memory footprint enough to run smaller parameter models locally.

What is the best way to offload model layers to a GPU during CPU inference?▼

The best way to offload model layers to a non-NVIDIA GPU during CPU inference is using the hybrid CPU/GPU layering feature. This allows you to balance memory constraints and throughput by building with Metal or ROCm.

What are the limitations of running 70B parameter models locally without CUDA?▼

Running 70B parameter models locally without CUDA requires aggressive 2–8 bit GGUF quantization to fit within memory limits. Throughput will be constrained by CPU and non-NVIDIA GPU bandwidth, making hybrid offloading necessary.

Related Skills

Explore More Software Engineering Skills→
SignmanalSignmanal
1

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

Community
Intermediate
cabeznocabezno

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM.

Community
Advanced
anhvthanhvth
9

llm-integration

Cache OpenAI chat completion calls with memoized client utilities.

Community
Intermediate
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced