skills.rest
gztcode
gztcodeCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run large language model inference across CPU, Apple Silicon, and GPU platforms.

Updated May 8, 2026
Search Tags:#multi-platform#quantization#LLM inference#edge deployment#model conversion
One-click install
npx skills add https://github.com/gztcode/my-skills --skill llama-cpp-gztcode
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/gztcode/my-skills/tree/main/llama-cpp
Command: npx skills add https://github.com/gztcode/my-skills --skill llama-cpp-gztcode

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of running large language model inference on a variety of hardware platforms, including CPU, Apple Silicon, AMD/Intel GPUs, and NVIDIA GPUs. It provides flexible quantization options and model conversion capabilities, making it suitable for edge deployment, non-NVIDIA environments, and flexible local quantization.

Core Features & Use Cases

  • Multi-Platform Inference: Supports inference on CPU, Apple Silicon, AMD/Intel GPUs, and NVIDIA GPUs.
  • Flexible Quantization: Offers quantization options from 2 to 8 bits, with K-quants and imatrix for improved quality.
  • Model Conversion: Converts HuggingFace models to the GGUF format for use with llama.cpp.
  • Use Case: Ideal for developers who need to deploy LLMs on edge devices, M1/M2/M3/M4 Macs, or CUDA-less environments, and require flexible quantization options.

Quick Start

Run the llama-cpp skill to perform inference on the Llama-2-7B model using the CPU with the following command:

./llama-cli -m model.Q4_K_M.gguf -p "Explain quantum computing" -n 256

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on edge devices or non-NVIDIA hardware?▼

You can run LLM inference on edge devices or non-NVIDIA hardware by using flexible quantization and model conversion to deploy models across CPU, Apple Silicon, and AMD/Intel GPUs without requiring CUDA.

How do I convert HuggingFace models to GGUF format for local deployment?▼

Converting HuggingFace models to GGUF format allows them to be executed by llama.cpp, enabling flexible local quantization and multi-platform inference across various hardware environments.

What quantization options are available for reducing LLM memory footprint?▼

Available quantization options range from 2 to 8 bits, including K-quants and imatrix techniques, which help reduce memory footprint while maintaining improved inference quality on edge devices.

Does llama.cpp support LLM inference on Apple Silicon Macs?▼

Yes, llama.cpp supports LLM inference on Apple Silicon Macs, alongside CPU, AMD/Intel GPUs, and NVIDIA GPUs, making it highly suitable for edge deployment and CUDA-less environments.

What are the limitations of using llama.cpp for multi-platform LLM inference?▼

Limitations include requiring llama.cpp and Python bindings for execution, and while it supports multi-platform inference, performance and quality depend on the selected 2 to 8-bit quantization configurations.

Related Skills

Explore More Software Engineering Skills→
InternScienceInternScience
52

pka_predictor

Predict pKa values for small molecules from SMILES inputs.

Official
Advanced
ihatesea69ihatesea69
3

llama-cpp

Run GGUF quantized LLM inference on CPU, Apple Silicon, and consumer GPUs.

Community
Advanced
richardnguyen0715richardnguyen0715

llama-cpp

Execute local GGUF inference for llama.cpp models across multiple hardware platforms.

Community
Advanced
tangzheng202202tangzheng202202
1

local-llm-deployment-assessment

Assess local hardware and generate optimized LLM deployment configurations.

Community
Advanced