skills.rest
XyHalcyon
XyHalcyonCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local inference with llama.cpp GGUF models and discover models from Hugging Face Hub.

Updated Sep 28, 2021
Search Tags:#GGUF#model discovery#Hugging Face Hub#llama.cpp#local inference
One-click install
npx skills add https://github.com/XyHalcyon/config-files --skill llama-cpp-xyhalcyon
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/XyHalcyon/config-files/tree/main/hermes/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/XyHalcyon/config-files --skill llama-cpp-xyhalcyon

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies local inference with llama.cpp models and facilitates discovery of models from the Hugging Face Hub.

Core Features & Use Cases

  • Local Inference: Perform inference on llama.cpp models locally on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
  • Model Discovery: Find and download models from the Hugging Face Hub that support llama.cpp.
  • Use Case: If you need a lightweight model for an embedded device with limited resources, this Skill can help you find a Q3_K_M quantized model that fits your needs.

Quick Start

Use the llama-server command with the Hugging Face Hub model you've selected. For example:

llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local inference with GGUF models on my hardware?▼

Local GGUF inference supports multiple hardware configurations including CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs. You can execute the `llama-server` command directly with a Hugging Face Hub model to run inference locally without external dependencies.

What is the best way to find a quantized GGUF model for a device with limited resources?▼

Hugging Face Hub model discovery allows you to find specific quantized GGUF models, such as Q3_K_M, for devices with limited resources. Search the Hub for lightweight models that match your hardware constraints and download them directly.

Can I use llama.cpp with Hugging Face Hub models directly?▼

Yes, you can use llama.cpp with Hugging Face Hub models directly via the `llama-server` command. Specify the model repository and quantization tag, like `bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0`, to automatically download and run inference.

Do I need to manually download GGUF files before starting local inference?▼

Manual GGUF file downloads are not required because model discovery integrates directly with the Hugging Face Hub. The local inference process requires GGUF files, which the `llama-server` command can fetch automatically from a specified Hub repository.

What are the limitations of running local inference with llama.cpp?▼

Local inference with llama.cpp requires GGUF file formats and depends on the `llama-cpp-python` dependency. While it supports diverse hardware like Apple Silicon and ROCm, performance is constrained by the chosen quantization level and available device memory.

Related Skills

Explore More Software Engineering Skills→
PiaoxuemoliPiaoxuemoli

llama-cpp

Run local GGUF inference with llama.cpp models from Hugging Face Hub.

Community
Advanced
BermudaLocalsBermudaLocals
1

llama-cpp

Execute local GGUF inference and discover Hugging Face models with llama.cpp.

Community
Intermediate
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
ToqsickToqsick

llama-cpp

Run local GGUF inference for llama.cpp models with quantization selection and Hugging Face Hub discovery.

Community
Advanced