llama-cpp

Name: llama-cpp
Availability: InStock
Author: zangjeicy

Discover GGUF models on Hugging Face and run local llama.cpp inference.

Updated Jul 13, 2026

Search Tags:#quantization #inference #gguf #llama.cpp #local llm #hugging face

Ready to run

npx skills add https://github.com/zangjeicy/Hermes --skill llama-cpp-zangjeicy

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/zangjeicy/Hermes/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/zangjeicy/Hermes --skill llama-cpp-zangjeicy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve?

This skill simplifies the complex process of finding, downloading, and running local GGUF models, removing the technical friction of managing hardware compatibility and inference configurations.

Core Features & Use Cases

Model Discovery: Search and identify the best GGUF models on Hugging Face based on your specific hardware constraints.
Inference Management: Launch local servers or CLI tools for chat, embeddings, and speculative decoding without manual setup.
Use Case: If you need to run a 7B parameter model on a laptop with limited VRAM, this skill helps you select the optimal quantization level and provides the exact command to launch it efficiently.

Quick Start

Use the llama-cpp skill to find and run the best quantized version of Llama-3.2-3B for my current hardware.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run local GGUF models on hardware with limited VRAM?▼

You can run local GGUF models on limited VRAM by selecting an optimal quantization level to reduce memory footprint. This skill evaluates hardware constraints and provides exact CLI commands to launch inference efficiently on your device.

What's the best way to find and download quantized models on Hugging Face?▼

The best way to find quantized models on Hugging Face is using model discovery features that search repositories based on hardware constraints. This identifies optimal GGUF models for your specific edge-based execution requirements.

Can I launch a local server for llama.cpp inference without manual setup?▼

Yes, you can launch a local server for llama.cpp inference without manual setup. This skill manages inference configurations automatically, enabling chat, embeddings, and speculative decoding via local servers or CLI tools.

Does llama-cpp-python support grammar-constrained generation and batched inference?▼

Yes, llama-cpp-python supports grammar-constrained generation and batched inference. This skill facilitates these advanced features across CPU and GPU architectures, satisfying complex edge-based LLM execution requirements.

When do I need to adjust quantization levels for local LLM execution?▼

You need to adjust quantization levels when targeting specific hardware constraints or optimizing for limited VRAM. This skill helps select optimal quantization to ensure efficient local GGUF model inference across CPU and GPU architectures.