skills.rest
kwcantrell
kwcantrellCOMMUNITY
Explore all skills by this author →
GitHub

llama-server

Serve local GGUF LLMs with an OpenAI-compatible HTTP API.

Updated Jul 5, 2026
Search Tags:#HTTP API#OpenAI-compatible#model serving#llama-server#local LLM
One-click install
npx skills add https://github.com/kwcantrell/rusty-agent --skill llama-server
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-server
Source: https://github.com/kwcantrell/rusty-agent/tree/main/.agents/skills/llama-server
Command: npx skills add https://github.com/kwcantrell/rusty-agent --skill llama-server

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama.cpp, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a comprehensive solution for serving local Large Language Models (LLMs) using llama-server, enabling users to create their own OpenAI-compatible endpoint without relying on external services.

Core Features & Use Cases

  • Local LLM Serving: Serve a local GGUF LLM over HTTP using llama.cpp's llama-server.
  • HTTP API: Access and interact with the OpenAI-compatible REST API for local LLM inference.
  • Structured Output: Generate structured output using GBNF grammars, JSON schemas, and tool/function calls.
  • Use Case: For instance, you can set up a local LLM server to process and respond to user queries in real-time, without the need for an external API key.

Quick Start

Run the llama-server with the following command to serve a local model:

llama-server -m model.gguf -c 4096 -ngl 99

Frequently Asked Questions about llama-server

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve a local LLM with an OpenAI-compatible API?▼

Serve local LLMs with an OpenAI-compatible API by running llama-server with a GGUF model file. This provides an HTTP REST API endpoint for LLM inference without requiring external API keys or services.

Can I generate structured output like JSON from a local model?▼

Yes, you can generate structured output from a local model using GBNF grammars, JSON schemas, and tool/function calls provided by the llama-server HTTP API during inference.

How do I start llama-server for local LLM inference?▼

Start local LLM inference by running the command `llama-server -m model.gguf -c 4096 -ngl 99` in your terminal, which loads the specified GGUF model and initializes the server.

Does llama-server support tool and function calls?▼

Yes, llama-server supports tool and function calls. It provides OpenAI-compatible HTTP API endpoints that allow your local LLM to execute structured automated inference workflows.

What do I need to set up a local LLM server without an external API?▼

To set up a local LLM server without an external API, you need the llama.cpp dependency installed and your target model files in GGUF format to run the server.

What are the limitations of running local LLMs with llama-server?▼

Running local LLMs with llama-server is limited by your hardware capacity, requiring sufficient GPU resources to load GGUF model files and process HTTP API inference requests efficiently.

Related Skills

Explore More Software Engineering Skills→
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-local-models

Search Hugging Face Hub and launch GGUF models via llama.cpp.

Community
Intermediate
copilot-ldcopilot-ld
1

libllm

Expose an HTTP LLM API client for OpenAI-compatible chat completions and embeddings.

Official
Advanced
zengbaochengzengbaocheng

llama-cpp

Run local GGUF inference with llama.cpp using llama-server or llama-cli.

Community
Advanced
pmcdowallpmcdowall

serving-llms-vllm

Configure vLLM deployments with OpenAI-compatible APIs and performance tuning.

Community
Advanced