llama-server

What problem does it solve?

This Skill provides a comprehensive solution for serving local Large Language Models (LLMs) using llama-server, enabling users to create their own OpenAI-compatible endpoint without relying on external services.

Core Features & Use Cases

• Local LLM Serving: Serve a local GGUF LLM over HTTP using llama.cpp's llama-server.
• HTTP API: Access and interact with the OpenAI-compatible REST API for local LLM inference.
• Structured Output: Generate structured output using GBNF grammars, JSON schemas, and tool/function calls.
• Use Case: For instance, you can set up a local LLM server to process and respond to user queries in real-time, without the need for an external API key.

Quick Start

Run the llama-server with the following command to serve a local model:

llama-server -m model.gguf -c 4096 -ngl 99