run-llms

Guide local LLM setup and operation with the Harbor toolkit.

3.2k|219|Updated Jul 27, 2024

Search Tags:#Docker #LLM #Harbor #model management #AI setup #GPU optimization #local AI services

Ready to run

npx skills add https://github.com/av/harbor --skill run-llms

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: run-llms
Source: https://github.com/av/harbor/tree/main/skills/run-llms
Command: npx skills add https://github.com/av/harbor --skill run-llms

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires docker, docker-compose, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a comprehensive guide for setting up and running local Large Language Models (LLMs) using the Harbor toolkit. It helps users troubleshoot common issues and integrate various local AI services.

Core Features & Use Cases

LLM Setup: Detailed instructions for installing and configuring Harbor, Docker, and other necessary components.
Model Management: Steps to pull and manage models from Ollama, HuggingFace, and llama.cpp registries.
Service Integration: Integration with services like Open WebUI, SearXNG, and Open Terminal for a full AI toolkit experience.
GPU Optimization: Tips for troubleshooting GPU-related issues and optimizing VRAM usage.
Use Case: For a user looking to set up a local AI service to handle chat and code execution, this Skill provides a step-by-step guide to achieve that.

Quick Start

Run the 'run-llms' skill to start the setup process for local LLMs using Harbor.

Frequently Asked Questions about run-llms

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up local LLMs using Harbor and Docker Compose?▼

To set up local LLMs using Harbor, you need to install Docker and Docker Compose first. The run-llms skill then guides you through configuring Harbor and pulling models from Ollama, HuggingFace, and llama.cpp registries to build your local AI service.

What is needed to run local AI services with Harbor?▼

Running local AI services with Harbor requires Docker, Docker Compose, and basic knowledge of local AI environments. You will use these dependencies to deploy integrated services like Open WebUI, SearXNG, and Open Terminal for a complete local AI toolkit.

Can I manage models from HuggingFace and Ollama in one local setup?▼

Yes, you can manage models from HuggingFace and Ollama in one local setup. The Harbor toolkit integrates model management across multiple registries, allowing you to pull and configure models from Ollama, HuggingFace, and llama.cpp within a single environment.

How do I troubleshoot GPU optimization and VRAM usage for local LLMs?▼

To troubleshoot GPU optimization and VRAM usage for local LLMs, the Harbor setup guide provides specific tips for resolving common GPU-related issues. It helps you configure Docker and local AI services to maximize VRAM efficiency during model execution.

Does Harbor work with Open WebUI and SearXNG for local chat services?▼

Yes, Harbor works with Open WebUI and SearXNG for local chat services. The toolkit supports service integration with Open WebUI, SearXNG, and Open Terminal, allowing you to handle chat and code execution through a comprehensive local AI interface.