rkllama

Manage RKLLama model lifecycle on RK3588 NPUs with node pinning and explicit unload/load workflows.

32|5|Updated Oct 6, 2024

Search Tags:#llm #kubernetes #model-management #npu #rkllama #rk3588

Ready to run

npx skills add https://github.com/gilesknap/tpi-k3s-ansible --skill rkllama

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: rkllama
Source: https://github.com/gilesknap/tpi-k3s-ansible/tree/main/.claude/skills/rkllama
Command: npx skills add https://github.com/gilesknap/tpi-k3s-ansible --skill rkllama

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

RKLLama provides reliable operations for RKLLama on RK3588 NPUs with node pinning, clear model lifecycle management, and visibility into silent-failure modes of rkllama_client pull.

Core Features & Use Cases

In-pod CLI access via rkllama_client with a known path and host-side wrappers for convenience.
Clear model lifecycle: pinning to node04, explicit unloads before loading new models, and rules for one model per NPU.
Guidance on hardware envelope, quant settings, and model conversions for deployment and maintenance.

Quick Start

Inspect the current RKLLama deployment and unload any active model to prepare for a new load.

Frequently Asked Questions about rkllama

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I manage LLM model lifecycle on RK3588 NPU in Kubernetes?▼

You can manage the LLM lifecycle on an RK3588 NPU by using explicit unload and load workflows, ensuring a strict one-model-per-NPU constraint to switch models safely within your Kubernetes deployment.

How do I pin a Kubernetes node for RKLLama NPU workloads?▼

Pinning a Kubernetes node for RKLLama ensures reliable operations by dedicating specific hardware, such as node04, to run the RKLLama workload and manage its per-node model state.

Can I use rkllama_client inside a pod to pull and load models?▼

Yes, you can use rkllama_client via known in-pod command paths to pull and load models, utilizing host-side wrappers and NFS-backed model storage directories for reliable access.

Why does rkllama_client pull silently fail when loading a new model?▼

Silent failures during a pull or load often occur when the one-model-per-NPU constraint is violated; explicitly unloading the active model before loading a new one resolves this issue.

What are the constraints for deploying RK3588 NPU models with NFS storage?▼

Deploying RK3588 NPU models with NFS storage requires adhering to model directory conventions, applying strict one-model-per-NPU limits, and configuring appropriate quant settings for the hardware envelope.