jetson-llm-serve

What problem does it solve?

This Skill removes the guesswork from serving large language models and vision-language models on Jetson devices by choosing the right runtime, launch settings, and verification steps for the target hardware.

Core Features & Use Cases

• Jetson-aware runtime selection: Chooses upstream vLLM for newer Thor and Orin JetPack 7.2 systems, or the NVIDIA-AI-IOT vLLM image for older Orin releases.
• LLM and VLM serving recipes: Generates complete deployment guidance for text models and multimodal models using vLLM or SGLang.
• Operational safeguards: Includes MAXN power setup, memory-conscious defaults, OpenAI-compatible endpoints, and basic pre-flight checks so the server starts reliably.
• Use case: A developer wants to expose a model from a Jetson board to Open WebUI or an application without manually researching JetPack compatibility, container tags, or launch flags.

Quick Start

Ask the assistant to serve your chosen model on the Jetson device and provide the correct container, launch command, endpoint, and verification step.