jetson-inference-mem-tune

What problem does it solve?

This Skill helps you choose the right Jetson serving runtime and memory-related launch flags so LLM and VLM workloads fit available device memory and avoid out-of-memory failures.

Core Features & Use Cases

• Runtime Selection: Chooses between vLLM, SGLang, llama.cpp, and TensorRT Edge-LLM based on the Jetson SKU and workload.
• Memory Flag Tuning: Recommends concrete launch flags such as GPU utilization, context length, max sequences, and KV-cache-related settings.
• Low-Memory Planning: Helps with tight-memory scenarios like Orin Nano 8 GB, model downsizing, and switching to lower-footprint runtimes.
• Use Case: A developer can feed it a fresh Jetson memory audit and get a safe, runtime-specific launch recipe for serving, RAG, or embedding workloads.

Quick Start

Ask the skill to read a current Jetson memory audit and return the best inference runtime plus exact launch flags for your model-serving workload.