jetson-inference-mem-tune

You can prevent out-of-memory errors during LLM inference by selecting the right runtime and tuning memory flags like GPU utilization and KV-cache settings, which this skill recommends based on a live Jetson memory audit.

The best way to fit a large language model on a Jetson Orin Nano 8GB is to use low-memory planning techniques, such as switching to a lower-footprint runtime like llama.cpp and applying specific quantization notes and launch flags.

Choosing between vLLM, SGLang, and llama.cpp for Jetson serving depends on your specific Jetson SKU and workload, and this skill evaluates your memory audit to recommend the most suitable runtime with exact launch configurations.

No, this Jetson memory tuning process does not require starting or restarting services; it only returns runtime-specific launch flags, alternatives, and safety checks for you to apply manually to your live device.

To tune KV-cache and context length settings for VLM workloads on Jetson, you provide a fresh memory audit JSON, and the skill returns concrete, runtime-specific launch flags tailored to your available device memory.