skills.rest
atomicrajat
atomicrajatCOMMUNITY
Explore all skills by this author →
GitHub

jetson-inference-mem-tune

Recommends inference runtimes and memory launch flags for LLM/VLM serving on NVIDIA Jetson devices.

2|Updated Aug 20, 2026
Search Tags:#vllm#edge deployment#sglang#llama.cpp#llm inference#jetson#memory tuning
One-click install
npx skills add https://github.com/atomicrajat/industry_safety_monitoring_system --skill jetson-inference-mem-tune-atomicrajat
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: jetson-inference-mem-tune
Source: https://github.com/atomicrajat/industry_safety_monitoring_system/tree/main/.claude/skills/jetson-inference-mem-tune
Command: npx skills add https://github.com/atomicrajat/industry_safety_monitoring_system --skill jetson-inference-mem-tune-atomicrajat

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Deploying LLM or VLM servers on memory-constrained NVIDIA Jetson devices often leads to OOM failures and guesswork around runtime selection and memory flags. This Skill turns a live memory audit snapshot into concrete runtime and launch-flag recommendations for vLLM, SGLang, llama.cpp, and TensorRT Edge-LLM. ## Core Features & Use Cases - Runtime Selection: Picks the lightest suitable serving stack (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) based on Jetson SKU, memory headroom, and workload type (llm-server, vlm-server, embedding, rag). - Memory Flag Generation: Emits exact launch flags such as --gpu-memory-utilization, --max-model-len, --mem-fraction-static, -ngl, and -c, scaled to SKU defaults or a user-specified free-memory target. - Quantization and Install Guidance: Provides per-runtime quantization recommendations (W4A16, AWQ/GPTQ, NVFP4, GGUF Q4_K_M) and JetPack-matched container paths for Orin and Thor. - Use Case: A developer's vLLM server keeps OOMing on an AGX Orin while serving a 7B model. The Skill reads the audit JSON and returns concrete --gpu-memory-utilization and --max-model-len values plus lower-memory alternatives like llama.cpp with GGUF. ## Quick Start Ask the agent to run scripts/recommend.py against a fresh jetson-memory-audit JSON with your runtime and workload, for example: recommend vLLM memory flags for an llm-server workload on my Jetson using the latest audit snapshot.

Frequently Asked Questions about jetson-inference-mem-tune

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix vLLM OOM errors on a Jetson AGX Orin?▼

Run scripts/recommend.py with --runtime vllm --workload llm-server against a fresh jetson-memory-audit JSON. It returns concrete --gpu-memory-utilization and --max-model-len values scaled to your device, plus a --target-mb option to reserve free memory.

Which serving stack should I use for a 7B model on Orin Nano 8 GB?▼

On memory-tight Orin Nano-class devices, the recommendation script typically selects llama.cpp with GGUF INT4/Q4_K_M quantization for the lowest memory floor. Run it with --runtime auto --workload llm-server to get the selected runtime and flags like -ngl, -c, and --no-mmap.

vLLM vs llama.cpp vs SGLang for Jetson inference?▼

vLLM suits high-throughput serving with continuous batching, SGLang fits programmable RAG and tool-use workflows, and llama.cpp offers the tightest memory footprint via GGUF. The Skill picks among them based on SKU, memory headroom, and workload type.

Does this skill support Jetson Thor and older Orin JetPack versions?▼

Yes. For Thor it recommends upstream vLLM 0.20+ or NVIDIA SGLang 26.01 containers, and for Orin it distinguishes JetPack 7.2 / L4T r39+ (upstream vLLM) from older releases (NVIDIA-AI-IOT prebuilt images). It never applies Orin-specific container paths to Thor.

Can the skill start or restart my model server with the recommended flags?▼

No. The Skill is read-only and only emits launch flags as JSON; the user or an outer orchestration agent is responsible for launching or restarting the server. It also never invents fallback flags when the script returns empty launch_flags.

What are the limitations of the memory recommendations?▼

Recommendations are only as fresh as the audit JSON, so re-run jetson-memory-audit after stopping services or changing power modes. The script estimates pressure from SKU defaults and audit totals, so model-specific KV-cache and quantization behavior may still require benchmarking.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
3.2k

jetson-memory-audit

Measure Jetson DRAM and NvMap usage and verify memory reclamation with before/after audit snapshots.

Official
Intermediate
NVIDIA-AI-IOTNVIDIA-AI-IOT
110

jetson-llm-benchmark

Benchmark Jetson LLM serving and return structured latency and throughput metrics.

Official
Advanced
NVIDIANVIDIA
3.4k

quant-recipe-search

Design and manage iterative optimization searches for quantization recipes.

Official
Advanced
NVIDIANVIDIA
3.2k

jetson-inference-mem-tune

Recommends inference runtimes and memory launch flags for LLM workloads on NVIDIA Jetson.

Official
Intermediate