jetson-llm-serve

Serving VLMs on Jetson hardware requires selecting a multimodal model container and using vLLM or SGLang launch recipes configured for vision-language inference. You must apply memory-conscious defaults and verify the OpenAI-compatible endpoint through the /v1 API.

Yes, vLLM on Jetson requires different containers based on the hardware generation. Newer Thor and Orin systems with JetPack 7.2 use upstream vLLM, while older Orin releases require the specific NVIDIA-AI-IOT vLLM image for proper runtime compatibility.

A Jetson LLM server fails to start reliably when MAXN power mode is not configured or memory-conscious defaults are missing. Ensuring the proper Jetson generation, model-specific container selection, and host networking on port 8000 provides the necessary operational safeguards for reliable startup.

The best way to expose a local Jetson model to Open WebUI is to launch an OpenAI-compatible serving endpoint using vLLM or SGLang with host networking on port 8000. This allows Open WebUI to connect directly to your local deployment without manual compatibility research.

Yes, you can use SGLang for multimodal vision-language inference on Jetson devices. The deployment generates complete serving recipes for both text and multimodal models, applying the correct runtime and launch settings for your specific Thor or Orin hardware.