jetson-llm-benchmark

Benchmark Jetson LLM serving performance by executing runtime wrapper scripts against a live service to return structured latency and throughput metrics. It measures TTFT, ITL, TPOT, and end-to-end latency for vLLM, llama.cpp, and Ollama deployments.

Ollama performance checks benchmark a reachable daemon with single-stream requests to capture prompt, generation, and throughput metrics. This validates whether your model deployment is memory-bound or decode-bound on the live Jetson device.

vLLM concurrency sweeps measure TTFT, ITL, TPOT, throughput, and end-to-end latency for an already running OpenAI-compatible vLLM server. The benchmark requires the matching wrapper script, validated model identifiers, and live Jetson device detection.

llama.cpp GGUF benchmarking runs Jetson-appropriate measurements for local models through the NVIDIA-AI-IOT container workflow. You must provide valid GGUF paths and ensure live device detection for SKU, L4T, and container metadata.

Live device detection is required to identify SKU, L4T, and container metadata before benchmarking Jetson LLM serving performance. You also need the matching wrapper scripts and validated model identifiers or GGUF paths to execute the runs.

Benchmark Jetson LLM serving to compare before-and-after tuning changes, validate power-mode choices, and confirm memory or decode bottlenecks. This removes guesswork by producing repeatable latency and throughput metrics in a structured format.