jetson-llm-benchmark

What problem does it solve?

It removes the guesswork from measuring Jetson-hosted model serving by producing repeatable latency and throughput benchmarks in a structured format that is easy to compare across runs.

Core Features & Use Cases

• vLLM concurrency sweeps: Measure TTFT, ITL, TPOT, throughput, and end-to-end latency for an already running OpenAI-compatible vLLM server.
• llama.cpp GGUF benchmarking: Run Jetson-appropriate llama.cpp benchmarks for local GGUF models through the NVIDIA-AI-IOT container workflow.
• Ollama performance checks: Benchmark a reachable Ollama daemon with single-stream requests and capture prompt, generation, and throughput metrics.
• Use Case: Compare before-and-after tuning changes, validate power-mode choices, and confirm whether a model deployment is memory-bound or decode-bound on a live Jetson device.

Quick Start

Ask the agent to benchmark the running Jetson LLM service with the matching runtime wrapper, using the served model ID or GGUF path and then report the resulting JSON metrics.