jetson-speculative-decoding

Speculative decoding accelerates Jetson vLLM token generation by using an EAGLE-3 head or a small same-family draft model to verify multiple tokens per step. This reduces time-per-output-token latency specifically when generation speed, rather than initial prompt processing, is the bottleneck.

Yes, speculative decoding is supported on both Jetson AGX Orin and Thor deployments. The configuration applies Jetson-specific guardrails, including the correct runtime path, memory headroom adjustments, and device-specific limits tailored for these hardware platforms.

When an EAGLE-3 head is unavailable, you can configure speculative decoding using a small same-family draft model as a fallback. This draft model proposes tokens for the main model to verify, accelerating single-stream or low-concurrency serving.

Benchmark validation is required to verify performance gains, comparing before-and-after results at low concurrency. You must benchmark the vLLM server before applying the speculative decoding configuration and again after to confirm it improves performance.

Speculative decoding is targeted for single-stream or low-concurrency serving scenarios on Jetson devices. It is designed to reduce token generation latency when individual response time is the primary bottleneck rather than overall system throughput.