jetson-speculative-decoding

Configure speculative decoding for Jetson vLLM servers with EAGLE-3 heads or draft models.

110|19|Updated May 25, 2026

Search Tags:#benchmarking #speculative decoding #vllm #jetson #draft model #eagle-3

Ready to run

npx skills add https://github.com/NVIDIA-AI-IOT/jetson-device-skills --skill jetson-speculative-decoding

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: jetson-speculative-decoding
Source: https://github.com/NVIDIA-AI-IOT/jetson-device-skills/tree/main/skills/jetson-speculative-decoding
Command: npx skills add https://github.com/NVIDIA-AI-IOT/jetson-device-skills --skill jetson-speculative-decoding

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you reduce slow token generation on Jetson vLLM servers when time-per-output-token is the bottleneck, not initial response latency.

Core Features & Use Cases

EAGLE-3 Speculative Decoding: Enable a published EAGLE-3 head for supported Jetson deployments to improve decode speed.
Draft-Model Fallback: Use a small same-family draft model when no EAGLE-3 head is available.
Jetson-Specific Guardrails: Applies the right runtime path, memory headroom adjustments, and device-specific limits for Thor and AGX Orin systems.
Benchmark Validation: Verifies that speculation actually helps by comparing before-and-after benchmark results at low concurrency.

Quick Start

Ask the assistant to update your existing Jetson vLLM launch command with the correct speculative decoding configuration, then benchmark the server before and after the change to confirm it improves performance.

Frequently Asked Questions about jetson-speculative-decoding

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I speed up token generation latency on a Jetson vLLM server?▼

Speculative decoding accelerates Jetson vLLM token generation by using an EAGLE-3 head or a small same-family draft model to verify multiple tokens per step. This reduces time-per-output-token latency specifically when generation speed, rather than initial prompt processing, is the bottleneck.

Can I use speculative decoding with vLLM on Jetson AGX Orin and Thor devices?▼

Yes, speculative decoding is supported on both Jetson AGX Orin and Thor deployments. The configuration applies Jetson-specific guardrails, including the correct runtime path, memory headroom adjustments, and device-specific limits tailored for these hardware platforms.

How do I configure vLLM speculative decoding if an EAGLE-3 head is not available?▼

When an EAGLE-3 head is unavailable, you can configure speculative decoding using a small same-family draft model as a fallback. This draft model proposes tokens for the main model to verify, accelerating single-stream or low-concurrency serving.

What is the best way to verify speculative decoding improves vLLM performance on Jetson?▼

Benchmark validation is required to verify performance gains, comparing before-and-after results at low concurrency. You must benchmark the vLLM server before applying the speculative decoding configuration and again after to confirm it improves performance.

Does speculative decoding help with high-concurrency vLLM serving on Jetson?▼

Speculative decoding is targeted for single-stream or low-concurrency serving scenarios on Jetson devices. It is designed to reduce token generation latency when individual response time is the primary bottleneck rather than overall system throughput.