speculative-decoding

Accelerate LLM inference using draft models, Medusa heads, and lookahead decoding.

13.0k|930|Updated Nov 3, 2025

Search Tags:#latency reduction #speculative decoding #vllm #medusa #llm inference #lookahead decoding #draft models

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill speculative-decoding-orchestra-research

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: speculative-decoding
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/19-emerging-techniques/speculative-decoding
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill speculative-decoding-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, and includes references (resource) components.

What problem does it solve? Autoregressive LLM decoding generates tokens sequentially, making inference slow and expensive for real-time applications like chatbots and code generation. This Skill provides implementation guidance for speculative decoding techniques that achieve 1.5-3.6× speedup without quality loss. ## Core Features & Use Cases - Draft Model Speculative Decoding: Use a small draft model to generate candidate tokens that a large target model verifies in parallel via transformers' assisted generation. - Medusa Multiple Decoding Heads: Add trainable prediction heads to an existing LLM to predict multiple future tokens with tree-based attention verification. - Lookahead Decoding: Apply Jacobi iteration to generate and verify n-grams in parallel without any draft model or training. - Use Case: Deploy Llama-2-70b with Llama-2-7b as a draft model through vLLM to double serving throughput for a production chatbot while maintaining identical output quality. ## Quick Start Use the speculative-decoding skill to set up assisted generation with a draft model for my Llama-2-70b deployment to reduce inference latency.

Frequently Asked Questions about speculative-decoding

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I speed up LLM inference with speculative decoding?▼

Load a small draft model alongside your target model and pass it as assistant_model to the generate method in transformers 4.36+. The draft model proposes tokens that the target model verifies in parallel, giving 1.5-2× speedup with zero quality loss.

Medusa vs speculative decoding: which should I use?▼

Medusa achieves 2-3.6× speedup by adding prediction heads to a single model but requires minimal training. Draft model speculative decoding needs no training and works if you already have a smaller version of your model, but delivers lower speedup.

Does speculative decoding reduce output quality?▼

No, speculative decoding is mathematically equivalent to standard target model decoding. Tokens are only accepted when they match the target model's probability distribution, so outputs remain identical in quality.

How do I choose a draft model for speculative decoding?▼

Select a draft model 5-10× smaller than the target model, such as Llama-2-7b as draft for Llama-2-70b. If your target model is too small for a useful draft, use Medusa or Lookahead Decoding instead.

Can I use speculative decoding with vLLM in production?▼

Yes, vLLM supports speculative decoding through the speculative_model and num_speculative_tokens parameters when initializing the LLM class. This enables batched speculative generation for high-throughput serving.

What are the limitations of lookahead decoding?▼

Lookahead decoding achieves lower speedup (1.5-2.3×) than Medusa and has verification overhead that grows with window size. Larger models require smaller window and n-gram parameters because verification becomes more expensive.