speculative-decoding

Reduce LLM inference latency by combining speculative decoding, Medusa decoding heads, and Lookahead Decoding to verify candidate tokens in a single forward pass. Apply tree-structured verification to accelerate text generation.

Medusa decoding heads speed up transformer-based models by adding multiple parallel heads to predict several future tokens simultaneously. This eliminates the need for a separate draft model during speculative decoding.

The best way to accelerate latency-sensitive deployments like chatbots and code assistants is using speculative decoding with plug-and-play tree-based verification. This approach supports transformer-based models and optional draft components.

Yes, Lookahead Decoding works with transformer-based models by applying Jacobi-iteration-style parallelization. It generates and verifies n-grams efficiently without requiring separate draft heads or additional model components.

Speculative decoding can use a smaller draft model to generate candidate tokens verified in a single forward pass, but Medusa heads provide an alternative by predicting multiple future tokens in parallel without a draft model.

Use speculative decoding when a smaller draft model is available, or switch to Medusa heads to predict multiple future tokens in parallel without separate draft models. Lookahead Decoding offers Jacobi-iteration-style n-gram verification as another alternative.