speculative_decoding

Reduce LLM inference latency by using speculative decoding to verify tokens from a smaller draft model in parallel, achieving 1.5-3.6× speedup without compromising output quality. This avoids quality loss by verifying draft predictions against the target model.

Yes, speculative decoding works with vLLM and Hugging Face transformers because the Skill depends on both frameworks alongside PyTorch and Accelerate. This integration allows you to accelerate models like Llama-2-70b using a Llama-2-7b draft model directly.

The best way to speed up LLM generation on constrained hardware is deploying Medusa's multiple decoding heads or Lookahead Decoding, which reformulates decoding as an equation-solving problem using Jacobi iteration for parallel n-gram verification.

Lookahead Decoding differs from draft model inference acceleration by eliminating the need for a separate model, instead using Jacobi iteration to solve decoding equations and verify n-grams in parallel. Draft models require training a smaller model to predict tokens.

No, you do not need to train Medusa heads to use speculative decoding, as the Skill supports multiple independent techniques. You can choose standard draft model verification, Medusa with its multiple decoding heads, or Lookahead Decoding based on your deployment needs.

Speculative decoding requires a smaller draft model to efficiently predict future tokens, which the larger target LLM then verifies in parallel. This asynchronous verification process drastically reduces generation latency and increases overall throughput.