speculative-decoding

Speculative decoding accelerates autoregressive LLM inference by using a smaller draft model to propose tokens and a larger target model to verify them in parallel, reducing latency without sacrificing quality.

Yes, speculative decoding integrates natively with PyTorch and Transformers, supporting standard generation APIs to load target models and optional draft heads or lookahead modules for accelerated text generation.

Medusa integration adds multiple decoding heads to the target model to predict future tokens, enabling faster parallel generation and accelerating chatbot or code completion workloads without changing the output distribution.

Lookahead Decoding reformulates autoregressive decoding to generate n-grams in parallel and verify them against the target model, enabling plug-and-play speedups for latency-sensitive deployments across varying hardware compute budgets.

The best approach is initializing a small draft model to propose candidate tokens while the main target model validates them in parallel, satisfying integration with standard Transformers generation APIs for real-time chat deployments.

You can use a separate draft model, but it is optional; the Skill also supports Medusa heads or Lookahead modules to achieve parallel generation and inference speedups without a standalone draft model.