vllm-speculative-decoding

What problem does it solve?

Production-grade operators need a reliable way to deploy and optimize vLLM speculative decoding across multiple model families, balancing latency and throughput while maintaining correctness.

Core Features & Use Cases

• Supports multiple speculative methods (eagle3, dflash, mtp, ngram, ngram_gpu, suffix, draft_model, extract_hidden_states) to cover a wide range of production workloads and target families.
• Provides guidance on method selection, parameter tuning (e.g., num_speculative_tokens, parallel_drafting), and metrics interpretation to help diagnose domain/tokenizer mismatches, batch-size effects, and deployment gating.
• Includes operational guidance for validation, monitoring, and troubleshooting to sustain production reliability and performance.

Quick Start

Smoke-test a vLLM endpoint to verify spec-dec is active and report the current acceptance rate and drafts/accepted token counts.