The Real Speed of Speculative Decoding: Why LLMs Don't Accelerate as Promised in Papers

Scientific publications on speculative decoding methods (such as EAGLE-3) frequently report inference acceleration of up to 6.5x without any loss in generation quality. However, independent testing across real-world scenarios shows that results can range from a comfortable 2.6x on code generation to an actual slowdown (0.95x) on queries that are atypical for the model. The real-world gain is determined not so much by the algorithm itself, but by hardware architectural limitations and the parameters of the draft tree.

Why Speculative Decoding Works

Text generation in large language models (LLMs) at a batch size of one (batch = 1) is limited by memory bandwidth rather than chip computing power. At each step, the model is forced to read all of its weights from memory just to produce a single token.

Experimental measurements confirm this behavior:

• A forward pass of the target model for 1 token takes about 44.7 ms.
• A pass for a batch of 192 tokens requires 51.8 ms.
• At 512 tokens, the time increases to 125.6 ms.

Since validating a tree of several dozen draft tokens takes almost the same amount of time as generating a single token, hypothesis validation is practically free. If even a few positions from the generated draft are accepted, the overall generation time decreases.

Benchmark Results Across Different Task Types

The efficiency of the method is measured by the acceptance length (τ) — the average number of tokens approved by the target model in a single validation cycle. Testing a combination of the Qwen3 model and a specialized draft head on an Nvidia T4 GPU showed a strong dependence of acceleration on the nature of the task:

The more structured the text (code, formulas), the easier it is for the draft model to anticipate the next steps of the target network. On open-ended dialogues and atypical data, the accuracy of draft branches drops, and the acceptance length shrinks.

The Draft Tree Depth Trap

Constructing draft variants is not completely cost-free. Unlike tree width (which is verified in parallel in a single pass), each level of depth requires an additional sequential run of the draft model.

The data domain determines the maximum potential gain, while excessive tree depth determines the risk of running into a performance penalty.

If the draft model generates a tree that is too deep (for example, 7–10 steps deep) with low prediction accuracy, the total time spent building rejected chains begins to exceed the savings from verification. Setting up a shallow tree (depth 2) eliminates performance degradation on complex data, boosting performance to 1.04–1.10x even with a modest acceptance length.

Why Model Scaling Doesn't Deliver the Expected Leap

It is widely believed that speculative decoding works more efficiently on larger target models due to the relatively low computational weight of the draft head. However, experiments transitioning from 1.7B to 4B parameters show that the draft head also scales up in size (from 137M to 218M parameters due to the increased hidden dimension). Consequently, the absolute time of a draft step increases, keeping the overall acceleration at practically the same level.

Checklist for Evaluating LLM Acceleration Claims

When analyzing inference performance benchmarks, key factors should be taken into account:

1. Baseline for comparison: measuring against an unoptimized loop often masks framework overhead under the guise of pure algorithmic gains.
2. Batch size: the maximum speculative effect is observed only at batch = 1. With larger batches, the GPU compute units are already saturated, and gains drop sharply.
3. Data specifics: third-party speedup metrics cannot be automatically transferred to tasks with a different lexical structure.
4. Draft configuration: the shape and depth of the tree must be calibrated for the specific GPU and target domain.