[Bugfix] Enable speculative decoding for models with nearly-identical vocab sizes

[benchislett]

Models such as Qwen 2.5 can have identical tokenizers but slightly different vocab sizes due to padding. For Qwen 2.5, the tokenizer is identical but the embedding is padded to a multiple of 128 for the smaller models and a multiple of 256 for the larger models, resulting in a mismatch. This disables speculative decoding, which requires the tokens and respective probabilities match between both models.

This PR addresses #5203, #10913, and #12323

To resolve this I check for a small positive delta between the size of the target and draft model (<= 128). This should maintain correctness with many tokenizer mismatches, but allow the Qwen models to be used in speculative decoding with each other. If there is a better way to check for similar tokenizers, I am open to suggestions.

During inference, the probability output of the draft model is simply padded to the embedding size of the target model.

I have measured the performance gained by using speculative decoding for Qwen 2.5 Coder 7B with a draft model of Qwen 2.5 Coder 0.5B on a consumer RTX 4090:

Batch Size	No Spec TPOT	NGram (k=4) TPOT	Spec (k=4) TPOT
1	15.73ms	12.67ms	9.15ms
16	18.48ms	19.41ms	14.69ms

Run with the following commands respectively:

vllm serve "Qwen/Qwen2.5-Coder-7B-Instruct" --disable-log-requests --max-model-len 8192 --max-num-seqs 128 --tensor-parallel-size 1 --gpu-memory-utilization 0.8 --num-scheduler-steps 8

vllm serve "Qwen/Qwen2.5-Coder-7B-Instruct" --disable-log-requests --max-model-len 8192 --max-num-seqs 128 --tensor-parallel-size 1 --gpu-memory-utilization 0.8 --num-speculative-tokens 4 --speculative-model "[ngram]" --ngram-prompt-lookup-min 1 --ngram-prompt-lookup-max 4

vllm serve "Qwen/Qwen2.5-Coder-7B-Instruct" --disable-log-requests --max-model-len 8192 --max-num-seqs 128 --tensor-parallel-size 1 --gpu-memory-utilization 0.8 --num-speculative-tokens 4 --speculative-model "Qwen/Qwen2.5-Coder-0.5B-Instruct"

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ShangmingCai]

Does this change affect the draft acceptance rate? I know it is not for Qwen, but I am not sure about other models. If it is not, then it would be a great general fix.

[benchislett]

This change will not affect any existing behaviour. It simply modifies the exclusion condition for speculative compatibility checking, and only for models which were not previously compatible there is a small padding added.

[DefinitlyEvil]

Any updates on this? This one could be a helpful PR. This might fix #13759 .

[benchislett]

Awaiting review. To my knowledge the PR is ready to go and only enables new compatibility without affecting existing workloads in any way.

[DefinitlyEvil]

Awaiting review. To my knowledge the PR is ready to go and only enables new compatibility without affecting existing workloads in any way.

Cool, thanks so much for your effort! This could help me a lot! May I install your fork for now somehow?

[benchislett]

I have just updated the fork and branch with the latest commits from vLLM. The remote branch can be found here and should (at the time of writing) be functionally identical to the latest vLLM main, plus this fix.

[DefinitlyEvil]

I have just updated the fork and branch with the latest commits from vLLM. The remote branch can be found here and should (at the time of writing) be functionally identical to the latest vLLM main, plus this fix.

Cool! I gonna check this out!