[V1] APC + prompt logprobs unsupported (PR 2/N for v1 sample and prompt logprobs support) #11910

afeldman-nm · 2025-01-10T00:46:02Z

This PR forces the v1 LLMEngine and AsyncLLM to fail in generate() if a request with prompt logprobs enabled arrives and the engine has prefix caching enabled. Note that while it is possible for prompt logprobs and APC to be compatible, this is left to future work.

This is part of a broader vllm v1 + logprobs workstream:

PR no. 1 (#9880 ) is a dependency of this PR; it adds the infrastructure for sample and prompt logprobs support with limited unit tests

PR no. 2 is this PR

PR no. 3 will port test_completion.py logprobs tests to v1, make any adjustments required to make these tests pass, and fix a compatibility issue with Mistral tokenizer

Signed-off-by: mzusman <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…#10719) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…nAI compatible server (vllm-project#10635) Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: youkaichao <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…roject#10682) Signed-off-by: Ubuntu <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…t#10701) Signed-off-by: spacewander <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…ject#10724) Signed-off-by: rickyx <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Richard Liu <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…project#10561) Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…-project#10742) Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…ject#10733) Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: NickLucche <[email protected]> Signed-off-by: Cyrus Leung <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…-project#10771) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: youkaichao <[email protected]> Co-authored-by: youkaichao <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…0769) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…ject#10763) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…0805) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

…oject#10804) Signed-off-by: Roger Wang <[email protected]> Co-authored-by: Chen Zhang <[email protected]> Co-authored-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Andrew Feldman <[email protected]>

…t#10502) This PR provides initial support for single-node disaggregated prefill in 1P1D scenario. Signed-off-by: KuntaiDu <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: YaoJiayi <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

Signed-off-by: Andrew Feldman <[email protected]>

github-actions · 2025-01-10T00:46:13Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

Signed-off-by: Andrew Feldman <[email protected]>

mzusman and others added 30 commits December 2, 2024 11:36

[Bugfix][Mamba] Fix Multistep on Mamba-like models (vllm-project#10705)

4e53851

Signed-off-by: mzusman <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Bugfix] Ignore lm_head when loading embedding models (vllm-project…

8239c6f

…#10719) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Frontend] don't block event loop in tokenization (preprocess) in Ope…

5a3a0eb

…nAI compatible server (vllm-project#10635) Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[misc] upgrade filelock version (vllm-project#10731)

b22e27c

Signed-off-by: youkaichao <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Model] support bitsandbytes quantization with minicpm3 model (vllm-p…

b5864e2

…roject#10682) Signed-off-by: Ubuntu <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Doc] Update model in arch_overview.rst to match comment (vllm-projec…

b9cabc9

…t#10701) Signed-off-by: spacewander <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Bug][CLI] Allow users to disable prefix caching explicitly (vllm-pro…

d61d661

…ject#10724) Signed-off-by: rickyx <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[V1] Do not allocate beyond the max_model_len (vllm-project#10730)

39f4494

Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Kernel] Update vllm-flash-attn version (vllm-project#10736)

dcdf2f3

Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[TPU] Update requirements-tpu (vllm-project#10726)

ea6ed6b

Signed-off-by: Richard Liu <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Model] Added GLM-4 series hf format model support vllm==0.6.4 (vllm-…

ac0b495

…project#10561) Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Kernel] Update vllm-flash-attn version to reduce CPU overheads (vllm…

1362dac

…-project#10742) Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[V1] Optimize the CPU overheads in FlashAttention custom op (vllm-pro…

bc6637c

…ject#10733) Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Model] Add Internlm2 LoRA support (vllm-project#5064)

3733796

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Model] Clean up MiniCPMV (vllm-project#10751)

170a30c

Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Misc] typo find in sampling_metadata.py (vllm-project#10740)

8d83244

Signed-off-by: Andrew Feldman <[email protected]>

[Bugfix] Fix Idefics3 bug (vllm-project#10778)

d8499c0

Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[platform] Add verify_quantization in platform. (vllm-project#10757)

3c8ced2

Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Bugfix] Fix OpenVino/Neuron driver_worker init (vllm-project#10779)

5146352

Signed-off-by: NickLucche <[email protected]> Signed-off-by: Cyrus Leung <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Model] Refactor Molmo weights loading to use AutoWeightsLoader (vllm…

d95da87

…-project#10771) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Interleaved ATTN] Support for Mistral-8B (vllm-project#10591)

7831672

Signed-off-by: youkaichao <[email protected]> Co-authored-by: youkaichao <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[doc] format fix (vllm-project#10789)

a877540

Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Model] Replace embedding models with pooling adapter (vllm-project#1…

cbf1489

…0769) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Misc] Improve type annotations for support_torch_compile (vllm-pro…

db1ca39

…ject#10763) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Misc] Rename embedding classes to pooling (vllm-project#10801)

d198e8f

Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[doc] add warning about comparing hf and vllm outputs (vllm-project#1…

cf04e11

…0805) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

[Misc] Adding MMMU-Pro vision dataset to serving benchmark (vllm-pr…

b58062b

…oject#10804) Signed-off-by: Roger Wang <[email protected]> Co-authored-by: Chen Zhang <[email protected]> Co-authored-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

removed fast tests from pipeline

bcdb5b8

Signed-off-by: Andrew Feldman <[email protected]>

[Model] Add BNB support to Llava and Pixtral-HF (vllm-project#10795)

02eb179

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

afeldman-nm added 16 commits January 8, 2025 11:52

Merge branch 'main' into v1_logprobs_merge

ce4f081

Merge branch 'main' into v1_logprobs_merge

62d648a

wip

3546639

Signed-off-by: Andrew Feldman <[email protected]>

Merge branch 'main' into v1_logprobs_merge

a8c0167

Merge branch 'v1_logprobs' into v1_logprobs_prompt

0bba8f9

GPU returns num_prompt_logprobs + 1 prompt logprobs

69218ab

Signed-off-by: Andrew Feldman <[email protected]>

now prompt logprobs include prompt token

2505244

Signed-off-by: Andrew Feldman <[email protected]>

wip making prompt logprobs line up with tok ids

e1058ac

Signed-off-by: Andrew Feldman <[email protected]>

partial req peek token

5f33902

Signed-off-by: Andrew Feldman <[email protected]>

refactoring

199a834

Signed-off-by: Andrew Feldman <[email protected]>

refactoring; non-blocking cpu->gpu transfer

879fc44

Signed-off-by: Andrew Feldman <[email protected]>

wip detokenizer tests

0f425fe

Signed-off-by: Andrew Feldman <[email protected]>

detok test fix

1089127

Signed-off-by: Andrew Feldman <[email protected]>

passing detok tests

d2742d8

Signed-off-by: Andrew Feldman <[email protected]>

Merge branch 'main' into v1_logprobs

cf28c9b

Merge branch 'main' into v1_logprobs_merge

749be5a

afeldman-nm changed the title ~~[V1] APC + prompt logprobs unsupported (PR 2/3 for v1 sample and prompt logprobs support)~~ [V1] APC + prompt logprobs unsupported (PR 2/N for v1 sample and prompt logprobs support) Jan 10, 2025

afeldman-nm mentioned this pull request Jan 10, 2025

[V1] PR 1/N for v1 sample and prompt logprobs support #9880

Open

afeldman-nm added 4 commits January 10, 2025 01:48

LLMEngine test working, wip AsyncLLM test

a55e679

Signed-off-by: Andrew Feldman <[email protected]>

reverted unwanted changes

b2c0c95

Signed-off-by: Andrew Feldman <[email protected]>

success

9a40c5f

Signed-off-by: Andrew Feldman <[email protected]>

Merge branch 'main' into v1_logprobs_apc_merge

ca94fd4

afeldman-nm marked this pull request as ready for review January 10, 2025 15:56

afeldman-nm requested review from WoosukKwon, robertgshaw2-neuralmagic, njhill, ywang96, comaniac and alexm-neuralmagic as code owners January 10, 2025 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[V1] APC + prompt logprobs unsupported (PR 2/N for v1 sample and prompt logprobs support) #11910

[V1] APC + prompt logprobs unsupported (PR 2/N for v1 sample and prompt logprobs support) #11910

afeldman-nm commented Jan 10, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Jan 10, 2025

[V1] APC + prompt logprobs unsupported (PR 2/N for v1 sample and prompt logprobs support) #11910

Are you sure you want to change the base?

[V1] APC + prompt logprobs unsupported (PR 2/N for v1 sample and prompt logprobs support) #11910

Conversation

afeldman-nm commented Jan 10, 2025 • edited by github-actions bot Loading

github-actions bot commented Jan 10, 2025

afeldman-nm commented Jan 10, 2025 •

edited by github-actions bot

Loading