Skip to content

[P/D][V1] MLA support in KV Connector #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 10, 2025
Merged

[P/D][V1] MLA support in KV Connector #6

merged 1 commit into from
Apr 10, 2025

Conversation

Flechman
Copy link

@Flechman Flechman commented Apr 10, 2025 •
edited by github-actions bot
Loading

PR vllm-project#15960 assumes that the KV cache always contains different values for K and V and thus doesn't support MLA.
This PR adds support for MLA where the KV cache is a single same value for K and V.

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@Flechman Flechman changed the base branch from v1-disagg to local-dev/v1-disagg April 10, 2025 19:37
@Flechman Flechman mentioned this pull request Apr 10, 2025
Merged
3 tasks
hasB4K
hasB4K approved these changes Apr 10, 2025
View reviewed changes
Copy link

@hasB4K hasB4K left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ApostaC
ApostaC approved these changes Apr 10, 2025
View reviewed changes
Copy link
Owner

@ApostaC ApostaC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@robertgshaw2-redhat PTAL!

@ApostaC ApostaC merged commit 406d6bf into ApostaC:local-dev/v1-disagg Apr 10, 2025
3 checks passed
ApostaC added a commit that referenced this pull request Apr 11, 2025
@ApostaC
Squashed commit of the following:
1cea76f 
commit 406d6bf
Author: Rémi Delacourt <[email protected]>
Date:   Fri Apr 11 00:47:40 2025 +0200

    Add MLA support for v1 disagg connector (#6)

    Signed-off-by: remi <[email protected]>

commit 1d8415d
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:59:54 2025 +0000

    rename

    Signed-off-by: [email protected] <[email protected]>

commit 9c4159c
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:41:20 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 54e1491
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:31:35 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 8e1eadc
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:26:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 05349a5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 22:10:50 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7f57f3c
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:13:31 2025 +0000

    update lifecycle

    Signed-off-by: [email protected] <[email protected]>

commit 7c31e29
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:03:55 2025 +0000

    nits

    Signed-off-by: [email protected] <[email protected]>

commit 74af233
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:44:01 2025 +0000

    done with nits

    Signed-off-by: [email protected] <[email protected]>

commit e64f745
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:28:51 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 40e5d81
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:25:04 2025 +0000

    refactor

    Signed-off-by: [email protected] <[email protected]>

commit 25c9592
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:20:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit fc58dd5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:13:39 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 20decdf
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:06:15 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 5145566
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:52:03 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 62e1421
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:47:40 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 689379e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:39:40 2025 +0000

    updaed

    Signed-off-by: [email protected] <[email protected]>

commit b1310fd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:36:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7b64acb
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:32:47 2025 +0000

    clean up code

    Signed-off-by: [email protected] <[email protected]>

commit 7766ca5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:31:33 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit b0629bd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:27:58 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit eca7a49
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:25:24 2025 +0000

    cleaning

    Signed-off-by: [email protected] <[email protected]>

commit 1881aa5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:24:55 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7833645
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:11:18 2025 +0000

    updared

    Signed-off-by: [email protected] <[email protected]>

commit e72e5e4
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:08:34 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 48c2eb2
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:07:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit de1e487
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:01:14 2025 +0000

    fix nit

    Signed-off-by: [email protected] <[email protected]>

commit 0163070
Merge: e2ecc14 8b3f606
Author: Robert Shaw <[email protected]>
Date:   Wed Apr 9 10:39:46 2025 -0400

    Merge pull request #4 from robertgshaw2-redhat/rob-changes

    Rob changes

commit 8b3f606
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:26 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 90e8c53
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit da019df
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:23:47 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 4ebcc3e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:44:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 00df670
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:20:14 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit a73721a
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:19:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 31d807e
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 20:58:28 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

commit 5accb53
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 16:00:29 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

Signed-off-by: ApostaC <[email protected]>
ApostaC added a commit that referenced this pull request Apr 14, 2025
@ApostaC
Squashed commit of the following:
ea8df07 
commit 3a24897
Author: ApostaC <[email protected]>
Date:   Thu Apr 10 18:31:47 2025 -0700

    [Fix] memory leak problem by proper clean up

    Signed-off-by: ApostaC <[email protected]>

commit 406d6bf
Author: Rémi Delacourt <[email protected]>
Date:   Fri Apr 11 00:47:40 2025 +0200

    Add MLA support for v1 disagg connector (#6)

    Signed-off-by: remi <[email protected]>

commit 1d8415d
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:59:54 2025 +0000

    rename

    Signed-off-by: [email protected] <[email protected]>

commit 9c4159c
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:41:20 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 54e1491
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:31:35 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 8e1eadc
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:26:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 05349a5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 22:10:50 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7f57f3c
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:13:31 2025 +0000

    update lifecycle

    Signed-off-by: [email protected] <[email protected]>

commit 7c31e29
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:03:55 2025 +0000

    nits

    Signed-off-by: [email protected] <[email protected]>

commit 74af233
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:44:01 2025 +0000

    done with nits

    Signed-off-by: [email protected] <[email protected]>

commit e64f745
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:28:51 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 40e5d81
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:25:04 2025 +0000

    refactor

    Signed-off-by: [email protected] <[email protected]>

commit 25c9592
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:20:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit fc58dd5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:13:39 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 20decdf
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:06:15 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 5145566
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:52:03 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 62e1421
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:47:40 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 689379e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:39:40 2025 +0000

    updaed

    Signed-off-by: [email protected] <[email protected]>

commit b1310fd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:36:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7b64acb
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:32:47 2025 +0000

    clean up code

    Signed-off-by: [email protected] <[email protected]>

commit 7766ca5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:31:33 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit b0629bd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:27:58 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit eca7a49
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:25:24 2025 +0000

    cleaning

    Signed-off-by: [email protected] <[email protected]>

commit 1881aa5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:24:55 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7833645
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:11:18 2025 +0000

    updared

    Signed-off-by: [email protected] <[email protected]>

commit e72e5e4
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:08:34 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 48c2eb2
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:07:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit de1e487
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:01:14 2025 +0000

    fix nit

    Signed-off-by: [email protected] <[email protected]>

commit 0163070
Merge: e2ecc14 8b3f606
Author: Robert Shaw <[email protected]>
Date:   Wed Apr 9 10:39:46 2025 -0400

    Merge pull request #4 from robertgshaw2-redhat/rob-changes

    Rob changes

commit 8b3f606
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:26 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 90e8c53
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit da019df
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:23:47 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 4ebcc3e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:44:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 00df670
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:20:14 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit a73721a
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:19:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 31d807e
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 20:58:28 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

commit 5accb53
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 16:00:29 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

Signed-off-by: ApostaC <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hasB4K hasB4K hasB4K approved these changes

@ApostaC ApostaC ApostaC approved these changes

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Flechman @hasB4K @ApostaC