Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Efficient CPU prefix caching for the prefill step #10888

Closed
wants to merge 11 commits into from
Closed

[Core] Efficient CPU prefix caching for the prefill step #10888

wants to merge 11 commits into from

Conversation

lixiaobai09
Copy link

@lixiaobai09 lixiaobai09 commented Dec 4, 2024 •
edited by github-actions bot
Loading

We implement an efficient CPU prefix caching for the prefill step, including:

  1. A naive implementation of CPU KV Block Cache Manager.
  2. A data transmission optimization to overlap layer-wise block swapping and forward computing.
  3. A data transmission optimization to overlap request-level block swapping and computing by delaying the transmission requests one step.

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@lixiaobai09 lixiaobai09 changed the title [Misc] Efficient CPU prefix caching for the prefill step [Core] Efficient CPU prefix caching for the prefill step Dec 4, 2024
Dahai Tang added 8 commits December 4, 2024 09:50
Feat(kv store): add a CPU KV Cache Block Store
531de0b 
Conflicts:
	vllm/attention/backends/abstract.py
	vllm/attention/backends/flash_attn.py
	vllm/core/scheduler.py
	vllm/model_executor/models/qwen2.py
	vllm/sequence.py
	vllm/worker/worker.py
	vllm/worker/worker_base.py

Signed-off-by: Dahai Tang <[email protected]>
Fix(kv store): fix some building and running bugs
8613037 
Signed-off-by: Dahai Tang <[email protected]>
Refactor: pass lint check
f47bbce 
Signed-off-by: Dahai Tang <[email protected]>
Refactor: pass lint check
3e9fd6a 
Signed-off-by: Dahai Tang <[email protected]>
Feat(lint): pass the clang format checker
f272bd7 
Signed-off-by: Dahai Tang <[email protected]>
Refactor: pass more detail lint checker
243e4f9 
Signed-off-by: Dahai Tang <[email protected]>
isort the import order
2a9deb6 
Signed-off-by: Dahai Tang <[email protected]>
Refactor: lint checker
d9b6509 
Signed-off-by: Dahai Tang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123 zhuohan123 is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@alexm-neuralmagic alexm-neuralmagic Awaiting requested review from alexm-neuralmagic alexm-neuralmagic is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

Assignees
No one assigned
Labels
ci/build frontend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@lixiaobai09