Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bugfix] Remove xformers requirement for Pixtral #9597

Merged
merged 3 commits into from
Oct 24, 2024

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented Oct 22, 2024

This PR adds a torch SDPA implementation for PixtralHF attention

FIX #9036 (comment)

I'm trying to run LlavaNextForConditionalGeneration on a non-CUDA hardware platform, but when I import llava_next, it pulls in pixtral, which in turn imports xformers. From my understanding, xformers is required only for CUDA support. Is there a way to avoid this dependency or run LlavaNextForConditionalGeneration without xformers on non-CUDA platforms?

Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mgoin mgoin force-pushed the remove-xformers-req-pixtral branch from c5c736c to a2eccfc Compare October 22, 2024 20:42
@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 24, 2024
@mgoin mgoin changed the title Remove xformers requirement for Pixtral [Bugfix] Remove xformers requirement for Pixtral Oct 24, 2024
@simon-mo simon-mo merged commit c91ed47 into main Oct 24, 2024
68 of 70 checks passed
WoosukKwon pushed a commit that referenced this pull request Oct 25, 2024
MErkinSag pushed a commit to MErkinSag/vllm that referenced this pull request Oct 26, 2024
cooleel pushed a commit to cooleel/vllm that referenced this pull request Oct 28, 2024
cooleel pushed a commit to cooleel/vllm that referenced this pull request Oct 28, 2024
@simon-mo simon-mo deleted the remove-xformers-req-pixtral branch October 28, 2024 16:51
FerdinandZhong pushed a commit to FerdinandZhong/vllm that referenced this pull request Oct 29, 2024
NickLucche pushed a commit to NickLucche/vllm that referenced this pull request Oct 31, 2024
NickLucche pushed a commit to NickLucche/vllm that referenced this pull request Oct 31, 2024
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 4, 2024
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024
tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants