-
-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ray[default] to wget to run distributed inference out of box #11265
Add ray[default] to wget to run distributed inference out of box #11265
Conversation
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
…de case This helps user to run vLLM with ray distributed executor using vLLM default image out of box. Signed-off-by: Jiaxin Shan <[email protected]>
2a7d504
to
a4328a4
Compare
this change is not directly related to above issue. I did verify the testing on my end and it's working fine. vllm/tests/entrypoints/openai/test_chat.py Lines 469 to 503 in 2d1b9ba
|
…m-project#11265) Signed-off-by: Jiaxin Shan <[email protected]> Signed-off-by: lucast2021 <[email protected]>
…m-project#11265) Signed-off-by: Jiaxin Shan <[email protected]>
part of #11137
Why do we need such PR?
ray[default]
package provides more capabilities like submission api and dashboard which is essential to run vLLM distributed inference in cluster environmentwget
is used in kubeRay prob.https://github.com/ray-project/kuberay/blob/e595ee4c6297fb6b385421f7ca34fbd7c1c0b49f/ray-operator/controllers/ray/common/pod.go#L253 and https://github.com/ray-project/kuberay/blob/e595ee4c6297fb6b385421f7ca34fbd7c1c0b49f/ray-operator/controllers/ray/utils/constant.go#L178. This can be definitely changed tocurl
but I feel adding a new package won't take that many room here.wget
andray[default]
won't be risky. I notice some other files already include this change.vllm/requirements-tpu.txt
Line 11 in f9ecbb1
vllm/Dockerfile.arm
Line 11 in f9ecbb1
FIX