Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ray[default] to wget to run distributed inference out of box #11265

Merged
merged 1 commit into from
Dec 20, 2024

Conversation

Jeffwan
Copy link
Contributor

@Jeffwan Jeffwan commented Dec 17, 2024

part of #11137

Why do we need such PR?

  1. User wants to leverage vllm default image to manage RayCluster version instead of managing separate ray images and vLLM distributions which frequently surface version compatibility problems
  2. ray[default] package provides more capabilities like submission api and dashboard which is essential to run vLLM distributed inference in cluster environment
  3. wget is used in kubeRay prob.https://github.com/ray-project/kuberay/blob/e595ee4c6297fb6b385421f7ca34fbd7c1c0b49f/ray-operator/controllers/ray/common/pod.go#L253 and https://github.com/ray-project/kuberay/blob/e595ee4c6297fb6b385421f7ca34fbd7c1c0b49f/ray-operator/controllers/ray/utils/constant.go#L178. This can be definitely changed to curl but I feel adding a new package won't take that many room here.
  4. Adding wget and ray[default] won't be risky. I notice some other files already include this change.

FIX

Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mergify mergify bot added the ci/build label Dec 17, 2024
…de case

This helps user to run vLLM with ray distributed executor using vLLM default image out of box.

Signed-off-by: Jiaxin Shan <[email protected]>
@Jeffwan Jeffwan force-pushed the jiaxin/add-ray-related-libs branch from 2a7d504 to a4328a4 Compare December 17, 2024 18:14
@Jeffwan
Copy link
Contributor Author

Jeffwan commented Dec 18, 2024

image

this change is not directly related to above issue. I did verify the testing on my end and it's working fine.

async def test_guided_choice_chat(client: openai.AsyncOpenAI,
guided_decoding_backend: str,
sample_guided_choice):
messages = [{
"role": "system",
"content": "you are a helpful assistant"
}, {
"role":
"user",
"content":
"The best language for type-safe systems programming is "
}]
chat_completion = await client.chat.completions.create(
model=MODEL_NAME,
messages=messages,
max_completion_tokens=10,
extra_body=dict(guided_choice=sample_guided_choice,
guided_decoding_backend=guided_decoding_backend))
choice1 = chat_completion.choices[0].message.content
assert choice1 in sample_guided_choice
messages.append({"role": "assistant", "content": choice1})
messages.append({
"role": "user",
"content": "I disagree, pick another one"
})
chat_completion = await client.chat.completions.create(
model=MODEL_NAME,
messages=messages,
max_completion_tokens=10,
extra_body=dict(guided_choice=sample_guided_choice,
guided_decoding_backend=guided_decoding_backend))
choice2 = chat_completion.choices[0].message.content
assert choice2 in sample_guided_choice
assert choice1 != choice2

image

@simon-mo simon-mo merged commit 47a0b61 into vllm-project:main Dec 20, 2024
20 of 22 checks passed
@Jeffwan Jeffwan deleted the jiaxin/add-ray-related-libs branch December 20, 2024 23:34
lucas-tucker pushed a commit to lucas-tucker/vllm-lucas-tucker that referenced this pull request Dec 21, 2024
BKitor pushed a commit to BKitor/vllm that referenced this pull request Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants