Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc][CI/Build] Update docs and tests to use vllm serve #6431

Merged
merged 12 commits into from
Jul 17, 2024

Conversation

DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Jul 15, 2024

Follow-up to #5090. As a sanity check, I have also updated the entrypoints tests to use the new CLI.

After this, we can update the Docker images and performance benchmarks to use the new CLI.

cc @EthanqX

Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only trigger fastcheck CI to run, which consists only a small and essential subset of tests to quickly catch errors with the flexibility to run extra individual tests on top (you can do this by unblocking test steps in the Buildkite run).

Full CI run is still required to merge this PR so once the PR is ready to go, please make sure to run it. If you need all test signals in between PR commits, you can trigger full CI as well.

To run full CI, you can do one of these:

  • Comment /ready on the PR
  • Add ready label to the PR
  • Enable auto-merge.

🚀

@simon-mo
Copy link
Collaborator

I want to hold this off until the release. So people visiting the nightly docs can directly use the CLI

@simon-mo
Copy link
Collaborator

In the meantime, please feel free to start improving it!!!

@DarkLight1337 DarkLight1337 changed the title [Doc] Update docs to use vllm serve [Doc][CI/Build] Update docs and tests to use vllm serve Jul 15, 2024
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does --model still work or will it cause issues with vllm serve? I'm curious if it is sufficient to replace python -m vllm.entrypoints.openai.api_server with vllm serve or if it specifically replaces python -m vllm.entrypoints.openai.api_server --model

@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Jul 16, 2024

Does --model still work or will it cause issues with vllm serve?

model_tag is a required positional arg in vllm serve so you have to pass it like vllm serve <model_tag>, regardless of whether --model is passed. The value passed to model is actually overwritten by model_tag (we should not allow both to be passed at the same time tbh).

Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarity, this is a nice PR to update usage LGTM!

@DarkLight1337
Copy link
Member Author

@simon-mo since v0.5.2 has been released, can we merge this?

@simon-mo simon-mo enabled auto-merge (squash) July 16, 2024 15:33
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 16, 2024
@simon-mo simon-mo merged commit 5bf35a9 into vllm-project:main Jul 17, 2024
71 of 72 checks passed
@DarkLight1337 DarkLight1337 deleted the vllm-serve-docs branch July 17, 2024 07:43
dtrifiro pushed a commit to opendatahub-io/vllm that referenced this pull request Jul 17, 2024
fialhocoelho pushed a commit to opendatahub-io/vllm that referenced this pull request Jul 19, 2024
xjpang pushed a commit to xjpang/vllm that referenced this pull request Jul 24, 2024
gnpinkert pushed a commit to gnpinkert/vllm that referenced this pull request Jul 26, 2024
Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants