-
-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[V1] Use input_ids as input for text-only models #11032
Conversation
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Chatted offline with @ywang96, while we also need to support input embeddings for text-only models, this PR is good to go since the API is still not ready for V1. |
Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Akshat Tripathi <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Currently, the model runner in V1 always uses
input_embeds
rather thaninput_ids
for the compatibility with multi-modal models. However, this excludes the embedding layer from the CUDA graph, and thus causes slight performance regression (and larger performance regression when TP > 1). This PR addresses this by using theinput_ids
for text-only models while keeping usinginput_embeds
for multi-modal models.