Skip to content

[ModelRunner] Support embedding inputs #916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Potabk
Copy link
Contributor

@Potabk Potabk commented May 21, 2025

What this PR does / why we need it?

  • Adds support for passing prompt_embeds to LLM.generate as
llm.generate({"prompt_embeds": input_embeds}, sampling_params)

or

llm.generate(
    [{"prompt_embeds": input_embeds} for input_embeds in inputs_embeds], sampling_params
)
  • Add prompt_embeds to examples

Does this PR introduce any user-facing change?

How was this patch tested?

CI passed with new added/existing test.
and I have test with the example script in this pr, and the output seems looks good:

[Single Inference Output]
------------------------------
The capital of France is Paris. Paris is the largest city in France and is
------------------------------
Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 3966.87it/s]
Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.99it/s, est. speed input: 177.08 toks/s, output: 63.91 toks/s]

[Batch Inference Outputs]
------------------------------
Q1: Please tell me about the capital of France.
A1: The capital of France is Paris. It is located in the northern part of the

Q2: When is the day longest during the year?
A2: The day is longest during the year at the summer solstice. This typically occurs

Q3: Where is bigger, the moon or the sun?
A3: The sun is significantly bigger than the moon. 

The sun has a diameter of

------------------------------

@Potabk Potabk force-pushed the dev branch 3 times, most recently from 1532649 to 0bd6624 Compare May 22, 2025 07:19
@Potabk
Copy link
Contributor Author

Potabk commented May 22, 2025

@wangxiyuan this is ready for review

@Potabk Potabk changed the title [Worker][ModelRunner][WIP] Support embedding inputs [ModelRunner] Support embedding inputs May 22, 2025
Copy link
Collaborator

@wangxiyuan wangxiyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's good to add a test as well

@@ -0,0 +1,83 @@
import torch
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice example

@wangxiyuan wangxiyuan added the ready read for review label May 22, 2025
@Potabk Potabk force-pushed the dev branch 2 times, most recently from 2fd64bf to 727f97f Compare May 23, 2025 07:43
@Potabk Potabk force-pushed the dev branch 3 times, most recently from 6d57499 to c0f3cc4 Compare May 28, 2025 02:55
Signed-off-by: wangli <[email protected]>
Potabk added 2 commits May 28, 2025 03:00
Signed-off-by: wangli <[email protected]>
Signed-off-by: wangli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:tests ready read for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants