Chat method for offline llm #5049

nunjunj · 2024-05-25T08:23:02Z

Apply chat template for the offline LLM class. Adding new method generate_chat(), which generates chat responses for the input messages.

FIX #1582
FIX #3718
FIX #6191
FIX #6416

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

simon-mo

Great first step

Let's call it just chat because generate_chat has duplicated meaning. Chat method is generating.
Let's also support just a string or list of string for simplicity. We will apply it as a single turn conversation in this case. We should also take a parameter specifying the global system prompt. This is done for usability and it's simple to adopt the chat method.
Let's add support for specifying custom chat template via a string. This is supported in online api server.
Please add a good example script in the examples directory showcase the many different ways the chat method can be used with string comments.

DarkLight1337 · 2024-05-31T02:45:38Z

This should also fix #1582. I have edited the OP accordingly.

DarkLight1337 · 2024-05-31T14:25:20Z

Let's leave out multi_modal_data from the function signature; just focus on text input for now. We can later pass that in as an item of message (where message is a dictionary) to be more similar to OpenAI API.

Also, I would rename test_chat.py to test_llm_chat.py to match the naming convention of the other tests.

tests/entrypoints/test_chat.py

simon-mo · 2024-06-06T00:20:50Z

vllm/entrypoints/llm.py

+            messages = [{
+                'role':
+                'system',
+                'content':
+                global_system_prompt or default_system_prompt
+            }, {


some model (like Mixtral) doesn't support system prompt

Should I remove the default_system_prompt, and only include a system prompt when it is explicitly passed in?
Or Is there a method or property I can check to see if a model supports system prompts? Thank you for your guidance!

examples/offline_inference_chat.py

llmpros · 2024-07-08T19:56:07Z

hi @nunjunj kindly want to check if you may want to continue on this PR? Thanks

nunjunj · 2024-07-08T20:11:27Z

Yes, I am working on some merge conflicts and it should be done in a couple of days.

…

On Tue, Jul 9, 2024 at 2:56 AM ning.zhang ***@***.***> wrote: hi @nunjunj <https://github.com/nunjunj> kindly want to check if you may want to continue on this PR? Thanks — Reply to this email directly, view it on GitHub <#5049 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZLBZ7VMFFVUCPMM6WR3T2DZLLVG3AVCNFSM6AAAAABIIWS5SCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJVGA4DKOBWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

mgoin · 2024-07-15T14:43:45Z

examples/offline_inference_chat.py

+# str and list[str] inputs are treated as text completions, similar to LLM.generate(),
+message = "Entropy is"
+outputs = llm.chat(
+    message,
+    sampling_params=sampling_params,
+    use_tqdm=False,
+)
+print_outputs(outputs)
+
+messages = [
+    "Relativity is a theory about", "Bangkok is a city in",
+    "One of the best programming languages is"
+]
+outputs = llm.chat(
+    messages,
+    sampling_params=sampling_params,
+    use_tqdm=False,
+)
+print_outputs(outputs)


I'm unsure if llm.chat should accept unformatted text completions. It is possible users will use this and be unaware that no chat templating is happening even though they are using the chat interface. I believe HF's apply_chat_template fails if a non-dictionary is passed in, so we may want to follow that lead.

vllm/entrypoints/llm.py

llmpros · 2024-07-17T17:09:50Z

Yes, I am working on some merge conflicts and it should be done in a couple of days.
…
On Tue, Jul 9, 2024 at 2:56 AM ning.zhang @.> wrote: hi @nunjunj https://github.com/nunjunj kindly want to check if you may want to continue on this PR? Thanks — Reply to this email directly, view it on GitHub <#5049 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZLBZ7VMFFVUCPMM6WR3T2DZLLVG3AVCNFSM6AAAAABIIWS5SCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJVGA4DKOBWHE . You are receiving this because you were mentioned.Message ID: @.>

Hello @nunjunj just want to kindly ping and circle back to see if there may be any update on this feature. Thanks

DarkLight1337 · 2024-07-24T06:20:03Z

You may consider using the utilities in chat_utils.py (#6602) to avoid code duplication.

vllm/entrypoints/llm.py

DarkLight1337 · 2024-08-03T04:36:02Z

Also, please avoid unnecessarily editing whitespace.

…-llm

vllm/entrypoints/llm.py

DarkLight1337

LGTM now, thanks for implementing this! We can address multi-modal support in a future PR.

vllm/entrypoints/llm.py

…line-llm

mgoin

LGTM as well!

Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>

Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]> Signed-off-by: Alvant <[email protected]>

Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>

simon-mo reviewed May 25, 2024

View reviewed changes

DarkLight1337 mentioned this pull request Jun 3, 2024

[Feature]: Option to override HuggingFace's configurations #5205

Closed

simon-mo reviewed Jun 6, 2024

View reviewed changes

This was referenced Jul 3, 2024

[Doc]: Is the prompts feed into LLM will be automatically adding chat_template or not ? #6101

Closed

[Usage]: OpenAI-like API in offline inference #6191

Closed

DarkLight1337 mentioned this pull request Jul 14, 2024

[Feature]: Apply chat template through LLM class #6416

Closed

mgoin reviewed Jul 15, 2024

View reviewed changes

This was referenced Jul 20, 2024

[Misc] Manage HTTP connections in one place #6600

Merged

[Frontend] Move chat utils #6602

Merged

nunjunj and others added 2 commits July 23, 2024 23:04

add chat method

e4b1d87

add chat_template and add_generation_prompt params

189b100

nunjunj force-pushed the chat-method-for-offline-llm branch 4 times, most recently from e26dc5d to c49532c Compare July 23, 2024 17:34

run format.sh

7fcc1bf

nunjunj force-pushed the chat-method-for-offline-llm branch 2 times, most recently from 7fcc1bf to 7530495 Compare July 23, 2024 17:49

mgoin mentioned this pull request Jul 30, 2024

[Frontend]: Add apply_chat_template method and update generate method in LLM class #6936

Closed

nunjunj force-pushed the chat-method-for-offline-llm branch from 7530495 to 7fcc1bf Compare August 1, 2024 15:19

DarkLight1337 mentioned this pull request Aug 2, 2024

[Frontend] Factor out chat message parsing #7055

Merged

DarkLight1337 reviewed Aug 3, 2024

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

Merge remote-tracking branch origin/main into chat-method-for-offline…

6ee29e8

…-llm

nunjunj force-pushed the chat-method-for-offline-llm branch from a8f42d2 to c4ca2f3 Compare August 8, 2024 18:13

Merge branch 'main' into chat-method-for-offline-llm

b7c4031

nunjunj force-pushed the chat-method-for-offline-llm branch 2 times, most recently from 7b7b481 to f54c406 Compare August 15, 2024 15:23

fix lint

c64cc8c

nunjunj force-pushed the chat-method-for-offline-llm branch from f54c406 to c64cc8c Compare August 15, 2024 15:25

nunjunj added 2 commits August 15, 2024 22:41

revert format changes

3cd737e

fix lint

9cead6b

DarkLight1337 reviewed Aug 15, 2024

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

Reduce diffs

20b38d9

DarkLight1337 reviewed Aug 15, 2024

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

Clean

415193d

DarkLight1337 approved these changes Aug 15, 2024

View reviewed changes

DarkLight1337 enabled auto-merge (squash) August 15, 2024 16:21

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 15, 2024

DarkLight1337 reviewed Aug 15, 2024

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

DarkLight1337 and others added 3 commits August 16, 2024 00:27

Clean 2

7eb64e2

Fix docs

64993b6

Merge remote-tracking branch 'upstream/main' into chat-method-for-off…

465b65e

…line-llm

mgoin approved these changes Aug 16, 2024

View reviewed changes

youkaichao disabled auto-merge August 16, 2024 02:41

youkaichao merged commit 3b19e39 into vllm-project:main Aug 16, 2024
67 of 69 checks passed

DarkLight1337 mentioned this pull request Sep 3, 2024

[Frontend] Multimodal support in offline chat #8098

Merged

AlpinDale mentioned this pull request Nov 17, 2024

feat: add chat method for LLM class aphrodite-engine/aphrodite-engine#822

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chat method for offline llm #5049

Chat method for offline llm #5049

nunjunj commented May 25, 2024 •

edited by DarkLight1337

Loading

simon-mo left a comment

DarkLight1337 commented May 31, 2024

DarkLight1337 commented May 31, 2024 •

edited

Loading

simon-mo Jun 6, 2024

nunjunj Jun 6, 2024 •

edited

Loading

llmpros commented Jul 8, 2024

nunjunj commented Jul 8, 2024 via email

mgoin Jul 15, 2024

llmpros commented Jul 17, 2024

DarkLight1337 commented Jul 24, 2024

DarkLight1337 commented Aug 3, 2024 •

edited

Loading

DarkLight1337 left a comment

mgoin left a comment

Chat method for offline llm #5049

Chat method for offline llm #5049

Conversation

nunjunj commented May 25, 2024 • edited by DarkLight1337 Loading

PR Title and Classification

Code Quality

Notes for Large Changes

What to Expect for the Reviews

Thank You

simon-mo left a comment

Choose a reason for hiding this comment

DarkLight1337 commented May 31, 2024

DarkLight1337 commented May 31, 2024 • edited Loading

simon-mo Jun 6, 2024

Choose a reason for hiding this comment

nunjunj Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

llmpros commented Jul 8, 2024

nunjunj commented Jul 8, 2024 via email

mgoin Jul 15, 2024

Choose a reason for hiding this comment

llmpros commented Jul 17, 2024

DarkLight1337 commented Jul 24, 2024

DarkLight1337 commented Aug 3, 2024 • edited Loading

DarkLight1337 left a comment

Choose a reason for hiding this comment

mgoin left a comment

Choose a reason for hiding this comment

nunjunj commented May 25, 2024 •

edited by DarkLight1337

Loading

DarkLight1337 commented May 31, 2024 •

edited

Loading

nunjunj Jun 6, 2024 •

edited

Loading

DarkLight1337 commented Aug 3, 2024 •

edited

Loading