Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI/Build] Add Model Tests for PixtralHF #9813

Merged
merged 4 commits into from
Nov 1, 2024
Merged

[CI/Build] Add Model Tests for PixtralHF #9813

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f62f57c
Create test_pixtral_hf.py
mgoin Oct 29, 2024
1d5b6ce
Format
mgoin Oct 29, 2024
4c8f29c
Merge branch 'vllm-project:main' into pixtral-hf-testing
mgoin Oct 31, 2024
e39d977
Swap test to new structure
mgoin Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions tests/models/decoder_only/vision_language/test_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,15 @@
# vllm_output_post_proc=model_utils.phi3v_vllm_to_hf_output,
# num_logprobs=10,
# ),
"pixtral_hf": VLMTestInfo(
models=["nm-testing/pixtral-12b-FP8-dynamic"],
test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),
prompt_formatter=lambda img_prompt: f"<s>[INST]{img_prompt}[/INST]",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prompt_formatter=lambda img_prompt: f"<s>[INST]{img_prompt}[/INST]",
prompt_formatter=lambda img_prompt: f"<s>[INST]\n{img_prompt}[/INST]",

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually based on the example script, there should be a newline here

Copy link
Member Author

@mgoin mgoin Oct 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DarkLight1337 From the HF example, the newline is actually supposed to be between the prompt and the images i.e.

PROMPT = "<s>[INST]Describe the images.\n[IMG][IMG][IMG][IMG][/INST]"

vllm/examples/offline_inference_vision_language.py

Line 295 in 77f7ef2

prompt = f"<s>[INST]{question}\n[IMG][/INST]"

I wasn't sure how to do this with this structure, but it wasn't important to get matching vLLM vs HF

Copy link
Member

@DarkLight1337 DarkLight1337 Nov 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see now. You can use the img_idx_to_prompt argument for using [IMG] token instead of <image>. We need to update the test framework to be able to support custom joining between text and image parts though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I did use img_idx_to_prompt=lambda idx: "[IMG]", and that worked thankfully. Luckily the newline isn't crucial for correctness, so that feature can wait

img_idx_to_prompt=lambda idx: "[IMG]",
max_model_len=8192,
max_num_seqs=2,
auto_cls=AutoModelForVision2Seq,
),
"qwen": VLMTestInfo(
models=["Qwen/Qwen-VL"],
test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),
Expand Down