[Model] Add GLM-4v support and meet vllm==0.6.2

[sixsixcoder]

Overview

This PR support the glm-4v-9b multimodal model while maintaining compatibility with chatglm.
This PR was inspired and reused some code here #5358
This PR is updated from #8663

FIX #5417
FIX #6097

Changes

1. Add vision_config for ChatGLMConfig
2. Add glm4 vision encoder in vllm/model_executor/models/glm4_vision_encoder.py.
3. Add optional vision module for ChatGLMModel, making ChatGLMForCausalLM multimodal capable.
4. Added support for receiving and processing image_embeds parameters
5. Added the weight loading logic
6. Add pytest
7. Add example

Development Environment

vllm==0.6.2.post2+cu123
vllm-flash-attn==2.6.1
transformers==4.44.2
torch==2.4.0
torchvision==0.19.0
cuda==12.3
python==3.10

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

---

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Model] for adding a new model or improving an existing model. Model name should appear in the title.
• [Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
• [Kernel] for changes affecting CUDA kernels or other compute kernels.
• [Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
• [Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• We adhere to Google Python style guide and Google C++ style guide.
• Pass all linter checks. Please use format.sh to format your code.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
• Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Adding or changing kernels

Each custom kernel needs a schema and one or more implementations to be registered with PyTorch.

• Make sure custom ops are registered following PyTorch guidelines: Custom C++ and CUDA Operators and The Custom Operators Manual
• Custom operations that return Tensors require meta-functions. Meta-functions should be implemented and registered in python so that dynamic dims can be handled automatically. See above documents for a description of meta-functions.
• Use torch.libary.opcheck() to test the function registration and meta-function for any registered ops. See tests/kernels for examples.
• When changing the C++ signature of an existing op, the schema must be updated to reflect the changes.
• If a new custom type is needed, see the following document: Custom Class Support in PT2.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

• After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
• After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
• After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
• Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[Isotr0py]

Suggested change

	def run_glm4v(question):
	def run_glm4v(question: str, modality: str):
	assert modality == "image"

[Isotr0py]

Suggested change

	return {'pixel_values': pixel_values}
	return MultiModalInputs({'pixel_values': pixel_values})

[Isotr0py]

Please address the lints errors with bash format.sh as well.

Update: Almost forgot, you need to add glm-4v to docs/source/models/supported_models.rst as well.

[sixsixcoder]

bash format.sh请同时解决 lints 错误。

更新：差点忘了，您还需要添加glm-4v到docs/source/models/supported_models.rst 。

Thanks for your checking and replying, I have updated all the details

[Isotr0py]

Please fix the failed ruff and yapf check by running bash format.sh, otherwise LGTM now.

[DarkLight1337]

Thanks for your patience! I have fixed the lint errors for you.

[sixsixcoder]

Thanks for your patience! I have fixed the lint errors for you.

Thanks for your help

[sixsixcoder]

Please fix the failed ruff and yapf check by running bash format.sh, otherwise LGTM now.

Thank you for your long-term help

[DarkLight1337]

Can this model run under shorter max_model_len (e.g. 2048)? It is causing OOM in the CI.

[sixsixcoder]

此模型可以在更短的时间max_model_len（例如 2048）下运行吗？它会导致 CI​​ 中的 OOM。

sure

[DarkLight1337]

After updating the test so that the tokenizer can work with transformers v4.45, I found some discrepancies between the HF and vLLM output which cause the tests to fail. Did you encounter similar issues before?

Matched tokens: [785, 2168]
hf:     'The image captures a street scene in an area that appears to be of Chinese cultural influence, judging by the traditional Chinese architectural elements such as the red and gold colors, the upturned eaves, and the Chinese characters on the signage. In the foreground, there is a red octagonal stop sign, which is a traffic control device instructing drivers to bring their vehicles to a complete stop before proceeding. The stop sign is mounted on a red pole with a decorative top.\n\nBehind the stop sign, there is a black SUV in motion, passing by the camera. On the left, there is a building with a sign that reads "yes,"'   {69552: -1.1920928244535389e-07, 5568: -16.0, 90520: -32.0, 53327: -48.0, 57500: -128.0}
vllm:   'The image depicts a street scene with a focus on a red octagonal sign that reads "STOP" in white capital letters. The sign is mounted on a post with a decorative top, and there are two white lion statues on either side of the sign, which are often found in Chinese culture as symbols of guardianship and courage.\n\nIn the background, there is a traditional Chinese-style gate with red columns and a black and gold roof, which is a common architectural feature in Chinese temples, palaces, and other significant buildings. The gate is adorned with Chinese characters, and there are red lanterns hanging from the top, which are often used'      {61457: Logprob(logprob=-1.1776821613311768, rank=1, decoded_token=' depicts'), 4933: Logprob(logprob=-1.4276821613311768, rank=2, decoded_token=' shows'), 39934: Logprob(logprob=-1.6776821613311768, rank=3, decoded_token=' captures'), 4419: Logprob(logprob=-2.6776821613311768, rank=4, decoded_token=' features'), 18656: Logprob(logprob=-2.9276821613311768, rank=5, decoded_token=' displays')}

[DarkLight1337]

Nevertheless, I will skip this test in CI because it OOMs even with the shorter max_model_len, and the output is still reasonable.

[sixsixcoder]

Nevertheless, I will skip this test in CI because it OOMs even with the shorter max_model_len, and the output is still reasonable.

Thank you for your help in merging this PR smoothly. We look forward to more cooperation in the future.

[tuanhe]

After updating the test so that the tokenizer can work with transformers v4.45, I found some discrepancies between the HF and vLLM output which cause the tests to fail. Did you encounter similar issues before?

Matched tokens: [785, 2168]
hf:     'The image captures a street scene in an area that appears to be of Chinese cultural influence, judging by the traditional Chinese architectural elements such as the red and gold colors, the upturned eaves, and the Chinese characters on the signage. In the foreground, there is a red octagonal stop sign, which is a traffic control device instructing drivers to bring their vehicles to a complete stop before proceeding. The stop sign is mounted on a red pole with a decorative top.\n\nBehind the stop sign, there is a black SUV in motion, passing by the camera. On the left, there is a building with a sign that reads "yes,"'   {69552: -1.1920928244535389e-07, 5568: -16.0, 90520: -32.0, 53327: -48.0, 57500: -128.0}
vllm:   'The image depicts a street scene with a focus on a red octagonal sign that reads "STOP" in white capital letters. The sign is mounted on a post with a decorative top, and there are two white lion statues on either side of the sign, which are often found in Chinese culture as symbols of guardianship and courage.\n\nIn the background, there is a traditional Chinese-style gate with red columns and a black and gold roof, which is a common architectural feature in Chinese temples, palaces, and other significant buildings. The gate is adorned with Chinese characters, and there are red lanterns hanging from the top, which are often used'      {61457: Logprob(logprob=-1.1776821613311768, rank=1, decoded_token=' depicts'), 4933: Logprob(logprob=-1.4276821613311768, rank=2, decoded_token=' shows'), 39934: Logprob(logprob=-1.6776821613311768, rank=3, decoded_token=' captures'), 4419: Logprob(logprob=-2.6776821613311768, rank=4, decoded_token=' features'), 18656: Logprob(logprob=-2.9276821613311768, rank=5, decoded_token=' displays')}

Same issue 😭

[WangErXiao]

I found this will effect prefix cache function for text glm model. @sixsixcoder

[DarkLight1337]

I found this will effect prefix cache function for text glm model. @sixsixcoder

Could you explain in more detail? Does enabling prefix cache explicitly not work?

[ywang96]

I found this will effect prefix cache function for text glm model. @sixsixcoder

Could you explain in more detail? Does enabling prefix cache explicitly not work?

@DarkLight1337 We currently disable prefix caching for all multimodal models, and the change of this PR will disable it even when the user uses this model only for text inference

[WangErXiao]

I found this will effect prefix cache function for text glm model. @sixsixcoder

Could you explain in more detail? Does enabling prefix cache explicitly not work?

Yes, I open prefix cache explicitly. It still log the disable prefix cache.

[DarkLight1337]

I found this will effect prefix cache function for text glm model. @sixsixcoder

Could you explain in more detail? Does enabling prefix cache explicitly not work?

@DarkLight1337 We currently disable prefix caching for all multimodal models, and the change of this PR will disable it even when the user uses this model only for text inference

Oh, I thought we only disabled it if prefix caching is not set.