Skip to content

[vertexai] Fix image support for ToolMessage in langchain-vertexai #950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rong-xyz
Copy link

@rong-xyz rong-xyz commented May 28, 2025

Basically, try send a base64 image (say from MCP) in ToolMessage you will easily get 900k token usage, which a lot of the user would assume it will just work if they transfermed from anthropic
This is a fix where the model behaves relatively good by faking role: function into a user but with tags
I made a similar fix to langchain-aws , where the model will simply skip translation if it is a ToolMessage, here everything is to_str() and image is easily 900k token usage langchain-ai/langchain-aws#495

As confimed by someone worked in Google:

Right now this is somewhat undefined, but model will probably figure it out if you send the image as a user role, and document this in the dev instruction. We have done bunch of work where wejust drop the image right after theFunctionResponse (so right after the\n and that seems to work well. More specifically the image has a nameso we print Wrote image: foo.pngas the textual function output then do
\n
foo.jpg

end quote.

The google data structure is a pain to use and work with compared to other providers, It will be too much work to write a test, as I will use this to patch our code base anyways, I will submit the PR here

Copy link
Collaborator

@lkuligin lkuligin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you fix linter and add a corresponding unit test, please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants