vertex ai openai-compatible api

[krrishdholakia]

Just looked at the vertex ai openai compatible api - https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library

Realized they only let you pass images in the 'image_url' param.

LiteLLM supports passing pdf/video/etc. through this to gemini, so this would be a regression for us to support.

Leaving this discussion open, in case things change

[Manouchehri]

This is what I would do (this would also make for a better caching experience IMO):

1. LiteLLM downloads the image defined at image_url.url, calculates the sha512 of it.

2. LiteLLM uploads it via the S3 API for GCS at like https://storage.googleapis.com/demo-bucket-leet-1/34e90c22537b17f44ac1db09e74b9022, and generates its own cache key based on that URL. When it forwards the request upstream, it does one of the following:

3a. (for Vertex AI) e.g. use gs://demo-bucket-leet-1/34e90c22537b17f44ac1db09e74b9022 as the file_uri/image_url.

3b. (for GPT-4, GPT-3.5, etc) Using the S3 API (which GCS supports), generate a presigned URL of it as the image_url. https://gist.github.com/gleicon/2b8acb9f9c0f22753eaac227ff997b34?permalink_comment_id=4721658#gistcomment-4721658 e.g. https://storage.googleapis.com/demo-bucket-leet-1/34e90c22537b17f44ac1db09e74b9022?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1E2NHHGLJNVFHX3VBWMRUJMAN7Q6AZSY2VPOWBUNENYHOFSR5HRYMJ6FJ%2F20240613%2Fus-east4%2Fs3%2Faws4_request&X-Amz-Date=20240613T164848Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=3dad22351c2da39218f754ee8e227406e0d81f3ec70d305ff617e30d9b67d9e6

This would allow the uploading of files to be united across different providers, since we would be always using the S3 API for uploading files. It would only differ at the end.

[krrishdholakia]

hey @Manouchehri , what problem is this solving?

[Manouchehri]

It solves multiple problems.

1. Helps simplifying adding support for image_url when using the Vertex AI OpenAI-compatible API endpoint. i.e. it helps solve what this discussion is about.
2. It allows deduplication of the same image files that might have different URLs. e.g. without my proposed solution, https://litellm.vercel.app/img/docusaurus-social-card.png and https://docs.litellm.ai/img/docusaurus-social-card.png are treated and cached as two completely different requests, despite being identical.
3. It separates out the storage of media from the text prompts. i.e. no more huge 1MB base64 blobs shoved into Langfuse; just a pretty URL instead.
4. Improves reliability. When using https://docs.litellm.ai/img/docusaurus-social-card.png, there is zero promise that that URL won't change. We can be 100% sure that https://storage.googleapis.com/demo-bucket-leet-1/25ca0995f576b910b60bc53cb94a9bbc won't change ever, since we are storing a copy and referencing it by the hash.

There's probably a few other reasons why this is a good idea, but these seem like enough reasons to consider it? :)