feat - add vision chat for ollama_chat

[linznin]

Title

Implement vision chat with ollama_chat provider

Relevant issues

Fixes 6808

Type

🆕 New Feature

Changes

• Formatted the request to include image content for the ollama_chat provider.

[REQUIRED] Testing - Attach a screenshot of any new tests passing locall

If UI changes, send a screenshot/GIF of working UI fixes

Before the fix:

completion(
model="ollama_chat/llava:7b",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Whats in this image?"},
{
"type": "image_url",
"image_url": {"url": jpeg_image},
},
],
}
],
api_base="http://localhost:11434")

LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Traceback (most recent call last):
  File "$HOME\litellm\litellm\main.py", line 2798, in completion
    generator = ollama_chat.get_ollama_response(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$HOME\litellm\litellm\llms\ollama_chat.py", line 325, in get_ollama_response
    raise OllamaError(status_code=response.status_code, message=response.text)
litellm.llms.ollama_chat.OllamaError: {"error":"json: cannot unmarshal array into Go struct field ChatRequest.messages of type string"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "$HOME\litellm\main.py", line 18, in <module>
    response = completion(
               ^^^^^^^^^^^
  File "$HOME\litellm\litellm\utils.py", line 960, in wrapper
    raise e
  File "$HOME\litellm\litellm\utils.py", line 849, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$HOME\litellm\litellm\main.py", line 3059, in completion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "$HOME\litellm\litellm\litellm_core_utils\exception_mapping_utils.py", line 2136, in exception_type
    raise e
  File "$HOME\litellm\litellm\litellm_core_utils\exception_mapping_utils.py", line 2105, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: Ollama_chatException - {"error":"json: cannot unmarshal array into Go struct field ChatRequest.messages of type string"}

After the fix it works fine.

Description by Korbit AI

What change is being made?

Add functionality for handling and converting images within the ollama_chat module by introducing the _convert_image function and testing this functionality with new unit tests.

Why are these changes being made?

This change enhances the ollama_chat module by allowing it to process inline images from user messages, converting non-JPEG/PNG formats to JPEG. This feature ensures better compatibility with the image handling requirements of the ollama platform. The included unit tests confirm that the image conversion works as expected, which is essential for maintaining robustness.

Is this description stale? Ask me to generate a new description by commenting /korbit-generate-pr-description

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 23, 2024 1:55pm

[korbit-ai]

I've completed my review and didn't find any issues... but I did find this bunny.

(\(\
( -.-)
o_(")(")

Files scanned

​

File Path	Reviewed
litellm/llms/ollama_chat.py	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Need a new review? Comment /korbit-review on this PR and I'll review your latest changes.

Korbit Guide: Usage and Customization

Interacting with Korbit

• You can manually ask Korbit to review your PR using the /korbit-review command in a comment at the root of your PR.
• You can ask Korbit to generate a new PR description using the /korbit-generate-pr-description command in any comment on your PR.
• Too many Korbit comments? I can resolve all my comment threads if you use the /korbit-resolve command in any comment on your PR.
• Chat with Korbit on issues we post by tagging @korbit-ai in your reply.
• Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.

Customizing Korbit

• Check out our docs on how you can make Korbit work best for you and your team.
• Customize Korbit for your organization through the Korbit Console.

Current Korbit Configuration

General Settings

​

Setting	Value
Review Schedule	Automatic excluding drafts
Max Issue Count	10
Automatic PR Descriptions	✅

Issue Categories

​

Category	Enabled
Naming	✅
Database Operations	✅
Documentation	✅
Logging	✅
Error Handling	✅
Systems and Environment	✅
Objects and Data Structures	✅
Readability and Maintainability	✅
Asynchronous Processing	✅
Design Patterns	✅
Third-Party Libraries	✅
Performance	✅
Security	✅
Functionality	✅

Feedback and Support

• Tell us what you think of Korbit
• Schedule a call with our team
• Email us @ [email protected]