How to enable Vision enhancements for Azure Open AI?

[kishorek]

How can we enable Vision enhancements for Azure Open AI GPT 4 Vision preview? It is suggested to enable it for enhancing the OCR - https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision#use-vision-enhancement-with-images

The below parameters need to be added to the payload for chat/completions:

"enhancements": {"ocr": {"enabled": true}, "grounding": {"enabled": false}},
    "dataSources": [
        {
            "type": "AzureComputerVision",
            "parameters": {
                "endpoint": "https://<endpoint>.cognitiveservices.azure.com",
                "key": "***",
            },
        }
    ]

[ishaan-jaff]

@kishorek tracking this here: #1438

[ishaan-jaff]

Here's how (live on 1.17.6+): https://docs.litellm.ai/docs/providers/azure#usage---with-azure-vision-enhancements

import os 
from litellm import completion

os.environ["AZURE_API_KEY"] = "your-api-key"

# azure call
response = completion(
    model = "azure/<your deployment name>", 
    messages=[
        {
            "role": "user",
            "content": [
                            {
                                "type": "text",
                                "text": "What’s in this image?"
                            },
                            {
                                "type": "image_url",
                                "image_url": {
                                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                                }
                            }
                        ]
        }
    ],
    enhancements = {"ocr": {"enabled": True}, "grounding": {"enabled": True}},
    dataSources = [
            {
                "type": "AzureComputerVision",
                "parameters": {
                    "endpoint": "<your_computer_vision_endpoint>",
                    "key": "<your_computer_vision_key>",
                },
            }
    ],
)