Demo Repo from session 5: GPT-4 with Vision

[multispark]

Demo Repo: https://github.com/mattgotteiner/AI-Chat-App-Hack-Vision
Video: https://www.youtube.com/watch?v=0g8BJwRRKAA

[zzulueta]

Is there a way to upload an image in the RAG Chat app and type a prompt as well to instruct the system to do a specific task? For example, you upload recipes in the system and then you upload images of what's in your pantry and prompt the system to suggest what recipe to cook.

[pamelafox]

You could try implementing that, it'd be a neat hack! It'd require a decent amount of custom code, as you need to add a file upload to the form, receive the binary data on the server, and then send the embedding of that image to Azure AI search. There may be React components to make the image upload easier. You could start by trying it in a notebook first, perhaps.
This notebook may also be helpful:
https://github.com/pamelafox/vector-search-demos/blob/main/image_search.ipynb

[zzulueta]

If my PDFs in the RAG Chat have text and images, does the existing python code (https://github.com/Azure-Samples/azure-search-openai-demo/) also get info on the images such as tags or description?

[pamelafox]

I believe it OCRs the images: https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept-read?view=doc-intel-4.0.0

That's been my experience. So if there is text in the images itself, it should grab it. I don't actually know if PDF images have hidden text (like alt text for images), I imagine it might grab that too as well.