Add streaming

[pamelafox]

Related to #26, customers would like streaming. Can use Chat Protocol SDK

[john0isaac]

I would like to work on that!

[pamelafox]

Awesome, I will assign it to you!

Since we're using the Chat Protocol SDK, you'll want to adhere the backend to that and then use the streaming method on the frontend. The azure-search-openai-demo already conforms to the protocol backend, so you can take a similar approach there, adding a /stream endpoint that returns json-lines.

The azure-search-openai-demo frontend doesn't yet use the Chat Protocol SDK in the main branch, but I did bring it in in a branch here:
Azure-Samples/azure-search-openai-demo@main...pamelafox:azure-search-openai-demo:chatprotocolsdk#diff-506debba46b93087dc46a916384e56392808bcc02a99d9291557f3e674d4ad6cR177

You can also compare that with the protocol SDK react sample here:
https://github.com/microsoft/ai-chat-protocol/tree/main/samples/frontend/js/react

However, azure-search-openai-demo implements a setTimeout that works better for token-by-token streaming than the sample in that repo.

Let me know if any questions along the way.

[pamelafox]

Oh, and confession: there are no true tests for this repo yet. A bit embarassed to admit that since you saw my workshop yesterday on testing! So you will need to manual testing, or be the first to add tests.

[john0isaac]

Thanks for the explanation! ❤️
I was following you on GitHub when you ported to chat protocol. So don't worry about that.

I finally managed to add streaming to my sample yesterday that's why I wanted to add it here too. Azure-Samples/Cosmic-Food-RAG-app#39

I didn't show up here earlier as I just figured this out yesterday.

I will work on some tests too.