OpenAI endpoints don't support canceling request

[Tostino]

Using the latest development branch, I noticed in testing the OpenAI endpoints (chat_completion) that there is an issue with canceling the request if the user cancels the query for example.

If a user kills their query, the inference request is never interrupted and the inference continues unabated and further requests to the API seem to just sit there forever.

Edit: After more research, the issue is related to: https://www.postgresql.org/message-id/flat/CAFYwGJ3%2BXg7EcL2nU-MxX6p%2BO6c895Pm3mYZ-b%2B9n9DffEh5MQ%40mail.gmail.com

I saw there was another thread a couple years later, but it seems like nothing ever was done: https://www.postgresql.org/message-id/flat/CAB0yrek0C9ioVoaTiPUXy4GKzHxuf8aSc3%2BkrCCSK_Emxwf4nQ%40mail.gmail.com

So, at this point I am researching ways to use the async client to get the desired behavior:

1. It will cancel the request on the inference server when the user requests to cancel their query,
2. It will react quickly to the users request to cancel their query.

Will update when I have more info.

[Tostino]

Alright, have something working finally: Tostino@6163939

This moves things to use the async client and handles checking for cancelled queries.

This also properly supports timeout now.

The behavior is much better when things aren't on the "happy path" as far as my testing goes.

[alejandrodnm]

Hey @Tostino thanks for this. As I said in my previous message, we are finishing a new big feature. Once we finish the release (we are almost done) we are going to look into your issues.

[alejandrodnm]

Hey @Tostino , can you open a PR with your changes? That way it'll be easier to iterate over it. Thanks :)

[Tostino]

Yeah, I need to rebase on the main branch first and then i'll open a PR.