Cancel pipeline. Doesn't work for request canlleations yet

[marius-baseten]

No description provided.

[squidarth]

I wonder if we can keep this sync, and use something like this to check for disconnection. ie:

from anyio import from_thread
def predict(self, model_input, request: fastapi.Request):
     is_disconnected = from_thread.run(request.is_disconnected)
     if is_disconnected:
         raise HTTPException(...)    

cancelling the ongoing inference is tricky though