Allow add middleware after app has started #16758
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
allow add middleware after app has started
this should completely fix RuntimeError("Cannot add middleware after an application has started")
which can occur due to a race condition
explanation of race condition
By design
Starlette / FastAPI
middleware can only be added before the the server has started, technically it can still be added until the server receive a first request, at which point it builds theapp.middleware_stack
after this point you normally cannot add middleware
BUT we do add and remove middlewares after the app is launched
stable-diffusion-webui/webui.py
Line 79 in 82a973c
stable-diffusion-webui/webui.py
Lines 104 to 112 in 82a973c
stable-diffusion-webui/modules/initialize_util.py
Lines 192 to 198 in 82a973c
in this section of code we first remove the gradio default
CORSMiddleware
then added our ownCORSMiddleware
with different config along withGZipMiddleware
and after this if
--api
is enabled we also add some more middlewaresstable-diffusion-webui/modules/api/api.py
Lines 135 to 196 in 82a973c
we are able to add new middlewares because in
setup_middleware()
we setapp.middleware_stack
toNone
, which would makeStarlette / FastAPI
rebuild theapp.middleware_stack
on the next recived requestIn most cases this works fine, but if one is unlucky enough it is totally possible for the server to receive a request after we set
app.middleware_stack
toNone
and between the last middleware is addedif this were to happen it will trigger a runtime error which would then crash the server
reproducing the race condition
key point
if ther is a request between
app.middleware_stack = None
but andadd_middleware()
then the issue will be triggerFor testing purposes this issue can be manually triggered by placeing a breakpoin in
setup_middleware()
afterapp.middleware_stack = None
but before anyapp.add_middleware
for example on
stable-diffusion-webui/modules/initialize_util.py
Line 196 in 82a973c
When the breakpoint is hit, make a request the server (visit the webui webpage or make any request on any endpoint
or perform any request on any API route)
after request continued execution
you should see
RuntimeError("Cannot add middleware after an application has started")
been triggerthis is because even though we have set
app.middleware_stack = None
if a request has been made to the server it will rebuildmiddleware_stack
, makeingmiddleware_stack
no longer None, triggeing the errorIs this an issue that needs addressing?
Yes
well this doesn't happen often, it does happen frequent enough that we have seen multiple report of this issue
also remember seeing some reports of the server dying soon after launch, this may be the cause of those issues,
when runtime error is triggered the server only shutsdown after the model is loaded, resulting in a delay between the runtime error and the actual shutdown
browser auto launch after app start, would significantly increase the chance of this happening
as it essentially is sending lots of requests to the serve at around the same time as we add new middleare
The Fix
the fix I came up with is to patch
Starlette.add_middleware
, adding some logic that ensures themiddleware_stack
isNone
when we add new middlewaresthe resulting of this is that I believe it is now impossible for it to trigger a RuntimeError due to this issue
this even makes it possible to add new middleware at any time
Remarks
while setting
middleware_stack
toNone
makeing it rebuild the stack is a but unorthodox so could be considered dangerous and could break in the futureBUT, but considering we have been doing this operation for a long time and it has worked I think it's good enough
The only difference that this PR does is add better logic in performing this operation
Checklist: