Closing SSE transport during server shutdown #2635

vxgmichel · 2025-05-13T11:28:37Z

vxgmichel
May 13, 2025

I noticed that ASGI servers running an app with an active SSE connection will not shutdown when a ctrl+c is emitted.

This can easily reproduced using this fastapi+htmx app:

import asyncio
import itertools
from fastapi import FastAPI
from fastapi.responses import HTMLResponse, StreamingResponse

app = FastAPI()


@app.get("/", response_class=HTMLResponse)
async def index():
    return """
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>SSE ASGI tests</title>
  <script src="https://unpkg.com/[email protected]"></script>
  <script src="https://unpkg.com/[email protected]"></script>
  <link rel="stylesheet" href="https://cdn.simplecss.org/simple.min.css">
</head>
<body>
<a hx-ext="sse" sse-connect="/events" sse-swap="message">0</a>
</body>
</html>
"""


@app.get("/events")
async def event_source():

    async def event_generator():
        for x in itertools.count():
            yield f"data: {x}\n\n"
            await asyncio.sleep(1)

    return StreamingResponse(event_generator(), media_type="text/event-stream")

Simply run uvicorn app:app and browse to http://127.0.0.1:8000/, then ctrl+c the server and note that it hangs with:

INFO:     127.0.0.1:47008 - "GET /events HTTP/1.1" 200 OK
^CINFO:     Shutting down
INFO:     Waiting for connections to close. (CTRL+C to force quit)

Note that uvicorn is not the only ASGI server that behaves this way, I've tried Daphne, Hypercorn and Granian and they all behave the same. I also noticed that the sse-starlette had to patch the uvicorn Server.handle_exit during import to work around this problem. This breaks the separation between the ASGI app and server, which makes this solution not ideal. However, it's hard to think of a better solution since the ASGI specification does not provide any event for when the shutdown is required: the lifespan.shutdown event only occurs when all active connections have been closed.

In comparison, ASGI servers do close active websocket connections when an exit signal is received. Here's the WebSocketProtocol.shutdown method in uvicorn:

uvicorn/uvicorn/protocols/websockets/websockets_impl.py

Lines 147 to 153 in 66b9b58

    
           def shutdown(self) -> None: 
        
               self.ws_server.closing = True 
        
               if self.handshake_completed_event.is_set(): 
        
                   self.fail_connection(1012) 
        
               else: 
        
                   self.send_500_response() 
        
               self.transport.close()

The same thing could be implemented for SSE, discriminating for the text/event-stream content type specifically. For instance, this rough patch seems to do the job:

diff --git a/uvicorn/protocols/http/httptools_impl.py b/uvicorn/protocols/http/httptools_impl.py
index e8795ed..77f8f17 100644
--- a/uvicorn/protocols/http/httptools_impl.py
+++ b/uvicorn/protocols/http/httptools_impl.py
@@ -337,7 +337,7 @@ class HttpToolsProtocol(asyncio.Protocol):
         """
         Called by the server to commence a graceful shutdown.
         """
-        if self.cycle is None or self.cycle.response_complete:
+        if self.cycle is None or self.cycle.response_complete or (self.cycle.response_started and self.cycle.is_sse):
             self.transport.close()
         else:
             self.cycle.keep_alive = False
@@ -402,6 +402,7 @@ class RequestResponseCycle:
         self.response_complete = False
         self.chunked_encoding: bool | None = None
         self.expected_content_length = 0
+        self.is_sse = False
 
     # ASGI exception wrapper
     async def run_asgi(self, app: ASGI3Application) -> None:
@@ -502,6 +503,12 @@ class RequestResponseCycle:
                     self.keep_alive = False
                 content.extend([name, b": ", value, b"\r\n"])
 
+                # Handle special case for text/event-stream
+                if name == b"content-type" and value.lower().split(";")[0].strip() == b"text/event-stream":
+                    self.is_sse = True
+                else:
+                    self.is_sse = False
+
             if self.chunked_encoding is None and self.scope["method"] != "HEAD" and status_code not in (204, 304):
                 # Neither content-length nor transfer-encoding specified
                 self.chunked_encoding = True

I imagine that the difference between SSE and websockets is that websockets can be seen as a different protocol with its own rules, while SSE is merely a thin layer on top of long running HTTP response, which we might generally want to run to completion when the server shutdowns. In that sense, SSE would be an exception. Also, notice that the ASGI specification does mention websockets specifically, but doesn't mention SSE at all.

It seems like this is a situation where it's unclear whether it's the responsibility of the ASGI servers, frameworks or specification to take this problem into account. However I think it would be very valuable for the community to tackle this issue since SSE are still a popular solution for unidirectional event-based communication.

Kludex · 2025-05-13T11:46:01Z

Kludex
May 13, 2025
Maintainer

I think --timeout-graceful-shutdown works in this situation, right?

14 replies

nggit May 14, 2025

Following your discussion it seems your idea is against proper resource management.

up to the application framework to decide whether ongoing responses should terminate or not when the shutdown is requested.

Please clarify, do you want to make the shutdown not absolute? as a server, it should ensure the application terminates. The application must not say no.

I see this bug in Uvicorn and some (not all) implementations because they doesn't handle Ctrl-C perfectly.

vxgmichel May 15, 2025
Author

Please clarify, do you want to make the shutdown not absolute?

Sure, maybe this sentence was unclear. What I meant was that one possible solution could be to let the application framework decide which ongoing response should be interrupted and which one should run to completion when a graceful shutdown is requested.

The application must not say no.

Obviously if the graceful shutdown takes too long, the server is free to not wait for the ongoing requests to finish. But at least the application had the chance to stop the requests that can be interrupted, such as those corresponding to SSE.

Also note that this is only one possible solution I could think of, the other one being to let the servers close all the SSE connections when a shutdown is requested, similar to what is done for websckets. But I think it has to be one those: it's either the server or the framework responsibility.

nggit May 16, 2025

Sorry if I misinterpreted your sentences, so do you really expect the app to know the "graceful" shutdown event (before the connection is lost)?. I believe this method is not legit and it didn't even cross my mind.

close all the SSE connections when a shutdown is requested, similar to what is done for websckets

Note that for WebSocket behavior is likely specific to Uvicorn or others. This is because it's actually not graceful under the hood (as you seem to know). I do this gracefully on WebSocket as well as SSE, in my another ASGI server implementation.

vxgmichel May 16, 2025
Author

do you really expect the app to know the "graceful" shutdown event (before the connection is lost)?. I believe this method is not legit and it didn't even cross my mind.

Yes, sse-starlette actually does something along those lines. The problem is that since there is no dedicated event to get notified that the shutdown is requested, sse-starlette has to patch the handle_exit method of the uvicorn server, which ties it to a specific ASGI server implementation.

nggit May 16, 2025

Well, I tend not to prefer such hackish practices,

but also on the other hand I see this as unnatural to fix. Adding http.shutdown.requested in the middle of receive() seems redundant. but I'm not against it. Maybe you can raise the issue on asgiref to see what others think.

nggit · 2025-05-13T12:42:36Z

nggit
May 13, 2025

This is not SSE specific. This is because Uvicorn fails to shutdown while the client is still connected.

Here is a simple app that is slow to send the body to the client:

import asyncio

async def app(scope, receive, send):
    assert scope['type'] == 'http'

    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [
            (b'content-type', b'text/plain'),
        ],
    })
    await asyncio.sleep(1e9) # can be forever
    await send({
        'type': 'http.response.body',
        'body': b'Hello, world!',
    })

A second Ctrl-C would do a "force-quit", but that doesn't work. But the worst thing is that there is no timeout on the handler. It's a DoS vulnerability.

2 replies

vxgmichel May 13, 2025
Author

I thought this was more a feature than a bug in the case of standard HTTP requests, since the shutdown workflow is:

stop accepting new connections
wait for pending requests to finish
send the lifespan.shutdown event
wait for the lifespan.shutdown.complete or lifespan.shutdown.failed event
stop the server

I think it makes sense in general to let the requests terminate, I see the SSE as the exception rather than the norm in that regard.

nggit May 13, 2025

"stop accepting new connections" has a downside, if all workers "wait for pending requests to finish" then the website is basically inaccessible forever. due to consider "let the requests terminate" is a feature.

gi0baro · 2025-05-15T06:35:41Z

gi0baro
May 15, 2025

Granian maintainer here.
The point is that a server needs to decide whether:

gracefully shutdown
forcefully shutdown

SSE just exposes this behaviour, but from my experience forcefully close every connection on ctrl+c/signals is worse. So that's why every server waits for existing connection to shut down and avoid to abruptly close them.

4 replies

nggit May 15, 2025

But there should be a timeout right, or pressing the second Ctrl-C to force.

gi0baro May 15, 2025

Can't speak for uvicorn, but in Granian that's exactly what --workers-kill-timeout does.

nggit May 15, 2025

It should be, if you try my example app here only Uvicorn and Hypercorn are not correct. This implies there is a hidden problem that must be addressed first.

Edit:
Uvicorn has --timeout-graceful-shutdown and strangely enough without it means no timeout at all. But that doesn't take away from the fact that "Ctrl+C to force quit" doesn't work as promised (bug).

Hypercorn has --graceful-timeout, but does not react at all to Ctrl-C (bug).

vxgmichel May 15, 2025
Author

but from my experience forcefully close every connection on ctrl+c/signals is worse.

I agree with that. My point is that at the moment, websocket connections do get closed during a graceful shutdown, while SSE connections do not. I understand the reasons for that: websocket is a different protocol, while SSE is just built on top of HTTP responses, which are typically not interrupted during a graceful shutdown. This leads to this weird situation where most developers are surprised when their server hangs, waiting for the SSE connections to terminate in order to gracefully shutdown.

Uh oh!

Closing SSE transport during server shutdown #2635

Uh oh!

Uh oh!

Replies: 3 comments · 20 replies

Uh oh!

Kludex May 13, 2025 Maintainer

Uh oh!

Uh oh!

vxgmichel May 15, 2025 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vxgmichel May 16, 2025 Author

Uh oh!

Uh oh!

Uh oh!

vxgmichel May 13, 2025 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vxgmichel May 15, 2025 Author

Replies: 3 comments 20 replies

Kludex
May 13, 2025
Maintainer

vxgmichel May 15, 2025
Author

vxgmichel May 16, 2025
Author

vxgmichel May 13, 2025
Author

vxgmichel May 15, 2025
Author