Delayed shutdown

[toini]

Hi! We are seeing some issues during deployments when the websocket clients connected to old instances are all closed at the same time. As typically the clients try to reconnect immediately (depending on how the community users have implemented their clients in different languages) this results in bursts to newer instances and they may not be able handle the connections and subscriptions before timeouts as there are external dependencies.

To counter that I'm planning to add some delays in the old instance shutdowns but as I've seen a lot of instructive discussions here I thought I'd ask for advice on best practices.

Actually, only after grabbing the code I wrote I realized this has nothing to do with the protocol implementation. This is anyhow where I'm landing at:

const graphqlWs = new WebSocketServer({
	noServer: true,
	path: SUBSCRIPTION_PATH
});

const closeClientsWithDelay = async (delayMs = 200) => {
	const count = graphqlWs.clients.size;
	logger.info(`[websocket] closing ${count} clients with ${delayMs} ms delay; this should take ${count * delayMs / 1000} seconds`);

	// Close a client after a delay
	const closeClientWithDelay = (client: WebSocket, delay: number) => {
		return new Promise((resolve) => {
			setTimeout(() => {
				logger.info(`[websocket] closing client`);
				// This also closes the async iterator which has a hook to call iot service to stop streaming
				client.close(1001, 'Server shutdown');
				resolve();
			}, delay);
		});
	};

	// Iterate over the clients and close them with a delay
	for (const client of graphqlWs.clients) {
		await closeClientWithDelay(client, delayMs);
	}

	graphqlWs.removeAllListeners();
	await new Promise<void>((resolve, reject) => {
		graphqlWs.close((err?: Error) => (err ? reject(err) : resolve()));
	});
};

Any suggestion on how to best handle this situation?

[enisdenjo]

Hey there, this is an interesting case! And I expected to have mitigated it already by defaulting the retry mechanism to randomised exponential backoff in graphql-ws clients to avoid the thundering herd problem. Are the graphql-ws clients posing a problem or is it mostly clients from different implementations?

[toini]

Yes, the clients we have control over do have an exponential backoff in place but unfortunately that's only a fraction of the clients. Majority of them reconnect immediately even though we advice against that in our API spec. So the issue is definitely not about the implementation in this library but maybe rather about how to build protection from other, possibly missbehaving, client implementations.

[enisdenjo]

I mean, there's not much you can do except reject the connections until the server is ready. The connections will come in regardless.

To reduce the load as much as possible, you can reject the connections on HTTP level instead of establishing an WS connection and then closing it. Basically if the incoming request is:

Connection: Upgrade
Upgrade: websocket

respond with a 5XX or whatever.

[toini]

I would not want to start rejecting clients attempting to connect (as they might not know how to react to 5XX or they might just retry immediately). Rather I can alleviate the issue by closing the clients at the old instance slower, so they also connect to new instances slower. Any problems you would spot at the code snippet above?

[enisdenjo]

The snippet looks good. You close the clients one by one after a delay so they should also connect to the new instance one by one - smart!

[toini]

Thanks for the input :)