Kubernetes integration and graceful shutdown

[Askaholic]

Adds a graceful shutdown mode where the server will stop allowing new games to be created and wait until all active games have completed before shutting down normally. The grace period time before a hard shutdown happens can be configured or short circuited by sending a second SIGTERM or SIGINT to the process.

Also adds a kubernetes health check server on port 2000 by default that will ensure kubernetes waits long enough for the server to start before sending over new traffic.

Closes #469

[p4block]

This is great stuff, reminds me of how LiveKit works.

terminationGracePeriodSeconds to 5 hours in the helm chart, ensuring Kubernetes gives sufficient time for the pod to gracefully shut down.

Extremely doable :D

[Askaholic]

Could rename the field to something else. Maybe request_command?

[Askaholic]

Looks like the queue pop time delta becomes negative and the pop_time being in the past causes the client UI to bug out a little bit. We should probably handle that on the server and just send a 0 or null or something.

{'command': 'matchmaker_info', 'queues': [{'queue_name': 'ladder1v1', 'queue_pop_time': '2023-04-16T22:13:07.857983+00:00', 'queue_pop_time_delta': -1249.9953863620758, 'num_players': 0, 'boundary_80s': [], 'boundary_75s': [], 'team_size': 1}, {'queue_name': 'tmm2v2', 'queue_pop_time': '2023-04-16T22:13:07.858472+00:00', 'queue_pop_time_delta': -1249.9949333667755, 'num_players': 0, 'boundary_80s': [], 'boundary_75s': [], 'team_size': 2}, {'queue_name': 'tmm4v4_full_share', 'queue_pop_time': '2023-04-16T22:13:07.828933+00:00', 'queue_pop_time_delta': -1250.0244863033295, 'num_players': 0, 'boundary_80s': [], 'boundary_75s': [], 'team_size': 4}, {'queue_name': 'tmm4v4', 'queue_pop_time': '2023-04-16T22:13:07.831161+00:00', 'queue_pop_time_delta': -1250.0222806930542, 'num_players': 0, 'boundary_80s': [], 'boundary_75s': [], 'team_size': 4}]}

We might want the matchmaker_info message to actually return an empty list when the queues are disabled.