Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: graceful shutdown #204

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: graceful shutdown #204

wants to merge 3 commits into from

Conversation

Nabil-Salah
Copy link
Contributor

Description

added graceful shutdown

Changes

src/relay/mod.rs, src/relay/federation/mod.rs, src/events/mod.rs files

Related Issues

issue#203

Checklist

  • Tests included
  • Build pass
  • Documentation
  • Code format and docstring

@Nabil-Salah Nabil-Salah marked this pull request as ready for review December 12, 2024 11:04
select! {
_ = signal::ctrl_c() => {
log::info!("shutting down relay gracefully");
break;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to close the listener here?
something like

tcp_lister.close()...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tokio tcp_listener doesn't implement close or shutdown function to terminate the listener.
but using this tokio select simultaneously awaits for both the two actions and whenever one hits first it goes with it
So once the Ctrl+C signal is received, the listener stops accepting new TCP connections, and the application exits the loop

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So once the Ctrl+C signal is received, the listener stops accepting new TCP connections, and the application exits the loop

Yes, but this is not graceful. All in-flight requests will be forcefully closed as well.

There are two alternatives i can think of:

  1. see the example at tokio mini-redis https://github.com/tokio-rs/mini-redis/blob/e186482ca00f8d884ddcbe20417f3654d03315a4/src/server.rs#L172-L197

  2. If client can handle it, then it is probably OK to not shutdown it gracefully.

@iwanbk
Copy link
Member

iwanbk commented Dec 17, 2024

i wonder why there are a lot of clippy warning but the clippy github actions still green.

Copy link
Member

@sameh-farouk sameh-farouk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @Nabil-Salah. This is a good start, but I have highlighted some important issues in this PR below:

  • Multiple signal handlers

    • Having multiple signal handlers (listening for the same signal in multiple tasks) can lead to different issues, redundant handling, and unexpected behavior, such only one of the tasks will "see" the signal (whichever consumes it first), and the others will be left waiting).
      It is generally recommended that a single, centralized signal handler (in src/bins/rmb-relay.rs) coordinate the shutdown process. A clean approach would be to use cancellation tokens to signal different tasks to shut down gracefully.
  • Handling SIGTERM and SIGINT

    • It is important to handle both SIGTERM and SIGINT signals for graceful shutdowns.

      • SIGTERM is the standard signal used to terminate a process (e.g., from kill commands or system shutdowns).
      • SIGINT is triggered by the user pressing Ctrl+C. (the only one you are covering here)

Handling both ensures the application can shut down cleanly in various scenarios.

  • Avoiding std::process::exit()
    • Using std::process::exit() for shutdown is generally discouraged because it abruptly terminates the process without running cleanup logic and executing Drop trait implementations.
    • Instead, it is preferable to allow the application to exit naturally, ensuring all cleanup tasks are performed properly. If you need to return a specific numeric value to indicate the exit status, consider using ExitCode as the return type for the main function.

Lastly, from a functionality perspective, while it requires careful handling since it wasn't initially implemented during the code planning, we should ensure that in-flight messages (received via a connected WebSocket or the federation HTTP endpoint, but not yet forwarded to the destination) are properly processed before tasks return upon receiving the shutdown notification. This can be challenging and demands a thorough understanding of the full relay messaging flow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants