Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful shutdown #878

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

silvestrpredko
Copy link
Contributor

@silvestrpredko silvestrpredko commented Jun 13, 2024

Type of change

New feature (non-breaking change which adds functionality)

Checklist:

  • Formatted with cargo fmt
  • Make an entry to CHANGELOG.md if it's relevant to the users of the library. If it's not relevant mention why.

@silvestrpredko
Copy link
Contributor Author

Short description:
I wanna make a graceful shutdown without a significant redesign of the system. For me watch seems like a good solution. Please don't hesitate to suggest improvements to the initial design.

@silvestrpredko silvestrpredko force-pushed the silvestr/graceful-shutdown branch from 21c3949 to 588b90c Compare June 13, 2024 14:50
@swanandx
Copy link
Member

i also made a poc for graceful shutdown using cancellation token, you can check it here main...rumqttd-shutdown !

we can compare both approaches and combine good stuff from both :)

@coveralls
Copy link

coveralls commented Jun 13, 2024

Pull Request Test Coverage Report for Build 9501924124

Details

  • 0 of 71 (0.0%) changed or added relevant lines in 4 files are covered.
  • 4 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.1%) to 36.033%

Changes Missing Coverage Covered Lines Changed/Added Lines %
rumqttd/src/link/bridge.rs 0 1 0.0%
rumqttd/src/link/timer.rs 0 1 0.0%
rumqttd/src/link/console.rs 0 4 0.0%
rumqttd/src/server/broker.rs 0 65 0.0%
Files with Coverage Reduction New Missed Lines %
rumqttd/src/link/timer.rs 1 0.0%
rumqttd/src/server/broker.rs 3 0.0%
Totals Coverage Status
Change from base Build 9501879560: -0.1%
Covered Lines: 5970
Relevant Lines: 16568

💛 - Coveralls

@silvestrpredko
Copy link
Contributor Author

i also made a poc for graceful shutdown using cancellation token, you can check it here main...rumqttd-shutdown !

we can compare both approaches and combine good stuff from both :)

I like that handle returns in start. The reason why I was ignoring CancellationToken it's that tokio-utils is an optional dependency and I just don't want to change it. So it's more up to you what changes you would like to make, and what it should be used at the end. Please write below your decision.

@silvestrpredko silvestrpredko force-pushed the silvestr/graceful-shutdown branch from eddc878 to 40612fb Compare June 14, 2024 18:57
@coveralls
Copy link

coveralls commented Jun 14, 2024

Pull Request Test Coverage Report for Build 9521027502

Details

  • 0 of 73 (0.0%) changed or added relevant lines in 5 files are covered.
  • 5 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.09%) to 36.042%

Changes Missing Coverage Covered Lines Changed/Added Lines %
rumqttd/src/link/bridge.rs 0 1 0.0%
rumqttd/src/link/timer.rs 0 1 0.0%
rumqttd/src/link/console.rs 0 4 0.0%
rumqttd/src/main.rs 0 8 0.0%
rumqttd/src/server/broker.rs 0 59 0.0%
Files with Coverage Reduction New Missed Lines %
rumqttd/src/link/timer.rs 1 0.0%
rumqttd/src/main.rs 1 0.0%
rumqttd/src/server/broker.rs 3 0.0%
Totals Coverage Status
Change from base Build 9501879560: -0.09%
Covered Lines: 5970
Relevant Lines: 16564

💛 - Coveralls

@silvestrpredko silvestrpredko force-pushed the silvestr/graceful-shutdown branch from 40612fb to 3222431 Compare June 14, 2024 21:06
@silvestrpredko
Copy link
Contributor Author

Hey @swanandx. I redesigned the approach a little bit. It seems like a good solution. Please check it out.

P.S I also added join API call for situation when you don't want to handle a graceful shutdown

@coveralls
Copy link

coveralls commented Jun 14, 2024

Pull Request Test Coverage Report for Build 9522368042

Details

  • 0 of 80 (0.0%) changed or added relevant lines in 5 files are covered.
  • 5 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.1%) to 36.027%

Changes Missing Coverage Covered Lines Changed/Added Lines %
rumqttd/src/link/bridge.rs 0 1 0.0%
rumqttd/src/link/timer.rs 0 1 0.0%
rumqttd/src/link/console.rs 0 4 0.0%
rumqttd/src/main.rs 0 8 0.0%
rumqttd/src/server/broker.rs 0 66 0.0%
Files with Coverage Reduction New Missed Lines %
rumqttd/src/link/timer.rs 1 0.0%
rumqttd/src/main.rs 1 0.0%
rumqttd/src/server/broker.rs 3 0.0%
Totals Coverage Status
Change from base Build 9501879560: -0.1%
Covered Lines: 5970
Relevant Lines: 16571

💛 - Coveralls

@silvestrpredko
Copy link
Contributor Author

Hey, @swanandx do you have a chance to look at this PR?

  - Add a `shutdown_rx` parameter to the `start` function to receive shutdown signals.
  - Handle the shutdown signal in the main loop and break out cleanly.

- console.rs:
  - Add a `shutdown_rx` parameter to the `start` function to receive shutdown signals.
  - Use `axum::serve` with `with_graceful_shutdown` to handle shutdown signals.

- timer.rs:
  - Add a `shutdown_rx` parameter to the `start` function to receive shutdown signals.
  - Handle the shutdown signal in the main loop and break out cleanly.

- broker.rs:
  - Pass the `shutdown_rx` to various components like bridge, console, timer, and servers.
  - Handle shutdown signals in the server's main loop and break out cleanly.
  - Add a `ShutdownHandler` and `ShutdownDropGuard` for managing shutdown signals.

These changes allow for graceful shutdown of the rumqtt broker and its components, ensuring clean termination and resource cleanup when a shutdown signal is received.
@silvestrpredko silvestrpredko force-pushed the silvestr/graceful-shutdown branch from 3222431 to d39e149 Compare July 22, 2024 14:41
@coveralls
Copy link

Pull Request Test Coverage Report for Build 10043003235

Details

  • 0 of 80 (0.0%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on silvestr/graceful-shutdown at 36.033%

Changes Missing Coverage Covered Lines Changed/Added Lines %
rumqttd/src/link/bridge.rs 0 1 0.0%
rumqttd/src/link/timer.rs 0 1 0.0%
rumqttd/src/link/console.rs 0 4 0.0%
rumqttd/src/main.rs 0 8 0.0%
rumqttd/src/server/broker.rs 0 66 0.0%
Totals Coverage Status
Change from base Build 9675366490: 36.0%
Covered Lines: 5970
Relevant Lines: 16568

💛 - Coveralls

@silvestrpredko
Copy link
Contributor Author

@swanandx Just I kindly reminder. Could you please take a look?

@swanandx
Copy link
Member

@swanandx Just I kindly reminder. Could you please take a look?

Hey, I sincerely apologise for the delay, but I'm not having any bandwidth as of now or in near future. So this might be delayed 😕

@baxterjo
Copy link

baxterjo commented Dec 5, 2024

@swanandx Any updates on this? Is there another member that can take over the review?

#[tracing::instrument(skip(self))]
pub fn start(&mut self) -> Result<(), Error> {
pub fn start(self) -> Result<BrokerHandler, Error> {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would make me weary about breaking current functionality. Current start function does not return, so those currently using broker start and expecting it to go forever could potentially exit their programs prematurely with this change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, when I made this change I didn't think about breaking of current API. In my case, I only need something that will stop the broker when it goes out of execution scope(tests).

I think that it would be better to control how far execution goes by a handler, rather than spawn it and then block execution till it exits. Maybe could be implemented something like that https://github.com/tokio-rs/axum/blob/main/examples/graceful-shutdown/src/main.rs

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries. Sorry if I'm stepping on your toes but I submitted #929 last week so I could keep moving on my work. I think an ideal future state would be to make a new start_async() method for the Broker that would look lot what you've built here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem, I just hope that one of our PRs will be merged.

@baxterjo baxterjo mentioned this pull request Dec 5, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants