Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature request]: Indicate time of possible circuit closure in BrokenCircuitException #2284

Closed
DL444 opened this issue Sep 10, 2024 · 5 comments · Fixed by #2324
Closed

Comments

@DL444
Copy link
Contributor

DL444 commented Sep 10, 2024

Is your feature request related to a specific problem? Or an existing feature?

Related to #506.

Imagine a worker service pulling messages from a queue (e.g. Azure Service Bus) and issuing circuit-breaker protected calls to downstream HTTP endpoints. If the downstream services go down and the circuit breaks, the worker has no easy way to know when the circuit breaker would possibly become closed again.

#506 recommends applications to stop processing messages for some time, and I would agree. The problem is "for how long". Knowing when the circuit breaker is likely still open without relying on actually performing and failing the request is important for message queue scenarios, because excessive retries and failures can quickly exhaust message delivery limit and deadletter the messages unnecessarily.

Applications can surely take a guess, but it still needs this piece of information to make an informed decision on how long to cease message processing. It wouldn't help if the circuit breaker is configured to break for 2 days and the application blindly guesses 2 hours.

Describe the solution you'd like

Indicate the time when the circuit breaker can possibly become closed again in BrokenCircuitException. Possible API:

  public class BrokenCircuitException : ExecutionRejectedException
  {
+     public TimeSpan? RetryAfter { get; }
  }

This matches RetryAfter from RateLimiterRejectedException. Pulling it up to ExecutionRejectedException is not advisable because it does not apply to all derived exception types.

For IsolatedCircuitException, set this to null.

Additional context

No response

@martintmk
Copy link
Contributor

@DL444

I think this would be great addition and relatively straightforward to implement! Are you willing tho give this a shot?

Basically, you just need to update this code:

private BrokenCircuitException CreateBrokenCircuitException() => _breakingException switch

And to retrieve the value until the CB is blocked you can use:

@DL444
Copy link
Contributor Author

DL444 commented Sep 11, 2024

Are you willing tho give this a shot

Yes, I would like to try that this week.

@peter-csala
Copy link
Contributor

Are you willing tho give this a shot

Yes, I would like to try that this week.

Hey if you need some help, please let us know.

@DL444
Copy link
Contributor Author

DL444 commented Sep 23, 2024

Hey if you need some help, please let us know.

Not at this time, I've just been busy. I actually already have a working implementation here, but still need some time to sort out testing.

Copy link
Contributor

Thanks for creating this issue @DL444 - the associated changes have been published as part of version 8.5.0 📦, which is now available from NuGet.org 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants