Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a log message when the logger enters burst protection mode #8852

Open
PragTob opened this issue Sep 24, 2024 · 2 comments
Open

Add a log message when the logger enters burst protection mode #8852

PragTob opened this issue Sep 24, 2024 · 2 comments
Assignees
Labels
enhancement help wanted Issue not worked on by OTP; help wanted from the community team:VM Assigned to OTP team VM

Comments

@PragTob
Copy link
Contributor

PragTob commented Sep 24, 2024

Hi, it's me - logger documentation PR barrage guy 😅

Is your feature request related to a problem? Please describe.
I recently spent a lot of time trying to debug why our system had seemingly swallowed log messages. First of course I thought the application was broken for real.
Anyhow, after reading the logger docs I figured we'd likely either hit :drop or :flush mode or maybe even the burst mode protection. I figured we hit drop mode (see #8849 ), but I was unsure if we had hit "burst protection mode". There is no log message that I'm aware of (checked the code) or any other way I could directly know that burst protection mode was turned on. So, I turned it off and hoped that this was it - but having the certainty before would have been good. As well as a way that I could have seen it while perusing the logs.

Describe the solution you'd like
I'd like there to be a log emitted when burst mode protection is entered. I know it's one more log event to emit in an already high stress situation for the logger but I think the tradeoff is worth it.

I think it's also in line with a log message being emitted when :drop mode is entered.

Something like: Logger :default is entering burst protecting mode, dropping logs

Describe alternatives you've considered

  • keep as is - possible, but I think this is better for the reasons above
  • other notification mechanism - maybe a log message isn't the right mechanism? I have thought of telemetry but I don't think we want OTP to depend on it. We might also just set a piece of state in the logger itself so that you could ask the logger like "hey, have you encountered burst mode?" and maybe it'd be visible in observer. Unsure where we'd do that, though. I also think that logging that information is more natural - as when you're debugging the logs you are likely already in the logs and so you might stumble upon that message, helping you. Like, I'm sure I went through more than 500 log messages and so encountering that message would have immensely helped me figuring out what's wrong.

Additional context
Thank you for all your work on erlang and the new logger 💚

@IngelaAndin IngelaAndin added the team:VM Assigned to OTP team VM label Sep 24, 2024
@PragTob
Copy link
Contributor Author

PragTob commented Sep 25, 2024

If we agree on a way to tackle this and it's a log message I'm confident I can also send a PR despite not being the best erlang programmer myself :) I have already read the burst limiting code in search for a log message and think I know where it should be emitted.

@lucioleKi lucioleKi added the help wanted Issue not worked on by OTP; help wanted from the community label Oct 2, 2024
@jhogberg
Copy link
Contributor

Sorry for the late reply! Yes, we think this is a good idea. Feel free to make a PR :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement help wanted Issue not worked on by OTP; help wanted from the community team:VM Assigned to OTP team VM
Projects
None yet
Development

No branches or pull requests

4 participants