Rate limited response error causes infinite loop

[jborichevskiy]

I am seeing an infinite notification loop happen when hitting any exception reported by Honeybadger while having exceeded our account rate limit.

Upon reporting, the first response is [warn] [Honeybadger] API failure: "{\"error\":\"Rate Limit Exceeded\"}. It looks like this failure is then being caught at client.ex:162, but then Logger.error is called - sending this new notification and continuing the loop. This continues until the server is restarted.

Proposed fixes

Switching the calls from Logger.error to Logger.warn stops the loop from starting. I'm happy to submit a PR with this but I'm not sure this is the ideal fix for everyone.

An alternative might be to use @sezaru's exclude_error config option PR #342 but it would need to be merged in first.

Open to other solutions or ideas here! Thanks.

Version info

Using honeybadger 0.16.0.

Reproduced on multiple systems (Elixir 1.11.0-dev (82dca55) (compiled with Erlang/OTP 23) on MacOS 10.15.7) and (elixir 1.10.2 / Erlang 22.2 on Ubuntu 18.04.5 LTS).

Front conversations

[joshuap]

@jborichevskiy sorry for the trouble, and thanks for the offer to help! Email [email protected] if you need us to temporarily increase your monthly error quota as a result of this.

@sorentwo what solution do you like best?

[sorentwo]

@joshuap I think we should use Logger.warn for this in the immediate term.

[joshuap]

I think we should use Logger.warn for this in the immediate term.

Sounds good, thanks! @jborichevskiy would you like to submit that PR?

[jborichevskiy]

I think we should use Logger.warn for this in the immediate term.

Sounds good, thanks! @jborichevskiy would you like to submit that PR?

Sure thing - will do in a few!

[jborichevskiy]

@joshuap I made an oops - it is already Logger.warn; I must've lost track of my local changes when submitting the issue; sorry for the confusion.

Logger.info also seems to be triggering the loop. I'll dig around some more and follow up!

[zmagajna]

Hi, any update on that? Having the same issue with this.

[jborichevskiy]

Hi, any update on that? Having the same issue with this.

Apologies - been sidetracked with some other work. Haven't figured out a fix here yet but will update when I do.

[joshuap]

We had another user report something similar—they received thousands of "Honeybadger rate limit exceeded" errors.

I'm looking to get this wrapped up in a few weeks.

[joshuap]

From @rabidpraxis:

We could add a :honeybadger domain to each internal logger call and then include that domain to the list of ignored ones https://github.com/honeybadger-io/honeybadger-elixir/blob/master/test/honeybadger/logger_test.exs#L156-L163