Breaks the cycle where reporting low memory can trigger more reporting #1000
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reporting low memory via log_warn() consumes some heap. A decrease in heap can trigger another low memory message. This can run away and produce a burst of low memory messages that consume a lot more memory than necessary.
This causes the overall system to be more sensitive to low memory conditions because the warnings can chew up the last bit of memory at the worst time.
Through separate testing I found that channel->write() on a websocket temporarily consumes about 1.5k. Writing to a uart is much less.
This PR keeps track of a separate "true" memory low watermark and "last-reported" low watermark and delays (200 ms) reporting if the true low watermark is not too much (< 2k) below the last reported low watermark. This breaks the runaway cycle of messages triggering more low-watermark events. The true low watermark is still reported eventually (up to 200ms late), and larger drops in heap are not subject to the delay, in case the system is about to become nonresponsive.
I tested this with a modified build that deliberately leaks memory on command. With the system idle, depleting the memory 1k at a time eventually triggered a long burst of Low memory warnings that consumed the rest of the memory. With the delay in place to break the cycle, only two messages were generated and very little extra memory was consumed by the messages.