Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaks the cycle where reporting low memory can trigger more reporting #1000

Merged
merged 1 commit into from
Aug 15, 2023

Conversation

vector76
Copy link
Contributor

Reporting low memory via log_warn() consumes some heap. A decrease in heap can trigger another low memory message. This can run away and produce a burst of low memory messages that consume a lot more memory than necessary.

This causes the overall system to be more sensitive to low memory conditions because the warnings can chew up the last bit of memory at the worst time.

Through separate testing I found that channel->write() on a websocket temporarily consumes about 1.5k. Writing to a uart is much less.

This PR keeps track of a separate "true" memory low watermark and "last-reported" low watermark and delays (200 ms) reporting if the true low watermark is not too much (< 2k) below the last reported low watermark. This breaks the runaway cycle of messages triggering more low-watermark events. The true low watermark is still reported eventually (up to 200ms late), and larger drops in heap are not subject to the delay, in case the system is about to become nonresponsive.

I tested this with a modified build that deliberately leaks memory on command. With the system idle, depleting the memory 1k at a time eventually triggered a long burst of Low memory warnings that consumed the rest of the memory. With the delay in place to break the cycle, only two messages were generated and very little extra memory was consumed by the messages.

@MitchBradley MitchBradley changed the base branch from main to Devt August 15, 2023 16:48
@MitchBradley MitchBradley merged commit b18497f into bdring:Devt Aug 15, 2023
12 checks passed
@vector76 vector76 deleted the low_mem_feedback_cycle branch August 15, 2023 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants