-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cleanup old checkpoints from finalization state #1083
Conversation
The lint step failed in a weird way: I have a feeling this has something to do with the scripted diff merged in #1072 as the messages refers to a52d88e @cornelius Can you have a look? |
Concept ACK 0bc5e57 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ConceptACK 0bc5e57
I see it as a great quick win! This checkpoint design grows n^2, and with the low effort, we did it linearly. I have in mind to completely delete checkpoints as I don't see a need for them if we can look back for the previous finalization state, but that requires a significantly larger change/refactoring, and after this PR, it won't give us a drastic improvement so it can wait.
I'm looking at As a bare minimum I think we should change anyway the log messages in |
The weird lint failure should be fixed by #1084 now. |
Fixed/checked in 8fe4a24. |
This fix significantly reduces individual finalization state size. In a test, I run 1 proposer with 4 finalizers with configuration epoch_length=2. After processing ~9k blocks and ~4.5k epochs, finalization state size was 767599. After applying this PR, it has been reduced to 196002. More finalizers we have, more memory we save, because checkpoints map holds votes from every finalizer. Signed-off-by: Stanislav Frolov <[email protected]>
Signed-off-by: Stanislav Frolov <[email protected]>
8fe4a24
to
d751b58
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
except my last comment, utACK d751b58
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ConceptACK d751b58
Tested resource usage - significantly lowers memory usage.
Signed-off-by: Stanislav Frolov <[email protected]>
This fix significantly reduces individual finalization state size.
In a test, I run 1 proposer with 4 finalizers with configuration epoch_length=2.
After processing ~9k blocks and ~4.5k epochs, finalization state size was 767599.
After applying this PR, it has been reduced to 196002. More finalizers we have,
more memory we save, because checkpoints map holds votes from every finalizer.
Relates to #1073.