-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[STORAGE USE REDUCTION] Commitlog segment compression #1592
Comments
Please keep in mind that closed commitlog segments may need to be accessed by tx offset, at least until the most recent snapshot. We maintain an offset index, mapping tx- to byte-offsets, for that purpose. So either we compress segments when moving them to cold storage (i.e. only those before a snapshot). |
It was decided that we can do this later in a non-breaking way by looking for either our magic number or the zstd magic number to determine whether this segment file is compressed or not. |
MVP / definition of done, as I see it:
|
|
I contend that we don't actually care, as long as replaying from the most recent snapshot is still fast. I am not aware of any other performance-constrained case in which we traverse commitlog segments. |
@gefjon replication will need to be able to randomly seek in segments, at least back to the latest snapshot. |
Ack. This is a significantly weaker constraint than the one you wrote originally. E.g. I believe we would accept a solution where replaying from or seeking within a compressed segment was slow, but where the segment(s) after the most recent snapshot were kept uncompressed and were therefore fast. |
No description provided.
The text was updated successfully, but these errors were encountered: