ScheduledMerges: make the write buffer size configurable
#716
+421
−94
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jorisdral
force-pushed
the
jdral/restructure-prototypes
branch
from
3d24f3d to
e9f008f
Compare
jorisdral
force-pushed
the
jdral/prototypes-configurable-writebuffersize
branch
from
8c8c7ee to
ffc0607
Compare
jorisdral
force-pushed
the
jdral/restructure-prototypes
branch
from
e9f008f to
bb50a83
Compare
jorisdral
force-pushed
the
jdral/prototypes-configurable-writebuffersize
branch
2 times, most recently
from
5244165 to
32aa054
Compare
jorisdral
requested review from
dcoutts,
mheinzel,
recursion-ninja and
wenkokke
as code owners
|
I've put the PR back into draft mode because I found a bug |
jorisdral
force-pushed
the
jdral/prototypes-configurable-writebuffersize
branch
2 times, most recently
from
1b4f9f7 to
75a5d2b
Compare
…uffer size This config value is largely ignored for now, but it is passed to places where we'll need it in the next few commits.
jorisdral
force-pushed
the
jdral/prototypes-configurable-writebuffersize
branch
from
75a5d2b to
2b67d45
Compare
The new functions can compute the maximum run size for a tiering/levelling run given a level number, or conversely, can compute the level number of a tiering/levelling run given the run's size. These alternative calculations are parameterised by the maximum write buffer size, which is configured using the `configMaxWriteBufferSize` field of `LSMConfig`. As a sanity check for the run size calculations, some properties are added to the prototypes test suite. These alternative calculations are also closer to the real implementation of the merge schedule.
And remove the old calculations.
jorisdral
force-pushed
the
jdral/prototypes-configurable-writebuffersize
branch
from
2b67d45 to
44a8197
Compare
|
Ready for review again! |
And add a test, and immediately start using the alternative calculations in the prototype.
jorisdral
force-pushed
the
jdral/prototypes-configurable-writebuffersize
branch
from
44a8197 to
ad1e471
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We'd need to configure the write buffer size in the
ScheduledMergesprototype if we want to compare traces for the prototype and real implementation (see #445).This PR also changes the run size calculations to be somewhat closer to the calculations in the real implementation.
This PR builds on top of #715 to reduce the number of merge conflicts