rptest: add log_compaction_test.py
#24187
Conversation
WillemKauf
force-pushed
the
tombstones_testing
branch
from
2703ac2
to
3f68c4a
Compare
WillemKauf
force-pushed
the
tombstones_testing
branch
2 times, most recently
from
def5f7d
to
fc193af
Compare
WillemKauf
force-pushed
the
tombstones_testing
branch
from
fc193af
to
379a1e1
Compare
WillemKauf
force-pushed
the
tombstones_testing
branch
from
379a1e1
to
119dd18
Compare
WillemKauf
force-pushed
the
tombstones_testing
branch
from
119dd18
to
8243701
Compare
| assert consumer.consumer_status.validator.tombstones_consumed > 0 | ||
| assert consumer.consumer_status.validator.invalid_reads == 0 | ||
|
|
||
| def part_two(self): |
There was a problem hiding this comment.
nit: part_one/part_two are not the most convenient names for future readers
There was a problem hiding this comment.
Renamed these functions.
|
|
||
| # Sleep until the log has definitely been fully compacted. | ||
| timeout = self.extra_rp_conf['log_compaction_interval_ms'] / 1000 * 10 | ||
| time.sleep(timeout) |
There was a problem hiding this comment.
I think it doesn't guarantee that the log was compacted. It's better to read a metric to validate that compaction happened.
There was a problem hiding this comment.
I have added a number of metrics to the storage::probe that are now used in this test case, and will be of future use for compaction observability:
_segment_cleanly_compacted, the number of segments that have been cleanly compacted (i.e, had their keys de-duplicated with all previous segments before them to the front of the log)._segments_marked_tombstone_free, the number of segments that have beenverified through the compaction process to be tombstone free._num_rounds_window_compaction, the number of rounds of sliding window compaction that have been driven to completion.
I have removed this sleep() in place of a deterministic check that compaction has settled using some of these metrics.
| consumer.wait(timeout_sec=60) | ||
|
|
||
| partition_move_thread.join() | ||
| self._stop_stress_fibers() |
There was a problem hiding this comment.
why stress is needed only here?
There was a problem hiding this comment.
Moved stress fiber to run continuously throughout the test.
A metric that measures the number of segments that have been cleanly compacted (i.e, had their keys de-duplicated with all previous segments before them to the front of the log).
A metric that measures the number of segments that have been verified through the compaction process to be tombstone free.
A metric that measures the number of rounds of sliding window compaction that have been driven to completion.
See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted`, `--validate-latest-values` as input parameters.
WillemKauf
force-pushed
the
tombstones_testing
branch
from
8243701
to
6df7e4b
Compare
|
the below tests from https://buildkite.com/redpanda/redpanda/builds/58820#019369b8-b74d-4c4f-9b4a-3da1ca75260f have failed and will be retried |
WillemKauf
force-pushed
the
tombstones_testing
branch
from
6df7e4b
to
43595ba
Compare
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58843#01936ab4-c4e4-4370-95d8-dc5827a7ec9e |
|
non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58843#01936ace-5008-4e64-98c4-5d82dd412623: |
Retry command for Build#58843please wait until all jobs are finished before running the slash command |
Serves as a stress test for compaction and tombstone removal.
Depends on changes made in redpanda-data/kgo-verifier#60, and so the commit SHA for
ducktape-deps/kgo-verifierhas been updated (CI will be red if this isn't merged yet)Backports Required
Release Notes