Add transactions to the many_partitions scale test #7588
Conversation
| raise | ||
| # non-txn background traffic | ||
| repeater_msg_size = 16384 | ||
| with repeater_traffic(context=self._ctx, |
There was a problem hiding this comment.
It's a bit odd to pick out the restart stress portion of the test and only run that bit with transactions.
The approach for compaction is to have a top level set of repeater_kwargs, and two different test cases (compacted vs. non) -- can we do the same for transactions? Maybe rather than each feature independently, we should have just two variants of the test: vanilla (no compaction or transactions) and "fully loaded" (compaction, transactions enabled).
I think that'll be useful in identifying which failures result from issues in the primary data path, vs. which ones relate to particular optional features.
|
Not sure where you're running this in development, but would be good to try it on i3en.6xlarge instances. The nightly runs are on i3en.xlarge for cost, but that's only ~10k partitions, the i3en.6xlarge instances will get you up to the When you try the version with+without transactions, it may be that we get an interesting result in terms of what the hard partition limit can be with transactions on vs. off. |
This PR extends the many_partitions test to use transactions in background traffic during a restart stress test. Not sure if this is the best place to insert transactions into our scale tests. However, I thought it'd be a good starting place for a discussion about where adding in transactions would be appropriate.
This PR requires redpanda-data/kgo-verifier#16 to be merged for the changed test to pass.
Backports Required
UX Changes
N/a
Release Notes
N/a