Datalake in Random Node Operations test #24179

mmaslankaprv · 2024-11-19T13:28:28Z

Enabling Iceberg in Random Node Operations test to validate its correctness in face of failures and node operations.

Backports Required

Release Notes

none

mmaslankaprv · 2024-11-19T14:01:12Z

/dt

mmaslankaprv · 2024-11-20T09:25:36Z

/dt

vbotbuildovich · 2024-11-20T13:08:35Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58327#01934945-6ead-458a-ba3d-adba5f914e44
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58327#01934945-6eb2-4f74-9a10-550455d20848
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58552#019353bc-a1d1-4417-9900-2757680edc34
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58677#01936308-8324-4009-9483-c3648fad5caf
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58699#01936456-fa32-4b79-b47b-c8dbce237408
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58699#01936456-fa37-49b9-8236-c55d79db293c

bharathv · 2024-11-20T18:19:49Z

tests/rptest/tests/random_node_operations_test.py

+            cloud_storage_bucket=self._si_settings.cloud_storage_bucket,
+            cloud_storage_access_key=self._si_settings.
+            cloud_storage_access_key,
+            cloud_storage_secret_key=self._si_settings.
+            cloud_storage_secret_key,
+            cloud_storage_region=self._si_settings.cloud_storage_region,
+            cloud_storage_api_endpoint=self._si_settings.endpoint_url,
+            filesystem_wrapper_mode=False)


probably needs a rebase ( NV just merged something that affects this)

vbotbuildovich · 2024-11-21T10:41:32Z

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58436#01934df0-96ea-4560-ba1b-870db0bfc356:

"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_license_upload_and_query"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58436#01934df0-96e8-41bd-8f4b-6ee6b32e663a:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=False.with_tiered_storage=False.with_iceberg=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58552#019353c1-7090-41da-8ba7-f1e471cd82d8:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=False.with_iceberg=True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=False.with_iceberg=True.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58552#019353c1-708f-4c06-9d59-8837cd3d0c6f:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58552#019353c1-7092-4e64-a734-d79e9a173300:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58552#019353c1-708c-48c5-ab8b-3ed16be62d0c:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=True.with_iceberg=True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=True.with_iceberg=True.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58677#01936308-8321-488b-98e5-4dab01961c6b:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58677#01936308-8324-4009-9483-c3648fad5caf:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58699#01936471-e21e-41e2-8250-ff6d3116c7d3:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58699#01936471-e21f-4439-95f0-0b5dee690c4d:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=False.with_tiered_storage=True.with_iceberg=True.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58699#01936471-e222-4a29-8b98-00cf2647acee:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58776#01936807-c27d-420b-b11f-ac29fb296b1d:

"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=False.with_tiered_storage=False.with_iceberg=True.cloud_storage_type=CloudStorageType.S3"

vbotbuildovich · 2024-11-21T10:41:43Z

Retry command for Build#58436

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/cluster_features_test.py::FeaturesMultiNodeTest.test_license_upload_and_query
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"enable_failures":false,"mixed_versions":false,"with_iceberg":false,"with_tiered_storage":false}

vbotbuildovich · 2024-11-22T13:39:35Z

Retry command for Build#58552

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":true,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":true,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":true,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":true,"with_tiered_storage":true}

vbotbuildovich · 2024-11-25T12:52:47Z

Retry command for Build#58677

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}

vbotbuildovich · 2024-11-25T18:07:26Z

the below tests from https://buildkite.com/redpanda/redpanda/builds/58699#01936415-5294-4558-a6c6-9e4d478aa3e5 have failed and will be retried

gtest_raft_rpunit

the below tests from https://buildkite.com/redpanda/redpanda/builds/58699#01936415-5293-4076-bef5-06d65681c0cc have failed and will be retried

storage_e2e_single_thread_rpunit

the below tests from https://buildkite.com/redpanda/redpanda/builds/58776#019367c1-77a8-4b65-9aaf-efff1ae837ae have failed and will be retried

storage_e2e_single_thread_rpunit

vbotbuildovich · 2024-11-25T19:37:31Z

Retry command for Build#58699

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":false,"with_iceberg":true,"with_tiered_storage":true}

Added assert validating if Redpanda memory reservation was correctly configured when creating `datalake_manager` service. As an addition to that the log entry informing about configured memory parameters was added. Signed-off-by: Michał Maślanka <[email protected]>

Added a parameter that enables Iceberg in RandomNodeOperations test to validate the behavior with node operations and failures. Signed-off-by: Michał Maślanka <[email protected]>

vbotbuildovich · 2024-11-26T12:20:40Z

Retry command for Build#58776

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":false,"with_iceberg":true,"with_tiered_storage":false}

github-actions bot added area/build area/redpanda labels Nov 19, 2024

mmaslankaprv force-pushed the rno-test branch 2 times, most recently from dea809b to 5da2d68 Compare November 19, 2024 13:33

mmaslankaprv force-pushed the rno-test branch from 5da2d68 to b1b6ca6 Compare November 20, 2024 09:17

mmaslankaprv marked this pull request as ready for review November 20, 2024 13:32

mmaslankaprv requested review from bharathv, ztlpn and andrwng November 20, 2024 13:32

bharathv reviewed Nov 20, 2024

View reviewed changes

mmaslankaprv force-pushed the rno-test branch from b1b6ca6 to 8c8cb91 Compare November 21, 2024 07:40

mmaslankaprv force-pushed the rno-test branch from 8c8cb91 to 0f22815 Compare November 22, 2024 10:44

mmaslankaprv force-pushed the rno-test branch 2 times, most recently from dd6f6a3 to 724ee16 Compare November 25, 2024 09:31

mmaslankaprv force-pushed the rno-test branch from 724ee16 to 97a060a Compare November 25, 2024 16:07

mmaslankaprv added 2 commits November 26, 2024 10:13

tests: enable datalake in random node operation tests

57c06f5

Added a parameter that enables Iceberg in RandomNodeOperations test to validate the behavior with node operations and failures. Signed-off-by: Michał Maślanka <[email protected]>

mmaslankaprv force-pushed the rno-test branch from 97a060a to 57c06f5 Compare November 26, 2024 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datalake in Random Node Operations test #24179

Datalake in Random Node Operations test #24179

mmaslankaprv commented Nov 19, 2024 •

edited

Loading

mmaslankaprv commented Nov 19, 2024

mmaslankaprv commented Nov 20, 2024

vbotbuildovich commented Nov 20, 2024 •

edited

Loading

bharathv Nov 20, 2024

vbotbuildovich commented Nov 21, 2024 •

edited

Loading

vbotbuildovich commented Nov 21, 2024 •

edited

Loading

vbotbuildovich commented Nov 22, 2024 •

edited

Loading

vbotbuildovich commented Nov 25, 2024 •

edited

Loading

vbotbuildovich commented Nov 25, 2024 •

edited

Loading

vbotbuildovich commented Nov 25, 2024 •

edited

Loading

vbotbuildovich commented Nov 26, 2024

Datalake in Random Node Operations test #24179

Are you sure you want to change the base?

Datalake in Random Node Operations test #24179

Conversation

mmaslankaprv commented Nov 19, 2024 • edited Loading

Backports Required

Release Notes

mmaslankaprv commented Nov 19, 2024

mmaslankaprv commented Nov 20, 2024

vbotbuildovich commented Nov 20, 2024 • edited Loading

bharathv Nov 20, 2024

Choose a reason for hiding this comment

vbotbuildovich commented Nov 21, 2024 • edited Loading

vbotbuildovich commented Nov 21, 2024 • edited Loading

Retry command for Build#58436

vbotbuildovich commented Nov 22, 2024 • edited Loading

Retry command for Build#58552

vbotbuildovich commented Nov 25, 2024 • edited Loading

Retry command for Build#58677

vbotbuildovich commented Nov 25, 2024 • edited Loading

vbotbuildovich commented Nov 25, 2024 • edited Loading

Retry command for Build#58699

vbotbuildovich commented Nov 26, 2024

Retry command for Build#58776

mmaslankaprv commented Nov 19, 2024 •

edited

Loading

vbotbuildovich commented Nov 20, 2024 •

edited

Loading

vbotbuildovich commented Nov 21, 2024 •

edited

Loading

vbotbuildovich commented Nov 21, 2024 •

edited

Loading

vbotbuildovich commented Nov 22, 2024 •

edited

Loading

vbotbuildovich commented Nov 25, 2024 •

edited

Loading

vbotbuildovich commented Nov 25, 2024 •

edited

Loading

vbotbuildovich commented Nov 25, 2024 •

edited

Loading