cluster: Addition of new Scrubbing test #246

amathuria · 2021-10-12T14:32:45Z

Signed-off-by: Aishwarya Mathuria [email protected]

cluster/ceph.py

Add 2 new tests similar to the background Recovery test for running Scrub load with client IO and the other one for running Scrub, Recovery, and Client load. The Scrub and client IO test performs the following steps: - Create a pool and image to populate scrub objects (scrub pool) - Create scrub thread - Populate the scrub pool with objects using radosbench - Initiate deep-scrub on the scrub pool - Create a second pool and an image in it to run client IO - Initiate fio job on the second image at the same time the deep-scrub starts In the second test, we have an additional recovery pool that is populated after an OSD is marked down and out. Once the pool is populated we mark the OSD up and in which starts backfill. At the same time, we begin deep-scrub on the scrub pool and client IO. Signed-off-by: Aishwarya Mathuria <[email protected]>

sseshasa

For the scrubbing test, the name of the states like 'osdout' and 'osdin' can confuse other readers. Can this be changed to more appropriate states like for e.g., 'fillscrubpool', 'initscrub' to make it more intuitive?

cluster/ceph.py

sseshasa

I forgot to add this. The scrub tests can be factored out into a separate file under the same directory to avoid ceph.py from bloating up.

amathuria · 2022-08-23T13:46:35Z

For the scrubbing test, the name of the states like 'osdout' and 'osdin' can confuse other readers. Can this be changed to more appropriate states like for e.g., 'fillscrubpool', 'initscrub' to make it more intuitive?

Makes sense, I'll work on that and shift the scrub tests into a separate file.

sseshasa · 2022-09-07T06:06:50Z

benchmark/librbdfio.py

@@ -80,9 +82,17 @@ def initialize(self):

        common.sync_files('%s/*' % self.run_dir, self.out_dir)

+        if 'scrubbing_test' in self.cluster.config:


Can this just be called 'scrub_test'. I think all instances of 'scrubbing' in function names and other places can be replaced with 'scrub'. This makes the code much neater and consistent to read. What do you think?

sseshasa · 2022-09-07T06:07:08Z

benchmark/librbdfio.py

@@ -231,6 +267,9 @@ def recovery_callback_blocking(self):
    def recovery_callback_background(self):
        logger.info('Recovery thread completed!')

+    def scrubbing_callback(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:07:13Z

client_endpoints/ceph_client_endpoints.py

@@ -114,6 +116,15 @@ def create_rbd_recovery(self):
                rbd_name = '%s-%s' % (self.pool, self.get_rbd_name(node, ep_num))
                self.cluster.mkimage(rbd_name, self.endpoint_size, self.pool, self.data_pool, self.order)

+    def create_rbd_scrubbing(self):


'create_rbd_scrub_pool()' sounds better.

sseshasa · 2022-09-07T06:07:25Z

client_endpoints/client_endpoints.py

@@ -45,3 +45,6 @@ def remove(self):

    def create_recovery_image(self):
        pass
+
+    def create_scrubbing_image(self):


'scrubbing' to 'scrub' here and in the other *_endpoint.py files.

sseshasa · 2022-09-07T06:07:34Z

cluster/ceph.py

@@ -9,6 +9,7 @@
 import json

 from .cluster import Cluster
+from .scrub_tests import ScrubbingTestThreadBackground, ScrubRecoveryThreadBackground


Looks much better now. To be consistent you can consider changing class name from "ScrubbingTestThreadBackground" to "ScrubTestThreadBackground".

sseshasa · 2022-09-07T06:07:43Z

cluster/ceph.py

@@ -563,6 +578,44 @@ def check_health(self, check_list=None, logfile=None, recstatsfile=None):

        return ret

+    def log_scrubbing_stats(self, scrubstatsfile=None, pgid=None):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:07:51Z

cluster/ceph.py

@@ -654,6 +731,21 @@ def create_recovery_test(self, run_dir, callback, test_type='blocking'):
            self.rt = RecoveryTestThreadBackground(rt_config, self, callback, self.stoprequest, self.haltrequest, self.startiorequest)
        self.rt.start()

+    def create_scrubbing_test(self, run_dir, callback):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:07:58Z

cluster/ceph.py

@@ -672,6 +764,32 @@ def wait_recovery_done(self):
                break
            self.rt.join(1)

+    def maybe_populate_scrubbing_pool(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:08:04Z

cluster/ceph.py

+            common.pdsh(settings.getnodes('head'), 'sudo %s -p %s bench %s write -b %s --max-objects %s --no-cleanup' % (self.rados_cmd, self.scrub_pool_name, self.prefill_scrub_time, self.prefill_scrub_object_size, self.prefill_scrub_objects)).communicate()
+            #self.check_health()
+
+    def initiate_scrubbing(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:08:10Z

cluster/ceph.py

+        logger.info("Initiating scrub on pool %s" % self.scrub_pool_name)
+        common.pdsh(settings.getnodes('head'), '%s osd pool deep-scrub %s' % (self.ceph_cmd, self.scrub_pool_name)).communicate()
+
+    def wait_scrubbing_done(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:09:11Z

benchmark/fio.py

@@ -239,6 +263,12 @@ def recovery_callback_blocking(self):
    def recovery_callback_background(self):
        logger.info('Recovery thread completed!')

+    def scrubbing_callback(self):


'scrubbing' to 'scrub'

Signed-off-by: Aishwarya Mathuria <[email protected]>

amathuria force-pushed the wip-cbt-addscrubtest branch 3 times, most recently from 2071234 to 52e8dae Compare February 8, 2022 15:06

amathuria marked this pull request as ready for review February 8, 2022 15:07

amathuria requested review from sseshasa and markhpc February 10, 2022 14:11

sseshasa reviewed May 9, 2022

View reviewed changes

cluster/ceph.py Outdated Show resolved Hide resolved

amathuria force-pushed the wip-cbt-addscrubtest branch from 52e8dae to 9b2e21a Compare August 16, 2022 14:23

sseshasa requested changes Aug 17, 2022

View reviewed changes

sseshasa reviewed Aug 17, 2022

View reviewed changes

cluster/ceph.py Outdated Show resolved Hide resolved

sseshasa reviewed Aug 17, 2022

View reviewed changes

amathuria force-pushed the wip-cbt-addscrubtest branch from 1a2c63c to da9ec6d Compare September 6, 2022 14:13

sseshasa reviewed Sep 7, 2022

View reviewed changes

Moving out scrub tests to a separate file

fd4de2d

Signed-off-by: Aishwarya Mathuria <[email protected]>

amathuria force-pushed the wip-cbt-addscrubtest branch from da9ec6d to fd4de2d Compare September 20, 2022 11:30

cluster: Add option for shallow scrub for scrub tests

4291252

Signed-off-by: Aishwarya Mathuria <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cluster: Addition of new Scrubbing test #246

cluster: Addition of new Scrubbing test #246

amathuria commented Oct 12, 2021

sseshasa left a comment

sseshasa left a comment

amathuria commented Aug 23, 2022 •

edited

Loading

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

sseshasa Sep 7, 2022

		@@ -80,9 +82,17 @@ def initialize(self):

		common.sync_files('%s/*' % self.run_dir, self.out_dir)

		if 'scrubbing_test' in self.cluster.config:

		@@ -563,6 +578,44 @@ def check_health(self, check_list=None, logfile=None, recstatsfile=None):

		return ret

		def log_scrubbing_stats(self, scrubstatsfile=None, pgid=None):

cluster: Addition of new Scrubbing test #246

Are you sure you want to change the base?

cluster: Addition of new Scrubbing test #246

Conversation

amathuria commented Oct 12, 2021

sseshasa left a comment

Choose a reason for hiding this comment

sseshasa left a comment

Choose a reason for hiding this comment

amathuria commented Aug 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amathuria commented Aug 23, 2022 •

edited

Loading