Test for Graceful node shutdown #8233

avd-sagare · 2023-08-14T08:44:11Z

Test to verify that data is accessible and uncorrupted as well as
operational cluster and osd are up after graceful nodes shutdown

ocs-ci

PR validation on existing cluster

Cluster Name: asagare-414aws
Cluster Configuration:
PR Test Suite: system_test
PR Test Path: tests/e2e/system_test/test_graceful_nodes_shutdown.py
Additional Test Params:
OCP VERSION: 4.14
OCS VERSION: 4.14
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: asagare-414aws
Cluster Configuration:
PR Test Suite: system_test
PR Test Path: tests/e2e/system_test/test_graceful_nodes_shutdown.py
Additional Test Params:
OCP VERSION: 4.14
OCS VERSION: 4.14
tested against branch: master

Job UNSTABLE (some or all tests failed).

avd-sagare · 2023-08-22T05:42:04Z

Blocked Validation due to https://bugzilla.redhat.com/show_bug.cgi?id=2227835

ocs-ci

PR validation on existing cluster

Cluster Name: asagare-odf-hugep
Cluster Configuration:
PR Test Suite: system_test
PR Test Path: tests/e2e/system_test/test_graceful_nodes_shutdown.py
Additional Test Params:
OCP VERSION: 4.14
OCS VERSION: 4.14
tested against branch: master

Job FAILED (installation failed, tests not executed).

ocs-ci

PR validation on existing cluster

Cluster Name: asagare-odf-hugepages
Cluster Configuration:
PR Test Suite: system_test
PR Test Path: tests/e2e/system_test/test_graceful_nodes_shutdown.py
Additional Test Params:
OCP VERSION: 4.14
OCS VERSION: 4.14
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: asagare-odf-hugepages
Cluster Configuration:
PR Test Suite: system_test
PR Test Path: tests/e2e/system_test/test_graceful_nodes_shutdown.py
Additional Test Params:
OCP VERSION: 4.14
OCS VERSION: 4.14
tested against branch: master

Job UNSTABLE (some or all tests failed).

hnallurv · 2023-09-21T08:04:15Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        # S3 bucket
+        self.setup_s3_bucket(mcg_obj=mcg_obj, bucket_factory=bucket_factory)
+
+    def setup_amq_kafka_notification(self, bucket_factory):


see if you can use existing amq common function instead of writing new one here.

PrasadDesala · 2023-09-21T07:44:43Z

ocs_ci/helpers/longevity_helpers.py

+            #            (
+            #                "OC",
+            #                {
+            #                    "interface": "OC",
+            #                    "namespace_policy_dict": {
+            #                        "type": "Cache",
+            #                        "ttl": 3600,
+            #                        "namespacestore_dict": {"aws": [(1, None)]},
+            #                    },
+            #                    "placement_policy": {
+            #                        "tiers": [
+            #                            {"backingStores": [constants.DEFAULT_NOOBAA_BACKINGSTORE]}
+            #                        ]
+            #                    },
+            #                },
+            #            ),


please do not comment the core functions, instead handle it in the test script.

PrasadDesala · 2023-09-21T07:45:21Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+from ocs_ci.helpers.helpers import default_storage_class
+from ocs_ci.ocs.bucket_utils import s3_put_object, retrieve_verification_mode
+
+# from ocs_ci.ocs.resources.fips import check_fips_enabled


you can remove the comment code

PrasadDesala · 2023-09-21T07:57:17Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+from ocs_ci.ocs.utils import get_pod_name_by_pattern
+from ocs_ci.helpers.helpers import default_storage_class
+from ocs_ci.ocs.bucket_utils import s3_put_object, retrieve_verification_mode
+


you can delete this empty line

PrasadDesala · 2023-09-21T08:03:51Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        # S3 bucket
+        self.setup_s3_bucket(mcg_obj=mcg_obj, bucket_factory=bucket_factory)
+
+    def setup_amq_kafka_notification(self, bucket_factory):


you are not calling it in setup

tests/e2e/system_test/test_graceful_nodes_shutdown.py

PrasadDesala · 2023-09-21T08:09:35Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        multi_obc_lifecycle_factory(num_of_obcs=20, bulk=True, measure=False)
+
+        # OCP Workloads
+        start_ocp_workload(workloads_list=["registry", "monitoring"], run_in_bg=True)


you will have to handle the teardown of ocp workloads too

tests/e2e/system_test/test_graceful_nodes_shutdown.py

hnallurv · 2023-09-21T08:28:06Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        tree_output = ct_pod.exec_ceph_cmd(ceph_cmd="ceph osd tree")
+        logger.info("ceph osd tree output:")
+        logger.info(tree_output)
+


Please cover other post verification steps from https://docs.google.com/document/d/129IreytPAPPQW1sov1JyQ6_SHXfgpVF4EtD9ZP9m10s/edit#bookmark=id.w6l0liw6o9rr

tests/e2e/system_test/test_graceful_nodes_shutdown.py

PrasadDesala · 2023-11-20T06:27:48Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        """
+        nodes = get_nodes()
+        for node in nodes:
+            assert (
+                node.get()["status"]["allocatable"]["hugepages-2Mi"] == "64Mi"
+            ), f"Huge pages is not applied on {node.name}"
+        """


If this code is not needed, please remove it

It is needed. On testing cluster it was not enabled so for testing purpose its commented.

PrasadDesala · 2023-11-20T06:28:25Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+
+        def teardown():
+            logger.info("cleanup the environment")
+            """ cleanup logging workload """


this is a single line comment. pls consider adding with hash

PrasadDesala · 2023-11-20T06:28:39Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+                logger.info("Logging is configured")
+                uninstall_cluster_logging()
+
+            """ cleanup monitoring workload """


same as above

PrasadDesala · 2023-11-20T06:29:43Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+
+        request.addfinalizer(teardown)
+
+        logger.info("Starting the test setup")


Should we add the setup related func to a setup func?

I think if we are not achieving optimisation or resource saving thne its ok to be as it is.

PrasadDesala · 2023-11-20T06:30:22Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        ##################################### non encrypted pvc (ne_pvc)
+        create + clone
+        non-encrypted block/fs pvc
+        Returns:
+            pvc object
+            pod object
+            file_name
+            origin md5sum
+            snapshot
+        """


please fix doc string format issues

PrasadDesala · 2023-11-20T06:31:54Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        if not check_if_monitoring_stack_exists():
+            assert "monitoring workload not started after reboot"
+        if not check_if_registry_stack_exists():
+            assert "registry workload not started after reboot"


the workload exists does not mean that the workload is running fine without any issues after reboot

Ok. what additional steps you suggest to check?

PrasadDesala · 2023-11-20T06:36:16Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+        Sanity().health_check(tries=60)
+
+        self.validate_data_integrity()
+        self.validate_snapshot_restore(snapshot_restore_factory)


where are we checking the below step-
PVC expansion is possible on the restored snapshot- validate_pvc_expansion

https://github.com/red-hat-storage/ocs-ci/pull/8233/files#diff-56d19df1f041e2768208d6a3ec334de5933efc60cdd389ec07e0332b787243fcR274

calling validate_pvc_expansion inside validate_snapshot_restore

PrasadDesala · 2023-11-20T06:36:51Z

tests/e2e/system_test/test_graceful_nodes_shutdown.py

+
+        self.validate_data_integrity()
+        self.validate_snapshot_restore(snapshot_restore_factory)
+        validate_mcg_bg_features(skip_any_features=["expiration", "rgw kafka", "nsfs"])


where are checking for the below steps-

get operation on the bucket gets the data that was stored before shutdown

delete operation on the bucket can delete the data that was stored before shutdown

put and list operations work correctly

S3 operations on NSFS bucket on files is not implemented yet.

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

Signed-off-by: Avdhoot <[email protected]>

avd-sagare · 2024-07-22T12:11:14Z

PR validation job-
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/39907/

avd-sagare · 2024-07-22T14:10:13Z

Tested PR after restoring BS size to 17GB-
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/39987/

openshift-ci · 2024-07-24T14:22:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: avd-sagare, mashetty330, PrasadDesala

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Avdhoot <[email protected]>

avd-sagare requested review from PrasadDesala and hnallurv August 14, 2023 08:44

avd-sagare requested a review from a team as a code owner August 14, 2023 08:44

pull-request-size bot added the size/XL label Aug 14, 2023

avd-sagare force-pushed the System_test-Shutdown_Restart branch from 4ab31cf to ae3545e Compare August 14, 2023 13:10

ocs-ci reviewed Aug 16, 2023

View reviewed changes

avd-sagare requested a review from Shrivaibavi August 17, 2023 13:28

ocs-ci reviewed Aug 21, 2023

View reviewed changes

avd-sagare force-pushed the System_test-Shutdown_Restart branch from ae3545e to 9bed8b5 Compare August 23, 2023 05:40

ocs-ci reviewed Aug 23, 2023

View reviewed changes

avd-sagare force-pushed the System_test-Shutdown_Restart branch from 9bed8b5 to 9b61886 Compare August 23, 2023 12:49

ocs-ci reviewed Aug 23, 2023

View reviewed changes

avd-sagare force-pushed the System_test-Shutdown_Restart branch from 9b61886 to 85bd0fd Compare September 21, 2023 06:55

hnallurv reviewed Sep 21, 2023

View reviewed changes

PrasadDesala reviewed Sep 21, 2023

View reviewed changes

PrasadDesala added the team/e2e E2E team related issues/PRs label Sep 21, 2023

hnallurv reviewed Sep 21, 2023

View reviewed changes

avd-sagare force-pushed the System_test-Shutdown_Restart branch from 85bd0fd to 1177594 Compare November 9, 2023 08:37

pull-request-size bot added size/L PR that changes 100-499 lines size/XXL and removed size/XL size/L PR that changes 100-499 lines labels Nov 9, 2023

avd-sagare force-pushed the System_test-Shutdown_Restart branch from c45c408 to f4d7a20 Compare November 17, 2023 14:16

pull-request-size bot added size/L PR that changes 100-499 lines and removed size/XXL labels Nov 17, 2023

avd-sagare force-pushed the System_test-Shutdown_Restart branch from f4d7a20 to 5b706fe Compare November 20, 2023 05:43

PrasadDesala reviewed Nov 20, 2023

View reviewed changes

avd-sagare force-pushed the System_test-Shutdown_Restart branch from 5b706fe to fed1568 Compare November 20, 2023 12:54

Avdhoot and others added 16 commits July 22, 2024 16:01

Removed shutdownc cmd changes

8b3c935

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

Added platform based start node

6b4b225

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

Added changes

d72e102

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

Added timeout

ba77748

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

Added sc

42a66ca

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

Added new changes

ac2aefc

Signed-off-by: Avdhoot <[email protected]> Signed-off-by: Avdhoot <[email protected]>

changed sequence of factory function and added teardwon factory

97cf162

Signed-off-by: Avdhoot <[email protected]>

Increased vsphere vm timeout

6b506d5

Signed-off-by: Avdhoot <[email protected]>

Increased volume size from 17 to 50

3186056

Signed-off-by: Avdhoot <[email protected]>

Increased timeout

afa172a

Signed-off-by: Avdhoot <[email protected]>

Increased timeout

78fcc04

Signed-off-by: Avdhoot <[email protected]>

Added arg in validate_mcg function call

265619b

Signed-off-by: Avdhoot <[email protected]>

restored timeout value for vspeher

87cadaf

Signed-off-by: Avdhoot <[email protected]>

Set BS timeout value to default

2593a25

Signed-off-by: Avdhoot <[email protected]>

Removed pvc encryption for cephfs

1ae58dd

Signed-off-by: Avdhoot <[email protected]>

Added fixture scale_noobaa_resources_session

4fa1e16

Signed-off-by: Avdhoot <[email protected]>

avd-sagare force-pushed the System_test-Shutdown_Restart branch from 39a0e2b to 4fa1e16 Compare July 22, 2024 10:32

restored value to 17

cebb346

Signed-off-by: Avdhoot <[email protected]>

pull-request-size bot added size/L PR that changes 100-499 lines and removed size/XL labels Jul 22, 2024

mashetty330 approved these changes Jul 24, 2024

View reviewed changes

openshift-ci bot assigned mashetty330 Jul 24, 2024

openshift-ci bot added the lgtm label Jul 24, 2024

PrasadDesala approved these changes Jul 24, 2024

View reviewed changes

openshift-ci bot assigned PrasadDesala Jul 24, 2024

PrasadDesala merged commit ba411e7 into red-hat-storage:master Jul 24, 2024
5 of 6 checks passed

amr1ta pushed a commit to amr1ta/ocs-ci that referenced this pull request Jul 25, 2024

Test for Graceful node shutdown (red-hat-storage#8233)

28fe7a9

Signed-off-by: Avdhoot <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test for Graceful node shutdown #8233

Test for Graceful node shutdown #8233

avd-sagare commented Aug 14, 2023

ocs-ci left a comment

ocs-ci left a comment

avd-sagare commented Aug 22, 2023

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

hnallurv Sep 21, 2023

PrasadDesala Sep 21, 2023

PrasadDesala Sep 21, 2023

PrasadDesala Sep 21, 2023

PrasadDesala Sep 21, 2023

PrasadDesala Sep 21, 2023

hnallurv Sep 21, 2023

PrasadDesala Nov 20, 2023

avd-sagare Nov 20, 2023

PrasadDesala Nov 20, 2023

PrasadDesala Nov 20, 2023

PrasadDesala Nov 20, 2023

avd-sagare Nov 22, 2023

PrasadDesala Nov 20, 2023

PrasadDesala Nov 20, 2023

avd-sagare Nov 20, 2023

PrasadDesala Nov 20, 2023

avd-sagare Nov 20, 2023

avd-sagare Nov 20, 2023

PrasadDesala Nov 20, 2023

avd-sagare Nov 20, 2023

avd-sagare commented Jul 22, 2024

avd-sagare commented Jul 22, 2024

openshift-ci bot commented Jul 24, 2024


		request.addfinalizer(teardown)

		logger.info("Starting the test setup")

Test for Graceful node shutdown #8233

Test for Graceful node shutdown #8233

Conversation

avd-sagare commented Aug 14, 2023

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

avd-sagare commented Aug 22, 2023

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avd-sagare commented Jul 22, 2024

avd-sagare commented Jul 22, 2024

openshift-ci bot commented Jul 24, 2024