Added post validation steps for NS loadbalancing scenarios

[HaruChebrolu]

Description

Added post validation steps for NS loadbalancing scenarios.
Logs: http://magna002.ceph.redhat.com/cephci-jenkins/harika/ns_loadbalancing/cephci-run-NLZQ8M/

click to expand checklist

• Create a test case in Polarion reviewed and approved.
• Create a design/automation approach doc. Optional for tests with similar tests already automated.
• Review the automation design
• Implement the test script and perform test runs
• Submit PR for code review and approve
• Update Polarion Test with Automation script details and update automation fields
• If automation is part of Close loop, update BZ flag qe-test_coverage “+” and link Polarion test

[rahullepakshi]

gw info logs too many times, if possible lets try to avoid duplication and remove it if its part of GW initialization

[rahullepakshi]

@HaruChebrolu It is better to add all listeners first and then move to new namespace addition in interest of time and IO getting stuck . Though they are new GWs, it is always advised to add all listeners before adding namespaces

[rahullepakshi]

The order below would be more beneficial for scale up new Gws

1. Run IO on old namespaces
2. ceph orch apply new GW nodes
3. Add listeners on all subsystems for alll new GW nodes immediately
4. Then add namespaces
5. Disconnect, discover and connect again from initiators

Currently it is 2,1,4-3,5

[HaruChebrolu]

Will update the loop to add listeners to all subsystems and then move to adding namespaces.
And currently I running IO on old namespaces and then I am applying new GW nodes, please let me know if I am missing something.

[rahullepakshi]

Thanks @HaruChebrolu
I got confused with

cephci/tests/nvmeof/test_ceph_nvmeof_loadbalancing.py

Lines 280 to 283 in a16e3a8

	ha.scale_up(scaleup_nodes, gw_nodes, old_namespaces)
	# Validate IO for old namespaces
	ha.validate_scaleup(scaleup_nodes, old_namespaces)

i think you are validating IO for old namespaces after orch apply new nodes? If its case then its fine. thanks
Another suggestionis you can validate IO for old namespaces as well as new namespaces together after listener add and namespace add. We can do this way too

[HaruChebrolu]

Done

[HaruChebrolu]

gw info logs too many times, if possible lets try to avoid duplication and remove it if its part of GW initialization

I will try to mitigate as much as I can, but I am little skeptical if I can completely avoid it.

[Manohar-Murthy]

lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: HaruChebrolu, Manohar-Murthy, rahullepakshi

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment