Automate performance testing for CCR plugin

[ankitkala]

Is your feature request related to a problem? Please describe

We want to to be able to run performance tests for cross cluster repliacation plugin using opensearch-benchmark.

Describe the solution you'd like

We'd like to add the support for replication performance testing as part of OS performance testing framework. Eventually, We'd want to have periodic runs for the replication plugin as part of Jenkins CI.

Describe alternatives you've considered

We explored whether we should have a separate dedicated Jenkins setup for CCR plugin but it seems like a duplication of effort and thus decided to just focus on making it work with OS performance framework instead.

Tasks:

• https://github.com/opensearch-project/opensearch-infra/issues/146
• [CCR Perf]Add support for Multi node cluster  #1496
• Add support for creating multiple cluster and setup CCR #1498
• [CCR Perf] Add support for multiple test scenarios in a PerfSuite  #1499
• Run CCR Perf tests as part of Jenkins CI #1500

[dblock]

After #126 we should be able to onboard plugins into whatever we come up with, including CCR.

[ankitkala]

Yep, I've been following up that issue closely. Let me come up with the list of tasks required to support CCR perf suite and update here.

[ankitkala]

@dblock I had an offline discussion with @bbarani and based on the discussion, here is the plan for automating the performance tests:

• We'll add a script in replication repo that can be used by everyone contributing to CCR as well as opensearch build.
• This script would be invoked as part of distribution build regularly within Jenkins.
• Script would use PerfTestSuite & PerfTestCluster to setup multiple clusters and run the replication perf tests. If there are any changes required to support these on OS infra side, I'll take those up and raise the PRs.
• The clusters would be created in build team's account.
• For manually triggers, we'll be using our team's account (which we'd have to whitelisted with perf testing team).
• We'd have read only access to Jenkins to monitor the jobs.
• We'd be running perf tests only for latest and next release.
• We'll start testing single node clusters. For testing on multinode cluster, perf testing team would have to remove the hardcoded ports so that we can send IP for a loadbalancer.
• I've also been following up with perf testing team and they're planning to support multiple IPs incase of multinode clusters. It in a existing feature in OSB and only perf tool to add support in the APIs.

[dblock]

• We'll add a script in replication repo that can be used by everyone contributing to CCR as well as opensearch build.
• This script would be invoked as part of distribution build regularly within Jenkins.
• Script would use PerfTestSuite & PerfTestCluster to setup multiple clusters and run the replication perf tests. If there are any changes required to support these on OS infra side, I'll take those up and raise the PRs.

What will a script do that existing scripts don't?
I would say this is already test.sh and it takes a manifest as input.

[ankitkala]

These are the steps we have for running a replication perf test:

1. Create 2 clusters(Leader and follower).
2. Connect the two clusters and start replication.
3. Start data ingestion on leader using OSB and monitor replication metrics.
4. Stop the replication and remove the connection(Can be skipped in this case since these clusters won't be reused).

We need to execute the additional logic to setup the replication prior to running the tests. I'm assuming you'd prefer that we keep this logic(same for other plugin owners) in our own repo. If this is not the case, let me know, then we don't really need any separate script we can just call test.sh with an input manifest.

[gaiksaya]

[Triage] Hi @ankitkala , In which repository is the work related to this issue is gonna happen?
We wanted to know where the code will reside for this.

[ankitkala]

Sorry for late reply. The code is expected to be in build repo itself.

[bbarani]

Hi @ankitkala , thanks for the update. I dont see any updates on the tasks listed on this META issue. Can you provide an update?

[ankitkala]

I'll be resuming on this task by end of this week. Expect the PRs by early next week or so.

[ankitkala]

https://github.com/opensearch-project/opensearch-infra/pull/151

[bbarani]

@ankitkala I see that all the tasks associated with this issue are completed. Can we close this issue?