OCPCLOUD-3042: Create control-plane-machine-set-tests-ext command for origin e2e #363

sunzhaohua2 · 2025-07-29T08:38:17Z

Per design of Openshift Test Extensions create a Control Plane MachineSet Operator OpenShift tests extension.

Get extension usage

$ bin/control-plane-machine-set-tests-ext                                                                      
cluster control plane machine set Operator tests extension for OpenShift

Usage:
   [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  images      List test images
  info        Display extension metadata
  list        List items
  run-suite   Run a group of tests by suite. This is more limited than origin, and intended for light local development use. Orchestration parameters, scheduling, isolation, etc are not obeyed, and Ginkgo tests are executed serially.
  run-test    Runs tests by name
  update      Update test metadata

Flags:
  -h, --help   help for this command

Use " [command] --help" for more information about a command.

Get extension definition info

$ bin/control-plane-machine-set-tests-ext info                                                                   
{
    "apiVersion": "v1.1",
    "source": {
        "commit": "",
        "build_date": "",
        "git_tree_state": ""
    },
    "component": {
        "product": "openshift",
        "type": "payload",
        "name": "cluster-control-plane-machine-set-operator"
    },
    "suites": [
        {
            "name": "cpmso/periodic",
            "description": "",
            "qualifiers": [
                "(source == \"openshift:payload:cluster-control-plane-machine-set-operator\") \u0026\u0026 (!labels.exists(l, l == \"Periodic\"))"
            ]
        },
        {
            "name": "cpmso/presubmit",
            "description": "",
            "qualifiers": [
                "(source == \"openshift:payload:cluster-control-plane-machine-set-operator\") \u0026\u0026 (labels.exists(l, l == \"PreSubmit\") )"
            ]
        }
    ],
    "images": null
}

List test cases 

$ bin/control-plane-machine-set-tests-ext list tests --suite cpmso/presubmit                                  
[
  {
    "name": "ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine",
    "labels": {
      "PreSubmit": {}
    },
    "resources": {
      "isolation": {}
    },
    "source": "openshift:payload:cluster-control-plane-machine-set-operator",
    "codeLocations": [
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:34",
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:35",
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:39",
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:40",
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:44",
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:45",
      "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129"
    ],
    "lifecycle": "blocking",
    "environmentSelector": {}
  },
  ...

Run test suites

$ bin/control-plane-machine-set-tests-ext run-suite cpmso/presubmit                                         
  Running Suite:  - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator
  =====================================================================================================
  Random Seed: 1753839740 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and ControlPlaneMachineSet is updated to set MachineNamePrefix [OCPFeatureGate:CPMSMachineNamePrefix] and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit]
  /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129
    STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 09:42:21.898
    STEP: Checking the control plane machine set exists @ 07/30/25 09:42:22.173
    STEP: Checking the control plane machine set is active @ 07/30/25 09:42:22.41
    STEP: Updating the machine name prefix of the control plane machine set to "master-prefix" @ 07/30/25 09:42:23.06
    STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 09:42:25.214
    STEP: Waiting for the index 1 to be replaced @ 07/30/25 09:42:25.959
    STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 09:42:25.959
    STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 09:42:25.961
    STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 09:42:25.962
    STEP: Checking that index 1 has 2 replicas @ 07/30/25 09:42:26.39

make e2e

$ make e2e-presubmit                                                                                            
GINKGO_EXTRA_ARGS=--label-filter=PreSubmit ./hack/e2e.sh
go run ./hack/../vendor/github.com/onsi/ginkgo/v2/ginkgo -r -v --fail-fast --trace --timeout=4h --label-filter=PreSubmit ./test/e2e
Running Suite: E2E Suite - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e
=======================================================================================================================
Random Seed: 1753841589

Will run 15 of 22 specs
------------------------------
[BeforeSuite] 
/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/suite_test.go:43
[BeforeSuite] PASSED [0.001 seconds]
------------------------------
ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit]
/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129
  STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 10:13:20.961
  STEP: Checking the control plane machine set exists @ 07/30/25 10:13:21.188
  STEP: Checking the control plane machine set is active @ 07/30/25 10:13:21.42
  STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 10:13:21.865
  STEP: Waiting for the index 1 to be replaced @ 07/30/25 10:13:22.751
  STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 10:13:22.751
  STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 10:13:22.752
  STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 10:13:22.752
  STEP: Checking that index 1 has 2 replicas @ 07/30/25 10:13:22.977
  STEP: Correct index is being replaced @ 07/30/25 10:13:23.194
  STEP: Index 1 replacement created @ 07/30/25 10:13:23.194
  STEP: Checking the replacement machine for index 1 @ 07/30/25 10:13:23.194

openshift-ci-robot · 2025-07-30T02:47:09Z

@sunzhaohua2: This pull request references OCPQE-28828 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.20.0" version, but no target version was set.

In response to this:

Per design of Openshift Test Extensions create a Control Plane MachineSet Operator OpenShift tests extension.

Get extension usage

$ bin/control-plane-machine-set-tests-ext                                                                      
cluster control plane machine set Operator tests extension for OpenShift

Usage:
  [command]

Available Commands:
 completion  Generate the autocompletion script for the specified shell
 help        Help about any command
 images      List test images
 info        Display extension metadata
 list        List items
 run-suite   Run a group of tests by suite. This is more limited than origin, and intended for light local development use. Orchestration parameters, scheduling, isolation, etc are not obeyed, and Ginkgo tests are executed serially.
 run-test    Runs tests by name
 update      Update test metadata

Flags:
 -h, --help   help for this command

Use " [command] --help" for more information about a command.

Get extension definition info

$ bin/control-plane-machine-set-tests-ext info                                                                   
{
   "apiVersion": "v1.1",
   "source": {
       "commit": "",
       "build_date": "",
       "git_tree_state": ""
   },
   "component": {
       "product": "openshift",
       "type": "payload",
       "name": "cluster-control-plane-machine-set-operator"
   },
   "suites": [
       {
           "name": "cpmso/periodic",
           "description": "",
           "qualifiers": [
               "(source == \"openshift:payload:cluster-control-plane-machine-set-operator\") \u0026\u0026 (!labels.exists(l, l == \"Periodic\"))"
           ]
       },
       {
           "name": "cpmso/presubmit",
           "description": "",
           "qualifiers": [
               "(source == \"openshift:payload:cluster-control-plane-machine-set-operator\") \u0026\u0026 (labels.exists(l, l == \"PreSubmit\") )"
           ]
       }
   ],
   "images": null
}

List test cases 

$ bin/control-plane-machine-set-tests-ext list tests --suite cpmso/presubmit                                  
[
 {
   "name": "ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine",
   "labels": {
     "PreSubmit": {}
   },
   "resources": {
     "isolation": {}
   },
   "source": "openshift:payload:cluster-control-plane-machine-set-operator",
   "codeLocations": [
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:34",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:35",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:39",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:40",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:44",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:45",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129"
   ],
   "lifecycle": "blocking",
   "environmentSelector": {}
 },
 ...

Run test suites

$ bin/control-plane-machine-set-tests-ext run-suite cpmso/presubmit                                         
 Running Suite:  - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator
 =====================================================================================================
 Random Seed: 1753839740 - will randomize all specs

 Will run 1 of 1 specs
 ------------------------------
 ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and ControlPlaneMachineSet is updated to set MachineNamePrefix [OCPFeatureGate:CPMSMachineNamePrefix] and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit]
 /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129
   STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 09:42:21.898
   STEP: Checking the control plane machine set exists @ 07/30/25 09:42:22.173
   STEP: Checking the control plane machine set is active @ 07/30/25 09:42:22.41
   STEP: Updating the machine name prefix of the control plane machine set to "master-prefix" @ 07/30/25 09:42:23.06
   STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 09:42:25.214
   STEP: Waiting for the index 1 to be replaced @ 07/30/25 09:42:25.959
   STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 09:42:25.959
   STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 09:42:25.961
   STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 09:42:25.962
   STEP: Checking that index 1 has 2 replicas @ 07/30/25 09:42:26.39

make e2e

$ make e2e-presubmit                                                                                            
GINKGO_EXTRA_ARGS=--label-filter=PreSubmit ./hack/e2e.sh
go run ./hack/../vendor/github.com/onsi/ginkgo/v2/ginkgo -r -v --fail-fast --trace --timeout=4h --label-filter=PreSubmit ./test/e2e
Running Suite: E2E Suite - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e
=======================================================================================================================
Random Seed: 1753841589

Will run 15 of 22 specs
------------------------------
[BeforeSuite] 
/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/suite_test.go:43
[BeforeSuite] PASSED [0.001 seconds]
------------------------------
ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit]
/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129
 STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 10:13:20.961
 STEP: Checking the control plane machine set exists @ 07/30/25 10:13:21.188
 STEP: Checking the control plane machine set is active @ 07/30/25 10:13:21.42
 STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 10:13:21.865
 STEP: Waiting for the index 1 to be replaced @ 07/30/25 10:13:22.751
 STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 10:13:22.751
 STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 10:13:22.752
 STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 10:13:22.752
 STEP: Checking that index 1 has 2 replicas @ 07/30/25 10:13:22.977
 STEP: Correct index is being replaced @ 07/30/25 10:13:23.194
 STEP: Index 1 replacement created @ 07/30/25 10:13:23.194
 STEP: Checking the replacement machine for index 1 @ 07/30/25 10:13:23.194

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

mdbooth

I'm just learning about this tool from reading about your review, so if my comments make no sense just let me know! Looks cool.

cmd/control-plane-machine-set-tests-ext/main.go

mdbooth · 2025-07-31T13:31:45Z

cmd/control-plane-machine-set-tests-ext/main.go

+	})
+
+	var err error
+	framework.GlobalFramework, err = framework.NewFramework()


Here and in suite_test.go we're calling a framework module method and using the returned value to set a framework module global.

Could we rename framework.NewFramework() to framework.InitFramework() and have it:

only return error

set its internal global directly

?

mdbooth · 2025-07-31T13:35:44Z

cmd/control-plane-machine-set-tests-ext/main.go

+
+	ext.AddSuite(e.Suite{
+		Name:       "cpmso/periodic",
+		Qualifiers: []string{`!labels.exists(l, l == "Periodic")`},


This looks different to an example expressions in the enhancement:

"(test.tags.suite==\"fips\" || test.name.contains(\"fips\")) && !test.labels.has(\"Disruptive\")",

From that I'd have expected something like:

`test.labels.has("Periodic")`

Have we tested it?

@mdbooth thanks for your review. yes, I have tested, I refered to https://github.com/openshift/machine-api-operator/blob/main/cmd/machine-api-tests-ext/main.go#L37

List test cases 

$ bin/control-plane-machine-set-tests-ext list tests --suite cpmso/presubmit [ { "name": "ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine", "labels": { "PreSubmit": {} }, "resources": { "isolation": {} }, "source": "openshift:payload:cluster-control-plane-machine-set-operator", "codeLocations": [ "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:34", "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:35", "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:39", "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:40", "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:44", "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:45", "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129" ], "lifecycle": "blocking", "environmentSelector": {} }, ...

Run test suites

$ bin/control-plane-machine-set-tests-ext run-suite cpmso/presubmit Running Suite: - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator ===================================================================================================== Random Seed: 1753839740 - will randomize all specs Will run 1 of 1 specs ------------------------------ ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and ControlPlaneMachineSet is updated to set MachineNamePrefix [OCPFeatureGate:CPMSMachineNamePrefix] and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit] /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129 STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 09:42:21.898 STEP: Checking the control plane machine set exists @ 07/30/25 09:42:22.173 STEP: Checking the control plane machine set is active @ 07/30/25 09:42:22.41 STEP: Updating the machine name prefix of the control plane machine set to "master-prefix" @ 07/30/25 09:42:23.06 STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 09:42:25.214 STEP: Waiting for the index 1 to be replaced @ 07/30/25 09:42:25.959 STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 09:42:25.959 STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 09:42:25.961 STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 09:42:25.962 STEP: Checking that index 1 has 2 replicas @ 07/30/25 09:42:26.39

openshift-ci-robot · 2025-08-04T07:34:08Z

@sunzhaohua2: This pull request references OCPCLOUD-3042 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

Per design of Openshift Test Extensions create a Control Plane MachineSet Operator OpenShift tests extension.

Get extension usage

$ bin/control-plane-machine-set-tests-ext                                                                      
cluster control plane machine set Operator tests extension for OpenShift

Usage:
  [command]

Available Commands:
 completion  Generate the autocompletion script for the specified shell
 help        Help about any command
 images      List test images
 info        Display extension metadata
 list        List items
 run-suite   Run a group of tests by suite. This is more limited than origin, and intended for light local development use. Orchestration parameters, scheduling, isolation, etc are not obeyed, and Ginkgo tests are executed serially.
 run-test    Runs tests by name
 update      Update test metadata

Flags:
 -h, --help   help for this command

Use " [command] --help" for more information about a command.

Get extension definition info

$ bin/control-plane-machine-set-tests-ext info                                                                   
{
   "apiVersion": "v1.1",
   "source": {
       "commit": "",
       "build_date": "",
       "git_tree_state": ""
   },
   "component": {
       "product": "openshift",
       "type": "payload",
       "name": "cluster-control-plane-machine-set-operator"
   },
   "suites": [
       {
           "name": "cpmso/periodic",
           "description": "",
           "qualifiers": [
               "(source == \"openshift:payload:cluster-control-plane-machine-set-operator\") \u0026\u0026 (!labels.exists(l, l == \"Periodic\"))"
           ]
       },
       {
           "name": "cpmso/presubmit",
           "description": "",
           "qualifiers": [
               "(source == \"openshift:payload:cluster-control-plane-machine-set-operator\") \u0026\u0026 (labels.exists(l, l == \"PreSubmit\") )"
           ]
       }
   ],
   "images": null
}

List test cases 

$ bin/control-plane-machine-set-tests-ext list tests --suite cpmso/presubmit                                  
[
 {
   "name": "ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine",
   "labels": {
     "PreSubmit": {}
   },
   "resources": {
     "isolation": {}
   },
   "source": "openshift:payload:cluster-control-plane-machine-set-operator",
   "codeLocations": [
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:34",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:35",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:39",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:40",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:44",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/presubmit.go:45",
     "/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129"
   ],
   "lifecycle": "blocking",
   "environmentSelector": {}
 },
 ...

Run test suites

$ bin/control-plane-machine-set-tests-ext run-suite cpmso/presubmit                                         
 Running Suite:  - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator
 =====================================================================================================
 Random Seed: 1753839740 - will randomize all specs

 Will run 1 of 1 specs
 ------------------------------
 ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and ControlPlaneMachineSet is updated to set MachineNamePrefix [OCPFeatureGate:CPMSMachineNamePrefix] and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit]
 /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129
   STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 09:42:21.898
   STEP: Checking the control plane machine set exists @ 07/30/25 09:42:22.173
   STEP: Checking the control plane machine set is active @ 07/30/25 09:42:22.41
   STEP: Updating the machine name prefix of the control plane machine set to "master-prefix" @ 07/30/25 09:42:23.06
   STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 09:42:25.214
   STEP: Waiting for the index 1 to be replaced @ 07/30/25 09:42:25.959
   STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 09:42:25.959
   STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 09:42:25.961
   STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 09:42:25.962
   STEP: Checking that index 1 has 2 replicas @ 07/30/25 09:42:26.39

make e2e

$ make e2e-presubmit                                                                                            
GINKGO_EXTRA_ARGS=--label-filter=PreSubmit ./hack/e2e.sh
go run ./hack/../vendor/github.com/onsi/ginkgo/v2/ginkgo -r -v --fail-fast --trace --timeout=4h --label-filter=PreSubmit ./test/e2e
Running Suite: E2E Suite - /Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e
=======================================================================================================================
Random Seed: 1753841589

Will run 15 of 22 specs
------------------------------
[BeforeSuite] 
/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/suite_test.go:43
[BeforeSuite] PASSED [0.001 seconds]
------------------------------
ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the provider spec of index 1 is not as expected should rolling update replace the outdated machine [PreSubmit]
/Users/zhsun/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/test/e2e/helpers/cases.go:129
 STEP: Waiting for the cluster operators to stabilise (minimum availability time: 1m0s, timeout: 10m0s, polling interval: 10s) @ 07/30/25 10:13:20.961
 STEP: Checking the control plane machine set exists @ 07/30/25 10:13:21.188
 STEP: Checking the control plane machine set is active @ 07/30/25 10:13:21.42
 STEP: Updating the provider spec of the control plane machine at index 1 @ 07/30/25 10:13:21.865
 STEP: Waiting for the index 1 to be replaced @ 07/30/25 10:13:22.751
 STEP: Checking the number of control plane machines never goes above 4 replicas @ 07/30/25 10:13:22.751
 STEP: Checking that other indexes (not 1) do not have 2 replicas @ 07/30/25 10:13:22.752
 STEP: Waiting for the updated replicas to equal desired replicas @ 07/30/25 10:13:22.752
 STEP: Checking that index 1 has 2 replicas @ 07/30/25 10:13:22.977
 STEP: Correct index is being replaced @ 07/30/25 10:13:23.194
 STEP: Index 1 replacement created @ 07/30/25 10:13:23.194
 STEP: Checking the replacement machine for index 1 @ 07/30/25 10:13:23.194

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2025-11-02T09:00:45Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-ci · 2025-11-04T02:14:58Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelspeed for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sunzhaohua2 · 2025-11-04T02:15:05Z

/remove-lifecycle stale

sunzhaohua2 · 2025-11-04T03:02:39Z

/testwith openshift/cluster-control-plane-machine-set-operator/main/e2e-aws-ovn openshift/origin#30452

sunzhaohua2 · 2025-11-04T07:25:37Z

/retest-required

…or origin e2e

sunzhaohua2 · 2025-11-05T07:39:51Z

/retest

damdo

Thanks @sunzhaohua2

i think we need to slightly move the pkg to a separate module if possible

damdo · 2025-11-05T08:18:52Z

go.mod

+replace (
+	github.com/onsi/ginkgo/v2 => github.com/openshift/onsi-ginkgo/v2 v2.6.1-0.20251001123353-fd5b1fb35db1
+	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc => go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0
+	k8s.io/api => github.com/openshift/kubernetes/staging/src/k8s.io/api v0.0.0-20251017123720-96593f323733
+	k8s.io/apiextensions-apiserver => github.com/openshift/kubernetes/staging/src/k8s.io/apiextensions-apiserver v0.0.0-20251017123720-96593f323733
+	k8s.io/apimachinery => github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery v0.0.0-20251017123720-96593f323733
+	k8s.io/apiserver => github.com/openshift/kubernetes/staging/src/k8s.io/apiserver v0.0.0-20251017123720-96593f323733
+	k8s.io/cli-runtime => github.com/openshift/kubernetes/staging/src/k8s.io/cli-runtime v0.0.0-20251017123720-96593f323733
+	k8s.io/client-go => github.com/openshift/kubernetes/staging/src/k8s.io/client-go v0.0.0-20251017123720-96593f323733
+	k8s.io/cloud-provider => github.com/openshift/kubernetes/staging/src/k8s.io/cloud-provider v0.0.0-20251017123720-96593f323733
+	k8s.io/cluster-bootstrap => github.com/openshift/kubernetes/staging/src/k8s.io/cluster-bootstrap v0.0.0-20251017123720-96593f323733
+	k8s.io/code-generator => github.com/openshift/kubernetes/staging/src/k8s.io/code-generator v0.0.0-20251017123720-96593f323733
+	k8s.io/component-base => github.com/openshift/kubernetes/staging/src/k8s.io/component-base v0.0.0-20251017123720-96593f323733
+	k8s.io/component-helpers => github.com/openshift/kubernetes/staging/src/k8s.io/component-helpers v0.0.0-20251017123720-96593f323733
+	k8s.io/controller-manager => github.com/openshift/kubernetes/staging/src/k8s.io/controller-manager v0.0.0-20251017123720-96593f323733
+	k8s.io/cri-api => github.com/openshift/kubernetes/staging/src/k8s.io/cri-api v0.0.0-20251017123720-96593f323733
+	k8s.io/cri-client => github.com/openshift/kubernetes/staging/src/k8s.io/cri-client v0.0.0-20251017123720-96593f323733
+	k8s.io/csi-translation-lib => github.com/openshift/kubernetes/staging/src/k8s.io/csi-translation-lib v0.0.0-20251017123720-96593f323733
+	k8s.io/dynamic-resource-allocation => github.com/openshift/kubernetes/staging/src/k8s.io/dynamic-resource-allocation v0.0.0-20251017123720-96593f323733
+	k8s.io/endpointslice => github.com/openshift/kubernetes/staging/src/k8s.io/endpointslice v0.0.0-20251017123720-96593f323733
+	k8s.io/externaljwt => github.com/openshift/kubernetes/staging/src/k8s.io/externaljwt v0.0.0-20251017123720-96593f323733
+	k8s.io/kube-aggregator => github.com/openshift/kubernetes/staging/src/k8s.io/kube-aggregator v0.0.0-20251017123720-96593f323733
+	k8s.io/kube-controller-manager => github.com/openshift/kubernetes/staging/src/k8s.io/kube-controller-manager v0.0.0-20251017123720-96593f323733
+	k8s.io/kube-proxy => github.com/openshift/kubernetes/staging/src/k8s.io/kube-proxy v0.0.0-20251017123720-96593f323733
+	k8s.io/kube-scheduler => github.com/openshift/kubernetes/staging/src/k8s.io/kube-scheduler v0.0.0-20251017123720-96593f323733
+	k8s.io/kubectl => github.com/openshift/kubernetes/staging/src/k8s.io/kubectl v0.0.0-20251017123720-96593f323733
+	k8s.io/kubelet => github.com/openshift/kubernetes/staging/src/k8s.io/kubelet v0.0.0-20251017123720-96593f323733
+	k8s.io/kubernetes => github.com/openshift/kubernetes v1.30.1-0.20251017123720-96593f323733
+	k8s.io/metrics => github.com/openshift/kubernetes/staging/src/k8s.io/metrics v0.0.0-20251017123720-96593f323733
+	k8s.io/mount-utils => github.com/openshift/kubernetes/staging/src/k8s.io/mount-utils v0.0.0-20251017123720-96593f323733
+	k8s.io/pod-security-admission => github.com/openshift/kubernetes/staging/src/k8s.io/pod-security-admission v0.0.0-20251017123720-96593f323733
+	k8s.io/sample-apiserver => github.com/openshift/kubernetes/staging/src/k8s.io/sample-apiserver v0.0.0-20251017123720-96593f323733
+	k8s.io/sample-cli-plugin => github.com/openshift/kubernetes/staging/src/k8s.io/sample-cli-plugin v0.0.0-20251017123720-96593f323733
+	k8s.io/sample-controller => github.com/openshift/kubernetes/staging/src/k8s.io/sample-controller v0.0.0-20251017123720-96593f323733
+)


I think we don't want this in the main module, as this becomes a nightmare to bump every release while also keeping all the dependencies happy.

The idea we had is to follow this approach: openshift/machine-api-operator#1423
Let me know if that makes sense and if we can trial that

damdo · 2025-11-05T08:20:22Z

Makefile

+build: operator tests-ext  ## Build all binaries
+
+operator:  ## Build main operator binary
 	go build -o bin/manager ./cmd/control-plane-machine-set-operator

+tests-ext:  ## Build tests extension binary
+	go build -o bin/control-plane-machine-set-tests-ext ./cmd/control-plane-machine-set-tests-ext
+


These will need to change if we move this to a separate module

damdo · 2025-11-05T08:21:45Z

cmd/control-plane-machine-set-tests-ext/main.go

+	extensionRegistry.Register(ext)
+
+	root := &cobra.Command{
+		Long: "cluster control plane machine set Operator tests extension for OpenShift",


Suggested change

Long: "cluster control plane machine set Operator tests extension for OpenShift",

Long: "cluster-control-plane-machine-set-operator tests extension for OpenShift",

damdo · 2025-11-05T08:22:27Z

Makefile

 	go build -o bin/manager ./cmd/control-plane-machine-set-operator

+tests-ext:  ## Build tests extension binary
+	go build -o bin/control-plane-machine-set-tests-ext ./cmd/control-plane-machine-set-tests-ext


Do we want to use the cluster-control-plane-machine-set-operator-ext for naming?

damdo · 2025-11-05T08:22:51Z

Dockerfile

 FROM registry.ci.openshift.org/ocp/4.21:base-rhel9
 COPY --from=builder /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/bin/manager .
 COPY --from=builder /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/manifests manifests
+COPY --from=builder /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/bin/control-plane-machine-set-tests-ext.gz .


Do we want to use the cluster-control-plane-machine-set-operator-ext for naming?

damdo · 2025-11-05T08:26:39Z

/assign @damdo @RadekManak

/cc @vr4manta

openshift-ci · 2025-11-05T11:26:51Z

@sunzhaohua2: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-etcd-scaling	`e110a18`	link	true	`/test e2e-aws-ovn-etcd-scaling`
ci/prow/unit	`e110a18`	link	true	`/test unit`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

sunzhaohua2 · 2025-11-06T05:20:55Z

cmd/control-plane-machine-set-tests-ext/main.go

+		},
+		Qualifiers: []string{`labels.exists(l, l == "Disruptive")`},
+	})
+


openshift/disruptive allows limited parallelism, but cpms tests must run serially, this causes resource conflicts and test failures in job https://storage.googleapis.com/test-platform-results/logs/multi-pr-openshift-origin-30452-openshift-cluster-control-plane-machine-set-operator-363-e2e-gcp-disruptive/1985909492008620032/build-log.txt

I will remove this suite, there is no suitable parent suite for cpms tests.

sunzhaohua2 · 2025-11-06T05:25:20Z

i think we need to slightly move the pkg to a separate module if possible

I raised another pr #372 to move the tests-extension pkg to a separate module, @damdo PTAL if this is we wanted

RadekManak · 2025-11-06T15:16:21Z

/close
lgtm'd the replacement pr

openshift-ci · 2025-11-06T15:20:06Z

@RadekManak: Closed this PR.

In response to this:

/close
lgtm'd the replacement pr

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 29, 2025

openshift-ci bot requested review from mdbooth and racheljpg July 29, 2025 08:40

sunzhaohua2 force-pushed the ote1 branch from 20adff2 to 406098d Compare July 30, 2025 02:37

sunzhaohua2 changed the title ~~[WIP] Create control-plane-machine-set-tests-ext command for origin e2e~~ OCPQE-28828: Create control-plane-machine-set-tests-ext command for origin e2e Jul 30, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 30, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2025

sunzhaohua2 force-pushed the ote1 branch 2 times, most recently from 811abcd to de60708 Compare July 30, 2025 04:38

openshift-ci bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 30, 2025

mdbooth reviewed Jul 31, 2025

View reviewed changes

sunzhaohua2 force-pushed the ote1 branch from de60708 to d9f4482 Compare August 1, 2025 14:54

sunzhaohua2 changed the title ~~OCPQE-28828: Create control-plane-machine-set-tests-ext command for origin e2e~~ OCPCLOUD-3042: Create control-plane-machine-set-tests-ext command for origin e2e Aug 4, 2025

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 2, 2025

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 2, 2025

vendor update

c2f6cee

sunzhaohua2 force-pushed the ote1 branch from d9f4482 to be2cf17 Compare November 4, 2025 02:13

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 4, 2025

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 4, 2025

sunzhaohua2 force-pushed the ote1 branch from be2cf17 to 289490e Compare November 4, 2025 02:58

sunzhaohua2 mentioned this pull request Nov 4, 2025

OCPCLOUD-3042: Added OTE binary for cpmso openshift/origin#30452

Open

Create cluster-control-plane-machine-set-operator-tests-ext command f…

e110a18

…or origin e2e

sunzhaohua2 force-pushed the ote1 branch from 289490e to e110a18 Compare November 5, 2025 03:10

damdo reviewed Nov 5, 2025

View reviewed changes

openshift-ci bot assigned damdo and RadekManak Nov 5, 2025

openshift-ci bot requested a review from vr4manta November 5, 2025 08:26

sunzhaohua2 mentioned this pull request Nov 6, 2025

OCPCLOUD-3042: Create cluster-control-plane-machine-set-operator-ext command #372

Merged

sunzhaohua2 commented Nov 6, 2025

View reviewed changes

openshift-ci bot closed this Nov 6, 2025

	Long: "cluster control plane machine set Operator tests extension for OpenShift",
	Long: "cluster-control-plane-machine-set-operator tests extension for OpenShift",

OCPCLOUD-3042: Create control-plane-machine-set-tests-ext command for origin e2e #363

OCPCLOUD-3042: Create control-plane-machine-set-tests-ext command for origin e2e #363

Uh oh!

Conversation

sunzhaohua2 commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 30, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdbooth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Aug 4, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Nov 2, 2025

Uh oh!

openshift-ci bot commented Nov 4, 2025

Uh oh!

sunzhaohua2 commented Nov 4, 2025

Uh oh!

sunzhaohua2 commented Nov 4, 2025

Uh oh!

sunzhaohua2 commented Nov 4, 2025

Uh oh!

sunzhaohua2 commented Nov 5, 2025

Uh oh!

damdo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damdo commented Nov 5, 2025

Uh oh!

openshift-ci bot commented Nov 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunzhaohua2 commented Nov 6, 2025

Uh oh!

RadekManak commented Nov 6, 2025

Uh oh!

openshift-ci bot commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sunzhaohua2 commented Jul 29, 2025 •

edited

Loading

openshift-ci-robot commented Jul 30, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 4, 2025 •

edited by openshift-ci bot

Loading