SPLAT-2337: tests/ote: Introduce openshift test extension binary #117
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
/test ? |
openshift-ci
bot
added
the
do-not-merge/invalid-owners-file
|
@mtulio: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| // Initialization for kube ginkgo test framework needs to run before all tests execute | ||
| // specs.AddBeforeAll(func() { | ||
| // if err := initializeTestFramework(os.Getenv("TEST_PROVIDER")); err != nil { | ||
| // panic(err) | ||
| // } | ||
| // }) |
There was a problem hiding this comment.
Still trying to understand if we'll need to include all tests in the suite or only what we wanted (this proposal), then the openshift-tests will merge it with all binaries when building the suite.
| // filter only loadbalancer and nodes tests | ||
| // We must skip unsupported tests on OpenShift, such as ECR. | ||
| loadbalancerSpecs := specs.Select(extensiontests.NameContains("[cloud-provider-aws-e2e] loadbalancer")) | ||
| nodesSpecs := specs.Select(extensiontests.NameContains("[cloud-provider-aws-e2e] nodes")) | ||
| kubeTestsExtension.AddSpecs(loadbalancerSpecs) | ||
| kubeTestsExtension.AddSpecs(nodesSpecs) |
There was a problem hiding this comment.
@elmiko @JoelSpeed this is how we are filtering upstream tests inheriting only "well-known passing" from upstream. I wonder if it could be a way to start inheriting healthy tests to OCP core automatically (when added to upstream which matches this pattern). Thoughts?
There was a problem hiding this comment.
inheriting only "well-known passing" from
Are they tagging the upstream tests in some way that we can exclude anything that is known flaky? I see this is filtering node and LB specs, but what stops them adding something unstable here?
There was a problem hiding this comment.
Are they tagging the upstream tests in some way that we can exclude anything that is known flaky?
No tagging as far as I know, I believe because they/us are ensuring all upstream tests works/pass in existing CI e2e job. The 'well-known' refers to the current state that we've been working recently to fix all those that weren't working on OCP, that's reason we are removing ecr specs from the list.
but what stops them adding something unstable here?
I believe us making sure we review upstream (PRs), and those will be triggered in downstream sync PR, right?
IMO if we have an allow list we may lose signals from features added in upstream, we could have an exclude list if there are tests not working on OCP, but it's not the case currently.
Thoughts?
There was a problem hiding this comment.
I know that upstream K/K they filter out tech preview features tests by default, and enable those only in specific jobs, do you know if there will be a similar pattern for the CCM tests where new feature tests are marked initially as gated and then only promoted later? Or are we not expecting feature gated changes in the CCMs since they are mature and not really changing?
There was a problem hiding this comment.
do you know if there will be a similar pattern for the CCM tests where new feature tests are marked initially as gated and then only promoted later?
I don't know to be honest, my CCM experience is limited to AWS, and as fair as I can tell, gates aren't broadly implemented on that controller. @elmiko do you have some thoughts of gates in controller and test framework?
not really changing
it seems not the case of this controller.
|
/test e2e-aws-ovn |
$ ./openshift-tests-ext-ccm-aws list tests | jq .[].name
"[cloud-provider-aws-e2e] loadbalancer CLB should be reachable with default configurations"
"[cloud-provider-aws-e2e] loadbalancer NLB should be reachable with default configurations"
"[cloud-provider-aws-e2e] loadbalancer NLB should be reachable with target-node-labels"
"[cloud-provider-aws-e2e] loadbalancer CLB internal should be reachable with hairpinning traffic"
"[cloud-provider-aws-e2e] loadbalancer NLB internal should be reachable with hairpinning traffic"
"[cloud-provider-aws-e2e] nodes should set zone-id topology label"
"[cloud-provider-aws-e2e] nodes should label nodes with topology network info if instance is supported"
|
mtulio
force-pushed
the
SPLAT-2488-ocp-ote
branch
from
831f31f to
ee77be3
Compare
|
/test e2e-aws-ovn |
mtulio
force-pushed
the
SPLAT-2488-ocp-ote
branch
from
ee77be3 to
4647f1c
Compare
mtulio
changed the title
|
/testwith openshift/origin#30235 |
I may missed the test name: /testwith e2e-aws-ovn openshift/origin#30235 |
|
/testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/origin#30235 |
|
I wonder if it the pod was randomly killed as git fetch worked, re-trying: /testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/origin#30235 |
|
/testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/origin#30235 |
|
/testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/cluster-cloud-controller-manager-operator#391 |
|
/testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/cluster-cloud-controller-manager-operator#391 |
|
/test ? |
|
@mtulio: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-aws-ovn-techpreview |
|
re-testing after openshift/release#69692: /test e2e-aws-ovn-techpreview |
|
/testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/origin#30235 |
|
@mtulio: This pull request references SPLAT-2337 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@mtulio: This pull request references SPLAT-2337 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/testwith openshift/cloud-provider-aws/main/e2e-aws-ovn openshift/origin#30235 |
|
/test ? |
|
@mtulio: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-aws-ovn-techpreview |
|
Considering You can filter to review only the commit UPSTREAM: : Create OCP tests integration (OTE) (skip the vendor). |
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: nrb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
Thanks Nolan, thanks for review! Would you mind reviewing openshift/origin#30235 too? I am blocking there until this binary is available in the image. |
|
/lgtm |
|
unholding after lgtm labels! Thanks you all for review! |
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
Done, I don't think I have approval in origin |
Thanks, no problem, review labels are enough from component owner :) I asked TRT folks! |
|
/verified by @huali9 Based in the results on job https://prow.ci.openshift.org/view/gs/test-platform-results/logs/multi-pr-openshift-origin-30235-openshift-cloud-provider-aws-117-e2e-aws-ovn/1983393761152995328 The tests prefixed with "[cloud-provider-aws-e2e]", implemented and maintained by upstream project, is now incorporated and passing to the conformance suite openshift/conformance. |
openshift-ci-robot
added
the
verified
|
@huali9: This PR has been marked as verified by In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-merge-bot
bot
merged commit 42c152f
into
openshift:main
9 participants
Introduces the OpenShift test extension (OTE) binary to this project/CCM-AWS downstream.
The OTE is used to incorporate existing CCM-AWS upstream e2e tests to the OpenShift conformance test suite, so that we can increase the test scope of CCM on OpenShift ecosystem automatically inheriting new, matched, e2e tests added by upstream, ensuring recent features are added to the OpenShift conformance workflow.
The proposal adds the test binary to the following commit to carry on:
UPSTREAM <carry>: Create OCP tests integration (OTE).This proposal is blocked by the following changes:
This is part of the workforce to deliver the OpenShift changes: