[ODH] Seperate e2e tests in smaller groups

[atheo89]

Reopen this pr to fix the mess of the rebase to this: #53010

[openshift-ci-robot]

[REHEARSALNOTIFIER]
@atheo89: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-opendatahub-io-notebooks-main-anaconda-ubi8-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-codeserver-notebook-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-intel-notebooks-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-notebooks-ubi8-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-rstudio-notebook-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-runtimes-ubi8-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed
pull-ci-opendatahub-io-notebooks-main-runtimes-ubi9-e2e-tests	opendatahub-io/notebooks	presubmit	Presubmit changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[jiridanek]

/pj-rehearse max
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: atheo89, jiridanek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ci-operator/config/opendatahub-io/notebooks/OWNERS [atheo89]
• ci-operator/jobs/opendatahub-io/notebooks/OWNERS [atheo89]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[atheo89]

/pj-rehearse pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests
/pj-rehearse pull-ci-opendatahub-io-notebooks-main-codeserver-notebook-e2e-tests
/pj-rehearse pull-ci-opendatahub-io-notebooks-main-rstudio-notebook-e2e-tests

Let's run some of them

[openshift-ci-robot]

@atheo89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@atheo89: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command.

[openshift-ci-robot]

@atheo89: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command.

[jiridanek]

/pj-rehearse pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests pull-ci-opendatahub-io-notebooks-main-codeserver-notebook-e2e-tests pull-ci-opendatahub-io-notebooks-main-rstudio-notebook-e2e-tests

Let's see some results

[openshift-ci-robot]

@jiridanek: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[jiridanek]

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    intel-extension-for-tensorflow-lib==2.14.0.0.2 from https://files.pythonhosted.org/packages/bb/49/ae01a967453ebe2882c761837d0867940322d419671bfd9cb8f7ac3fd294/intel_extension_for_tensorflow_lib-2.14.0.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from -r /tmp/requirements_micropipenv-hf1sggme.txt (line 2)):
        Expected sha256 3d9a285c90fa5de8c4bc8d9ca12495da3868eb00dd36a94f748a8145d311999a
        Expected     or 4bf166e933885a767ce30e657462d5528ef9f3c54e75cbec7a7b28900dfb4683
        Expected     or 9db2da2ac7901ae0b4145533fe002af053e6cab5721366839ff359a42130a642
             Got        faf2cd03b41e4bf5588594a9ef33c2678255914f451d7313786a3af11c032c1f 

what witchcraft is this? https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/53476/rehearse-53476-pull-ci-opendatahub-io-notebooks-main-codeserver-notebook-e2e-tests/1803766230801190912#1:build-log.txt%3A5868

[atheo89]

Niahhh, this is new!
So, another one intel build error to log it on the tracker 🤣

[jiridanek]

/pj-rehearse more

[openshift-ci-robot]

@jiridanek: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[jiridanek]

looks ok to me,
/pj-rehearse ack

---

COMMIT temp.builder.openshift.io/ci-op-nc8gt59z/amd-c9s-python-3.9-amd64:1b97f910
Error: received unexpected terminate signal, the build amd-c9s-python-3.9-amd64 failed after 19m47s with reason BuildPodEvicted: The node was low on resource: ephemeral-storage. Threshold quantity: 32129992137, available: 29153240Ki. Container docker-build was using 48105624Ki, request is 0, has larger consumption of ephemeral-storage. 

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/53476/rehearse-53476-pull-ci-opendatahub-io-notebooks-main-rstudio-notebook-e2e-tests/1804070125482020864#1:build-log.txt%3A6510, looks to be https://issues.redhat.com/browse/RHOAIENG-3466, we don't request any ephemeral-storage for the build

>>> 48105624/1024/1024
45.877098083496094

Something like 60GiB is a reasonable initial request value for the buildconfig to build this! Alternatively, it would be nice if image could consume less disk space.

---

 E0621 11:10:14.289920      89 portforward.go:406] an error occurred forwarding 8888 -> 8888: error forwarding port 8888 to pod f4e17d6110a830547db06f68759f0f0aee2ea41f5629d37667f18abe8bc25d3e, uid : port forward into network namespace "/var/run/netns/09bcdb01-30e4-4b1f-a729-cec3058a0d41": failed to connect to localhost:8888 inside namespace f4e17d6110a830547db06f68759f0f0aee2ea41f5629d37667f18abe8bc25d3e: dial tcp [::1]:8888: connect: connection refused

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server
E0621 11:10:14.290825      89 portforward.go:234] lost connection to pod 

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/53476/rehearse-53476-pull-ci-opendatahub-io-notebooks-main-intel-notebooks-e2e-tests/1804070117915496448, this should be a known issue

• Intel tensorflow notebook failed to get tested on OCP-CI opendatahub-io/notebooks#562

[openshift-ci-robot]

@jiridanek: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

@atheo89: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/opendatahub-io/notebooks/main/intel-notebooks-e2e-tests	4f6022a	link	unknown	/pj-rehearse pull-ci-opendatahub-io-notebooks-main-intel-notebooks-e2e-tests
ci/rehearse/opendatahub-io/notebooks/main/rstudio-notebook-e2e-tests	4f6022a	link	unknown	/pj-rehearse pull-ci-opendatahub-io-notebooks-main-rstudio-notebook-e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.