Initial e2e tests

[astefanutti]

This PR adds e2e tests for the CodeFlare stack, more specifically, MCAD, KubeRay/Ray and the CodeFlare SDK.

It includes:

• A e2e test suite with 3 initial tests:
  • MNIST training submitted as a RayJob to a RayCluster managed by MCAD
  • MNIST training submitted as a batch Job managed by MCAD
  • MNIST training on a Ray cluster via the CodeFlare SDK
    (only runs on OpenShift, requires K8s Support for SDK codeflare-sdk#146)
• A GitHub Actions workflow, that runs e2e tests against a KinD test cluster.
• The parameterisation of the components under tests, so the test suite can be reused by downstream projects.
• The documentation on how to run / debug the e2e tests locally.

The execution of the e2e workflow currently takes around 14 minutes on average. A third of that execution time corresponds to the build of the operator container image, which may be optimised in the future, by caching layers for example.

Fixes #4.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[sutaakar]

@astefanutti Would it have sense to add comments to the tests describing what functionality is tested in specific tests? Something like a description from test strategy ADR.

[astefanutti]

@astefanutti Would it have sense to add comments to the tests describing what functionality is tested in specific tests? Something like a description from test strategy ADR.

@sutaakar yes, absolutely. We can start with commenting the test case functions, and research ways to structure more that information in the next cycles.

[KPostOffice]

Some questions. I'm unfamiliar with golang testing.

[astefanutti]

@astefanutti Would it have sense to add comments to the tests describing what functionality is tested in specific tests? Something like a description from test strategy ADR.

@sutaakar yes, absolutely. We can start with commenting the test case functions, and research ways to structure more that information in the next cycles.

Done with 2045c7a.

[sutaakar]

Looks good to me

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anishasthana, sutaakar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [anishasthana]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment