Expand mc-bootstrap tests to validate installed apps

[yulianedyalkova]

Currently we only ensure that mc-bootstrap runs successfully but we have no further validation of what happens with the apps installed after.

Acceptance criteria:

• Goal is not to change the MC and test it (like app testing, upgrades etc) but rather validate that it comes up successfully as it is.
• The mechanism of validation should be as similar as possible to WC testing (ideally using the same tools as clustertest framework).

Task:

• Implement test framework that can deal with MCs
• Consume framework from tekton pipeline

[AverageMarcus]

Ok, I've been able to think about this and have a rough idea what it might look like...

Some considerations

1. This shouldn't be directly tied to the current mc-bootstrap setup as there is work being done to replace it with a CLI, instead this should be complementary and not rely on either the current or new implementation of mc-bootstrap
2. It should be possible to use these tests to validate production MCs as well as the ephemeral test MCs (likely with less tests run against prod)
3. The tests need to be runnable from both Tekton CI and from local dev laptops.

Approach

• Create a new repo for the test suites (likely managementcluster-test-suite to align with existing)

• The test suite will follow the similar approach to cluster-test-suites and apptest-framework and be built on top of clustertest. I suspect that only a single test suite will exist though rather than the per-provider (+ variants) we have for WCs. There might be a separate upgrade tests but that's currently out of scope I think.

• The test suite should be packaged as a container image (like with WCs) so it can be called from Tekton easily

• The test suite should expect a E2E_KUBECONFIG env var pointing at the MC to test (the MC must already have been created)

• The test cases should make use of Spec Labels (or Label Sets) to indicate the following:

  • If a specific test is only applicable to a specific provider (hopefully theres not many of these cases)
  • If a test has side-effects (such as creating or deleting resources) that wouldn't be good to run against a production MC

  These labels will then be used to filter the tests that are run based on the MC that is being tested

• mc-bootstrap should be updated with a new Make target that triggers the test suite (from latest container image?) using the Kubeconfig of the created MC from the earlier step.

• The new CLI should be updated to also include the ability to trigger these tests (I'm not sure what the current state of this CLI is so not 100% sure about this bit)

• The mc-bootstrap Tekton Pipeline should be updated with a new Task between the generate and cleanup that runs the new test suite.

• Team Tinkerers will likely need to be the CODEOWNERS of the repo but it should be treated like the other test suites and contributed by all teams (including the app teams)

[ericgraf]

Here is an example on how the pipeline could be created. I put this together last Nov.
https://github.com/giantswarm/mc-bootstrap/pull/721

[AverageMarcus]

Relevant from on-site discussion (taken from Slack thread):

To summarise:
We create a new management-cluster-test-suites that runs the cluster-wide “validation” / integration tests against an MC created by mc-bootstrap. The kubeconfig will be passed through from mc-bootstrap and just run as part of the current pipeline before the teardown task.
For individual app testing we have a pool of (let’s say) 10 ephemeral MCs pre-configured that we can make use for with app test suites. We then have automation that ensures at least 1 MC is “pre-warmed” at start of business day that the first run of app tests can use. When an app test is triggered we (in the background) launch the next MC in the pool so it’s ready for the next tests. Because we have a limited number of available MCs to use we’d need to use a queue in case there are more than 10 test ru