Tracking issue for e2e testing framework and implementation

[chuckha]

Testing framework

Based on the testing proposal we have a lot of work to do! This issue tracks the work to be done

Initial work to set patterns

• Build framework
  • Collect feedback
  • Merge framework
• Implement CAPD tests

Parallel work after patterns are in place (in order of priority)

• Automate CAPD tests (Wire up CAPD e2e #1921)
• Implement CAPA tests
• Ensure satisfactory documentation for use
• Design and implement debugging when the environment being run is not accessible (prow). See ✨Add the e2e framework #1737 (comment) for more information.
• Write tests (see next section) in the framework for each behavior we want tested. Consider consolidating if there is overlap.
• Enable the framework build in CI test/framework isn't being build by CI #1772 (🏃 add e2e framework to CI build #1771)
• Use go 1.12 instead of 1.13 test/framework go.mod should have go 1.12, not 1.13 #1773 (🐛 Use go1.12 instead of go1.13 for the e2e framework #1778)

Tests to write in the framework that can be reused

• There is a Kubernetes cluster with one node that passes a healthcheck after creating a properly configured Cluster, InfraCluster, Machine, InfraMachine and BootstrapConfig. (https://github.com/kubernetes-sigs/cluster-api/blob/master/test/framework/one_node_cluster.go)
• Creating the resources necessary for three control planes will create a three node cluster. (https://github.com/kubernetes-sigs/cluster-api/blob/master/test/framework/multi_node_control_plane.go)
• Deleting a cluster deletes all resources associated with that cluster including Machines, BootstrapConfigs, InfraMachines, InfraCluster, and generated secrets. (🏃E2e delete resources #1915)
• Creating a cluster with one control plane and one worker node will result in a cluster with two nodes.
• The version fields in Machines are respected within the bounds of the Kubernetes skew policy.
• Creating a control plane machine and a MachineDeployment with two replicas will create a three node cluster with one control plane node and two worker nodes.
• MachineDeployments do their best to keep Machines in an expected state. For example:
  • Modifying a replica count on a MachineDeployment will modify the number of worker nodes and Machines in running state the cluster has.
  • Deleting a machine that is managed by a MachineDeployment will be recreated by the MachineDeployment
• Optionally, Machines report failures when their underlying InfraMachines report failures.
• Manage multiple workload clusters in different namespaces.
• Workload Clusters created pass Kubernetes Conformance. Please see CAPA's implementation.

/milestone v0.3.0
/priority important-soon
/assign

[chuckha]

Some ideas that came out of #1742

reduce boilerplate

• Have framework provide a scheme that has the required types registered that the framework uses. This allows implementors to grab that runtime.Scheme and add their own types to it so they don't need to figure out what the framework requires. https://github.com/kubernetes-sigs/cluster-api/pull/1742/files#r345443377
• There is more work to be done around Generators/Components/Dependencies https://github.com/kubernetes-sigs/cluster-api/pull/1742/files#r345447689
  • Least of all would be generating Cluster and Machine objects https://github.com/kubernetes-sigs/cluster-api/pull/1742/files#r345451092

consistency

• Have the generators/dependencies figure out the version that they need automatically based on the CAPI version being used. https://github.com/kubernetes-sigs/cluster-api/pull/1742/files#r345441607
• Figure out a better way to handle clean up, especially of failed tests https://github.com/kubernetes-sigs/cluster-api/pull/1742/files#r345894500

cc @detiber @wfernandes, thanks for the feedback

[sb1975]

What does this mean
"Creating the resources necessary for three control planes will create a three node cluster."

Since MachineDeployments do not support control-plane nodes yet, does this mean presently the three control-plane nodes will be created individually using just "kind: Machine" and not MachineDeployments ?

[sb1975]

"Creating a control plane machine and a MachineDeployment with two replicas will create a three node cluster with one control plane node and two worker nodes."
^I guess I already tried this.
In the MachineDeployment yaml file, I just changed the replicas: 2.
It resulted in the target(or workload) cluster with 3 nodes ( 1 master + 2 worker).
I also validated that I can run a nginx pod inside the cluster and it works as expected.

apiVersion: cluster.x-k8s.io/v1alpha2 kind: MachineDeployment metadata: name: capi-quickstart-worker labels: cluster.x-k8s.io/cluster-name: capi-quickstart # Labels beyond this point are for example purposes, # feel free to add more or change with something more meaningful. # Sync these values with spec.selector.matchLabels and spec.template.metadata.labels. nodepool: nodepool-0 spec: replicas: **2** selector: matchLabels: cluster.x-k8s.io/cluster-name: capi-quickstart nodepool: nodepool-0 template: metadata: labels: cluster.x-k8s.io/cluster-name: capi-quickstart nodepool: nodepool-0 spec: version: v1.15.3 bootstrap: configRef: name: capi-quickstart-worker apiVersion: bootstrap.cluster.x-k8s.io/v1alpha2 kind: KubeadmConfigTemplate infrastructureRef: name: capi-quickstart-worker apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2 kind: DockerMachineTemplate
root@capd:~# kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
capi-quickstart-capi-quickstart-controlplane-0 Ready master 4h6m v1.15.3
capi-quickstart-capi-quickstart-worker-85cbf8fd8c-mpr2l Ready 3h45m v1.15.3
capi-quickstart-capi-quickstart-worker-85cbf8fd8c-wx75g Ready 3h42m v1.15.3

root@capd:~# kubectl --kubeconfig=./capi-quickstart.kubeconfig get pods -n mynamespace -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINAT ED NODE READINESS GATES
nginx 1/1 Running 0 21m 192.168.61.193 capi-quickstart-capi-quickstart-worker-85cbf8fd8c-mpr2l

Can I do something more to help this one please ?

[chuckha]

@sb1975 I updated the issue to link to the actual tests in the framework as examples to get folks started. Take a look at the existing tests and try to follow the existing patterns or if you find something that needs improving, improve it.

[wfernandes]

I'm planning on working on this framework test: Deleting a cluster deletes all resources associated with that cluster including Machines, BootstrapConfigs, InfraMachines, InfraCluster, and generated secrets. I'm hoping to pair that work as I accomplish this story #1775.

[chuckha]

Closing this in favor of #2141

Other cases are not important right now and will be added in future as importance rises.

/close

[k8s-ci-robot]

@chuckha: Closing this issue.

In response to this:

Closing this in favor of #2141

Other cases are not important right now and will be added in future as importance rises.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.