Skip to content

Latest commit

 

History

History
307 lines (210 loc) · 12 KB

testing.md

File metadata and controls

307 lines (210 loc) · 12 KB
Raw

Testing

Tests description

Based on upstreams documentation https://github.com/thtanaka/kubernetes/blob/master/docs/devel/testing.md we use three levels of testing: unit, integration and e2e.

Before starting, run make tools to install the required dependencies.

Running make test executes all the test suites.

We use ginkgo for testing. Every package needs a suite_test.go for setup. It can be generated by running ginkgo bootstrap in the sub folder. Rename the generated file afterwards, to stay consistent. There is also ginkgo generate to create skeleton test files.

Unit

While unit testing we:

  • test classes in isolation
  • pass all dependencies to the constructor, so we can inject fakes for testing
  • use counterfeiter and gomock/mockgen to generate and update fakes and mocks
  • don't test private methods, tests are in a separate _test package
  • try not to nest ginkgo contexts too deep and keep tests DRY by extracting useful helpers
  • assert incoming messages produce the expected state
  • assert outgoing commands happened, like a file gets written
  • assert all handled error cases are triggered
  • can ignore outgoing queries, which only change internal state

Setup Ruby

Ruby gem for template rendering

gem install bosh-template

Integration

Integration tests formulate expectations on the interactions of several components. They require access to a Kubernetes, preferably minikube.

Integration tests start our operator directly, bypassing the command line. They do require the operator docker image and the bosh-template Ruby gem.

The environment package provides helpers to start the operator, get the kubeconfig and use the clients to create objects. In testing the catalog defines test objects.

Integration tests use a special logger, which does not log to stdout and whose messages can be accessed as a an array by calling env.AllLogMessages().

When using bin/test-integration the integration tests are run in parallel. Each Ginkgo test node has a separate namespace, log file and webhook server port and certificate.

The node index starts at 1 and is used as following to generate names:

namespace: $TEST_NAMESPACE + <node_index>
webhook port: $CF_OPERATOR_WEBHOOK_SERVICE_PORT + <node_index>
log file: $CF_OPERATOR_TESTING_TMP/cf-operator-tests-<node_index>.log

Integration tests use the TEST_NAMESPACE environment variable as a base to calculate the namespace name. Test namespaces are deleted automatically once the tests are completed.

CF_OPERATOR_TESTING_TMP can be used to set a tmp directory for storing logs and other files generated during testing. If this variable is not set /tmp will be used instead.

The tests will create some NodePort services; normally the test can detect an IP address automatically. CF_OPERATOR_NODE_IP can set to the node IP of any arbitrary node to override this (e.g. for OpenStack Kubernetes clusters).

Generated files will be cleand up after the test run unless SKIP_CF_OPERATOR_TESTING_TMP_CLEANUP is set to true.

Webhook Configuration

Quarks StatefulSet requires a k8s webhook to mutate the volumes of a pod. Kubernetes will call back to the operator for certain requests and use the modified pod manifest, which is returned. CF-Operator also uses a validating webhook to validate the BOSH deployment spec and the creation of reference resources specified in the spec. Secret validation admission webhook restricts the user from updating a versioned secret.

The cf-operator integration tests use CF_OPERATOR_WEBHOOK_SERVICE_PORT as a base value to calculate the port number to listen to on CF_OPERATOR_WEBHOOK_SERVICE_HOST.

The tests use a mutatingwebhookconfiguration and a validatingwebhookconfiguration to configure Kubernetes to connect to this address. The address needs to be reachable from the cluster.

The configuration only applies to a single namespace, by using a selector. It contains the URL of the webhooks, build from CF_OPERATOR_WEBHOOK_SERVICE_HOST and the calculated port. It also contains SSL certificates and CA, which are necessary to connect to the webhook.

The certificates and keys are written to disk, so the webhook server can use them. They are also cached in a k8s secret for production, but that is not being used in integration tests, since they delete the test namespaces.

Tests suites should clean up their, namespace dependant, webhook configuration automatically.

End-to-End

The e2e tests are meant to test acceptance scenarios. They are written from an end user perspective. They are split into two types, 'cli' and 'kube'.

The e2e CLI test exercise different command line options and commands which don't need a running Kubernetes, like template rendering. The CLI tests build the operator binary themselves.

The second type of e2e tests use helm to install the CF operator into the k8s cluster and use the files from docs/examples for testing.

Running tests

In minikube

The following steps are necessary to have a proper environment setup, where all types of tests can be executed:

  1. Start minikube

    minikube start --kubernetes-version v1.15.5

  2. Switch to minikube docker daemon

    eval $(minikube docker-env)

    Note: Template rendering for BOSH jobs is done at deployment time by the operator binary. Therefore the operator docker image needs to be made available to Kubernetes cluster.

  3. Export the CF_OPERATOR_WEBHOOK_SERVICE_HOST env variable

    export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip -4 a s dev $(ip r l 0/0 | cut -f5 -d' ') | grep -oP 'inet \K\S+(?=/)')

    Note: On Mac, use export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip a s $(ip r g 0/0 | cut -f5 -d' ') | grep -oE 'inet [^ /]+' | cut -f2 -d' '), because grep cannot handle perl regexs. Note: You can also find the correct IP, by running ip addr. The IP address under vboxnet1 is the IP that you need.

  4. Export the OPERATOR_TEST_STORAGE_CLASS env variable

    export OPERATOR_TEST_STORAGE_CLASS=standard

    Note: Require for the PVC test creation, in minikube.

  5. Ensure GO111MODULE is set

    export GO111MODULE=on

    Note: When you have a vendor folder (either from the submodule or manually created) settings this to off speeds up the build-image target.

  6. Build the cf-operator binary

    bin/build

  7. Build the cf-operator docker image

    bin/build-image

Note: Consider setting DOCKER_IMAGE_TAG to a fixed variable. This will avoid rebuilding the docker image everytime, when doing changes in files not related to the cf-operator binary.

Note: When not running in CI, nothing ensures a proper cleanup of resources after the deletion of the cf-operator in the environment. You can make sure to manually verify that none old resources will interfere with a future installation, by:

# Deleting old mutating webhooks configurations
kubectl get mutatingwebhookconfiguration -oname | xargs -n 1 kubectl delete

In KinD

The following steps are necessary to have a proper environment setup, where all types of tests can be executed:

  1. Install KinD

Follow the instructions from https://github.com/kubernetes-sigs/kind/

  1. Start cluster

    kind create cluster --image kindest/node:v1.15.6

  2. Export the CF_OPERATOR_WEBHOOK_SERVICE_HOST env variable. Use the IP of the docker bridge or your public IP. Firewall rules may interfere.

    export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip -4 a s dev $(ip r l 0/0 | cut -f5 -d' ') | grep -oP 'inet \K\S+(?=/)')

Note: On Mac, use export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip a s $(ip r g 0/0 | cut -f5 -d' ') | grep -oE 'inet [^ /]+' | cut -f2 -d' '), because grep cannot handle perl regexs.

  1. Export the OPERATOR_TEST_STORAGE_CLASS env variable

    export OPERATOR_TEST_STORAGE_CLASS=standard

    Note: Required for the PVC tests.

  2. Build the cf-operator docker image

    First set the version to something static, not dependant on git:

    export DOCKER_IMAGE_TAG=${DOCKER_IMAGE_TAG:-dev}
    bin/build-image

    Or if you have local changes and use go mod edit --replace, follow instructions from development.

  3. Load image into KinD

    kind load docker-image cfcontainerization/cf-operator:$DOCKER_IMAGE_TAG

  4. Set QuarksJob dependency. Choose a tag from docker.io.

    export QUARKS_JOB_IMAGE_TAG=${QUARKS_JOB_IMAGE_TAG:-dev}

    If using a locally built quarks-job image, load it via

    kind load docker-image cfcontainerization/quarks-job:$QUARKS_JOB_IMAGE_TAG

    (see development).

Makefile

The following are the make targets available and their actions.

General Targets

Name Action
all install dependencies, run tests and builds cf-operator binary.
up starts the operator using the binary created by build make target.
vet runs the code analyzing tool vet to identify problems in the source code.
lint runs go lintto identify style mistakes.
tools installs go dependencies required to cf-operator.
check-scripts runs shellcheck to identify syntax, semmantic and subtle caveats in shell scripts.

Build Targets

Name Action
build builds the cf-operator binary.
build-image builds the cf-operator docker image.
build-helm builds the cf-operator helm tar file.

Test Targets

Name Action
test runs unit,integration and e2e tests.
test-unit runs unit tests only.
test-integration runs integration tests only.
test-cli-e2e runs end to end tests for CLI.
test-helm-e2e runs end to end tests on k8s using helm install.
test-integration-storage runs integration storage tests.
test-helm-e2e-storage runs e2e storage tests.

Generate Targets

Name Action
generate runs gen-kube and gen-fakes.
gen-kube generates kube client,informers, lister code.
gen-fakes generates fake objects for unit testing.
gen-command-docs generates docs for all commands.
verify-gen-kube informs if you need to run gen-kube make target.

CI

Our Concourse pipeline definitions are kept in the cf-operator-ci repo.