Deleting fields from CR .spec not updating during tests

[anderssonw]

For context we are attempting to test a kubernetes operator. It's OS, see: https://github.com/kartverket/skiperator. The operator creates new resources if certain parts of the spec has been set. It should delete the same resources if you remove this field from the spec.

Currently the tests are somewhat lacking, so I've added a bunch of new tests in the branch test-improvements. However, we see that a lot of the error assertions which look for deleted resources do not work in a mocked control plane, in addition, a few of the tests do not even work in a kind context.

What happened:
The tests:

• probes
• gcp

Fail for both mocked and kind.

The tests:

• access-policy
• service-port

Only fail during mocked control plane.

When checking the application CR while running kuttl using kind, I can see that the CR is not updated, in spite of the step being completed, which should update the CR.

Running through each of the failing tests step by step locally matches the assert and errors files.

What you expected to happen:
CR is updated as expected, and tests are running fine.

How to reproduce it (as minimally and precisely as possible):
Run the tests in the test-improvements branch with the Make command make build-test. This should run the tests locally using a mocked control plane.

Run the tests using kubectl kuttl test --config tests/config-test-local.yaml --kind-context=<YOUR_KIND_CONTEXT> --start-control-plane=false --suppress-log=events to run the tests in a kind cluster.

Anything else we need to know?:
I get a lot of errors like this:

    logger.go:42: 09:51:16 |  | 2023-05-03T09:51:16+02:00       ERROR   Reconciler error        {"controller": "application", "controllerGroup": "skiperator.kartverket.no", "controllerKind": "Application", "Application": {"name":"additional-ports","namespace":"test"}, "namespace": "test", "name": "additional-ports", "reconcileID": "bada5b89-ff00-4d72-a889-fc0157ff993f", "error": "Operation cannot be fulfilled on applications.skiperator.kartverket.no \"additional-ports\": StorageError: invalid object, Code: 4, Key: /registry/skiperator.kartverket.no/applications/test/additional-ports, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8e5d734b-b530-4060-abc7-5b337b08af4a, UID in object meta: "}
    logger.go:42: 09:51:16 |  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    logger.go:42: 09:51:16 |  |         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
    logger.go:42: 09:51:16 |  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    logger.go:42: 09:51:16 |  |         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
    logger.go:42: 09:51:16 |  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    logger.go:42: 09:51:16 |  |         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

Which may be relevant, but it also happens during tests that run fine.

Environment:

• Kubernetes version (use kubectl version):
  • Client Version: v1.26.2
  • Kustomize Version: v4.5.7
  • Server Version: v1.25.3
• KUTTL version (use kubectl kuttl version): KUTTL Version: version.Info{GitVersion:"dev", GitCommit:"dev", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.20.1", Compiler:"gc", Platform:"darwin/arm64"}
• Cloud provider or hardware configuration: Apple M1 Max
• OS (e.g. from /etc/os-release): MacOS v12.6
• Kernel (e.g. uname -a): 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64

[porridge]

Just pointing at a branch in a different repo is asking a bit too much of us I'm afraid.
Please paste here the kuttl files which create the CR and the file(s) which is supposed to remove some fields.