Manual Secret Changes Not Automatically Synchronized to Desired State

[ehsan-hedayatpour]

What happened?

I've encountered an issue with the Crossplane Kubernetes Provider related to the synchronization of Kubernetes secrets when manually changed using kubectl edit command.

How can we reproduce it?

I created a Crossplane resource with the following specifications:

$ cat << EOF > ./secret_object.yaml
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
metadata:
  name: my-object
spec:
  managementPolicy: ObserveCreateUpdate
  forProvider:
    manifest:
      apiVersion: v1
      kind: Secret
      metadata:
        name: my-secret
        namespace: default
      data:
        field1: VGhlRmlyc3RGaWVsZA==
        field2: VGhlU2Vjb25kRmllbGQ=
  providerConfigRef:
    name: k8s-access
EOF

$ kubectl apply -f secret_object.yaml

When manually changing the secret using kubectl edit secret my-secret, the Provider doesn't recognize the modification, and the secret remains unchanged(for example, removing field2).
However, when I change the secret using kubectl apply -f secret_object.yml, the provider properly detects the changes and replaces the secret to align with the desired state.

What environment did it happen in?

Crossplane Version: v1.14.3
Provider-Kubernetes Version: v0.9.0
Kubernetes Version: v1.27 (on-premise, Installed using kubespray)
OS: Ubuntu 22.04

[phisco]

The provider should "reconcile" it after 60 seconds or according to the configured poll interval IIRC, there is another issue to make provider-kubernetes watch resources instead of polling.

[olzemal]

We encountered this too and noticed the provider only compares/observes the kubectl.kubernetes.io/last-applied-configuration annotation. https://github.com/crossplane-contrib/provider-kubernetes/blob/main/internal/controller/object/object.go#L634

Maybe kubectl edit ... does not set this annotation and therefore the provider did not realize the "current-state" changed.

Obviously it would be nice if the provider compared the resource itself instead of the annotation, but this is probably impossible because of mutating webhooks etc.