Upgrading kube-prometheus-stack raises conflicts with "helm-controller" using apiextensions.k8s.io/v1

[ecurbelo]

Scenario

We have installed in a EKS cluster kube-prometheus-stack v58.1.2 and we want to update it to v61.3.1.

We need to run manually a few commands first before the upgrade according to the release notes:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.75.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.75.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
[...]

$ flux check  
► checking prerequisites
✗ flux 2.0.0 <2.3.0 (new version is available, please upgrade)
✔ Kubernetes 1.30.2-eks-db838b0 >=1.24.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.36.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.1.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.1.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.1.2
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

The bug

But when we run the first command, got this error:

$ kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.75.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
error: Apply failed with 3 conflicts: conflicts with "helm-controller" using apiextensions.k8s.io/v1:
- .metadata.annotations.controller-gen.kubebuilder.io/version
- .metadata.annotations.operator.prometheus.io/version
- .spec.versions
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

I found these two threads (1 and 2) talking about this same issue and the solution provided was Controlling the lifecycle of Custom Resource Definitions.

I changed the kube-prometheus-stack Helm Release but i am still getting the same error. Also tried to restart the prometheus and operator pods:

$ kubectl rollout restart statefulset.apps/prometheus-kube-prometheus-stack-prometheus -n kube-prometheus-stack
$ kubectl rollout restart deployment.apps/kube-prometheus-stack-operator -n kube-prometheus-stack

kube-prometheus-stack Helm Release

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
  namespace: flux-system
spec:
  releaseName: kube-prometheus-stack
  targetNamespace: kube-prometheus-stack
  interval: 5m
  chart:
    spec:
      chart: ./deploy/kube-prometheus-stack/operator
      sourceRef:
        kind: GitRepository
        name: monitoring-stack
        namespace: flux-system
      valuesFiles:
        - ./deploy/kube-prometheus-stack/operator/values.yaml

  install:
    createNamespace: true
    remediation: &remediation
      retries: 10
      remediateLastFailure: true
    crds: CreateReplace    <------------------------------------------ Added this line
  upgrade:
    remediation: *remediation
    crds: CreateReplace    <------------------------------------------ Added this line

Do you know if i am doing something wrong or what i am missing?