render cleanup hook job using cluster chart

[glitchcrab]

towards https://github.com/giantswarm/giantswarm/issues/31417

This pr:

Uses the cluster chart to render the helmreleases cleanup hook.

Trigger e2e tests

/run cluster-test-suites

[vxav]

❤️

[glitchcrab]

😅

[tinkerers-ci]

cluster-test-suites

Run name	pr-cluster-vsphere-288-cluster-test-suitesb5rrd
Commit SHA	1e7e631
Result	Failed ❌

📋 View full results in Tekton Dashboard

Rerun trigger:
/run cluster-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run cluster-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

[glitchcrab]

i'm going to ignore the test failure there - it looks like tekton did something stupid for one suite - the rest passed just fine.

[github-actions]

There were differences in the rendered Helm template, please check! ⚠️

Output

=== Differences when rendered with values file helm/cluster-vsphere/ci/test-wc-values.yaml ===

(file level)
  - four documents removed:
    ---
    # Source: cluster-vsphere/templates/helmreleases/cleanup-helmreleases-hook-job.yaml
    # Because cluster provider resources are often deleted before flux has a chance
    # to uninstall helm releases for all deleted HelmRelease CRs they become
    # leftovers because there is still flux finalizer on them. This looks like
    # following:
    #
    #     $ kubectl get helmrelease -n org-multi-project
    #     NAME                           AGE     READY   STATUS
    #     pawe1-cilium                   99m     False   failed to get last release revision
    #     pawe1-cloud-provider-vsphere   99m     False   failed to get last release revision
    #
    # Both HelmRelease CRs in this case have deletionTimestamp and finalizers set,
    # e.g.:
    #
    #     deletionTimestamp: "2023-03-02T14:34:49Z"
    #     finalizers:
    #       - finalizers.fluxcd.io
    #
    # To work around this, post-delete Job deletes all finalizers on all HelmRelease
    # CRs created with this chart.
    #
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: release-name-cleanup-helmreleases-hook
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
        helm.sh/hook-weight: "-1"
      labels:
        app: cluster-vsphere
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: release-name
        giantswarm.io/cluster: release-name
        giantswarm.io/organization: giantswarm
        application.giantswarm.io/team: rocket
        app.kubernetes.io/version: 0.63.0
        helm.sh/chart: cluster-vsphere-0.63.0
    # Source: cluster-vsphere/templates/helmreleases/cleanup-helmreleases-hook-job.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: release-name-cleanup-helmreleases-hook
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
        helm.sh/hook-weight: "-1"
      labels:
        app: cluster-vsphere
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: release-name
        giantswarm.io/cluster: release-name
        giantswarm.io/organization: giantswarm
        application.giantswarm.io/team: rocket
        app.kubernetes.io/version: 0.63.0
        helm.sh/chart: cluster-vsphere-0.63.0
    rules:
    - apiGroups:
      - helm.toolkit.fluxcd.io
      resources:
      - helmreleases
      verbs:
      - get
      - list
      - patch
    - apiGroups:
      - source.toolkit.fluxcd.io
      resources:
      - helmcharts
      verbs:
      - get
      - list
      - patch
      - delete
    # Source: cluster-vsphere/templates/helmreleases/cleanup-helmreleases-hook-job.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: release-name-cleanup-helmreleases-hook
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
        helm.sh/hook-weight: "-1"
      labels:
        app: cluster-vsphere
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: release-name
        giantswarm.io/cluster: release-name
        giantswarm.io/organization: giantswarm
        application.giantswarm.io/team: rocket
        app.kubernetes.io/version: 0.63.0
        helm.sh/chart: cluster-vsphere-0.63.0
    subjects:
    - kind: ServiceAccount
      name: release-name-cleanup-helmreleases-hook
      namespace: org-giantswarm
    roleRef:
      kind: Role
      name: release-name-cleanup-helmreleases-hook
      apiGroup: rbac.authorization.k8s.io
    # Source: cluster-vsphere/templates/helmreleases/cleanup-helmreleases-hook-job.yaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: release-name-cleanup-helmreleases-hook
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: before-hook-creation
        helm.sh/hook-weight: 0
      labels:
        app: cluster-vsphere
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: release-name
        giantswarm.io/cluster: release-name
        giantswarm.io/organization: giantswarm
        application.giantswarm.io/team: rocket
        app.kubernetes.io/version: 0.63.0
        helm.sh/chart: cluster-vsphere-0.63.0
    spec:
      ttlSecondsAfterFinished: 86400 # 24h
      template:
        metadata:
          name: release-name-cleanup-helmreleases-hook
          namespace: org-giantswarm
          labels:
            app: cluster-vsphere
            app.kubernetes.io/managed-by: Helm
            cluster.x-k8s.io/cluster-name: release-name
            giantswarm.io/cluster: release-name
            giantswarm.io/organization: giantswarm
            application.giantswarm.io/team: rocket
            app.kubernetes.io/version: 0.63.0
            helm.sh/chart: cluster-vsphere-0.63.0
            cnp: release-name-cleanup-helmreleases-hook
        spec:
          restartPolicy: Never
          serviceAccountName: release-name-cleanup-helmreleases-hook
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
          containers:
          - name: post-delete-job
            image: "gsoci.azurecr.io/giantswarm/kubectl:1.27.14"
            command:
            - /bin/sh
            - "-xc"
            - |
              NAMESPACE="org-giantswarm"
              CLUSTER_NAME="release-name"
              
              for release in $(kubectl get HelmRelease -n ${NAMESPACE} -l "giantswarm.io/cluster=${CLUSTER_NAME}" -o json | jq -r '.items[].metadata.name'); do
                echo "Patching helmrelease $release to remove finalizers"
                kubectl patch -n ${NAMESPACE} helmrelease "${release}" --type=merge -p '{"metadata": {"finalizers": []}}'
              done
              
              for chart in $(kubectl get HelmChart -n ${NAMESPACE} -o json | jq -r ".items[] | select(.spec.sourceRef.name == \"${CLUSTER_NAME}-default\" or .spec.sourceRef.name == \"${CLUSTER_NAME}-cluster\") | .metadata.name"); do
                echo "Patching helmchart $chart to remove finalizers"
                kubectl patch -n ${NAMESPACE} helmchart "${chart}" --type=merge -p '{"metadata": {"finalizers": []}}'
                echo "Deleting helmchart $chart"
                kubectl delete -n ${NAMESPACE} helmchart "${chart}" --ignore-not-found=true
              done
              
              echo "Unpausing cluster ${CLUSTER_NAME} to allow deletion..."
              kubectl patch -n ${NAMESPACE} cluster.cluster.x-k8s.io "${CLUSTER_NAME}" --type=merge -p '{"spec": {"paused": false}}'
              
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              seccompProfile:
                type: RuntimeDefault
              readOnlyRootFilesystem: true
            resources:
              requests:
                memory: 64Mi
                cpu: 10m
              limits:
                memory: 256Mi
                cpu: 100m
    
  
    ---
    # Source: cluster-vsphere/charts/cluster/templates/apps/helmreleases-cleanup/serviceaccount.yaml
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: test-helmreleases-cleanup
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
        helm.sh/hook-weight: "-1"
      labels:
        # deprecated: "app: cluster-vsphere" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-vsphere
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.2.1
        app.kubernetes.io/part-of: cluster-vsphere
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.2.1
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test
        giantswarm.io/organization: giantswarm
        giantswarm.io/service-priority: highest
        cluster.x-k8s.io/cluster-name: test
        cluster.x-k8s.io/watch-filter: capi
    # Source: cluster-vsphere/charts/cluster/templates/apps/helmreleases-cleanup/role.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: test-helmreleases-cleanup
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
        helm.sh/hook-weight: "-1"
      labels:
        # deprecated: "app: cluster-vsphere" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-vsphere
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.2.1
        app.kubernetes.io/part-of: cluster-vsphere
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.2.1
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test
        giantswarm.io/organization: giantswarm
        giantswarm.io/service-priority: highest
        cluster.x-k8s.io/cluster-name: test
        cluster.x-k8s.io/watch-filter: capi
    rules:
    - apiGroups:
      - helm.toolkit.fluxcd.io
      resources:
      - helmreleases
      verbs:
      - list
      - get
      - patch
    # Source: cluster-vsphere/charts/cluster/templates/apps/helmreleases-cleanup/rolebinding.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: test-helmreleases-cleanup
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
        helm.sh/hook-weight: "-1"
      labels:
        # deprecated: "app: cluster-vsphere" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-vsphere
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.2.1
        app.kubernetes.io/part-of: cluster-vsphere
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.2.1
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test
        giantswarm.io/organization: giantswarm
        giantswarm.io/service-priority: highest
        cluster.x-k8s.io/cluster-name: test
        cluster.x-k8s.io/watch-filter: capi
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: test-helmreleases-cleanup
    subjects:
    - kind: ServiceAccount
      name: test-helmreleases-cleanup
      namespace: org-giantswarm
    # Source: cluster-vsphere/charts/cluster/templates/apps/helmreleases-cleanup/job.yaml
    #
    # Because cluster resources are often deleted before Flux has a chance to uninstall the Helm releases for all deleted HelmRelease CRs,
    # they become leftovers because there is still a Flux finalizer on them.
    #
    # This looks as follows:
    #
    #     $ kubectl get helmreleases --namespace org-multi-project
    #     NAME                           AGE     READY   STATUS
    #     pawe1-cilium                   99m     False   failed to get last release revision
    #     pawe1-cloud-provider-vsphere   99m     False   failed to get last release revision
    #
    # Both HelmRelease CRs in this case have a deletion timestamp and finalizer set, e.g.:
    #
    #     deletionTimestamp: "2023-03-02T14:34:49Z"
    #     finalizers:
    #     - finalizers.fluxcd.io
    #
    # To work around this, this post-delete hook suspends all HelmRelease CRs created with this chart.
    #
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: test-helmreleases-cleanup
      namespace: org-giantswarm
      annotations:
        helm.sh/hook: post-delete
        helm.sh/hook-delete-policy: before-hook-creation
        helm.sh/hook-weight: 0
      labels:
        # deprecated: "app: cluster-vsphere" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-vsphere
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.2.1
        app.kubernetes.io/part-of: cluster-vsphere
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.2.1
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test
        giantswarm.io/organization: giantswarm
        giantswarm.io/service-priority: highest
        cluster.x-k8s.io/cluster-name: test
        cluster.x-k8s.io/watch-filter: capi
    spec:
      template:
        metadata:
          labels:
            # deprecated: "app: cluster-vsphere" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-vsphere
            app.kubernetes.io/name: cluster
            app.kubernetes.io/version: 1.2.1
            app.kubernetes.io/part-of: cluster-vsphere
            app.kubernetes.io/instance: release-name
            app.kubernetes.io/managed-by: Helm
            helm.sh/chart: cluster-1.2.1
            application.giantswarm.io/team: turtles
            giantswarm.io/cluster: test
            giantswarm.io/organization: giantswarm
            giantswarm.io/service-priority: highest
            cluster.x-k8s.io/cluster-name: test
            cluster.x-k8s.io/watch-filter: capi
        spec:
          serviceAccountName: test-helmreleases-cleanup
          containers:
          - name: kubectl
            image: "gsoci.azurecr.io/giantswarm/kubectl:1.27.14"
            securityContext:
              runAsNonRoot: true
              runAsUser: 1000
              runAsGroup: 1000
              allowPrivilegeEscalation: false
              seccompProfile:
                type: RuntimeDefault
              capabilities:
                drop:
                - ALL
              readOnlyRootFilesystem: true
            env:
            - name: NAMESPACE
              value: org-giantswarm
            - name: CLUSTER
              value: test
            command:
            - /bin/sh
            args:
            - "-c"
            - |
              # Print namespace & cluster.
              echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
              
              # Get releases.
              releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
              
              # Check releases.
              if [ -n "${releases}" ]
              then
                # Patch releases.
                kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
              else
                # Print info.
                echo "No releases to patch found."
              fi
              
            resources:
              requests:
                cpu: 10m
                memory: 64Mi
              limits:
                cpu: 100m
                memory: 256Mi
          restartPolicy: Never
      ttlSecondsAfterFinished: 86400 # 24h