Toggle audit

[njuettner]

Towards: giantswarm/roadmap#3669

Checklist

• Updated CHANGELOG.md.

Trigger E2E tests

/run cluster-test-suites

[tinkerers-ci]

Note

As this is a draft PR no triggers from the PR body will be handled.

If you'd like to trigger them while draft please add them as a PR comment.

[github-actions]

There were differences in the rendered Helm template, please check! ⚠️

Output

=== Differences when rendered with values file helm/cluster-azure/ci/test-mc-custom-vnet-values.yaml ===

(file level)
  - one document removed:
    ---
    # Source: cluster-azure/charts/cluster/templates/clusterapi/workers/kubeadmconfigtemplate.yaml
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      name: test-mc-def00-86469
      namespace: org-giantswarm
      labels:
        giantswarm.io/machine-deployment: test-mc-def00
        # deprecated: "app: cluster-azure" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-azure
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.2.2
        app.kubernetes.io/part-of: cluster-azure
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.2.2
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test-mc
        giantswarm.io/organization: test
        giantswarm.io/service-priority: lowest
        cluster.x-k8s.io/cluster-name: test-mc
        cluster.x-k8s.io/watch-filter: capi
        release.giantswarm.io/version: 25.0.0
    spec:
      template:
        spec:
          format: ignition
          ignition:
            containerLinuxConfig:
              additionalConfig: |
                systemd:
                  units:      
                  - name: os-hardening.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Apply os hardening
                      [Service]
                      Type=oneshot
                      ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                      ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                      ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                      [Install]
                      WantedBy=multi-user.target
                  - name: update-engine.service
                    enabled: false
                    mask: true
                  - name: locksmithd.service
                    enabled: false
                    mask: true
                  - name: sshkeys.service
                    enabled: false
                    mask: true
                  - name: teleport.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Teleport Service
                      After=network.target
                      [Service]
                      Type=simple
                      Restart=on-failure
                      ExecStart=/opt/bin/teleport start --roles=node --config=/etc/teleport.yaml --pid-file=/run/teleport.pid
                      ExecReload=/bin/kill -HUP $MAINPID
                      PIDFile=/run/teleport.pid
                      LimitNOFILE=524288
                      [Install]
                      WantedBy=multi-user.target
                  - name: kubeadm.service
                    dropins:
                    - name: 10-flatcar.conf
                      contents: |
                        [Unit]
                        # kubeadm must run after coreos-metadata populated /run/metadata directory.
                        Requires=coreos-metadata.service
                        After=coreos-metadata.service
                        # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939.
                        After=containerd.service
                        # kubeadm requires having an IP
                        After=network-online.target
                        Wants=network-online.target
                        [Service]
                        # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                        Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                        # To make metadata environment variables available for pre-kubeadm commands.
                        EnvironmentFile=/run/metadata/*
                  - name: containerd.service
                    enabled: true
                    contents: |
                    dropins:
                    - name: 10-change-cgroup.conf
                      contents: |
                        [Service]
                        CPUAccounting=true
                        MemoryAccounting=true
                        Slice=kubereserved.slice
                  - name: audit-rules.service
                    enabled: true
                    dropins:
                    - name: 10-wait-for-containerd.conf
                      contents: |
                        [Service]
                        ExecStartPre=/bin/bash -c "while [ ! -f /etc/audit/rules.d/containerd.rules ]; do echo 'Waiting for /etc/audit/rules.d/containerd.rules to be written' && sleep 1; done"
                        Restart=on-failure      
                storage:
                  filesystems:      
                  directories:      
                  - path: /var/lib/kubelet
                    mode: 0750      
                
          joinConfiguration:
            nodeRegistration:
              name: ${HOSTNAME}
              kubeletExtraArgs:
                azure-container-registry-config: /etc/kubernetes/azure.json
                cloud-provider: external
                cloud-config: /etc/kubernetes/azure.json
                healthz-bind-address: 0.0.0.0
                node-ip: ${COREOS_AZURE_IPV4_DYNAMIC}
                node-labels: "ip=${COREOS_AZURE_IPV4_DYNAMIC},role=worker,giantswarm.io/machine-pool=test-mc-def00,"
                v: 2
            patches:
              directory: /etc/kubernetes/patches
          preKubeadmCommands:
          - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
          - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
          - "systemctl restart containerd"
          - "/bin/test ! -d /var/lib/kubelet && (/bin/mkdir -p /var/lib/kubelet && /bin/chmod 0750 /var/lib/kubelet)"
          - "sed -i -e 's/registry.k8s.io\/pause/quay.io\/giantswarm\/pause/' /etc/sysconfig/kubelet"
          files:
          - path: /etc/sysctl.d/hardening.conf
            permissions: 0644
            encoding: base64
            content: ZnMuaW5vdGlmeS5tYXhfdXNlcl93YXRjaGVzID0gMTYzODQKZnMuaW5vdGlmeS5tYXhfdXNlcl9pbnN0YW5jZXMgPSA4MTkyCmtlcm5lbC5rcHRyX3Jlc3RyaWN0ID0gMgprZXJuZWwuc3lzcnEgPSAwCm5ldC5pcHY0LmNvbmYuYWxsLmxvZ19tYXJ0aWFucyA9IDEKbmV0LmlwdjQuY29uZi5hbGwuc2VuZF9yZWRpcmVjdHMgPSAwCm5ldC5pcHY0LmNvbmYuZGVmYXVsdC5hY2NlcHRfcmVkaXJlY3RzID0gMApuZXQuaXB2NC5jb25mLmRlZmF1bHQubG9nX21hcnRpYW5zID0gMQpuZXQuaXB2NC50Y3BfdGltZXN0YW1wcyA9IDAKbmV0LmlwdjYuY29uZi5hbGwuYWNjZXB0X3JlZGlyZWN0cyA9IDAKbmV0LmlwdjYuY29uZi5kZWZhdWx0LmFjY2VwdF9yZWRpcmVjdHMgPSAwCiMgSW5jcmVhc2VkIG1tYXBmcyBiZWNhdXNlIHNvbWUgYXBwbGljYXRpb25zLCBsaWtlIEVTLCBuZWVkIGhpZ2hlciBsaW1pdCB0byBzdG9yZSBkYXRhIHByb3Blcmx5CnZtLm1heF9tYXBfY291bnQgPSAyNjIxNDQKIyBSZXNlcnZlZCB0byBhdm9pZCBjb25mbGljdHMgd2l0aCBrdWJlLWFwaXNlcnZlciwgd2hpY2ggYWxsb2NhdGVzIHdpdGhpbiB0aGlzIHJhbmdlCm5ldC5pcHY0LmlwX2xvY2FsX3Jlc2VydmVkX3BvcnRzPTMwMDAwLTMyNzY3Cm5ldC5pcHY0LmNvbmYuYWxsLnJwX2ZpbHRlciA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2lnbm9yZSA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2Fubm91bmNlID0gMgoKIyBUaGVzZSBhcmUgcmVxdWlyZWQgZm9yIHRoZSBrdWJlbGV0ICctLXByb3RlY3Qta2VybmVsLWRlZmF1bHRzJyBmbGFnCiMgU2VlIGh0dHBzOi8vZ2l0aHViLmNvbS9naWFudHN3YXJtL2dpYW50c3dhcm0vaXNzdWVzLzEzNTg3CnZtLm92ZXJjb21taXRfbWVtb3J5PTEKa2VybmVsLnBhbmljPTEwCmtlcm5lbC5wYW5pY19vbl9vb3BzPTEK
          - path: /etc/selinux/config
            permissions: 0644
            encoding: base64
            content: IyBUaGlzIGZpbGUgY29udHJvbHMgdGhlIHN0YXRlIG9mIFNFTGludXggb24gdGhlIHN5c3RlbSBvbiBib290LgoKIyBTRUxJTlVYIGNhbiB0YWtlIG9uZSBvZiB0aGVzZSB0aHJlZSB2YWx1ZXM6CiMgICAgICAgZW5mb3JjaW5nIC0gU0VMaW51eCBzZWN1cml0eSBwb2xpY3kgaXMgZW5mb3JjZWQuCiMgICAgICAgcGVybWlzc2l2ZSAtIFNFTGludXggcHJpbnRzIHdhcm5pbmdzIGluc3RlYWQgb2YgZW5mb3JjaW5nLgojICAgICAgIGRpc2FibGVkIC0gTm8gU0VMaW51eCBwb2xpY3kgaXMgbG9hZGVkLgpTRUxJTlVYPXBlcm1pc3NpdmUKCiMgU0VMSU5VWFRZUEUgY2FuIHRha2Ugb25lIG9mIHRoZXNlIGZvdXIgdmFsdWVzOgojICAgICAgIHRhcmdldGVkIC0gT25seSB0YXJnZXRlZCBuZXR3b3JrIGRhZW1vbnMgYXJlIHByb3RlY3RlZC4KIyAgICAgICBzdHJpY3QgICAtIEZ1bGwgU0VMaW51eCBwcm90ZWN0aW9uLgojICAgICAgIG1scyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1MZXZlbCBTZWN1cml0eQojICAgICAgIG1jcyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1DYXRlZ29yeSBTZWN1cml0eQojICAgICAgICAgICAgICAgICAgKG1scywgYnV0IG9ubHkgb25lIHNlbnNpdGl2aXR5IGxldmVsKQpTRUxJTlVYVFlQRT1tY3MK
          - path: /etc/containerd/config.toml
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-containerd-ed23c962
                key: config.toml
          - path: /etc/kubernetes/patches/kubeletconfiguration.yaml
            permissions: 0644
            encoding: base64
            content: YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24Kc2h1dGRvd25HcmFjZVBlcmlvZDogMzAwcwpzaHV0ZG93bkdyYWNlUGVyaW9kQ3JpdGljYWxQb2RzOiA2MHMKa2VybmVsTWVtY2dOb3RpZmljYXRpb246IHRydWUKZXZpY3Rpb25Tb2Z0OgogIG1lbW9yeS5hdmFpbGFibGU6ICI1MDBNaSIKZXZpY3Rpb25IYXJkOgogIG1lbW9yeS5hdmFpbGFibGU6ICIyMDBNaSIKICBpbWFnZWZzLmF2YWlsYWJsZTogIjE1JSIKZXZpY3Rpb25Tb2Z0R3JhY2VQZXJpb2Q6CiAgbWVtb3J5LmF2YWlsYWJsZTogIjVzIgpldmljdGlvbk1heFBvZEdyYWNlUGVyaW9kOiA2MAprdWJlUmVzZXJ2ZWQ6CiAgY3B1OiAzNTBtCiAgbWVtb3J5OiAxMjgwTWkKICBlcGhlbWVyYWwtc3RvcmFnZTogMTAyNE1pCmt1YmVSZXNlcnZlZENncm91cDogL2t1YmVyZXNlcnZlZC5zbGljZQpwcm90ZWN0S2VybmVsRGVmYXVsdHM6IHRydWUKc3lzdGVtUmVzZXJ2ZWQ6CiAgY3B1OiAyNTBtCiAgbWVtb3J5OiAzODRNaQpzeXN0ZW1SZXNlcnZlZENncm91cDogL3N5c3RlbS5zbGljZQp0bHNDaXBoZXJTdWl0ZXM6Ci0gVExTX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19BRVNfMjU2X0dDTV9TSEEzODQKLSBUTFNfQ0hBQ0hBMjBfUE9MWTEzMDVfU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNV9TSEEyNTYKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1X1NIQTI1NgotIFRMU19SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEKLSBUTFNfUlNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX1JTQV9XSVRIX0FFU18yNTZfQ0JDX1NIQQotIFRMU19SU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQKc2VyaWFsaXplSW1hZ2VQdWxsczogZmFsc2UKc3RyZWFtaW5nQ29ubmVjdGlvbklkbGVUaW1lb3V0OiAxaAphbGxvd2VkVW5zYWZlU3lzY3RsczoKLSAibmV0LioiCg==
          - path: /etc/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
            permissions: 0700
            encoding: base64
            content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
          - path: /etc/teleport-join-token
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-teleport-join-token
                key: joinToken
          - path: /opt/teleport-node-role.sh
            permissions: 0755
            encoding: base64
            content: IyEvYmluL2Jhc2gKCmlmIHN5c3RlbWN0bCBpcy1hY3RpdmUgLXEga3ViZWxldC5zZXJ2aWNlOyB0aGVuCiAgICBpZiBbIC1lICIvZXRjL2t1YmVybmV0ZXMvbWFuaWZlc3RzL2t1YmUtYXBpc2VydmVyLnlhbWwiIF07IHRoZW4KICAgICAgICBlY2hvICJjb250cm9sLXBsYW5lIgogICAgZWxzZQogICAgICAgIGVjaG8gIndvcmtlciIKICAgIGZpCmVsc2UKICAgIGVjaG8gIiIKZmkK
          - path: /etc/teleport.yaml
            permissions: 0644
            encoding: base64
            content: dmVyc2lvbjogdjMKdGVsZXBvcnQ6CiAgZGF0YV9kaXI6IC92YXIvbGliL3RlbGVwb3J0CiAgam9pbl9wYXJhbXM6CiAgICB0b2tlbl9uYW1lOiAvZXRjL3RlbGVwb3J0LWpvaW4tdG9rZW4KICAgIG1ldGhvZDogdG9rZW4KICBwcm94eV9zZXJ2ZXI6IHRlbGVwb3J0LmdpYW50c3dhcm0uaW86NDQzCiAgbG9nOgogICAgb3V0cHV0OiBzdGRlcnIKYXV0aF9zZXJ2aWNlOgogIGVuYWJsZWQ6ICJubyIKc3NoX3NlcnZpY2U6CiAgZW5hYmxlZDogInllcyIKICBjb21tYW5kczoKICAtIG5hbWU6IG5vZGUKICAgIGNvbW1hbmQ6IFtob3N0bmFtZV0KICAgIHBlcmlvZDogMjRoMG0wcwogIC0gbmFtZTogYXJjaAogICAgY29tbWFuZDogW3VuYW1lLCAtbV0KICAgIHBlcmlvZDogMjRoMG0wcwogIC0gbmFtZTogcm9sZQogICAgY29tbWFuZDogWy9vcHQvdGVsZXBvcnQtbm9kZS1yb2xlLnNoXQogICAgcGVyaW9kOiAxbTBzCiAgbGFiZWxzOgogICAgaW5zOiB0ZXN0LW1jCiAgICBtYzogdGVzdC1tYwogICAgY2x1c3RlcjogdGVzdC1tYwogICAgYmFzZURvbWFpbjogYXp1cmV0ZXN0LmdpZ2FudGljLmlvCnByb3h5X3NlcnZpY2U6CiAgZW5hYmxlZDogIm5vIgo=
          - path: /etc/audit/rules.d/99-default.rules
            permissions: 0640
            encoding: base64
            content: IyBPdmVycmlkZGVuIGJ5IEdpYW50IFN3YXJtLgotYSBleGl0LGFsd2F5cyAtRiBhcmNoPWI2NCAtUyBleGVjdmUgLWsgYXVkaXRpbmcKLWEgZXhpdCxhbHdheXMgLUYgYXJjaD1iMzIgLVMgZXhlY3ZlIC1rIGF1ZGl0aW5nCg==
          - path: /etc/kubernetes/azure.json
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-def00-191ddc5e-azure-json
                key: worker-node-azure.json
            owner: "root:root"
    
  
    ---
    # Source: cluster-azure/charts/cluster/templates/clusterapi/workers/kubeadmconfigtemplate.yaml
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      name: test-mc-def00-80da5
      namespace: org-giantswarm
      labels:
        giantswarm.io/machine-deployment: test-mc-def00
        # deprecated: "app: cluster-azure" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-azure
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.3.0
        app.kubernetes.io/part-of: cluster-azure
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.3.0
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test-mc
        giantswarm.io/organization: test
        giantswarm.io/service-priority: lowest
        cluster.x-k8s.io/cluster-name: test-mc
        cluster.x-k8s.io/watch-filter: capi
        release.giantswarm.io/version: 25.0.0
    spec:
      template:
        spec:
          format: ignition
          ignition:
            containerLinuxConfig:
              additionalConfig: |
                systemd:
                  units:      
                  - name: os-hardening.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Apply os hardening
                      [Service]
                      Type=oneshot
                      ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                      ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                      ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                      [Install]
                      WantedBy=multi-user.target
                  - name: update-engine.service
                    enabled: false
                    mask: true
                  - name: locksmithd.service
                    enabled: false
                    mask: true
                  - name: sshkeys.service
                    enabled: false
                    mask: true
                  - name: teleport.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Teleport Service
                      After=network.target
                      [Service]
                      Type=simple
                      Restart=on-failure
                      ExecStart=/opt/bin/teleport start --roles=node --config=/etc/teleport.yaml --pid-file=/run/teleport.pid
                      ExecReload=/bin/kill -HUP $MAINPID
                      PIDFile=/run/teleport.pid
                      LimitNOFILE=524288
                      [Install]
                      WantedBy=multi-user.target
                  - name: kubeadm.service
                    dropins:
                    - name: 10-flatcar.conf
                      contents: |
                        [Unit]
                        # kubeadm must run after coreos-metadata populated /run/metadata directory.
                        Requires=coreos-metadata.service
                        After=coreos-metadata.service
                        # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939.
                        After=containerd.service
                        # kubeadm requires having an IP
                        After=network-online.target
                        Wants=network-online.target
                        [Service]
                        # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                        Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                        # To make metadata environment variables available for pre-kubeadm commands.
                        EnvironmentFile=/run/metadata/*
                  - name: containerd.service
                    enabled: true
                    contents: |
                    dropins:
                    - name: 10-change-cgroup.conf
                      contents: |
                        [Service]
                        CPUAccounting=true
                        MemoryAccounting=true
                        Slice=kubereserved.slice
                  - name: auditd.service
                    enabled: false      
                storage:
                  filesystems:      
                  directories:      
                  - path: /var/lib/kubelet
                    mode: 0750      
                
          joinConfiguration:
            nodeRegistration:
              name: ${HOSTNAME}
              kubeletExtraArgs:
                azure-container-registry-config: /etc/kubernetes/azure.json
                cloud-provider: external
                cloud-config: /etc/kubernetes/azure.json
                healthz-bind-address: 0.0.0.0
                node-ip: ${COREOS_AZURE_IPV4_DYNAMIC}
                node-labels: "ip=${COREOS_AZURE_IPV4_DYNAMIC},role=worker,giantswarm.io/machine-pool=test-mc-def00,"
                v: 2
            patches:
              directory: /etc/kubernetes/patches
          preKubeadmCommands:
          - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
          - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
          - "systemctl restart containerd"
          - "/bin/test ! -d /var/lib/kubelet && (/bin/mkdir -p /var/lib/kubelet && /bin/chmod 0750 /var/lib/kubelet)"
          - "sed -i -e 's/registry.k8s.io\/pause/quay.io\/giantswarm\/pause/' /etc/sysconfig/kubelet"
          files:
          - path: /etc/sysctl.d/hardening.conf
            permissions: 0644
            encoding: base64
            content: ZnMuaW5vdGlmeS5tYXhfdXNlcl93YXRjaGVzID0gMTYzODQKZnMuaW5vdGlmeS5tYXhfdXNlcl9pbnN0YW5jZXMgPSA4MTkyCmtlcm5lbC5rcHRyX3Jlc3RyaWN0ID0gMgprZXJuZWwuc3lzcnEgPSAwCm5ldC5pcHY0LmNvbmYuYWxsLmxvZ19tYXJ0aWFucyA9IDEKbmV0LmlwdjQuY29uZi5hbGwuc2VuZF9yZWRpcmVjdHMgPSAwCm5ldC5pcHY0LmNvbmYuZGVmYXVsdC5hY2NlcHRfcmVkaXJlY3RzID0gMApuZXQuaXB2NC5jb25mLmRlZmF1bHQubG9nX21hcnRpYW5zID0gMQpuZXQuaXB2NC50Y3BfdGltZXN0YW1wcyA9IDAKbmV0LmlwdjYuY29uZi5hbGwuYWNjZXB0X3JlZGlyZWN0cyA9IDAKbmV0LmlwdjYuY29uZi5kZWZhdWx0LmFjY2VwdF9yZWRpcmVjdHMgPSAwCiMgSW5jcmVhc2VkIG1tYXBmcyBiZWNhdXNlIHNvbWUgYXBwbGljYXRpb25zLCBsaWtlIEVTLCBuZWVkIGhpZ2hlciBsaW1pdCB0byBzdG9yZSBkYXRhIHByb3Blcmx5CnZtLm1heF9tYXBfY291bnQgPSAyNjIxNDQKIyBSZXNlcnZlZCB0byBhdm9pZCBjb25mbGljdHMgd2l0aCBrdWJlLWFwaXNlcnZlciwgd2hpY2ggYWxsb2NhdGVzIHdpdGhpbiB0aGlzIHJhbmdlCm5ldC5pcHY0LmlwX2xvY2FsX3Jlc2VydmVkX3BvcnRzPTMwMDAwLTMyNzY3Cm5ldC5pcHY0LmNvbmYuYWxsLnJwX2ZpbHRlciA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2lnbm9yZSA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2Fubm91bmNlID0gMgoKIyBUaGVzZSBhcmUgcmVxdWlyZWQgZm9yIHRoZSBrdWJlbGV0ICctLXByb3RlY3Qta2VybmVsLWRlZmF1bHRzJyBmbGFnCiMgU2VlIGh0dHBzOi8vZ2l0aHViLmNvbS9naWFudHN3YXJtL2dpYW50c3dhcm0vaXNzdWVzLzEzNTg3CnZtLm92ZXJjb21taXRfbWVtb3J5PTEKa2VybmVsLnBhbmljPTEwCmtlcm5lbC5wYW5pY19vbl9vb3BzPTEK
          - path: /etc/selinux/config
            permissions: 0644
            encoding: base64
            content: IyBUaGlzIGZpbGUgY29udHJvbHMgdGhlIHN0YXRlIG9mIFNFTGludXggb24gdGhlIHN5c3RlbSBvbiBib290LgoKIyBTRUxJTlVYIGNhbiB0YWtlIG9uZSBvZiB0aGVzZSB0aHJlZSB2YWx1ZXM6CiMgICAgICAgZW5mb3JjaW5nIC0gU0VMaW51eCBzZWN1cml0eSBwb2xpY3kgaXMgZW5mb3JjZWQuCiMgICAgICAgcGVybWlzc2l2ZSAtIFNFTGludXggcHJpbnRzIHdhcm5pbmdzIGluc3RlYWQgb2YgZW5mb3JjaW5nLgojICAgICAgIGRpc2FibGVkIC0gTm8gU0VMaW51eCBwb2xpY3kgaXMgbG9hZGVkLgpTRUxJTlVYPXBlcm1pc3NpdmUKCiMgU0VMSU5VWFRZUEUgY2FuIHRha2Ugb25lIG9mIHRoZXNlIGZvdXIgdmFsdWVzOgojICAgICAgIHRhcmdldGVkIC0gT25seSB0YXJnZXRlZCBuZXR3b3JrIGRhZW1vbnMgYXJlIHByb3RlY3RlZC4KIyAgICAgICBzdHJpY3QgICAtIEZ1bGwgU0VMaW51eCBwcm90ZWN0aW9uLgojICAgICAgIG1scyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1MZXZlbCBTZWN1cml0eQojICAgICAgIG1jcyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1DYXRlZ29yeSBTZWN1cml0eQojICAgICAgICAgICAgICAgICAgKG1scywgYnV0IG9ubHkgb25lIHNlbnNpdGl2aXR5IGxldmVsKQpTRUxJTlVYVFlQRT1tY3MK
          - path: /etc/containerd/config.toml
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-containerd-ed23c962
                key: config.toml
          - path: /etc/kubernetes/patches/kubeletconfiguration.yaml
            permissions: 0644
            encoding: base64
            content: YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24Kc2h1dGRvd25HcmFjZVBlcmlvZDogMzAwcwpzaHV0ZG93bkdyYWNlUGVyaW9kQ3JpdGljYWxQb2RzOiA2MHMKa2VybmVsTWVtY2dOb3RpZmljYXRpb246IHRydWUKZXZpY3Rpb25Tb2Z0OgogIG1lbW9yeS5hdmFpbGFibGU6ICI1MDBNaSIKZXZpY3Rpb25IYXJkOgogIG1lbW9yeS5hdmFpbGFibGU6ICIyMDBNaSIKICBpbWFnZWZzLmF2YWlsYWJsZTogIjE1JSIKZXZpY3Rpb25Tb2Z0R3JhY2VQZXJpb2Q6CiAgbWVtb3J5LmF2YWlsYWJsZTogIjVzIgpldmljdGlvbk1heFBvZEdyYWNlUGVyaW9kOiA2MAprdWJlUmVzZXJ2ZWQ6CiAgY3B1OiAzNTBtCiAgbWVtb3J5OiAxMjgwTWkKICBlcGhlbWVyYWwtc3RvcmFnZTogMTAyNE1pCmt1YmVSZXNlcnZlZENncm91cDogL2t1YmVyZXNlcnZlZC5zbGljZQpwcm90ZWN0S2VybmVsRGVmYXVsdHM6IHRydWUKc3lzdGVtUmVzZXJ2ZWQ6CiAgY3B1OiAyNTBtCiAgbWVtb3J5OiAzODRNaQpzeXN0ZW1SZXNlcnZlZENncm91cDogL3N5c3RlbS5zbGljZQp0bHNDaXBoZXJTdWl0ZXM6Ci0gVExTX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19BRVNfMjU2X0dDTV9TSEEzODQKLSBUTFNfQ0hBQ0hBMjBfUE9MWTEzMDVfU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNV9TSEEyNTYKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1X1NIQTI1NgotIFRMU19SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEKLSBUTFNfUlNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX1JTQV9XSVRIX0FFU18yNTZfQ0JDX1NIQQotIFRMU19SU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQKc2VyaWFsaXplSW1hZ2VQdWxsczogZmFsc2UKc3RyZWFtaW5nQ29ubmVjdGlvbklkbGVUaW1lb3V0OiAxaAphbGxvd2VkVW5zYWZlU3lzY3RsczoKLSAibmV0LioiCg==
          - path: /etc/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
            permissions: 0700
            encoding: base64
            content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
          - path: /etc/teleport-join-token
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-teleport-join-token
                key: joinToken
          - path: /opt/teleport-node-role.sh
            permissions: 0755
            encoding: base64
            content: IyEvYmluL2Jhc2gKCmlmIHN5c3RlbWN0bCBpcy1hY3RpdmUgLXEga3ViZWxldC5zZXJ2aWNlOyB0aGVuCiAgICBpZiBbIC1lICIvZXRjL2t1YmVybmV0ZXMvbWFuaWZlc3RzL2t1YmUtYXBpc2VydmVyLnlhbWwiIF07IHRoZW4KICAgICAgICBlY2hvICJjb250cm9sLXBsYW5lIgogICAgZWxzZQogICAgICAgIGVjaG8gIndvcmtlciIKICAgIGZpCmVsc2UKICAgIGVjaG8gIiIKZmkK
          - path: /etc/teleport.yaml
            permissions: 0644
            encoding: base64
            content: dmVyc2lvbjogdjMKdGVsZXBvcnQ6CiAgZGF0YV9kaXI6IC92YXIvbGliL3RlbGVwb3J0CiAgam9pbl9wYXJhbXM6CiAgICB0b2tlbl9uYW1lOiAvZXRjL3RlbGVwb3J0LWpvaW4tdG9rZW4KICAgIG1ldGhvZDogdG9rZW4KICBwcm94eV9zZXJ2ZXI6IHRlbGVwb3J0LmdpYW50c3dhcm0uaW86NDQzCiAgbG9nOgogICAgb3V0cHV0OiBzdGRlcnIKYXV0aF9zZXJ2aWNlOgogIGVuYWJsZWQ6ICJubyIKc3NoX3NlcnZpY2U6CiAgZW5hYmxlZDogInllcyIKICBjb21tYW5kczoKICAtIG5hbWU6IG5vZGUKICAgIGNvbW1hbmQ6IFtob3N0bmFtZV0KICAgIHBlcmlvZDogMjRoMG0wcwogIC0gbmFtZTogYXJjaAogICAgY29tbWFuZDogW3VuYW1lLCAtbV0KICAgIHBlcmlvZDogMjRoMG0wcwogIC0gbmFtZTogcm9sZQogICAgY29tbWFuZDogWy9vcHQvdGVsZXBvcnQtbm9kZS1yb2xlLnNoXQogICAgcGVyaW9kOiAxbTBzCiAgbGFiZWxzOgogICAgaW5zOiB0ZXN0LW1jCiAgICBtYzogdGVzdC1tYwogICAgY2x1c3RlcjogdGVzdC1tYwogICAgYmFzZURvbWFpbjogYXp1cmV0ZXN0LmdpZ2FudGljLmlvCnByb3h5X3NlcnZpY2U6CiAgZW5hYmxlZDogIm5vIgo=
          - path: /etc/kubernetes/azure.json
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-def00-191ddc5e-azure-json
                key: worker-node-azure.json
            owner: "root:root"
    
  

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-mc-cert-manager-user-values)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-mc-cert-manager-user-values)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-mc-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-mc-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-mc-external-dns-user-values)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-mc-external-dns-user-values)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-mc-metrics-server-user-values)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-mc-metrics-server-user-values)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-mc-net-exporter-user-values)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-mc-net-exporter-user-values)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-mc-security-bundle-user-values)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-mc-security-bundle-user-values)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-capi-node-labeler)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-capi-node-labeler)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-cert-exporter)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-cert-exporter)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-cert-manager)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-cert-manager)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-chart-operator-extensions)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-chart-operator-extensions)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-cilium-servicemonitors)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-cilium-servicemonitors)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-etcd-k8s-res-count-exporter)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-etcd-k8s-res-count-exporter)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-external-dns)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-external-dns)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-k8s-audit-metrics)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-k8s-audit-metrics)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-k8s-dns-node-cache)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-k8s-dns-node-cache)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-metrics-server)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-metrics-server)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-net-exporter)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-net-exporter)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-node-exporter)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-node-exporter)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-observability-bundle)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-observability-bundle)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-observability-policies)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-observability-policies)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-prometheus-blackbox-exporter)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-prometheus-blackbox-exporter)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-security-bundle)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-security-bundle)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-teleport-kube-agent)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-teleport-kube-agent)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-vertical-pod-autoscaler)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-mc-vertical-pod-autoscaler)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-mc)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-mc)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-cilium)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-cilium)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-coredns)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-coredns)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-network-policies)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-network-policies)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-vertical-pod-autoscaler-crd)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2beta1/HelmRelease/org-giantswarm/test-mc-vertical-pod-autoscaler-crd)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-default)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-default)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-default-test)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-default-test)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-cluster)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-cluster)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-cluster-test)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1beta2/HelmRepository/org-giantswarm/test-mc-cluster-test)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-mc)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-mc)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/spec/machineTemplate/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-mc)
  ± value change
    - 1.2.2
    + 1.3.0

/spec/machineTemplate/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-mc)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/spec/kubeadmConfigSpec/ignition/containerLinuxConfig/additionalConfig  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-mc)
  ± value change in multiline text (one insert, one deletion)
    -   - name: audit-rules.service
    -     enabled: true
    -     dropins:
    -     - name: 10-wait-for-containerd.conf
    -       contents: |
    -         [Service]
    -         ExecStartPre=/bin/bash -c "while [ ! -f /etc/audit/rules.d/containerd.rules ]; do echo 'Waiting for /etc/audit/rules.d/containerd.rules to be written' && sleep 1; done"
    -         Restart=on-failure      
    +   - name: auditd.service
    +     enabled: false      
  
  

/spec/kubeadmConfigSpec/files  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-mc)
  - one list entry removed:
    - path: /etc/audit/rules.d/99-default.rules
      permissions: 0640
      encoding: base64
      content: IyBPdmVycmlkZGVuIGJ5IEdpYW50IFN3YXJtLgotYSBleGl0LGFsd2F5cyAtRiBhcmNoPWI2NCAtUyBleGVjdmUgLWsgYXVkaXRpbmcKLWEgZXhpdCxhbHdheXMgLUYgYXJjaD1iMzIgLVMgZXhlY3ZlIC1rIGF1ZGl0aW5nCg==
    
  

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineDeployment/org-giantswarm/test-mc-def00)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineDeployment/org-giantswarm/test-mc-def00)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/spec/template/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineDeployment/org-giantswarm/test-mc-def00)
  ± value change
    - 1.2.2
    + 1.3.0

/spec/template/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineDeployment/org-giantswarm/test-mc-def00)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/spec/template/spec/bootstrap/configRef/name  (cluster.x-k8s.io/v1beta1/MachineDeployment/org-giantswarm/test-mc-def00)
  ± value change
    - test-mc-def00-86469
    + test-mc-def00-80da5

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-mc-control-plane)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-mc-control-plane)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-mc-def00)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-mc-def00)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (v1/ServiceAccount/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (v1/ServiceAccount/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - 1.2.2
    + 1.3.0

/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0

/spec/template/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - 1.2.2
    + 1.3.0

/spec/template/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-mc-helmreleases-cleanup)
  ± value change
    - cluster-1.2.2
    + cluster-1.3.0



=== Differences when rendered with values file helm/cluster-azure/ci/test-mc-values.yaml ===

(file level)
  - one document removed:
    ---
    # Source: cluster-azure/charts/cluster/templates/clusterapi/workers/kubeadmconfigtemplate.yaml
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      name: test-mc-def00-3fbe6
      namespace: org-giantswarm
      labels:
        giantswarm.io/machine-deployment: test-mc-def00
        # deprecated: "app: cluster-azure" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-azure
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.2.2
        app.kubernetes.io/part-of: cluster-azure
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.2.2
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test-mc
        giantswarm.io/organization: test
        giantswarm.io/service-priority: lowest
        cluster.x-k8s.io/cluster-name: test-mc
        cluster.x-k8s.io/watch-filter: capi
        release.giantswarm.io/version: 25.0.0
    spec:
      template:
        spec:
          format: ignition
          ignition:
            containerLinuxConfig:
              additionalConfig: |
                systemd:
                  units:      
                  - name: os-hardening.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Apply os hardening
                      [Service]
                      Type=oneshot
                      ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                      ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                      ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                      [Install]
                      WantedBy=multi-user.target
                  - name: update-engine.service
                    enabled: false
                    mask: true
                  - name: locksmithd.service
                    enabled: false
                    mask: true
                  - name: sshkeys.service
                    enabled: false
                    mask: true
                  - name: teleport.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Teleport Service
                      After=network.target
                      [Service]
                      Type=simple
                      Restart=on-failure
                      ExecStart=/opt/bin/teleport start --roles=node --config=/etc/teleport.yaml --pid-file=/run/teleport.pid
                      ExecReload=/bin/kill -HUP $MAINPID
                      PIDFile=/run/teleport.pid
                      LimitNOFILE=524288
                      [Install]
                      WantedBy=multi-user.target
                  - name: kubeadm.service
                    dropins:
                    - name: 10-flatcar.conf
                      contents: |
                        [Unit]
                        # kubeadm must run after coreos-metadata populated /run/metadata directory.
                        Requires=coreos-metadata.service
                        After=coreos-metadata.service
                        # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939.
                        After=containerd.service
                        # kubeadm requires having an IP
                        After=network-online.target
                        Wants=network-online.target
                        [Service]
                        # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                        Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                        # To make metadata environment variables available for pre-kubeadm commands.
                        EnvironmentFile=/run/metadata/*
                  - name: containerd.service
                    enabled: true
                    contents: |
                    dropins:
                    - name: 10-change-cgroup.conf
                      contents: |
                        [Service]
                        CPUAccounting=true
                        MemoryAccounting=true
                        Slice=kubereserved.slice
                  - name: audit-rules.service
                    enabled: true
                    dropins:
                    - name: 10-wait-for-containerd.conf
                      contents: |
                        [Service]
                        ExecStartPre=/bin/bash -c "while [ ! -f /etc/audit/rules.d/containerd.rules ]; do echo 'Waiting for /etc/audit/rules.d/containerd.rules to be written' && sleep 1; done"
                        Restart=on-failure      
                storage:
                  filesystems:      
                  directories:      
                  - path: /var/lib/kubelet
                    mode: 0750      
                
          joinConfiguration:
            nodeRegistration:
              name: ${HOSTNAME}
              kubeletExtraArgs:
                azure-container-registry-config: /etc/kubernetes/azure.json
                cloud-provider: external
                cloud-config: /etc/kubernetes/azure.json
                healthz-bind-address: 0.0.0.0
                node-ip: ${COREOS_AZURE_IPV4_DYNAMIC}
                node-labels: "ip=${COREOS_AZURE_IPV4_DYNAMIC},role=worker,giantswarm.io/machine-pool=test-mc-def00,"
                v: 2
            patches:
              directory: /etc/kubernetes/patches
          preKubeadmCommands:
          - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
          - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
          - "systemctl restart containerd"
          - "/bin/test ! -d /var/lib/kubelet && (/bin/mkdir -p /var/lib/kubelet && /bin/chmod 0750 /var/lib/kubelet)"
          - "sed -i -e 's/registry.k8s.io\/pause/quay.io\/giantswarm\/pause/' /etc/sysconfig/kubelet"
          files:
          - path: /etc/sysctl.d/hardening.conf
            permissions: 0644
            encoding: base64
            content: ZnMuaW5vdGlmeS5tYXhfdXNlcl93YXRjaGVzID0gMTYzODQKZnMuaW5vdGlmeS5tYXhfdXNlcl9pbnN0YW5jZXMgPSA4MTkyCmtlcm5lbC5rcHRyX3Jlc3RyaWN0ID0gMgprZXJuZWwuc3lzcnEgPSAwCm5ldC5pcHY0LmNvbmYuYWxsLmxvZ19tYXJ0aWFucyA9IDEKbmV0LmlwdjQuY29uZi5hbGwuc2VuZF9yZWRpcmVjdHMgPSAwCm5ldC5pcHY0LmNvbmYuZGVmYXVsdC5hY2NlcHRfcmVkaXJlY3RzID0gMApuZXQuaXB2NC5jb25mLmRlZmF1bHQubG9nX21hcnRpYW5zID0gMQpuZXQuaXB2NC50Y3BfdGltZXN0YW1wcyA9IDAKbmV0LmlwdjYuY29uZi5hbGwuYWNjZXB0X3JlZGlyZWN0cyA9IDAKbmV0LmlwdjYuY29uZi5kZWZhdWx0LmFjY2VwdF9yZWRpcmVjdHMgPSAwCiMgSW5jcmVhc2VkIG1tYXBmcyBiZWNhdXNlIHNvbWUgYXBwbGljYXRpb25zLCBsaWtlIEVTLCBuZWVkIGhpZ2hlciBsaW1pdCB0byBzdG9yZSBkYXRhIHByb3Blcmx5CnZtLm1heF9tYXBfY291bnQgPSAyNjIxNDQKIyBSZXNlcnZlZCB0byBhdm9pZCBjb25mbGljdHMgd2l0aCBrdWJlLWFwaXNlcnZlciwgd2hpY2ggYWxsb2NhdGVzIHdpdGhpbiB0aGlzIHJhbmdlCm5ldC5pcHY0LmlwX2xvY2FsX3Jlc2VydmVkX3BvcnRzPTMwMDAwLTMyNzY3Cm5ldC5pcHY0LmNvbmYuYWxsLnJwX2ZpbHRlciA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2lnbm9yZSA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2Fubm91bmNlID0gMgoKIyBUaGVzZSBhcmUgcmVxdWlyZWQgZm9yIHRoZSBrdWJlbGV0ICctLXByb3RlY3Qta2VybmVsLWRlZmF1bHRzJyBmbGFnCiMgU2VlIGh0dHBzOi8vZ2l0aHViLmNvbS9naWFudHN3YXJtL2dpYW50c3dhcm0vaXNzdWVzLzEzNTg3CnZtLm92ZXJjb21taXRfbWVtb3J5PTEKa2VybmVsLnBhbmljPTEwCmtlcm5lbC5wYW5pY19vbl9vb3BzPTEK
          - path: /etc/selinux/config
            permissions: 0644
            encoding: base64
            content: IyBUaGlzIGZpbGUgY29udHJvbHMgdGhlIHN0YXRlIG9mIFNFTGludXggb24gdGhlIHN5c3RlbSBvbiBib290LgoKIyBTRUxJTlVYIGNhbiB0YWtlIG9uZSBvZiB0aGVzZSB0aHJlZSB2YWx1ZXM6CiMgICAgICAgZW5mb3JjaW5nIC0gU0VMaW51eCBzZWN1cml0eSBwb2xpY3kgaXMgZW5mb3JjZWQuCiMgICAgICAgcGVybWlzc2l2ZSAtIFNFTGludXggcHJpbnRzIHdhcm5pbmdzIGluc3RlYWQgb2YgZW5mb3JjaW5nLgojICAgICAgIGRpc2FibGVkIC0gTm8gU0VMaW51eCBwb2xpY3kgaXMgbG9hZGVkLgpTRUxJTlVYPXBlcm1pc3NpdmUKCiMgU0VMSU5VWFRZUEUgY2FuIHRha2Ugb25lIG9mIHRoZXNlIGZvdXIgdmFsdWVzOgojICAgICAgIHRhcmdldGVkIC0gT25seSB0YXJnZXRlZCBuZXR3b3JrIGRhZW1vbnMgYXJlIHByb3RlY3RlZC4KIyAgICAgICBzdHJpY3QgICAtIEZ1bGwgU0VMaW51eCBwcm90ZWN0aW9uLgojICAgICAgIG1scyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1MZXZlbCBTZWN1cml0eQojICAgICAgIG1jcyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1DYXRlZ29yeSBTZWN1cml0eQojICAgICAgICAgICAgICAgICAgKG1scywgYnV0IG9ubHkgb25lIHNlbnNpdGl2aXR5IGxldmVsKQpTRUxJTlVYVFlQRT1tY3MK
          - path: /etc/containerd/config.toml
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-containerd-ed23c962
                key: config.toml
          - path: /etc/kubernetes/patches/kubeletconfiguration.yaml
            permissions: 0644
            encoding: base64
            content: YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24Kc2h1dGRvd25HcmFjZVBlcmlvZDogMzAwcwpzaHV0ZG93bkdyYWNlUGVyaW9kQ3JpdGljYWxQb2RzOiA2MHMKa2VybmVsTWVtY2dOb3RpZmljYXRpb246IHRydWUKZXZpY3Rpb25Tb2Z0OgogIG1lbW9yeS5hdmFpbGFibGU6ICI1MDBNaSIKZXZpY3Rpb25IYXJkOgogIG1lbW9yeS5hdmFpbGFibGU6ICIyMDBNaSIKICBpbWFnZWZzLmF2YWlsYWJsZTogIjE1JSIKZXZpY3Rpb25Tb2Z0R3JhY2VQZXJpb2Q6CiAgbWVtb3J5LmF2YWlsYWJsZTogIjVzIgpldmljdGlvbk1heFBvZEdyYWNlUGVyaW9kOiA2MAprdWJlUmVzZXJ2ZWQ6CiAgY3B1OiAzNTBtCiAgbWVtb3J5OiAxMjgwTWkKICBlcGhlbWVyYWwtc3RvcmFnZTogMTAyNE1pCmt1YmVSZXNlcnZlZENncm91cDogL2t1YmVyZXNlcnZlZC5zbGljZQpwcm90ZWN0S2VybmVsRGVmYXVsdHM6IHRydWUKc3lzdGVtUmVzZXJ2ZWQ6CiAgY3B1OiAyNTBtCiAgbWVtb3J5OiAzODRNaQpzeXN0ZW1SZXNlcnZlZENncm91cDogL3N5c3RlbS5zbGljZQp0bHNDaXBoZXJTdWl0ZXM6Ci0gVExTX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19BRVNfMjU2X0dDTV9TSEEzODQKLSBUTFNfQ0hBQ0hBMjBfUE9MWTEzMDVfU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNV9TSEEyNTYKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1X1NIQTI1NgotIFRMU19SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEKLSBUTFNfUlNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX1JTQV9XSVRIX0FFU18yNTZfQ0JDX1NIQQotIFRMU19SU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQKc2VyaWFsaXplSW1hZ2VQdWxsczogZmFsc2UKc3RyZWFtaW5nQ29ubmVjdGlvbklkbGVUaW1lb3V0OiAxaAphbGxvd2VkVW5zYWZlU3lzY3RsczoKLSAibmV0LioiCg==
          - path: /etc/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
            permissions: 0700
            encoding: base64
            content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
          - path: /etc/teleport-join-token
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-teleport-join-token
                key: joinToken
          - path: /opt/teleport-node-role.sh
            permissions: 0755
            encoding: base64
            content: IyEvYmluL2Jhc2gKCmlmIHN5c3RlbWN0bCBpcy1hY3RpdmUgLXEga3ViZWxldC5zZXJ2aWNlOyB0aGVuCiAgICBpZiBbIC1lICIvZXRjL2t1YmVybmV0ZXMvbWFuaWZlc3RzL2t1YmUtYXBpc2VydmVyLnlhbWwiIF07IHRoZW4KICAgICAgICBlY2hvICJjb250cm9sLXBsYW5lIgogICAgZWxzZQogICAgICAgIGVjaG8gIndvcmtlciIKICAgIGZpCmVsc2UKICAgIGVjaG8gIiIKZmkK
          - path: /etc/teleport.yaml
            permissions: 0644
            encoding: base64
            content: dmVyc2lvbjogdjMKdGVsZXBvcnQ6CiAgZGF0YV9kaXI6IC92YXIvbGliL3RlbGVwb3J0CiAgam9pbl9wYXJhbXM6CiAgICB0b2tlbl9uYW1lOiAvZXRjL3RlbGVwb3J0LWpvaW4tdG9rZW4KICAgIG1ldGhvZDogdG9rZW4KICBwcm94eV9zZXJ2ZXI6IHRlbGVwb3J0LmdpYW50c3dhcm0uaW86NDQzCiAgbG9nOgogICAgb3V0cHV0OiBzdGRlcnIKYXV0aF9zZXJ2aWNlOgogIGVuYWJsZWQ6ICJubyIKc3NoX3NlcnZpY2U6CiAgZW5hYmxlZDogInllcyIKICBjb21tYW5kczoKICAtIG5hbWU6IG5vZGUKICAgIGNvbW1hbmQ6IFtob3N0bmFtZV0KICAgIHBlcmlvZDogMjRoMG0wcwogIC0gbmFtZTogYXJjaAogICAgY29tbWFuZDogW3VuYW1lLCAtbV0KICAgIHBlcmlvZDogMjRoMG0wcwogIC0gbmFtZTogcm9sZQogICAgY29tbWFuZDogWy9vcHQvdGVsZXBvcnQtbm9kZS1yb2xlLnNoXQogICAgcGVyaW9kOiAxbTBzCiAgbGFiZWxzOgogICAgaW5zOiB0ZXN0LW1jCiAgICBtYzogdGVzdC1tYwogICAgY2x1c3RlcjogdGVzdC1tYwogICAgYmFzZURvbWFpbjogYXp1cmV0ZXN0LmdpZ2FudGljLmlvCnByb3h5X3NlcnZpY2U6CiAgZW5hYmxlZDogIm5vIgo=
          - path: /etc/audit/rules.d/99-default.rules
            permissions: 0640
            encoding: base64
            content: IyBPdmVycmlkZGVuIGJ5IEdpYW50IFN3YXJtLgotYSBleGl0LGFsd2F5cyAtRiBhcmNoPWI2NCAtUyBleGVjdmUgLWsgYXVkaXRpbmcKLWEgZXhpdCxhbHdheXMgLUYgYXJjaD1iMzIgLVMgZXhlY3ZlIC1rIGF1ZGl0aW5nCg==
          - path: /etc/kubernetes/azure.json
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-def00-24ae8195-azure-json
                key: worker-node-azure.json
            owner: "root:root"
    
  
    ---
    # Source: cluster-azure/charts/cluster/templates/clusterapi/workers/kubeadmconfigtemplate.yaml
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      name: test-mc-def00-4d95e
      namespace: org-giantswarm
      labels:
        giantswarm.io/machine-deployment: test-mc-def00
        # deprecated: "app: cluster-azure" label is deprecated and it will be removed after upgrading
    # to Kubernetes 1.25. We still need it here because existing ClusterResourceSet selectors
    # need this label on the Cluster resource.
    app: cluster-azure
        app.kubernetes.io/name: cluster
        app.kubernetes.io/version: 1.3.0
        app.kubernetes.io/part-of: cluster-azure
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
        helm.sh/chart: cluster-1.3.0
        application.giantswarm.io/team: turtles
        giantswarm.io/cluster: test-mc
        giantswarm.io/organization: test
        giantswarm.io/service-priority: lowest
        cluster.x-k8s.io/cluster-name: test-mc
        cluster.x-k8s.io/watch-filter: capi
        release.giantswarm.io/version: 25.0.0
    spec:
      template:
        spec:
          format: ignition
          ignition:
            containerLinuxConfig:
              additionalConfig: |
                systemd:
                  units:      
                  - name: os-hardening.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Apply os hardening
                      [Service]
                      Type=oneshot
                      ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                      ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                      ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                      [Install]
                      WantedBy=multi-user.target
                  - name: update-engine.service
                    enabled: false
                    mask: true
                  - name: locksmithd.service
                    enabled: false
                    mask: true
                  - name: sshkeys.service
                    enabled: false
                    mask: true
                  - name: teleport.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Teleport Service
                      After=network.target
                      [Service]
                      Type=simple
                      Restart=on-failure
                      ExecStart=/opt/bin/teleport start --roles=node --config=/etc/teleport.yaml --pid-file=/run/teleport.pid
                      ExecReload=/bin/kill -HUP $MAINPID
                      PIDFile=/run/teleport.pid
                      LimitNOFILE=524288
                      [Install]
                      WantedBy=multi-user.target
                  - name: kubeadm.service
                    dropins:
                    - name: 10-flatcar.conf
                      contents: |
                        [Unit]
                        # kubeadm must run after coreos-metadata populated /run/metadata directory.
                        Requires=coreos-metadata.service
                        After=coreos-metadata.service
                        # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939.
                        After=containerd.service
                        # kubeadm requires having an IP
                        After=network-online.target
                        Wants=network-online.target
                        [Service]
                        # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                        Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                        # To make metadata environment variables available for pre-kubeadm commands.
                        EnvironmentFile=/run/metadata/*
                  - name: containerd.service
                    enabled: true
                    contents: |
                    dropins:
                    - name: 10-change-cgroup.conf
                      contents: |
                        [Service]
                        CPUAccounting=true
                        MemoryAccounting=true
                        Slice=kubereserved.slice
                  - name: auditd.service
                    enabled: false      
                storage:
                  filesystems:      
                  directories:      
                  - path: /var/lib/kubelet
                    mode: 0750      
                
          joinConfiguration:
            nodeRegistration:
              name: ${HOSTNAME}
              kubeletExtraArgs:
                azure-container-registry-config: /etc/kubernetes/azure.json
                cloud-provider: external
                cloud-config: /etc/kubernetes/azure.json
                healthz-bind-address: 0.0.0.0
                node-ip: ${COREOS_AZURE_IPV4_DYNAMIC}
                node-labels: "ip=${COREOS_AZURE_IPV4_DYNAMIC},role=worker,giantswarm.io/machine-pool=test-mc-def00,"
                v: 2
            patches:
              directory: /etc/kubernetes/patches
          preKubeadmCommands:
          - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
          - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
          - "systemctl restart containerd"
          - "/bin/test ! -d /var/lib/kubelet && (/bin/mkdir -p /var/lib/kubelet && /bin/chmod 0750 /var/lib/kubelet)"
          - "sed -i -e 's/registry.k8s.io\/pause/quay.io\/giantswarm\/pause/' /etc/sysconfig/kubelet"
          files:
          - path: /etc/sysctl.d/hardening.conf
            permissions: 0644
            encoding: base64
            content: ZnMuaW5vdGlmeS5tYXhfdXNlcl93YXRjaGVzID0gMTYzODQKZnMuaW5vdGlmeS5tYXhfdXNlcl9pbnN0YW5jZXMgPSA4MTkyCmtlcm5lbC5rcHRyX3Jlc3RyaWN0ID0gMgprZXJuZWwuc3lzcnEgPSAwCm5ldC5pcHY0LmNvbmYuYWxsLmxvZ19tYXJ0aWFucyA9IDEKbmV0LmlwdjQuY29uZi5hbGwuc2VuZF9yZWRpcmVjdHMgPSAwCm5ldC5pcHY0LmNvbmYuZGVmYXVsdC5hY2NlcHRfcmVkaXJlY3RzID0gMApuZXQuaXB2NC5jb25mLmRlZmF1bHQubG9nX21hcnRpYW5zID0gMQpuZXQuaXB2NC50Y3BfdGltZXN0YW1wcyA9IDAKbmV0LmlwdjYuY29uZi5hbGwuYWNjZXB0X3JlZGlyZWN0cyA9IDAKbmV0LmlwdjYuY29uZi5kZWZhdWx0LmFjY2VwdF9yZWRpcmVjdHMgPSAwCiMgSW5jcmVhc2VkIG1tYXBmcyBiZWNhdXNlIHNvbWUgYXBwbGljYXRpb25zLCBsaWtlIEVTLCBuZWVkIGhpZ2hlciBsaW1pdCB0byBzdG9yZSBkYXRhIHByb3Blcmx5CnZtLm1heF9tYXBfY291bnQgPSAyNjIxNDQKIyBSZXNlcnZlZCB0byBhdm9pZCBjb25mbGljdHMgd2l0aCBrdWJlLWFwaXNlcnZlciwgd2hpY2ggYWxsb2NhdGVzIHdpdGhpbiB0aGlzIHJhbmdlCm5ldC5pcHY0LmlwX2xvY2FsX3Jlc2VydmVkX3BvcnRzPTMwMDAwLTMyNzY3Cm5ldC5pcHY0LmNvbmYuYWxsLnJwX2ZpbHRlciA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2lnbm9yZSA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2Fubm91bmNlID0gMgoKIyBUaGVzZSBhcmUgcmVxdWlyZWQgZm9yIHRoZSBrdWJlbGV0ICctLXByb3RlY3Qta2VybmVsLWRlZmF1bHRzJyBmbGFnCiMgU2VlIGh0dHBzOi8vZ2l0aHViLmNvbS9naWFudHN3YXJtL2dpYW50c3dhcm0vaXNzdWVzLzEzNTg3CnZtLm92ZXJjb21taXRfbWVtb3J5PTEKa2VybmVsLnBhbmljPTEwCmtlcm5lbC5wYW5pY19vbl9vb3BzPTEK
          - path: /etc/selinux/config
            permissions: 0644
            encoding: base64
            content: IyBUaGlzIGZpbGUgY29udHJvbHMgdGhlIHN0YXRlIG9mIFNFTGludXggb24gdGhlIHN5c3RlbSBvbiBib290LgoKIyBTRUxJTlVYIGNhbiB0YWtlIG9uZSBvZiB0aGVzZSB0aHJlZSB2YWx1ZXM6CiMgICAgICAgZW5mb3JjaW5nIC0gU0VMaW51eCBzZWN1cml0eSBwb2xpY3kgaXMgZW5mb3JjZWQuCiMgICAgICAgcGVybWlzc2l2ZSAtIFNFTGludXggcHJpbnRzIHdhcm5pbmdzIGluc3RlYWQgb2YgZW5mb3JjaW5nLgojICAgICAgIGRpc2FibGVkIC0gTm8gU0VMaW51eCBwb2xpY3kgaXMgbG9hZGVkLgpTRUxJTlVYPXBlcm1pc3NpdmUKCiMgU0VMSU5VWFRZUEUgY2FuIHRha2Ugb25lIG9mIHRoZXNlIGZvdXIgdmFsdWVzOgojICAgICAgIHRhcmdldGVkIC0gT25seSB0YXJnZXRlZCBuZXR3b3JrIGRhZW1vbnMgYXJlIHByb3RlY3RlZC4KIyAgICAgICBzdHJpY3QgICAtIEZ1bGwgU0VMaW51eCBwcm90ZWN0aW9uLgojICAgICAgIG1scyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1MZXZlbCBTZWN1cml0eQojICAgICAgIG1jcyAgICAgIC0gRnVsbCBTRUxpbnV4IHByb3RlY3Rpb24gd2l0aCBNdWx0aS1DYXRlZ29yeSBTZWN1cml0eQojICAgICAgICAgICAgICAgICAgKG1scywgYnV0IG9ubHkgb25lIHNlbnNpdGl2aXR5IGxldmVsKQpTRUxJTlVYVFlQRT1tY3MK
          - path: /etc/containerd/config.toml
            permissions: 0644
            contentFrom:
              secret:
                name: test-mc-containerd-ed23c962
                key: config.toml
          - path: /etc/kubernetes/patches/kubeletconfiguration.yaml
            permissions: 0644
            encoding: base64
            content: YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24Kc2h1dGRvd25HcmFjZVBlcmlvZDogMzAwcwpzaHV0ZG93bkdyYWNlUGVyaW9kQ3JpdGljYWxQb2RzOiA2MHMKa2VybmVsTWVtY2dOb3RpZmljYXRpb246IHRydWUKZXZpY3Rpb25Tb2Z0OgogIG1lbW9yeS5hdmFpbGFibGU6ICI1MDBNaSIKZXZpY3Rpb25IYXJkOgogIG1lbW9yeS5hdmFpbGFibGU6ICIyMDBNaSIKICBpbWFnZWZzLmF2YWlsYWJsZTogIjE1JSIKZXZpY3Rpb25Tb2Z0R3JhY2VQZXJpb2Q6CiAgbWVtb3J5LmF2YWlsYWJsZTogIjVzIgpldmljdGlvbk1heFBvZEdyYWNlUGVyaW9kOiA2MAprdWJlUmVzZXJ2ZWQ6CiAgY3B1OiAzNTBtCiAgbWVtb3J5OiAxMjgwTWkKICBlcGhlbWVyYWwtc3RvcmFnZTogMTAyNE1pCmt1YmVSZXNlcnZlZENncm91cDogL2t1YmVyZXNlcnZlZC5zbGljZQpwcm90ZWN0S2VybmVsRGVmYXVsdHM6IHRydWUKc3lzdGVtUmVzZXJ2ZWQ6CiAgY3B1OiAyNTBtCiAgbWVtb3J5OiAzODRNaQpzeXN0ZW1SZXNlcnZlZENncm91cDogL3N5c3RlbS5zbGljZQp0bHNDaXBoZXJTdWl0ZXM6Ci0gVExTX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19BRVNfMjU2X0dDTV9TSEEzODQKLSBUTFNfQ0hBQ0hBMjBfUE9MWTEzMDVfU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBCi0gVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNV9TSEEyNTYKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBCi0gVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NgotIFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIw...*[Comment body truncated]*

[njuettner]

/run cluster-test-suites

[tinkerers-ci]

Oh No! 😱 At least one test suite has failed during the AfterSuite cleanup stage and might have left around some resources on the MC!

Be sure to check the full results in Tekton Dashboard to see which test suite has failed and then run the following on the associated MC to list all leftover resources:

PIPELINE_RUN="pr-cluster-azure-340-cluster-test-suiteslld4b"

NAMES="$(kubectl api-resources --verbs list -o name | tr '\n' ,)"
kubectl get "${NAMES:0:${#NAMES}-1}" --show-kind --ignore-not-found -l cicd.giantswarm.io/pipelinerun=${PIPELINE_RUN} -A 2>/dev/null

[tinkerers-ci]

cluster-test-suites

Run name	pr-cluster-azure-340-cluster-test-suiteslld4b
Commit SHA	af7a380
Result	Failed ❌

📋 View full results in Tekton Dashboard

Rerun trigger:
/run cluster-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run cluster-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

[njuettner]

CAPZ Private is currently expected to fail, see https://gigantic.slack.com/archives/C04TGHDEF/p1725548655300869