dualstack vmss cluster stuck at namespace deletion with old k8s version

[lzhecheng]

/kind bug

[Before submitting an issue, have you checked the Troubleshooting Guide?]

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
For dualstack vmss CI with k8s version 1.27.1, clusters were stuck at namespace deletion. I deployed the same cluster locally and it also fails.

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/cloud-provider-azure-master-dualstack-vmss-capz/1732852687860404224

What did you expect to happen:
Old k8s version is also supported.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[CecileRobertMichon]

@lzhecheng I'm seeing the following errors in the logs:

      --------------------------------------------------------------------------------
      RESPONSE 404: 404 Not Found
      ERROR CODE: ResourceGroupNotFound
      --------------------------------------------------------------------------------
      {
        "error": {
          "code": "ResourceGroupNotFound",
          "message": "Resource group 'capz-cfoe84' could not be found."
        }
      }
      --------------------------------------------------------------------------------

Is it possible something is deleting the resource group during the test?

[CecileRobertMichon]

Actually the rg is getting deleted before the test finishes running because the test is taking 5 hours due to all the test timeouts and all RGs older than 4 hours are deleted by the rg cleanup job

So the RG deletion doesn't seem to be the cause of the issue

It looks like the latest run with 1.28 is failing the same way: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/cloud-provider-azure-master-dualstack-vmss-capz/1733215008105435136

[CecileRobertMichon]

I believe you are hitting a bug with the newest version of containerd, you need the same workaround applied in #4209 to https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/tests/k8s-azure/manifest/cluster-api/cluster-template-prow-dual-stack-mp.yaml

Side note: would you be able to use the CAPZ template for this test (https://github.com/CecileRobertMichon/cluster-api-provider-azure/blob/main/templates/test/ci/cluster-template-prow-dual-stack.yaml) ? It has both VMSS and VMs

What is https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/tests/k8s-azure/manifest/cluster-api/cluster-template-prow-dual-stack-mp.yaml adding that is not already covered in the CAPZ template?

Edit: instead of adding the restart containerd workaround you should be able to do #4350 directly

[lzhecheng]

@CecileRobertMichon Thank you. I will try it.
I'm willing to use capz templates but I want to create a cluster with only vmss.
I think there're 2 solutions:

1. Use 2 variables for workload count in capz templates, and by default set both to be the same. Then I can specify one variable be 0 in my testing.
2. I need to update cloud-provider-azure test code to be able to run on only vmss nodes or vm nodes. This solution needs some effort to go through all GetPod/CreatePod functions for example.

[lzhecheng]

@CecileRobertMichon Another question:
I thought the only difference between these 2 templates is that the first support KUBERNETES_VERSION: latest. However, the first one has MachineDeployment but the second one has MachineDeployment and MachinePool. Why this difference?

templates/test/ci/cluster-template-prow-ci-version-ipv6.yaml
templates/test/ci/cluster-template-prow-ipv6.yaml

[CecileRobertMichon]

I don't think that difference is intentional, we should add a MachinePool to the ci-version template as well

[lzhecheng]

I don't think that difference is intentional, we should add a MachinePool to the ci-version template as well

I created an issue for it

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.