Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cluster: drop DisableLocalStorageCapacityIsolation #3360

Conversation

giuseppe
Copy link
Member

it was used to workaround a kubelet crash issue with rootless providers.

The Kubelet seems to work fine now with localStorageCapacityIsolation enabled in a user namespace so drop the special handling. After this change, ephemeral storage can be used in a rootless cluster.

Closes: #3359

it was used to workaround a kubelet crash issue with rootless
providers.

The Kubelet seems to work fine now with localStorageCapacityIsolation
enabled in a user namespace so drop the special handling.  After this
change, ephemeral storage can be used in a rootless cluster.

Closes: kubernetes-sigs#3359

Signed-off-by: Giuseppe Scrivano <[email protected]>
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: giuseppe
Once this PR has been reviewed and has the lgtm label, please assign aojea for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 12, 2023
Copy link
Member

@BenTheElder BenTheElder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be identifying what changed, we know it was broken in the past, in which cases is it still broken? Some combination of docker or podman version and/or kubernetes version must be problematic still.

@BenTheElder
Copy link
Member

I see you looked for failures in #3359 but it's not clear on what dimension(s).

@giuseppe
Copy link
Member Author

giuseppe commented Sep 12, 2023

I see you looked for failures in #3359 but it's not clear on what dimension(s).

I am running kind as rootless on Fedora 38. I am still digging in the old issue, but it refers Fedora 35 which is EOF.

I'll try on a fresh machine with the default configuration and see what happens there

@giuseppe
Copy link
Member Author

it seems to work fine on Fedora 38 with the default configuration:

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="38 (Cloud Edition)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Cloud Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Cloud Edition"
VARIANT_ID=cloud
$ stat -f /
  File: "/"
    ID: 191e36d272db01a8 Namelen: 255     Type: btrfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 31174139   Free: 30280728   Available: 30198797
Inodes: Total: 0          Free: 0
$ podman unshare grep . /proc/self/?id_map
/proc/self/gid_map:         0       1000          1
/proc/self/gid_map:         1     524288      65536
/proc/self/uid_map:         0       1000          1
/proc/self/uid_map:         1     524288      65536

trying with 3 nodes:

$ kubectl get nodes
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   3m50s   v1.27.3
kind-worker          Ready    <none>          3m27s   v1.27.3
kind-worker2         Ready    <none>          3m28s   v1.27.3
kind-worker3         Ready    <none>          3m29s   v1.27.3

$ kubectl describe node/kind-control-plane
Name:               kind-control-plane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kind-control-plane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 12 Sep 2023 08:19:34 +0000
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kind-control-plane
  AcquireTime:     <unset>
  RenewTime:       Tue, 12 Sep 2023 08:23:53 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 12 Sep 2023 08:19:54 +0000   Tue, 12 Sep 2023 08:19:30 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 12 Sep 2023 08:19:54 +0000   Tue, 12 Sep 2023 08:19:30 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 12 Sep 2023 08:19:54 +0000   Tue, 12 Sep 2023 08:19:30 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Tue, 12 Sep 2023 08:19:54 +0000   Tue, 12 Sep 2023 08:19:54 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  10.89.0.7
  Hostname:    kind-control-plane
Capacity:
  cpu:                2
  ephemeral-storage:  124696556Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3995420Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  124696556Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3995420Ki
  pods:               110
System Info:
  Machine ID:                 4965c1dfc8bd44489ea0a63a36dd5930
  System UUID:                b5568aa7-d0cb-4520-aeb5-2fe0255484c8
  Boot ID:                    58819097-b96f-4eb8-bb83-bb0761e0afac
  Kernel Version:             6.4.14-200.fc38.x86_64
  OS Image:                   Debian GNU/Linux 11 (bullseye)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.1
  Kubelet Version:            v1.27.3
  Kube-Proxy Version:         v1.27.3
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
ProviderID:                   kind://podman/kind/kind-control-plane
Non-terminated Pods:          (9 in total)
  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-5d78c9869d-5vnqm                      100m (5%)     0 (0%)      70Mi (1%)        170Mi (4%)     4m13s
  kube-system                 coredns-5d78c9869d-hmjlz                      100m (5%)     0 (0%)      70Mi (1%)        170Mi (4%)     4m13s
  kube-system                 etcd-kind-control-plane                       100m (5%)     0 (0%)      100Mi (2%)       0 (0%)         4m27s
  kube-system                 kindnet-7sgp8                                 100m (5%)     100m (5%)   50Mi (1%)        50Mi (1%)      4m13s
  kube-system                 kube-apiserver-kind-control-plane             250m (12%)    0 (0%)      0 (0%)           0 (0%)         4m25s
  kube-system                 kube-controller-manager-kind-control-plane    200m (10%)    0 (0%)      0 (0%)           0 (0%)         4m25s
  kube-system                 kube-proxy-6bxcm                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m13s
  kube-system                 kube-scheduler-kind-control-plane             100m (5%)     0 (0%)      0 (0%)           0 (0%)         4m25s
  local-path-storage          local-path-provisioner-6bc4bddd6b-7vnj4       0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m13s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                950m (47%)  100m (5%)
  memory             290Mi (7%)  390Mi (9%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type    Reason                   Age                    From             Message
  ----    ------                   ----                   ----             -------
  Normal  Starting                 4m10s                  kube-proxy       
  Normal  NodeHasSufficientMemory  4m39s (x8 over 4m40s)  kubelet          Node kind-control-plane status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    4m39s (x8 over 4m40s)  kubelet          Node kind-control-plane status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     4m39s (x7 over 4m40s)  kubelet          Node kind-control-plane status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  4m39s                  kubelet          Updated Node Allocatable limit across pods
  Normal  Starting                 4m25s                  kubelet          Starting kubelet.
  Normal  NodeHasSufficientMemory  4m25s                  kubelet          Node kind-control-plane status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    4m25s                  kubelet          Node kind-control-plane status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     4m25s                  kubelet          Node kind-control-plane status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  4m25s                  kubelet          Updated Node Allocatable limit across pods
  Normal  RegisteredNode           4m13s                  node-controller  Node kind-control-plane event: Registered Node kind-control-plane in Controller
  Normal  NodeReady                4m9s                   kubelet          Node kind-control-plane status is now: NodeReady

@BenTheElder
Copy link
Member

I'm wondering if it was a kubelet or podman/runc/crun change, we've had to do things like this conditionally on version before from either the "node runtime" (podman/docker) or k/k.

I do think we should merge this, but I'm hesitant to make cluster bringup entirely fail again versus this feature gap.

@giuseppe
Copy link
Member Author

any decisions on this PR? Do we accept the risk :-) or play safe and close it?

IMO, cluster failures are bad and we should avoid them as much as possible, on the other hand, it was also not easy to find the root reason for this failure and why the rootless cluster was behaving differently. It took a while to figure out what was happening. Something that a hard failure would have made easier to fix.

@giuseppe
Copy link
Member Author

@BenTheElder ping

@BenTheElder
Copy link
Member

IMO, cluster failures are bad and we should avoid them as much as possible, on the other hand, it was also not easy to find the root reason for this failure and why the rootless cluster was behaving differently. It took a while to figure out what was happening. Something that a hard failure would have made easier to fix.

The hard failure is NOT fixable though, short of upgrading something (??) which we don't seem to know yet.

I'm hesitant to ship disabling a workaround without knowing what change in the ecosystem made it no longer required (e.g. if it's a particular k8s or podman or docker version then we can version gate this workaround), at the same time I understand the desire to disable this no longer necessary workaround

FWIW: You should not expect parity in rootless yet. Rootless k8s has always had gaps AFAIK, kind or otherwise.

@giuseppe
Copy link
Member Author

since there is no interest in this change, I am closing the PR

@giuseppe giuseppe closed this Feb 28, 2024
@BenTheElder
Copy link
Member

FTR: I think we should ship dropping DisableLocalStorageCapacityIsolation, but preferably we'd understand when it's safe to drop.

I don't know if anyone ever dug into what the root issue was here.

That said, if we can be confident that this will only be broken on really outdated hosts and will benefit the rest, we've just gone with requiring more up to date hosts before (like cgroupns=private #3311)

@giuseppe
Copy link
Member Author

giuseppe commented Mar 1, 2024

The commit 968c842 that added DisableLocalStorageCapacityIsolation does not add any information why it was needed. It can be a mix of CRI runtime, OCI runtime, Kubernetes, Kernel.

I won't have time to dig further into this. If anyone is interested in taking this over (or just accepting the risk), I can reopen it.

@vially
Copy link

vially commented Apr 22, 2024

The DisableLocalStorageCapacityIsolation workaround seems to have been first mentioned in #2525 as an attempt to fix #2524 (even though it later landed in #2559).

However, in the end, #2524 turned out to be a duplicate (#2524 (comment)) of a different known issue.

I guess the main question is, was ab38ef8 actually needed as part of #2559, or did it just look like it was needed?

@damzog
Copy link

damzog commented May 9, 2024

Hi @giuseppe: For me it is NOT working:

kubectl describe nodes
Name:               kind-control-plane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kind-control-plane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 09 May 2024 08:23:54 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kind-control-plane
  AcquireTime:     <unset>
  RenewTime:       Thu, 09 May 2024 09:43:07 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 09 May 2024 09:41:46 +0200   Thu, 09 May 2024 08:23:54 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 09 May 2024 09:41:46 +0200   Thu, 09 May 2024 08:23:54 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 09 May 2024 09:41:46 +0200   Thu, 09 May 2024 08:23:54 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 09 May 2024 09:41:46 +0200   Thu, 09 May 2024 08:25:09 +0200   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.18.0.4
  Hostname:    kind-control-plane
Capacity:
  cpu:            4
  hugepages-1Gi:  0
  hugepages-2Mi:  0
  memory:         16255864Ki
  pods:           110
Allocatable:
  cpu:            4
  hugepages-1Gi:  0
  hugepages-2Mi:  0
  memory:         16255864Ki
  pods:           110
System Info:
  Machine ID:                 546064e27be1426ca3d3074b17bcd163
  System UUID:                7d956cc8-c787-4449-a7d3-f73c0592d20d
  Boot ID:                    1cb5e54c-5d73-4564-a824-64a2ac3d988f
  Kernel Version:             5.15.0-105-generic
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.13
  Kubelet Version:            v1.29.2
  Kube-Proxy Version:         v1.29.2
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
ProviderID:                   kind://docker/kind/kind-control-plane
Non-terminated Pods:          (6 in total)
  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---
  kube-system                 etcd-kind-control-plane                       100m (2%)     0 (0%)      100Mi (0%)       0 (0%)         79m
  kube-system                 kindnet-rkjf9                                 100m (2%)     100m (2%)   50Mi (0%)        50Mi (0%)      79m
  kube-system                 kube-apiserver-kind-control-plane             250m (6%)     0 (0%)      0 (0%)           0 (0%)         79m
  kube-system                 kube-controller-manager-kind-control-plane    200m (5%)     0 (0%)      0 (0%)           0 (0%)         79m
  kube-system                 kube-proxy-jsk2n                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         79m
  kube-system                 kube-scheduler-kind-control-plane             100m (2%)     0 (0%)      0 (0%)           0 (0%)         79m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                750m (18%)  100m (2%)
  memory             150Mi (0%)  50Mi (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>


Name:               kind-worker
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kind-worker
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 09 May 2024 08:24:17 +0200
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  kind-worker
  AcquireTime:     <unset>
  RenewTime:       Thu, 09 May 2024 09:43:06 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 09 May 2024 09:41:44 +0200   Thu, 09 May 2024 08:24:17 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 09 May 2024 09:41:44 +0200   Thu, 09 May 2024 08:24:17 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 09 May 2024 09:41:44 +0200   Thu, 09 May 2024 08:24:17 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 09 May 2024 09:41:44 +0200   Thu, 09 May 2024 08:25:09 +0200   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.18.0.2
  Hostname:    kind-worker
Capacity:
  cpu:            4
  hugepages-1Gi:  0
  hugepages-2Mi:  0
  memory:         16255864Ki
  pods:           110
Allocatable:
  cpu:            4
  hugepages-1Gi:  0
  hugepages-2Mi:  0
  memory:         16255864Ki
  pods:           110
System Info:
  Machine ID:                 9cf3f4969ba5490589c5690b22ac7ee4
  System UUID:                ba390bfc-a813-45fa-9387-275f02994ea9
  Boot ID:                    1cb5e54c-5d73-4564-a824-64a2ac3d988f
  Kernel Version:             5.15.0-105-generic
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.13
  Kubelet Version:            v1.29.2
  Kube-Proxy Version:         v1.29.2
PodCIDR:                      10.244.2.0/24
PodCIDRs:                     10.244.2.0/24
ProviderID:                   kind://docker/kind/kind-worker
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                       CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                       ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-76f75df574-g4dv4                   100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     79m
  kube-system                 coredns-76f75df574-wlwc8                   100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     79m
  kube-system                 kindnet-x2x75                              100m (2%)     100m (2%)   50Mi (0%)        50Mi (0%)      78m
  kube-system                 kube-proxy-xlmvt                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         78m
  local-path-storage          local-path-provisioner-7577fdbbfb-sj285    0 (0%)        0 (0%)      0 (0%)           0 (0%)         79m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                300m (7%)   100m (2%)
  memory             190Mi (1%)  390Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>


Name:               kind-worker2
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kind-worker2
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 09 May 2024 08:24:15 +0200
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  kind-worker2
  AcquireTime:     <unset>
  RenewTime:       Thu, 09 May 2024 09:43:07 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 09 May 2024 09:41:50 +0200   Thu, 09 May 2024 08:24:15 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 09 May 2024 09:41:50 +0200   Thu, 09 May 2024 08:24:15 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 09 May 2024 09:41:50 +0200   Thu, 09 May 2024 08:24:15 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 09 May 2024 09:41:50 +0200   Thu, 09 May 2024 08:25:09 +0200   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.18.0.3
  Hostname:    kind-worker2
Capacity:
  cpu:            4
  hugepages-1Gi:  0
  hugepages-2Mi:  0
  memory:         16255864Ki
  pods:           110
Allocatable:
  cpu:            4
  hugepages-1Gi:  0
  hugepages-2Mi:  0
  memory:         16255864Ki
  pods:           110
System Info:
  Machine ID:                 aff7268a69814f68966b78fccd4f76e0
  System UUID:                0bcb66b4-808c-4640-99f4-9e45e9afa149
  Boot ID:                    1cb5e54c-5d73-4564-a824-64a2ac3d988f
  Kernel Version:             5.15.0-105-generic
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.13
  Kubelet Version:            v1.29.2
  Kube-Proxy Version:         v1.29.2
PodCIDR:                      10.244.1.0/24
PodCIDRs:                     10.244.1.0/24
ProviderID:                   kind://docker/kind/kind-worker2
Non-terminated Pods:          (2 in total)
  Namespace                   Name                CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                ------------  ----------  ---------------  -------------  ---
  kube-system                 kindnet-ddbst       100m (2%)     100m (2%)   50Mi (0%)        50Mi (0%)      78m
  kube-system                 kube-proxy-5kcwv    0 (0%)        0 (0%)      0 (0%)           0 (0%)         78m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  100m (2%)
  memory             50Mi (0%)  50Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:              <none>

@giuseppe
Copy link
Member Author

giuseppe commented May 9, 2024

@damzog have you applied this patch? Does it work without it?

@damzog
Copy link

damzog commented May 9, 2024

@giuseppe no I have installed current version of kind and ran into the issue of not having ephemeral storage. Then I started searching for a reason and found this discussion.

@giuseppe
Copy link
Member Author

giuseppe commented May 9, 2024

@giuseppe no I have installed current version of kind and ran into the issue of not having ephemeral storage. Then I started searching for a reason and found this discussion.

thanks for confirming it. I still believe it would be better to have the current patch as it is solving a real issue, and if any problems come up, it can be analyzed and maybe fix it in a different way

@BenTheElder
Copy link
Member

You can set a kubeadmConfigPatch with something like this:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
  kind: KubeletConfiguration
  localStorageCapacityIsolation: true

And see how this change would work for you

@damzog
Copy link

damzog commented May 9, 2024

@BenTheElder Thanks, that actually worked. And sorry for being a newbie in this :) Maybe this issue would be worth to be added to the standard docs?

@BenTheElder
Copy link
Member

And sorry for being a newbie in this :)

Not at all. And FWIW the config patches are a pretty advanced feature not meant to be required for everyday usage versus tinkering with Kubernetes itself.

Maybe this issue would be worth to be added to the standard docs?

I think we need to revisit merging this change and inverting it to requiring the unknown broken environments to use a patch with localStorageCapacityIsolation: false or otherwise help us identify what host environments if any still require setting this to false.

I'd like to consider it for the release after the current pending one (to minimize disruption, we're overdue to cut one anytime now and already have a solid change set)

@BenTheElder BenTheElder reopened this May 9, 2024
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 9, 2024
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot
Copy link
Contributor

@giuseppe: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kind-build 39450c0 link true /test pull-kind-build
pull-kind-test 39450c0 link true /test pull-kind-test
pull-kind-verify 39450c0 link true /test pull-kind-verify
pull-kind-conformance-parallel-ipv6 39450c0 link true /test pull-kind-conformance-parallel-ipv6
pull-kind-e2e-kubernetes-1-29 39450c0 link true /test pull-kind-e2e-kubernetes-1-29
pull-kind-e2e-kubernetes-1-27 39450c0 link true /test pull-kind-e2e-kubernetes-1-27
pull-kind-e2e-kubernetes 39450c0 link true /test pull-kind-e2e-kubernetes
pull-kind-e2e-kubernetes-1-26 39450c0 link true /test pull-kind-e2e-kubernetes-1-26
pull-kind-e2e-kubernetes-1-28 39450c0 link true /test pull-kind-e2e-kubernetes-1-28
pull-kind-conformance-parallel-ga-only 39450c0 link true /test pull-kind-conformance-parallel-ga-only
pull-kind-conformance-parallel-dual-stack-ipv4-ipv6 39450c0 link true /test pull-kind-conformance-parallel-dual-stack-ipv4-ipv6

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@BenTheElder BenTheElder added this to the v0.24.0 milestone May 13, 2024
robertpatrick pushed a commit to oracle/weblogic-kubernetes-operator that referenced this pull request May 22, 2024
k8s-ci-robot added a commit that referenced this pull request Jun 11, 2024
[rebased #3360] cluster: drop DisableLocalStorageCapacityIsolation
@BenTheElder
Copy link
Member

merged rebased in #3651. thanks @giuseppe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ephemeral storage not available when using (docker/podman) rootless providers
5 participants