PVC should expand if we increase the values from invalid to valid value

[saurabhwani5]

Describe the bug

When we try to expand 1GB PVC size to beyond scale limit for example 9223372036854775807Gi and then we try to increase to 2 GB then in this condition PVC size does not increase and give error as The PersistentVolumeClaim "scale-advance-pvc-1" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

How to Reproduce?

Please list the steps to help development teams reproduce the behavior

1. Install CSI with following images :

Operator : quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:1e579401a52f4ae12eff2b0b61b83af12d37d354a80d2ee6b83b9ac6b7cc01e2
Driver : quay.io/badri_pathak/ibm-spectrum-scale-csi-driver:pvc_max_size_v1

2. Create 1 Gi PVC :

[root@ocppr1226]# cat pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: csi-scale-fsetdemo-pod-2
  labels:
    app: nginx
spec:
  containers:
   - name: web-server
     image: nginx:1.22.0
     volumeMounts:
       - name: mypvc
         mountPath: /usr/share/nginx/html/scale
     ports:
     - containerPort: 80
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: scale-advance-pvc-1
       readOnly: false

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-advance-pvc-1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: ibm-spectrum-scale-csi-advance

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: ibm-spectrum-scale-csi-advance
provisioner: spectrumscale.csi.ibm.com
parameters:
   volBackendFs: "fs0"
   version: "2"
reclaimPolicy: Delete
allowVolumeExpansion: true
[root@ocppr1226]# oc apply -f pvc.yaml
pod/csi-scale-fsetdemo-pod-2 created
persistentvolumeclaim/scale-advance-pvc-1 created
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance created
[root@ocppr1226]#
[root@ocppr1226]# oc get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
scale-advance-pvc-1   Bound    pvc-4ed38513-a27a-4693-9625-53cc8442f95e   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 59s
[root@ocppr1226]#

4. Expand PVC size to 9223372036854775807Gi :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-advance-pvc-1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 9223372036854775807Gi
  storageClassName: ibm-spectrum-scale-csi-advance
[root@ocppr1226]# oc apply -f pvc.yaml
pod/csi-scale-fsetdemo-pod-2 configured
persistentvolumeclaim/scale-advance-pvc-1 configured
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance unchanged

5. Check log error :

[root@ocppr1226]# oc describe pvc
Name:          scale-advance-pvc-1
Namespace:     ibm-spectrum-scale-csi
StorageClass:  ibm-spectrum-scale-csi-advance
Status:        Bound
Volume:        pvc-4ed38513-a27a-4693-9625-53cc8442f95e
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
               volume.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       csi-scale-fsetdemo-pod-2
Conditions:
  Type       Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----       ------  -----------------                 ------------------                ------  -------
  Resizing   True    Mon, 01 Jan 0001 00:00:00 +0000   Wed, 02 Oct 2024 23:42:37 -0700
Events:
  Type     Reason                 Age                 From                                                                                                                Message
  ----     ------                 ----                ----                                                                                                                -------
  Warning  ProvisioningFailed     2m10s               persistentvolume-controller                                                                                         storageclass.storage.k8s.io "ibm-spectrum-scale-csi-advance" not found
  Normal   Provisioning           2m8s                spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-7f698474dd-xmhjf_e8c67c0b-ed68-46ab-b105-d4f95529af26  External provisioner is provisioning volume for claim "ibm-spectrum-scale-csi/scale-advance-pvc-1"
  Normal   ExternalProvisioning   98s (x4 over 2m8s)  persistentvolume-controller                                                                                         Waiting for a volume to be created either by the external provisioner 'spectrumscale.csi.ibm.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   ProvisioningSucceeded  86s                 spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-7f698474dd-xmhjf_e8c67c0b-ed68-46ab-b105-d4f95529af26  Successfully provisioned volume pvc-4ed38513-a27a-4693-9625-53cc8442f95e
  Warning  ExternalExpanding      26s                 volume_expand                                                                                                       waiting for an external controller to expand this PVC
  Normal   Resizing               13s (x7 over 26s)   external-resizer spectrumscale.csi.ibm.com                                                                          External resizer is resizing volume pvc-4ed38513-a27a-4693-9625-53cc8442f95e
  Warning  VolumeResizeFailed     13s (x7 over 26s)   external-resizer spectrumscale.csi.ibm.com                                                                          resize volume "pvc-4ed38513-a27a-4693-9625-53cc8442f95e" by resizer "spectrumscale.csi.ibm.com" failed: rpc error: code = Internal desc = ControllerExpandVolume - Volume expansion volID:[1;1;2815005349453165873;614A0B0A:66FAF656;933d520f-c34b-4f75-a75a-3777f6c4551b-ibm-spectrum-scale-csi;pvc-4ed38513-a27a-4693-9625-53cc8442f95e;/mnt/fs0/933d520f-c34b-4f75-a75a-3777f6c4551b-ibm-spectrum-scale-csi/pvc-4ed38513-a27a-4693-9625-53cc8442f95e] with  requested volSize:[9223372036854775807] is not allowed beyond max PV size:[1023999368203599872]
[root@ocppr1226]#

6. Expand PVC size to 2 Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-advance-pvc-1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  storageClassName: ibm-spectrum-scale-csi-advance
[root@ocppr1226]# oc apply -f pvc.yaml
pod/csi-scale-fsetdemo-pod-2 configured
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance unchanged
The PersistentVolumeClaim "scale-advance-pvc-1" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

Expected behavior

PVC size should increase to 2GB as current PVC size is 1GB

Data Collection and Debugging

[root@ocp D.1227]# pwd
/scale-csi/D.1227
[root@ocp D.1227]# ls
mustgather.tar.gz

[badri-pathak]

@saurabhwani5 There are several open issue in the external-resizer regarding this same issue. I haven't found any resolution in the latest external-resizer as such.Still discussion has been added in the https://github.com/kubernetes-csi/external-resizer

We need to wait to get this fixed.
Reference : resize failed and can't recover due to pvc rejection "Forbidden: field can not be less than previous value "

There is an upcoming feature to handle such scenario as a part of feature-gate in 1.31
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#recovering-from-failure-when-expanding-volumes

https://github.com/kubernetes-csi/external-resizer?tab=readme-ov-file

[badri-pathak]

As we have discussed, this is going to be released later in kubernetes releases as this is feature-gates in 1.31 . Keeping this open for now. @saurabhwani5