Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

Open
sameshai opened this issue Aug 12, 2024 · 7 comments
Open

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

sameshai opened this issue Aug 12, 2024 · 7 comments

Comments

@sameshai
Copy link

What happened:
As per PR below PR If the feature-gate HonorPVReclaimPolicy is enabled, create a pvc with a delete relaim policy, then delete the pvc, the pv stuck in deleting status, the error message is: #1155

csi-mockplugin-0/[email protected]: I0206 11:08:41.536819       1 controller.go:1523] delete "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": failed to remove finalizer for persistentvolume: persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
csi-mockplugin-0/[email protected]: W0206 11:08:41.536850       1 controller.go:989] Retrying syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38", failure 0
csi-mockplugin-0/[email protected]: E0206 11:08:41.536880       1 controller.go:1007] error syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
I0206 19:08:

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

sameershaikh@Sameers-MBP Downloads % k describe pvc
Name:     pvc-block
Namespace:   default
StorageClass: ibmc-vpc-block-5iops-tier
Status:    Bound
Volume:    pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:    <none>
Annotations:  pv.kubernetes.io/bind-completed: yes
        pv.kubernetes.io/bound-by-controller: yes
        volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
        volume.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:  [kubernetes.io/pvc-protection]
Capacity:   10Gi
Access Modes: RWO
VolumeMode:  Filesystem
Used By:    <none>
Events:    <none>
sameershaikh@Sameers-MBP Downloads % k describe pv 
Name:       pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:      <none>
Annotations:    pv.kubernetes.io/provisioned-by: vpc.block.csi.ibm.io
          volume.kubernetes.io/provisioner-deletion-secret-name: 
          volume.kubernetes.io/provisioner-deletion-secret-namespace: 
Finalizers:    [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer]
StorageClass:   ibmc-vpc-block-5iops-tier
Status:      Bound
Claim:       default/pvc-block
Reclaim Policy:  Delete
Access Modes:   RWO
VolumeMode:    Filesystem
Capacity:     10Gi
Node Affinity:   
 Required Terms:  
  Term 0:    failure-domain.beta.kubernetes.io/region in [us-south]
          failure-domain.beta.kubernetes.io/zone in [us-south-3]
Message:      
Source:
  Type:       CSI (a Container Storage Interface (CSI) volume source)
  Driver:      vpc.block.csi.ibm.io
  FSType:      ext4
  VolumeHandle:   r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
  ReadOnly:     false
  VolumeAttributes:   clusterID=cqsr5km10bghfqv414rg
              failure-domain.beta.kubernetes.io/region=us-south
              failure-domain.beta.kubernetes.io/zone=us-south-3
              iops=3000
              storage.kubernetes.io/csiProvisionerIdentity=1723447099267-9715-vpc.block.csi.ibm.io
              tags=
              volumeCRN=crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
              volumeId=r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
Events:
 Type  Reason        Age  From                      Message
 ----  ------        ----  ----                      -------
 Normal VolumeMetaDataSaved 2m20s ibm-vpc-block-csi-controller-588c75b988-xpkbt Success

sameershaikh@Sameers-MBP file-csi-driver % k describe clusterRole vpc-block-provisioner-role
Name:     vpc-block-provisioner-role
Labels:    addonmanager.kubernetes.io/mode=Reconcile
       app=ibm-vpc-block-csi-driver
Annotations: razee.io/build-url: https://travis.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/builds/22121223
       razee.io/source-url: https://github.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/commit/d7c3406c7f992ef727f3e9e3ce60f4e33c4f20c2
       version: 5.2.20_579
PolicyRule:
 Resources                    Non-Resource URLs Resource Names Verbs
 ---------                    ----------------- -------------- -----
 persistentvolumes                []         []       [get list watch create delete]
 persistentvolumeclaims             []         []       [get list watch update]
 nodes                      []         []       [get list watch]
 secrets                     []         []       [get list watch]
 csinodes.storage.k8s.io             []         []       [get list watch]
 storageclasses.storage.k8s.io          []         []       [get list watch]
 volumeattachments.storage.k8s.io        []         []       [get list watch]
 configmaps                   []         []       [get list]
 volumesnapshotcontents.snapshot.storage.k8s.io []         []       [get list]
 volumesnapshots.snapshot.storage.k8s.io     []         []       [get list]
 events                     []         []       [list watch create update patch]
sameershaikh@Sameers-MBP file-csi-driver %




},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-attacher Update v1 2024-08-12 13:05:53 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:finalizers\":{\"v:\\\"external-attacher/vpc-block-csi-ibm-io\\\"\":{}}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:13.994031    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994053    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994064    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994078    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994098    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:13.994456    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:19.569875    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Watch close - *v1.PersistentVolume total 13 items received
I0812 13:09:24.904270    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:24.904325    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
W0812 13:09:24.966351    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].key: failure-domain.beta.kubernetes.io/region is deprecated since v1.17; use "topology.kubernetes.io/region" instead
W0812 13:09:24.966364    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[1].key: failure-domain.beta.kubernetes.io/zone is deprecated since v1.17; use "topology.kubernetes.io/zone" instead
I0812 13:09:24.966561    1 controller.go:1606] "PersistentVolume deleted succeeded" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966783    1 controller.go:1171] "handleProtectionFinalizer" PV="&PersistentVolume{ObjectMeta:{pvc-351e6980-ac05-4441-a482-6ea544b3aad5  3f0a9a59-3442-4171-b67a-efdd232c3664 19236 0 2024-08-12 07:31:18 +0000 UTC 2024-08-12 13:09:24 +0000 UTC 0xc0005b1210 map[] map[pv.kubernetes.io/provisioned-by:vpc.block.csi.ibm.io volume.kubernetes.io/provisioner-deletion-secret-name: volume.kubernetes.io/provisioner-deletion-secret-namespace:] [] [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer] [{csi-provisioner Update v1 2024-08-12 13:01:59 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pv.kubernetes.io/provisioned-by\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-name\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-namespace\":{}},\"f:finalizers\":{\"v:\\\"external-provisioner.volume.kubernetes.io/finalizer\\\"\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:capacity\":{},\"f:claimRef\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{},\"f:resourceVersion\":{},\"f:uid\":{}},\"f:csi\":{\".\":{},\"f:driver\":{},\"f:fsType\":{},\"f:volumeAttributes\":{\".\":{},\"f:clusterID\":{},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:24.966803    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966813    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966820    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966830    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966844    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:24.966899    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:25.194175    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:25.194222    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:25.207594    1 controller.go:1569] "Failed to delete persistentvolume" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" err="persistentvolumes \"pvc-351e6980-ac05-4441-a482-6ea544b3aad5\" not found"
I0812 13:09:25.207641    1 controller.go:1007] "Retrying syncing volume" key="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" failures=0
E0812 13:09:25.207795    1 controller.go:1025] error syncing volume "pvc-351e6980-ac05-4441-a482-6ea544b3aad5": persistentvolumes "pvc-351e6980-ac05-4441-a482-6ea544b3aad5" not found
I0812 13:09:48.412100    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.CSINode total 9 items received
I0812 13:10:11.430774    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.PersistentVolumeClaim total 15 items received
I0812 13:11:05.631537    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Watch close - *v1.StorageClass total 11 items received
I0812 13:11:58.431219    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.Node total 21 items received

What you expected to happen:
I was expecting RBAC error

How to reproduce it:

  1. Create PVC with external provisioner
  2. Delete PVC which will trigger the PV deletion
  3. There is no error even if we dont have patch permission.

Anything else we need to know?:

Environment:

  • Driver version:
  • Kubernetes version (use kubectl version):
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@Madhu-1
Copy link
Contributor

Madhu-1 commented Sep 18, 2024

@xing-yang @carlory In the above code we are still doing the update operation and in the RBAC we don't have update access rather as part of #1155 to address the same issue, can you please check this one and let me know if we need to add RBAC for update or change the code in provisioner to use patch as well

https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L984

https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L1676

we still need an update for adding the annotation and for updating finalizers we can use the patch

@Madhu-1
Copy link
Contributor

Madhu-1 commented Sep 18, 2024

/assign

@carlory
Copy link
Member

carlory commented Sep 18, 2024 •
edited
Loading

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

I will try to reproduce it with hostpath driver.

we still need an update for adding the annotation and for updating finalizers we can use the patch

Agreed.

According to xing-yang's comment, to add a new RBAC rule, we need to bump the major version.

@carlory
Copy link
Member

carlory commented Sep 18, 2024

we need to add some e2e tests to detect a similar issue. should the new tests be added to the external-provisioner repo? cc @xing-yang @jsafrane

@Madhu-1
Copy link
Contributor

Madhu-1 commented Sep 18, 2024

/unassign

sorry i was wrong i like to the PVC update code not the PV update, i don't see code in main that require PV update access.

@carlory
Copy link
Member

carlory commented Sep 18, 2024

@sameshai Can you execute the following command with your service account name? A service account may be bound with more than one cluster role.

kubectl auth can-i patch pv --as=system:serviceaccount:default:csi-hostpathplugin-sa

@carlory
Copy link
Member

carlory commented Sep 18, 2024

If the answer is "yes", it means that the external provisioner can patch the persistent volume resource even if the external-provisioner cluster role does not have permission. It is allowed via other cluster roles.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Madhu-1 @carlory @sameshai