volume leaks when PVC is deleted when associated PV is in the terminating state

[divyenpatel]

Steps

1. Create PVC and wait for it to get bound.

# kubectl create -f pvc.yaml 
persistentvolumeclaim/example-vanilla-block-pvc created

# kubectl get pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
example-vanilla-block-pvc   Bound    pvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            example-vanilla-block-sc   19s

# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS               REASON   AGE
pvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            Delete           Bound    default/example-vanilla-block-pvc   example-vanilla-block-sc            23s

2. Delete PV.

# kubectl delete pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
persistentvolume "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" deleted
^C

PV can not be deleted, so kubectl delete pv command is not giving up terminal prompt. So I typed ctl+c to exit.
Checked PV status. It went into terminating state.

# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                               STORAGECLASS               REASON   AGE
pvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            Delete           Terminating   default/example-vanilla-block-pvc   example-vanilla-block-sc            2m23s

3. When PV is in the terminating state, delete PVC.

# kubectl delete pvc example-vanilla-block-pvc
persistentvolumeclaim "example-vanilla-block-pvc" deleted

4. PV and PVC are both deleted from the system.

# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found

The controller is not getting calls to delete the volume from the datastore. This results into leaking volume on the datastore.

Above workflow is executed using very latest external-provisioner - v2.1.0. This issue is also present on prior release of the external provisioner.

Filing this bug to discuss about what fix we can make to prevent this orphan volume on the system.

This issue was also discussed here - #195 (comment)

cc: @xing-yang @SandeepPissay

[xing-yang]

The controller is not getting calls to delete the volume from the datastore

@divyenpatel Which controller? You meant the CSI Driver controller?

[pohly]

My take on this: the system works as designed at the moment. Admins should only delete a PV if they know that the underlying volume in the storage system is gone.

The PV protection controller only prevents that the PV object gets removed while there is a PVC that uses the PV. It has no knowledge about the underlying volume.

[divyenpatel]

The controller is not getting calls to delete the volume from the datastore

@divyenpatel Which controller? You meant the CSI Driver controller?

Yes

[xing-yang]

The Reclaim Policy on PV is "Delete" here. If volume is not deleted regardless of the Reclaim Policy, what is the difference between "Delete" and "Retain" in this case?

[xing-yang]

Discussed in today's sig-storage meeting.

• Work as expected but very confusing as behavior changes depending on whether PV is being deleted before PVC is deleted.
  If we fix this, it will break existing customers with workflow that depends on this behavior (@jsafrane).
• Should update the doc first: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming. Then look at how to fix this without breaking existing customers workflows.

[wjun]

IHAC who met this issue in their production environment. My customer provides k8s as a self service to its customers. In this service provider model, whatever change customers make to their clusters through valid k8s CL/APIs, the service provider should not have potential billing issues (they count disk usages into billing).

[xing-yang]

Discussed this with @jsafrane and @msau42. We could introduce an alpha feature in 1.22 and fix the code to always honor the deletion policy, in the same time, deprecate the existing behavior. After one year (4 releases), we can GA the feature and stop supporting the existing behavior.

[xing-yang]

CC @deepakkinni

[deepakkinni]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[divyenpatel]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pohly]

/reopen
/lifecycle frozen

[k8s-ci-robot]

@pohly: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pohly]

/reopen

Not fixed in external-provisioner yet, only in sig-storage-lib-external-provisioner.

[k8s-ci-robot]

@pohly: Reopened this issue.

In response to this:

/reopen

Not fixed in external-provisioner yet, only in sig-storage-lib-external-provisioner.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[igoratencompass]

Interesting, thought this was a security feature so you don't delete a PV by accident while still referenced by a PVC.