Deleting a PV before a PVC results in an orphaned volume and bdev #1269

kylebrooks-8451 · 2022-12-11T15:09:06Z

Edit:

I tried to delete a pv before deleting the pvc and the kubectl delete pv command hung. I stopped it using ctrl-c and deleted the pvc first and then the pv. This resulted in an orphaned volume and bdev. I then tried to use the io-engine-client to manually delete the bdev but it failed:

docker run openebs/mayastor-io-engine-client:release-2.0 -b talos-df7-egr bdev list
UUID                                 NUM_BLOCKS BLK_SIZE CLAIMED_BY NAME                                 SHARE_URI
8c1abb67-b96c-47f5-be95-4c6e96bbc4da 167772160  512      lvol       /dev/sdb                             bdev:////dev/sdb
f517fe5e-3f81-4484-b597-634483fe8d2d 2097152    512      Orphaned   f517fe5e-3f81-4484-b597-634483fe8d2d bdev:///f517fe5e-3f81-4484-b597-634483fe8d2d

docker run openebs/mayastor-io-engine-client:release-2.0 -b talos-df7-egr bdev destroy f517fe5e-3f81-4484-b597-634483fe8d2d
Error: GrpcStatus { source: Status { code: InvalidArgument, message: "Error parsing URI ''", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Sun, 11 Dec 2022 15:03:42 GMT"} }, source: None }, backtrace: Backtrace(()) }

Expected behavior
The block device is destroyed. For some reason this block device was not destroyed by Kubernetes even though the StorageClass had reclaimPolicy: Delete.

** OS info (please complete the following information):**

Distro: Docker images
MayaStor revision or container image: 1.0.4

The text was updated successfully, but these errors were encountered:

kylebrooks-8451 · 2022-12-11T21:15:26Z

I think the issue is that the uri is empty for this bdev:

{
  "aliases": "talos-df7-egr/f517fe5e-3f81-4484-b597-634483fe8d2d",
  "blk_size": 512,
  "claimed": false,
  "claimed_by": "Orphaned",
  "name": "f517fe5e-3f81-4484-b597-634483fe8d2d",
  "num_blocks": 2097152,
  "product_name": "Logical Volume",
  "share_uri": "bdev:///f517fe5e-3f81-4484-b597-634483fe8d2d",
  "uri": "",
  "uuid": "f517fe5e-3f81-4484-b597-634483fe8d2d"
}

tiagolobocastro · 2022-12-12T10:21:13Z

Using grpc to io-engine should be the last resort.
If you use kubectl-mayastor get volumes does it return any volumes?

kylebrooks-8451 · 2022-12-12T14:10:22Z

Yes, it still shows the volumes:

kubectl mayastor get volumes
 ID                                    REPLICAS  TARGET-NODE  ACCESSIBILITY  STATUS  SIZE
 327b4e55-589c-42e2-baf9-d1bdadf63366  1         <none>       <none>         Online  1073741824

tiagolobocastro · 2022-12-12T14:36:54Z

Great, and is the pvc and pv gone?

kylebrooks-8451 · 2022-12-12T14:39:39Z

Yes, they are both gone. If it helps here is the YAML for the volume:

---
spec:
  labels:
    local: "true"
  num_replicas: 1
  size: 1073741824
  status: Created
  uuid: 327b4e55-589c-42e2-baf9-d1bdadf63366
  topology:
    node_topology:
      explicit:
        allowed_nodes:
          - talos-proxmox-0
          - talos-proxmox-1
          - talos-tu8-r34
          - talos-b2m-7a0
          - talos-df7-egr
          - talos-j92-0u1
          - talos-srn-grj
          - talos-ixc-r5k
        preferred_nodes:
          - talos-df7-egr
          - talos-ixc-r5k
          - talos-j92-0u1
          - talos-proxmox-0
          - talos-proxmox-1
          - talos-srn-grj
          - talos-tu8-r34
          - talos-b2m-7a0
    pool_topology:
      labelled:
        exclusion: {}
        inclusion:
          openebs.io/created-by: msp-operator
  policy:
    self_heal: true
state:
  size: 1073741824
  status: Online
  uuid: 327b4e55-589c-42e2-baf9-d1bdadf63366
  replica_topology:
    f517fe5e-3f81-4484-b597-634483fe8d2d:
      node: talos-df7-egr
      pool: talos-df7-egr
      state: Online

tiagolobocastro · 2022-12-12T14:53:43Z

How strange, with retention set to delete then it should have deleted it, right @abhilashshetty04 ?
Could we get logs from the csi-controller?

kylebrooks-8451 · 2022-12-12T15:39:21Z

Here are the logs. One thing to note, I did try to delete the PV before the PVC and had to ctrl-c the kubectl delete pv command and delete the pvc first. I'm wonder if I created a race condition indicated in the cis-attacher-log.txt.

csi-attacher-log.txt
csi-provisioner-log.txt
csi-controller-log.txt

I1210 18:40:58.639219       1 csi_handler.go:276] Detaching "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1"
I1210 18:40:58.720361       1 csi_handler.go:583] Detached "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1"
I1210 18:40:58.749118       1 csi_handler.go:276] Detaching "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1"
I1210 18:40:58.755756       1 csi_handler.go:583] Detached "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1"
I1210 18:40:58.769362       1 csi_handler.go:283] Failed to save detach error to "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1": volumeattachments.storage.k8s.io "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1" not found
I1210 18:40:58.769780       1 csi_handler.go:228] Error processing "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1": failed to detach: could not mark as detached: volumeattachments.storage.k8s.io "csi-9128a61d8587a1d12b1e6a9091d79ecf6164e5e28f9a90271ee5f46cb8c51eb1" not found
I1210 18:41:17.331177       1 csi_handler.go:708] Removed finalizer from PV "pvc-327b4e55-589c-42e2-baf9-d1bdadf63366"

tiagolobocastro · 2022-12-12T15:42:30Z

Oh right, if you try to delete the pv manually I don't think we support that as things stand @abhilashshetty04 ?

tiagolobocastro · 2022-12-12T15:45:41Z

To recover from this you'd have to delete the mayastor volume using rest, something like this:
curl -X 'DELETE' 'http://node:30011/v0/volumes/327b4e55-589c-42e2-baf9-d1bdadf63366' -H 'accept: /'

kylebrooks-8451 · 2022-12-12T16:15:25Z

@tiagolobocastro That command deleted the volume and bdev, thank you!

abhilashshetty04 · 2022-12-13T06:41:26Z

How strange, with retention set to delete then it should have deleted it, right @abhilashshetty04 ? Could we get logs from the csi-controller?
Yes should have been deleted by CSI driver.

kvzn · 2023-04-11T09:31:31Z

How I deleted the volume:
I've tried on several pods of mayastor, all of them have no curl and apt, need to try other ways:

user@mbp2023 ~ % kubectl get svc -n mayastor
NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
mayastor-agent-core              ClusterIP   10.128.39.22     <none>        50051/TCP,50052/TCP   6h13m
mayastor-api-rest                ClusterIP   10.128.209.239   <none>        8080/TCP,8081/TCP     6h13m
mayastor-etcd                    ClusterIP   10.128.176.26    <none>        2379/TCP,2380/TCP     6h13m
mayastor-etcd-headless           ClusterIP   None             <none>        2379/TCP,2380/TCP     6h13m
mayastor-loki                    ClusterIP   10.128.186.128   <none>        3100/TCP              6h13m
mayastor-loki-headless           ClusterIP   None             <none>        3100/TCP              6h13m
mayastor-metrics-exporter-pool   ClusterIP   10.128.39.212    <none>        9502/TCP              6h13m

Realized the line of mayastor-api-rest, tried this:

user@mbp2023 ~ % kubectl port-forward deployment/mayastor-api-rest -n mayastor 8081:8081
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081
Handling connection for 8081

And this:

user@mbp2023 ~ % curl -X 'DELETE' 'http://127.0.0.1:8081/v0/volumes/327b4e55-589c-42e2-baf9-d1bdadf63366' -H 'accept: /'
{"details":"Volume '327b4e55-589c-42e2-baf9-d1bdadf63366' not found","message":"SvcError :: VolumeNotFound","kind":"NotFound"} 
user@mbp2023 ~ % curl -X 'DELETE' 'http://127.0.0.1:8081/v0/volumes/0a59e089-5db5-4221-9b53-ecdd854a99ec' -H 'accept: /'
user@mbp2023 ~ %

Bang! It’s done!
Thank you @tiagolobocastro

nullzone · 2023-11-24T11:15:37Z

I had exactly the same issue.
Could the garbage job be fixed to implement the cleaning of these orphaned volumes or even better, not to delete the entries when Pv is call for deletion until this really happens?
When a PV is called for deletion when a existing PVC is still bounded, this get stuck (or should) until the PVC is also deleted.

Anyhow, thank you for all the above-mentioned. Helped me to detect the orphaned volumes and gave me a way to clean up and free the blocked resources.

tiagolobocastro · 2023-11-24T11:26:36Z

I guess we need to try and repro this first.
So let me double check what the flow is:

create pvc ---> bound
delete pv ---> gets stuck because pvc is still bound
delete pvc ---> pv is deleted but mayastor volume remains!??

nullzone · 2023-11-24T12:22:50Z

Right.

tiagolobocastro · 2024-01-23T18:05:02Z

Looks like this is a known k8s bug:
kubernetes-csi/external-provisioner#546
https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/2644-honor-pv-reclaim-policy/README.md

tiagolobocastro · 2024-01-24T18:43:04Z

Automatic GC on openebs/mayastor-control-plane#724
WA on current release, please restart the csi-controller pod

kylebrooks-8451 added the NEW New issue label Dec 11, 2022

kylebrooks-8451 assigned GlennBullingham Dec 11, 2022

tiagolobocastro unassigned GlennBullingham Dec 12, 2022

kylebrooks-8451 changed the title ~~io-engine-client can't destroy orphaned bdevs~~ Deleting a PV before a PVC results in an orphaned volume and bdev Dec 12, 2022

tiagolobocastro added backlog and removed NEW New issue labels Jan 20, 2024

tiagolobocastro mentioned this issue Jan 20, 2024

msp still in use when no pv exist #1245

Closed

tiagolobocastro self-assigned this Jan 23, 2024

tiagolobocastro mentioned this issue Jan 24, 2024

Delete Orphaned Volumes on a timer openebs/mayastor-control-plane#724

Merged

tiagolobocastro added in progress and removed backlog labels Jan 24, 2024

tiagolobocastro closed this as completed Jan 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deleting a PV before a PVC results in an orphaned volume and bdev #1269

Deleting a PV before a PVC results in an orphaned volume and bdev #1269

kylebrooks-8451 commented Dec 11, 2022 •

edited

Loading

kylebrooks-8451 commented Dec 11, 2022 •

edited

Loading

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

abhilashshetty04 commented Dec 13, 2022

kvzn commented Apr 11, 2023 •

edited

Loading

nullzone commented Nov 24, 2023

tiagolobocastro commented Nov 24, 2023

nullzone commented Nov 24, 2023

tiagolobocastro commented Jan 23, 2024

tiagolobocastro commented Jan 24, 2024

Deleting a PV before a PVC results in an orphaned volume and bdev #1269

Deleting a PV before a PVC results in an orphaned volume and bdev #1269

Comments

kylebrooks-8451 commented Dec 11, 2022 • edited Loading

kylebrooks-8451 commented Dec 11, 2022 • edited Loading

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

tiagolobocastro commented Dec 12, 2022

kylebrooks-8451 commented Dec 12, 2022

abhilashshetty04 commented Dec 13, 2022

kvzn commented Apr 11, 2023 • edited Loading

nullzone commented Nov 24, 2023

tiagolobocastro commented Nov 24, 2023

nullzone commented Nov 24, 2023

tiagolobocastro commented Jan 23, 2024

tiagolobocastro commented Jan 24, 2024

kylebrooks-8451 commented Dec 11, 2022 •

edited

Loading

kylebrooks-8451 commented Dec 11, 2022 •

edited

Loading

kvzn commented Apr 11, 2023 •

edited

Loading