Mounting CephFS inside pod

[BonzTM]

I feel like an idiot starting this discussion, but I can't seem to find any answers.

Today, I'm using CephFS in-tree volume mounting to mount my CephFS inside my pods. This allows me to have several pods (along with VMs) that can interact with the filesystem as it is. Think of it like mounting an NFS share inside of a pod.

The in-tree plugin is deprecated with K8s 1.28. The documentation now points to this CSI driver instead. Everything I read about the CephFS implementation here is that it creates PVs on top of CephFS filesystems. I want to mount the filesystem, not create a PV on top of it. Am I understanding this wrong?

What is the appropriate way to mount the existing filesystem with the in-tree plugin deprecated? Mount it on each host and leverage hostpath?

[Rakshith-R]

@BonzTM you can make use of static provisioning https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md#cephfs-static-pvc .

[BonzTM]

I checked both csi-provisioner and csi-attacher containers on the csi-cephfsplugin-provisioner pods and they don't seem to log anything.

For reference here's my pv definition:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-static-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Pi
  csi:
    driver: cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-node
      namespace: r-ceph
    volumeAttributes:
      clusterID: "7a1b9c1e-64ff-476c-8081-18b9773f6cd3"
      fsName: "my-cephfs"
      pool: data-ec0
      staticVolume: "true"
      rootPath: /
    volumeHandle: cephfs-static-pv
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

The only 2 differences between your definition and mine, are the volumeAttributes field.

1. All my keys are quoted. You'll notice that they are in the docs as well (with the exception of the rootPath value)
2. I do not have a pool defined. Because it's CephFS, the pool is defined at the FS itself.

    volumeAttributes:
      "clusterID": "7a1b9c1e-64ff-476c-8081-18b9773f6cd3"
      "fsName": "my-cephfs"
      "staticVolume": "true"
      "rootPath": /

[bumarcell]

@Madhu-1 I updated my previous comment to include the pvc definition. They both have status Bound
As for the issue you linked, unfortunately I don't get such events. All I get is the event:
AttachVolume.Attach failed for volume "cephfs-static-pv" : timed out waiting for external-attacher of cephfs.csi.ceph.com CSI driver to attach volume cephfs-static-pv

The CSIDrivers do exist 👍

@BonzTM I quoted the keys and removed the pool key, but nothing changed unfortunately

[bumarcell]

Weirdly enough the only related log I could find was from container csi-resizer on csi-cephfsplugin-provisioner pod:

"PV bound to PVC not found" PV="cephfs-static-pv" PVC="my-namespace/cephfs-static-pvc"

[bumarcell]

I feel like an idiot.. The rook operator namespace was missing before the driver name driver: rook-ceph.cephfs.csi.ceph.com 🤦‍♂️

Thanks a lot in all cases! 🙏

[Madhu-1]

I never through you are using Rook :D good that finally you are able to figure it out.

[BonzTM]

Follow-up question. Does the requested size matter? Say I have 100TB of data in my CephFS root volume. Do I need to request 100TB, or is the field ignored?

[Rakshith-R]

Follow-up question. Does the requested size matter? Say I have 100TB of data in my CephFS root volume. Do I need to request 100TB, or is the field ignored or required?

AFAIK it is ignored but it is better to keep the request value the same as the actual size just to reduce ambiguity.
Changing this value after creation will trigger resize which is not supported for static cephfs PVC.