Statically compile "cephcsi" binary?

[leseb]

I know cephcsi is using go-ceph which internally uses a lot of cgo but are they any chances that we could try to statically link all the libraries we need?
As anyone attempted that in the past?

Thanks!

[phlogistonjohn]

I would suggest: probably not.

There's the first issue of how to force go to build statically with cgo enabled, but beyond that the libs that we're linking to would need to be capable of static linking: so all of the lib{rados,rbd,cephfs} etc... but even their dependencies too, right? Even down to the level of glibc? I think it would be very tough to manage.

Just for my own curiosity: what issue(s) are you running into with cephcsi being dynamically linked?

[phlogistonjohn]

OK, interesting idea, I can see why this makes it a problem. It sounds a bit odd to me we'd want to avoid a container as the "point" of a container is to have a self-contained bundle of stuff. :-)
If you've got that "root level" access to host to put binaries on there, would it be possible to run a container image outside of k8s? Say via a systemd service + podman/docker/crio/whatnot?

[leseb]

The DaemonSet we are talking about right now is the csi-cephfsplugin, so I'd not move it on the host and live on its own. It feels too much out of our control once it's done.
One thing I'm exploring now is delegating the mount/umount to a new daemon instead of cephcsi. Essentially, something must hold the mount and hold it until the node reboots, and until then nothing should happen to this pod (no restart).

[nixpanic]

With rbd-nbd it is possible for the mapped RBD-image to get re-connected to the Ceph cluster after a container restart.

This is (not yet) possible with fuse-cephfs, but there is some work in progress. @gman0 or @pkalever might have followed the Linux kernel FUSE work a little closer than I did.

[leseb]

Once this is available, are there any plans to deprecate all the other methods? (mount.cephfs etc). If not, we still need to implement a fix for those.

[gman0]

Regarding FUSE, #2634 was recently merged, but there is a huge caveat:

This feature allows CSI CephFS plugin to be able to detect if a ceph-fuse mount is corrupted during the volume publishing phase, and will attempt to recover it for the newly scheduled pod. Pods that already reside on a node whose ceph-fuse mountpoints were broken may still need to be restarted, however.

(source https://github.com/ceph/ceph-csi/blob/devel/docs/ceph-fuse-corruption.md)

Unless the FUSE kernel module is able to handle transient disconnects (given its server-client architecture), I think any attempts to solve these issues in userspace will fall short. You can check https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAJfpegvEppXZbX25Nage463biMjWPKthr%3D519PSJ61yZmTavCw%40mail.gmail.com/#msg37401044 but I haven't made any progress on this yet.

[nixpanic]

As @phlogistonjohn stated, static linking of the executable will be very difficult. This is not something that projects normally recommend, as it includes a whole tree of dependencies that need to provide static linkable versions of their libraries. In the case of Ceph-CSI, it already becomes problematic with the Ceph libraries.

Furthermore, the executable is not completely self-contained for everything. For certain operations other executables need to be called (mostly for things like rbd attach, mount -t cephfs, mount.cephfs, fuse/nbd pieces and cryptsetup), which again depend on shared libraries provided by the base container-image.

I do not think any problem is worth creating and maintaining a statically linked cephcsi executable.