File descriptor passthrough implementation - rTorrent (mmap) with cache.files=off

[nohajc]

Having trouble using rTorrent with mergerfs mount, I came up with a way to implement "FUSE passthrough" using a bit of LD_PRELOAD function hooking.

I'm aware this is by no means a universal solution, however, it does seem to work nicely for what I need.

https://github.com/nohajc/mergerfs-io-passthrough

Basically, for every call to open, my function hook will

• first call the original libc open obtaining a mergerfs file descriptor
• use fgetxattr with user.mergerfs.fullpath to get the real file path
• close the original file descriptor and call open again with the underlying path

This means I'm always getting the "native" file descriptors, completely bypassing mergerfs file IO.
I'd be glad for any feedback... Whether you think this is a good approach, what are the possible caveats, etc.

I'm already aware of some limitations such as the need to cover all possible ways of invoking file IO (currently, this won't work with fopen for example). This could be solved in general by using something like https://github.com/pmem/syscall_intercept.

Also if there are any special read/write features enabled (such as moveonenospc), I suspect they won't work.

Anyway, I consider this a useful workaround which could be selectively applied to applications, rTorrent being the obvious candidate.

[trapexit]

First I have to ask... why do you need page caching disabled? Performance?

As for a work around... sure. The usage of preload is limited to dynamically linked apps but if that works for people then it is a tried and true way to intercept calls. While I've built a number of preload libraries over the years, even a simple union filesystem like shim, I have to say I don't think I ever thought to do so for this purpose but for those wanting native speeds after open then this could be a solution. Even with the additional costs on open and with how often torrent apps tend to open and close files I wouldn't be surprised if it was still more performant on the whole by a decent margin in some workloads.

You're right that anything not going through mergerfs for reads and writes, closes, etc. then will not go through mergerfs and "break". moveonenospc for sure and perhaps some other things.

I'd not be opposed to incorporating something like this into mergerfs v3. Even if it is just a simple cli wrapper / launcher. Though the relevance will be determined on kernel developments.

As for some feedback. open and openat probably are fine but you may want to look at creat as well as the other opens. (see man 2 open).

Also for safety I wouldn't close the old fd before opening the new one. Just in case.

[nohajc]

Performance is part of the motivation. With caching enabled, I also had random crashes when hash checking torrents. This could be worth investigating but I haven't collected any useful information yet... mainly because it happened once or twice and I wasn't able to reproduce it since then.

Thanks for the tips. You're right about the safety and the other APIs that should be covered.
Otherwise, I'm mainly concerned about consistency, especially when multiple processes interact with the filesystem but only some of them use the passthrough.

[trapexit]

I also had random crashes when hash checking torrents.

What crashed?

[nohajc]

Ah, sorry for not specifying... rtorrent crashed.
Could be something like this: rakshasa/rtorrent#954

[nohajc]

Btw, I only noticed now that I could have used IOCTL_FILE_INFO instead of the file xattr. That way it should work even with xattr=nosys, right?

[trapexit]

Yup. I'm going to be removing or at least deprecating xattr API in v3. I'm still playing with stuff so I've not made the ioctl API "public" yet. But you can use it if you wish just know it could change but at the moment I don't think it will.