MergerFS mount point crashing

[tigerkzr]

I was wonder if anyone has a guide or suggestions to get me started in the right direction towards trouble shooting why my mergerfs mount point keeps crashing.

My setup is ubuntu 22.10 server on Dell server hardware and up until a few weeks ago, MergerFS was rock solid. Now I am experiencing a crash of the mount point every other day or so. Basically I get the "transport endpoint is not connected" error. I was originally using a self compiled binary from the latest source. Since the problems started, I have reverted to the binary available from the Debian install package with no improvement.

At this point, I am just not sure where to even begin digging into it. I have used grep to try to dig into the system and kernel logs for mergerfs and fuse but end up with no results.

Any help is appreciated

[trapexit]

If it is crashing then you won't really find anything in logs. If it was a known situation and logable it wouldn't be crashing :)

Could you provide basic details as requested here: https://github.com/trapexit/mergerfs/tree/latest-release#information-to-include-in-bug-reports

The only way to see exactly what is going on is to catch it when it fails but that would require attaching a debugger.

[trapexit]

Yes, attaching stops the execution of the app. You have to run "continue" or just "c" in the interface to get it to continue.

[tigerkzr]

ahhh.. lol ima noob

[tigerkzr]

here's what gdb produced.

gdb output.txt

[trapexit]

How many CPUs does your system have? That's a lot of threads. Not that it should matter.

This was built with make DEBUG=1? It's missing details which I'd expect to see.

[tigerkzr]

48 logical cores :). Dual processor dell t430.

I used "make DEBUG=1 deb"

[trapexit]

Oh, no. Just make DEBUG=1 and run build/mergerfs. Building the deb does its own thing.

[tigerkzr]

yah, it didn't break until one of the .38s

[trapexit]

OK. I have it installed but I guess haven't rebooted in a while. Once I get some time I can try booting up the 5.19.0-38.

[trapexit]

24+ hours of stressing and nothing. I'm busy getting ready for a major move but if I find time I'll try setting up a VM or a secondary machine to test with a newer kernel. Unfortunately, it could a very tight race condition or something in which case it could be I never hit it on my setup.

[tigerkzr]

Oh man, worry about your move. As I said, things are mostly stable as long I avoid certain apps so my setup is far from crippled by this. No crashes since the last one I reported. I definitely understand the complexities, and likely luck, in tracking down what's actually happening. Further, as I seem to be the only one experiencing this so far, I can't exactly say it's a major priority for your resources. Given that, I greatly appreciate all the attention you have given this so far.

So again, worry about your move, and if you find the time, awesome, if you're too busy with other things, no big deal. If you do find the time, please keep me updated and absolutely let me know if I can do anything to assist.

[tigerkzr]

Just an update: I haven't had a crash since the last one I posted about here. I have been avoiding using SAB, so I may give it a few more days to finish up some tasks then try to crash it with SAB again.