mergerfs pool mount fails. Best way to monitor and self-recovery via script? #1098
Answered
by
trapexit
TheLinuxGuy
asked this question in
Q&A
-
|
I'm running a tiered cache setup (ZFS cache pool, slow disk on second mergerfs pool). The primary pool is the NFS mount point of linux hosts that copy a lot of data via rsync. I have found that sometimes FUSE mount fails causing NFS to fail as well; it doesn't recover from this failure. On the host running mergerfs pools; this is what I see Here's /etc/fstab (edit: I just realized i misplaced "msplfs" in the wrong mount as I write this post. However my question about monitoring and self-healing is still something I wish to know. The ZFS pool and mount point is fine. How I recover from this is remounting and restarting NFS: Info This tells me /mnt/cached mergerfs cache pool failed. I am not super sure just yet to best troubleshoot the root-cause of the mergerfs crash but I would want to figure out a recommended way to have a monitoring and self-healing script as it seems that mergerfs is not catching itself failing. |
Beta Was this translation helpful? Give feedback.
Answered by
trapexit
Nov 23, 2022
Replies: 7 comments 14 replies
-
Why? It shouldn't fail and shouldn't need to be repaired. There is nothing special about mergerfs vs any other filesystem. |
Beta Was this translation helpful? Give feedback.
2 replies
-
It's possible that using
Where would you find any FUSE errors if any were to be triggered? I did try syslog 'grep fuse' but only a single entry there at boot. |
Beta Was this translation helpful? Give feedback.
-
|
If you're saying msplfs causing crashes then let me look into that. |
Beta Was this translation helpful? Give feedback.
-
|
There are no "errors" to be reported. Things mount or not. If there are config errors it gets reported like any other. If there is a bug and it crashes there wouldn't be something logged except maybe in kernel messages depending on the system setup. |
Beta Was this translation helpful? Give feedback.
1 reply
-
This has been my observation; the mergerfs cache pool mounts successfully. Works for awhile; then local filesystem will report "Input/output error" - if you were seeing this on your system, what steps would you consider doing to detect and debug? I do see some oddity here in syslog Seems like its triggered here |
Beta Was this translation helpful? Give feedback.
-
|
The odd things is that it keeps crashing.
edit 11/21, some kernel error show in logs. But this could be by FUSE mount no longer working? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Just to rule out an issue with ZFS 2.1.3 (stable on ubuntu 10.04) I have upgraded it to ZFS 2.1.6. Also upgraded nfs (2.6.1-1ubuntu1.2) from It's odd that out of my two mergerfs pools the one that keeps crashing is the one which has ZFS+mergerfs_slow_disks - but the later never crashes. BUT I am not sure this is the root cause because if the primary branch which is ZFS (/cache) were to fail, I think mergerfs would just handle the requests towards the secondary branch (/mnt/slow-storage) which is the other mergerfs pool that's been stable. The system hang continue
dmesg stack These crashes seem to be occurring more frequenty when NFS server is configured to operate in NFSv4 mode only. Here was the setup I had done for v4.2 force: https://github.com/TheLinuxGuy/free-unraid/blob/main/miscelaneous_sysadmin.md#nfs |
Beta Was this translation helpful? Give feedback.
2 replies
-
This isn't mergerfs' fault. As the message says mergerfs itself is blocked. Meaning the filesystem/device is holding it up. Or the kernel. Likely means bad hardware. And mergerfs doesn't use system libfuse so non of those libraries matter. |
Beta Was this translation helpful? Give feedback.
-
Thanks for help clarifying what that meant. I have a theory: ZFS + (non-zfs native) NFS make the ZFS mountpoint unresponsive or stuck, perhaps in the middle of mergerfs writes or file moves. This leads to the nfsd kernel messages and the filesystem doesn't recover unless its force unmounted and restarted the systemctl mount unit. I have upgraded all the software (kernel, nfs, zfs) so I don't think this instability has been fixed and its a bug in some non-mergerfs place. To see if I can get my system's NFS mounts to be stable - I destroyed my ZFS pool and converted it to mdadm XFS RAID1 I'll start doing the heavy I/O workloads via NFS from multiple clients and see if I can make the crash behavior occur - if things become stable then hooray for me. The bug is likely in NFS or the kernel... There must be a reason why ZFS has its on built-in NFS server functionality - I think I have been using the standalone/generic NFS server on my configuration since the ZFS mount exports are not set to NFS export (since it wouldn't be compatible with mergerfs) |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to try |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
If it's crashing then strace is pretty useless. Need a stack trace from gdb. gdb path/to/mergerfs run -f -o options branches mountpoint when it crashes thread apply all bt |
Beta Was this translation helpful? Give feedback.
7 replies
-
Got it - sorry I never ran gdb before to debug. When the /mnt/cached mount stops listing files the PID of mergerfs continues to run; as if its hung. For me to be able to run In cases of when mergerfs process doesn't die - would |
Beta Was this translation helpful? Give feedback.
-
|
You're over thinking this. I've given you exactly what is needed when. A debugger is an interactive thing. If you run the app then the interactive part stops because it is running. Like I mentioned when it crashes it will catch that and bring it back to the shell where you will input the command I mention to get the trace. There is no reason for you to break execution till that point because it's running normally. |
Beta Was this translation helpful? Give feedback.
-
|
If it's not dying then we're barking up the wrong tree and I'm not really sure what is going on here without more details or something reproducible. |
Beta Was this translation helpful? Give feedback.
-
Ok maybe I am missing something here; can you help me understand why this may not be a mergerfs binary issue? The mount that becomes inaccessible is created on the system by the mergerfs process. The PID that results of that mount remains to be reported in |
Beta Was this translation helpful? Give feedback.
-
|
#1004 is mergerfs segfaulting (crashing). It is segfaulting because the kernel was asking a question it isn't supposed to ask so mergerfs had not been designed to respond. If the process isn't exiting then it's not "crashing" in the normal sense and I don't know what's going on. Unless you can describe to me a minimal example of the problem the best I can do is make vague suggestions based on common things. |
Beta Was this translation helpful? Give feedback.
Answer selected by
TheLinuxGuy
-
|
Even with XFS + mdadm the crashes / instability on NFS continue. Not sure if this says anything new or different from the past one but here it goes. |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I keep seeing "btrfs" in these stack traces. Since I have already swapped out ZFS with XFS for NVME cache and these crashes continue I think I am going to try with XFS on my "slow disks" branch. Then restart my tests again. |
Beta Was this translation helpful? Give feedback.
-
|
The best thing to do is to minimize all things down as much as possible and build up till it breaks. Stress test just the filesystem alone. Then with nfs. then with mergerfs. or whatever. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If it's crashing then strace is pretty useless. Need a stack trace from gdb.
gdb path/to/mergerfs
run -f -o options branches mountpoint
when it crashes
thread apply all bt