Empty disk and moving data to other disks before removing it

[zombieagain]

Hi, I have an 8 disk snapraid configuration, 2 drives are parity and 6 are pooled with mergerfs. Mergerfs is amazing and I've been using it for several years with zero issues.

Today the SMART info is telling me that one of my disks is failing. I have plenty of space in the other disks to move the data, without the need to immediately replace the failing drive.

Is there any function in mergerfs that would allow me to empty this disk and basically "balance" the data across the other disks.

Or any other suggestions on how to proceed? For now I'm thinking:

1. unmount mergerfs
2. do "sudo rsync -av" one directory at the time from the failing disk to other disks (one of the time, because I don't have a single disk that has enough space, so probably I need to transfer half of the data to one disk and the other half to another disk)
3. remove the failing disk from mergerfs configuration
4. mount mergerfs and cross my fingers that everything works

Is this the best strategy and is rsync -av the best command or are there other suggestions/switches that I should use?

Thank you in advance for any suggestions!

[trapexit]

Glad it's been working for you.

Nothing built in.

You are in the ballpark but there is a better strategy.

1. Use the runtime control API of mergerfs to set the branch mode in the pool to RO. Or remount with it set to RO.
2. mount a new instance of mergerfs minus the branch to remove to some temporary location
3. rsync -avP /mnt/branch-to-remove/ /tmp/tmp-mergerfs/
4. do 3 again to make sure you got everything in case something changed
5. change the config for the main mergerfs pool to remove the branch in question and remount and/or use the runtime to do the same
6. umount /mnt/branch-to-remove/

done

This always seemed to me to be the most obvious and flexible method of performing this procedure but many don't think of it. I'm likely to write something that does this general process at some point. The reason I haven't up to this point is often people just have different ideas on how they want to replace a filesystem and it'd be a project in itself to accommodate for it all. We'll see if I can just build something simple that works for most people into the product. But that will come after v3 finishes likely.

[j4nl]

Can something along these lines be a starting point for building a tool for people to use?

#!/user/bin/python3
#Script name: remove_mergerfs_branch.py
import os
import subprocess
import sys
import shutil

def run_command(command):
    result = subprocess.run(command, shell=True, capture_output=True, text=True)
    if result.returncode != 0:
        print(f"Error: Command failed: {command}")
        print(result.stderr)
        sys.exit(result.returncode)
    return result.stdout

def check_privileges():
    if os.geteuid() != 0:
        print("Error: This script must be run as root or with sudo.")
        sys.exit(1)

def check_mount_point(mount_point):
    if not os.path.ismount(mount_point):
        print(f"Error: The specified mount point {mount_point} does not exist or is not a mount point.")
        sys.exit(1)

def check_storage(src, dest):
    total, used, free = shutil.disk_usage(dest)
    src_size = sum(os.path.getsize(os.path.join(dirpath, filename)) 
                   for dirpath, dirnames, filenames in os.walk(src) 
                   for filename in filenames)
    if src_size > free:
        print("Error: Not enough storage available on the destination mount point.")
        sys.exit(1)

def set_mergerfs_ro(pool):
    print("Setting mergerfs pool to read-only mode...")
    run_command(f"mergerfs.ctl {pool} category.action=RO")

def mount_new_mergerfs(exclude_branch, pool, tmp_mount):
    print(f"Mounting new instance of mergerfs excluding {exclude_branch}...")
    branches = run_command(f"mergerfs.ctl {pool} branches")
    branches = branches.strip().split(',')
    branches = [b for b in branches if exclude_branch not in b]
    new_pool = ":".join(branches)
    run_command(f"mergerfs -o defaults,ro {new_pool} {tmp_mount}")

def rsync_branch(src, dest):
    print(f"Rsyncing from {src} to {dest}...")
    run_command(f"rsync -avP {src}/ {dest}/")
    print("Repeating rsync to ensure all changes are captured...")
    run_command(f"rsync -avP {src}/ {dest}/")

def update_mergerfs_config_and_remount(pool, exclude_branch):
    print("Updating mergerfs configuration and remounting...")
    branches = run_command(f"mergerfs.ctl {pool} branches")
    branches = branches.strip().split(',')
    branches = [b for b in branches if exclude_branch not in b]
    new_pool = ":".join(branches)
    run_command(f"mergerfs.ctl {pool} branches={new_pool}")

def unmount_branch(branch):
    print(f"Unmounting {branch}...")
    run_command(f"umount {branch}")

def main():
    if len(sys.argv) != 3:
        print("Usage: python remove_mergerfs_branch.py <mergerfs_pool> <branch_to_remove>")
        sys.exit(1)

    pool = sys.argv[1]
    branch_to_remove = sys.argv[2]
    tmp_mount = "/tmp/tmp-mergerfs"

    check_privileges()
    check_mount_point(branch_to_remove)

    if not os.path.exists(tmp_mount):
        os.makedirs(tmp_mount)

    set_mergerfs_ro(pool)
    mount_new_mergerfs(branch_to_remove, pool, tmp_mount)
    check_storage(branch_to_remove, tmp_mount)
    rsync_branch(branch_to_remove, tmp_mount)
    update_mergerfs_config_and_remount(pool, branch_to_remove)
    unmount_branch(branch_to_remove)

if __name__ == "__main__":
    main()

[zombieagain]

Amazing! Thanks for the great and fast response! To be honest I couldn't figure out how to use the runtime control API (is that mergerfs.ctl?) so I ended up changing the mounting option for that drive in fstab and remount.
Rsync is running now and it will take a while. I'll drop a line when the process is complete.
Thanks again!

[trapexit]

The runtime API details are in the docs. Can use standard tooling to change values. mergerfs.ctl is just a wrapper.

[zombieagain]

Thanks again! Everything worked as you described.