Significantly slower read times against mergerfs

[itsmegb]

Hi,

I'm trying to troubleshoot why i'm getting significantly slower read and write times against the mergerfs mount path than via the branch path.

Setup:

• 4 HDDs in mergerfs mount.
• All disks are connected to a HBA card
• HBA card is connected via CPU PCIe lanes (not chipset).
• Disks are formatted with xfs

Currently, the 'best' performance i seem to be able to achieve is by setting cache.files=off. Which gives me about 235 MB/s against the mergerfs mount.

When i run the dd commands against the branches directly, i'm getting ~5-7GB/s read and ~230MB/s write.

Interestingly, when using nullrw the write speed against the mergerfs mount shows as being faster than to the branch mounts.

Current mergerfs configuration

# file: .mergerfs
user.mergerfs.async_read="true"
user.mergerfs.auto_cache="false"
user.mergerfs.branches="/mnt/disk1=RW:/mnt/disk2=RW:/mnt/disk3=RW:/mnt/disk4=RW"
user.mergerfs.branches-mount-timeout="300"
user.mergerfs.cache.attr="120"
user.mergerfs.cache.entry="120"
user.mergerfs.cache.files="off"
user.mergerfs.cache.files.process-names="qbittorrent-nox|rtorrent"
user.mergerfs.cache.negative_entry="0"
user.mergerfs.cache.readdir="true"
user.mergerfs.cache.statfs="10"
user.mergerfs.cache.symlinks="false"
user.mergerfs.cache.writeback="false"
user.mergerfs.category.action="epall"
user.mergerfs.category.create="pfrd"
user.mergerfs.category.search="ff"
user.mergerfs.direct_io="false"
user.mergerfs.dropcacheonclose="true"
user.mergerfs.flush-on-close="opened-for-write"
user.mergerfs.follow-symlinks="never"
user.mergerfs.fsname="jbod"
user.mergerfs.func.access="ff"
user.mergerfs.func.chmod="epall"
user.mergerfs.func.chown="epall"
user.mergerfs.func.create="pfrd"
user.mergerfs.func.getattr="ff"
user.mergerfs.func.getxattr="ff"
user.mergerfs.func.link="epall"
user.mergerfs.func.listxattr="ff"
user.mergerfs.func.mkdir="pfrd"
user.mergerfs.func.mknod="pfrd"
user.mergerfs.func.open="ff"
user.mergerfs.func.readdir="seq"
user.mergerfs.func.readlink="ff"
user.mergerfs.func.removexattr="epall"
user.mergerfs.func.rename="epall"
user.mergerfs.func.rmdir="epall"
user.mergerfs.func.setxattr="epall"
user.mergerfs.func.symlink="pfrd"
user.mergerfs.func.truncate="epall"
user.mergerfs.func.unlink="epall"
user.mergerfs.func.utimens="epall"
user.mergerfs.fuse_msg_size="256"
user.mergerfs.ignorepponrename="false"
user.mergerfs.inodecalc="hybrid-hash"
user.mergerfs.kernel_cache="false"
user.mergerfs.lazy-umount-mountpoint="false"
user.mergerfs.link-exdev="passthrough"
user.mergerfs.link_cow="true"
user.mergerfs.log.metrics="false"
user.mergerfs.minfreespace="268435456000"
user.mergerfs.mount="/mnt/jbod"
user.mergerfs.moveonenospc="mfs"
user.mergerfs.nfsopenhack="off"
user.mergerfs.nullrw="false"
user.mergerfs.parallel-direct-writes="false"
user.mergerfs.pid="1650"
user.mergerfs.pin-threads="false"
user.mergerfs.posix_acl="false"
user.mergerfs.process-thread-count="-1"
user.mergerfs.process-thread-queue-depth="0"
user.mergerfs.read-thread-count="0"
user.mergerfs.readahead="2048"
user.mergerfs.readdirplus="false"
user.mergerfs.rename-exdev="passthrough"
user.mergerfs.scheduling-priority="-10"
user.mergerfs.security_capability="true"
user.mergerfs.srcmounts="/mnt/disk1:/mnt/disk2:/mnt/disk3:/mnt/disk4"
user.mergerfs.statfs="base"
user.mergerfs.statfs_ignore="none"
user.mergerfs.symlinkify="false"
user.mergerfs.symlinkify_timeout="3600"
user.mergerfs.threads="0"
user.mergerfs.version="2.39.0"
user.mergerfs.xattr="passthrough"

Prior to this testing, i had all of the cache options set to their defaults and was getting roughly the same read/write speeds (~235MB/s Read&Write) that i get with cache.files=off.

I've done some testing with different configuration options:

nullrw=true

Processing /mnt/disk1...
Writing 1GB file to /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.59264 s, 234 MB/s
Reading 1GB file from /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.14315 s, 7.5 GB/s
Removing 1GB file from /mnt/disk1...
Finished processing /mnt/disk1

Processing /mnt/disk2...
Writing 1GB file to /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.52199 s, 237 MB/s
Reading 1GB file from /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.147214 s, 7.3 GB/s
Removing 1GB file from /mnt/disk2...
Finished processing /mnt/disk2

Processing /mnt/disk3...
Writing 1GB file to /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.7411 s, 226 MB/s
Reading 1GB file from /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.139102 s, 7.7 GB/s
Removing 1GB file from /mnt/disk3...
Finished processing /mnt/disk3

Processing /mnt/disk4...
Writing 1GB file to /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.20644 s, 206 MB/s
Reading 1GB file from /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.145806 s, 7.4 GB/s
Removing 1GB file from /mnt/disk4...
Finished processing /mnt/disk4

Processing /mnt/jbod...
Writing 1GB file to /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.424417 s, 2.5 GB/s
Reading 1GB file from /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.207492 s, 5.2 GB/s
Removing 1GB file from /mnt/jbod...
Finished processing /mnt/jbod

All operations completed!

cache.files=off

Processing /mnt/disk1...
Writing 1GB file to /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.53909 s, 237 MB/s
Reading 1GB file from /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.150837 s, 7.1 GB/s
Removing 1GB file from /mnt/disk1...
Finished processing /mnt/disk1

Processing /mnt/disk2...
Writing 1GB file to /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.53129 s, 237 MB/s
Reading 1GB file from /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.161004 s, 6.7 GB/s
Removing 1GB file from /mnt/disk2...
Finished processing /mnt/disk2

Processing /mnt/disk3...
Writing 1GB file to /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.76927 s, 225 MB/s
Reading 1GB file from /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.166955 s, 6.4 GB/s
Removing 1GB file from /mnt/disk3...
Finished processing /mnt/disk3

Processing /mnt/disk4...
Writing 1GB file to /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.14311 s, 209 MB/s
Reading 1GB file from /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.154278 s, 7.0 GB/s
Removing 1GB file from /mnt/disk4...
Finished processing /mnt/disk4

Processing /mnt/jbod...
Writing 1GB file to /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.09812 s, 211 MB/s
Reading 1GB file from /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.57378 s, 235 MB/s
Removing 1GB file from /mnt/jbod...
Finished processing /mnt/jbod

All operations completed!

cache.files=partial

Processing /mnt/disk1...
Writing 1GB file to /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.62398 s, 232 MB/s
Reading 1GB file from /mnt/disk1...

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.145382 s, 7.4 GB/s
Removing 1GB file from /mnt/disk1...
Finished processing /mnt/disk1

Processing /mnt/disk2...
Writing 1GB file to /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.51992 s, 238 MB/s
Reading 1GB file from /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.150044 s, 7.2 GB/s
Removing 1GB file from /mnt/disk2...
Finished processing /mnt/disk2

Processing /mnt/disk3...
Writing 1GB file to /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.70602 s, 228 MB/s
Reading 1GB file from /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.151255 s, 7.1 GB/s
Removing 1GB file from /mnt/disk3...
Finished processing /mnt/disk3

Processing /mnt/disk4...
Writing 1GB file to /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.13287 s, 209 MB/s
Reading 1GB file from /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.168546 s, 6.4 GB/s
Removing 1GB file from /mnt/disk4...
Finished processing /mnt/disk4

Processing /mnt/jbod...
Writing 1GB file to /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.37677 s, 200 MB/s
Reading 1GB file from /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.999 s, 71.6 MB/s
Removing 1GB file from /mnt/jbod...
Finished processing /mnt/jbod

All operations completed!

cache.files=libfuse

Processing /mnt/disk1...
Writing 1GB file to /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.59261 s, 234 MB/s
Reading 1GB file from /mnt/disk1...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.141418 s, 7.6 GB/s
Removing 1GB file from /mnt/disk1...
Finished processing /mnt/disk1

Processing /mnt/disk2...
Writing 1GB file to /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.63872 s, 231 MB/s
Reading 1GB file from /mnt/disk2...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.150533 s, 7.1 GB/s
Removing 1GB file from /mnt/disk2...
Finished processing /mnt/disk2

Processing /mnt/disk3...
Writing 1GB file to /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.28297 s, 203 MB/s
Reading 1GB file from /mnt/disk3...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.147454 s, 7.3 GB/s
Removing 1GB file from /mnt/disk3...
Finished processing /mnt/disk3

Processing /mnt/disk4...
Writing 1GB file to /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.12317 s, 210 MB/s
Reading 1GB file from /mnt/disk4...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.148803 s, 7.2 GB/s
Removing 1GB file from /mnt/disk4...
Finished processing /mnt/disk4

Processing /mnt/jbod...
Writing 1GB file to /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.67421 s, 189 MB/s
Reading 1GB file from /mnt/jbod...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 27.1975 s, 39.5 MB/s
Removing 1GB file from /mnt/jbod...
Finished processing /mnt/jbod

All operations completed!

Any suggestions for configuration changes to improve my read speeds would be awesome, if they help with write speeds as well thats a bonus!

[trapexit]

When i run the dd commands against the branches directly, i'm getting ~5-7GB/s read and ~230MB/s write.

This doesn't make sense. There is zero chance HDDs are giving you 5-7GB/s speeds. The max theoretical speed of a SATA 3 connection is just 6Gbit/s.

Your dd command is not correct and allowing for caching.

Interestingly, when using nullrw the write speed against the mergerfs mount shows as being faster than to the branch mounts.

This should not be surprising at all. The whole point of nullrw is to make actual read and writes not happen. The data is thrown away on a write and returns success. There is no interaction with underlying branches. The point of the feature is to test theoretical max perf. What the overhead of FUSE / mergerfs is.

Any suggestions ...

Your testing methodology is wrong. Please show what you're doing with dd. You should just use the examples I provide in the docs.

The cache.files=off values look totally normal and are in line with the likely actual performance of the drives.

[trapexit]

https://github.com/trapexit/mergerfs?tab=readme-ov-file#benchmarking

[itsmegb]

Thanks for coming back to me.

Am i misreading the output of the dd commands that i included above? Sorry, that's what i based my assumption on.

The dd commands i used are the ones from the benchmark section of the docs. This is the loop from the script i ran to test all the drives:

for MNTPOINT in "${MOUNT_POINTS[@]}"; do
    echo "Processing /mnt/$MNTPOINT..."

    # Write a 1GB file using dd
    echo "Writing 1GB file to /mnt/$MNTPOINT..."
    dd if=/dev/zero of=/mnt/$MNTPOINT/1GB.file bs=1M count=1024 oflag=nocache conv=fdatasync status=progress

    # Read the 1GB file using dd
    echo "Reading 1GB file from /mnt/$MNTPOINT..."
    dd if=/mnt/$MNTPOINT/1GB.file of=/dev/null bs=1M count=1024 iflag=nocache conv=fdatasync  status=progress

    # Remove the 1GB file
    echo "Removing 1GB file from /mnt/$MNTPOINT..."
    rm /mnt/$MNTPOINT/1GB.file

    echo "Finished processing /mnt/$MNTPOINT"
done

I removed the conv=fdatasync statement from the read because it threw an error:

dd: fsync failed for '/dev/null': Invalid argument

[trapexit]

Try putting oflag=dsync on the read test to get realistic values.