Trouble with NFS and ACL

[static-moonlight]

I've set up versitygw (v0.16) in a Kubernetes cluster. versitygw is configured to use an NFS share as posix backend. It looks good on a high level. However, when I started to execute some simple S3 commands (using rclone), like creating a bucket, it works but it also fails. The folder is created (in the filesystem, on the NFS share), but there are errors in the logs:

$ rclone mkdir s3:test1

2024/03/13 13:30:17 Internal Error, set acl: xattr.Set test1 user.acl: operation not supported
13:30:17 | 500 |   12.165176ms | [...] | PUT | /test1 | -
13:30:17 | 409 |     652.053µs | [...] | PUT | /test1 | -

As I said: The directory is created nonetheless. But when I try to upload a file it gets bad. rclone seems to hang for quite a while and I can see lots of errors in the logs:

$ rclone copy file s3:test1/

2024/03/13 13:41:15 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:14 | 500 |      192.81µs | [...] | HEAD | /test1/file | -
2024/03/13 13:41:15 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:14 | 500 |      166.78µs | [...] | HEAD | /test1/file | -
2024/03/13 13:41:15 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:15 | 500 |     154.784µs | [...] | HEAD | /test1/file | -
2024/03/13 13:41:15 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:15 | 500 |     188.069µs | [...] | HEAD | /test1/file | -
2024/03/13 13:41:15 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:15 | 500 |     172.913µs | [...] | HEAD | /test1/file | -
2024/03/13 13:41:16 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:16 | 500 |     556.628µs | [...] | HEAD | /test1/file | -
13:41:17 | 400 |      59.462µs | [...] | GET | /metrics | -
2024/03/13 13:41:18 Internal Error, get acl: xattr.get test1 user.acl: operation not supported
13:41:18 | 500 |     184.182µs | [...] | HEAD | /test1/file | -

These are the NFS options being used (server default options):

rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=[...],local_lock=none,addr=[...]

I could also successfully verify that ACL is working, using nfs4_setfacl and nfs4_getfacl. So I guess the NFS-side should be fine and the requirement "The filesystem must have the ability to store extended attributes" from the documentation should be satisfied. Additionally I played arrount with different NFS versions (3, 4, 4.2), with no change whatsoever.

Is there anything I can do to make this work? Do you have any ideas of things I could try?

I admit: I don't have a lot of experience with ACL, so I'm not exactly sure were to look for the root cause. This might as well be a Kubernetes volume mount issue or a root/non-root permission problem (even if the error states "operation not supported"). I running out of ideas.

I will try to do another test (soon), using the same NFS share but without Kubernetes. Maybe this helps to find the problem ...

[benmcclelland]

The posix filesystem backend requires xattr support. This is used to store object/bucket metadata such as s3 ACLs, tags, etag, etc. NFS has only fairly recently added xattr support. See this article: https://lwn.net/Articles/799185/. I haven't been able to try this out yet, but I think this requires linux kernel 5.9 or newer, and "user_xattr,vers=4.2" client mount options.

The ACL references in the errors are related to storing the S3 bucket ACL information on the directory in the xattr "user.acl", and are not related to NFS ACLs.

[benmcclelland]

The cli commands to test for xattr compatibility would be getfattr and setfattr in linux.

[benmcclelland]

I tested this on an EL8 system, and this version of NFS appears to work successfully.

# mount -t nfs localhost:/mnt/localfs /mnt/nfs
# mount | grep nfs
localhost:/mnt/localfs on /mnt/nfs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

host info:

# uname -a
Linux bentest-n0 4.18.0-513.18.1.el8_9.x86_64 #1 SMP Thu Feb 22 03:02:37 EST 2024 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
AlmaLinux release 8.9 (Midnight Oncilla)

The important part is the xattr support:

# cd /mnt/nfs
# touch file
# setfattr -n user.test file 
# getfattr -d file
# file: file
user.test=""

[benmcclelland]

#459 adds a check for better error messages when xattr is not supported for backend filesystem

[static-moonlight]

Thank you so much for testing this. I was troubleshooting all day, even considering that I might have to upgrade the cluster nodes (currently we are running Rocky 8 on those machines).

The good news is, we even tweaked some server settings and found a configuration that solved the ACL problem. At least it looks that way. I don't see those errors anymore.

The bad news it that file uploads are still unreliable (using rclone). Sometimes it works, and then suddenly it doesn't. I need to do more testing on this one, because I can't rule out IO issues yet. I keep you updated (if you want) ...

[benmcclelland]

Yes please. I can try some rclone tests over here as well.

[benmcclelland]

In case this helps with your testing, I tested rclone from yum install and got a bunch of errors. After some investigation, it appears that this older rclone was not correctly generating signatures all the time. I tested the latest version, and this resolved all of the errors I was seeing:

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64
./rclone copy /etc vgw:backup/

broken:

# rclone version
rclone v1.55.1-DEV
- os/type: linux
- os/arch: amd64
- go/version: go1.15.5
- go/linking: dynamic
- go/tags: none

works:

# ./rclone version
rclone v1.66.0
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.88.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

[static-moonlight]

This is not an rclone issue or a vgw problem. I can confirm low level I/O problems with NFS 4.1 and NFS 4.2. NFS 3 and 4.0 looks good though. I don't know when we can solve this, but for now this limits my options significantly. Without NFS 4.2 I won't be able to provide a filesystem with extended attributes.

[static-moonlight]

Since this is not a vgw problem, I am closing this topic. Thank you so much for the help!