Alternative ACL storage

[static-moonlight]

I have to admit, I like this project, but no matter where I go: the main reason why I have to repeatedly rule out versitygw is almost always the same: the requirement for extended attributes.

I'm sure that at some point during development you have thought about this. I'm sure you have good reasons why you decided to make them a requirement. However, if the architecture allows it, I'd love to see alternative ways to let versity gateway store the information (just acl?) that are currently written to extended file attributes. Even if that would reduce performance. In the simplest case maybe a file or a set of files somewhere?!

idk ... what do you think?

[benmcclelland]

The xattr design was chosen because it mapped best to the use case for object/bucket metadata. I assumed at the time that xattr support is pretty ubiquitous for modern filesystems. What filesystems are you trying to use that doesn't have this support?

It's not a requirement, there could be alternatives for where to store metadata such as special metadata files, external database, etc as you suggest. We would need to give this some thought for this design. It would be tempting to have a .<object>.meta file for each <object> file created. This would double the number of files for datasets, and would add some complexity for the list objects code needing to skip these for object listing, but use them for filling in the metadata info.

Another option is if the specific filesystem that doesn't have xattr support maybe has another method for storing file metadata. Then we could have a new backend specifically for this fs.

[static-moonlight]

What filesystems are you trying to use that doesn't have this support?

Most of the storage infrastructure available to me uses NFS. Sometime even hidden under a layer of filesystem virtualization. NFS v3 seems to be the most common. In some exceptions there is NFS 4.2 available, with no guarantee that this is going to stay enabled, because there have been some stability issues. So, not a lot of room to maneuver when it comes to storage ...

But that's just me. I'm sure most peoples out there don't have that problem ;)

[benmcclelland]

Ok, yeah I had a feeling this might be NFS. The NFS xattr support is relatively recent. We may need a more general answer for NFSv3. I'm certainly open to ideas on where to store object metadata. This does include ACLs, but also user defined object metadata, etags, tagging, etc. So there can be a handful of attributes needed per object.

[static-moonlight]

I'm certainly open to ideas on where to store object metadata

If that would be my party, i would suggest the following:

1. refactor the existing implementation (if necessary), by putting everything xattr-related behind an architectural boundary (repository pattern comes to mind)
2. add additional implementations (open-close-principal when done right)

How step 2 looks in detail (meaning the how to store these information and where) would be a problem for later. Making sure the architecture can handle it would be more important for now.

[benmcclelland]

Looking at some other filesystem projects that store metadata over NFS, appears they have two approaches:

• store metadata in sidecar file (so a .meta file for each real file written)
• store metadata in external database or key value store

Since there are multiple attributes per file, there might be some read-modify-write overhead to adding/modifying attributes in a single sidecar or datastore value. Most likely the storage for this would end up being JSON encoded attribute list.

I think we could refactor the code to abstract the metadata storage and allow a few options here. As long as one of the above (or other options?) would be acceptable.

I can start a github issue to track the request and design here.

[benmcclelland]

#511 created to track this, not sure when we will be able to make some progress on this though

[benmcclelland]

@static-moonlight The PR that added the metadata abstraction layer got merged. Here is a proposed possibility for an alternative to xattrs storage: #522.

Would you be interested in testing this to see what you think?

[static-moonlight]

That was quick :) I will take a look at it ...

[static-moonlight]

I would happily test this. Is that already possible with the latest version or do I need to get/compile a development version from somewhere?

[benmcclelland]

Not yet in a version. Its in a pull request still looking for feedback. You can try this out with the following:

git clone https://github.com/versity/versitygw.git
cd versitygw 
git co origin/ben/sidecar_meta
make

You will need: git, make, and the go compiler installed for this. Once built, there is a new --sidecar option to enable this, for example:

./versitygw -a myaccess -s mysecret posix --sidecar /tmp/gw

[static-moonlight]

Just a few general thoughts:

1. I like the idea of having a folder which stores one metadata file per object. I agree, I would not put everything into one file. That sounds not efficient in terms of read and write/update operations.
2. I assume the metadata folder is going to be located in the given data folder? So with a command like this: /versitygw [...] posix --sidecar /tmp/gw, the metadata folder would probably something like this: /tmp/gw/.vgw or /tmp/gw/.versitygw? I like that idea. I think I would have done the same. MinIO is doing this as well. versitygw only has to make absolutely sure that hidden (dot) folders in general and/or this .vgw folder in particular is never ever going to be served as a bucket.
3. I would not introduce a --sidecar parameter at this point. That's too specific. I would rather recommend to introduce a generic parameter that would allow later extension, like --metadata-repository or --metadata (or something like that, and don't forget the beautiful env variables ;) ). That parameter would default to xattr (when no explicitly defined). But it could also be set to something else, like folder. Example: /versitygw [...] posix --metadata=folder /tmp/gw. Potential values like xattr or folder would allow versitygw to select the correct metadata storage handler during startup. It would also allow other potential implementations (e.g. database(s) or magic potatoes or whatever), without the need to touch this mechanism every time. Think of it like a plugin-in system. The metadata storage is responsible for the "how" metadata are accessed and has a name, like "xattr" or "folder" so it can be identified internally. This would make this really flexible.

[howff]

PR #522 has been updated, see the comments there

[static-moonlight]

I see ...

/tmp/data/
/tmp/data/.vgw_meta.bucket2
/tmp/data/.vgw_meta.bucket2/acl
/tmp/data/bucket2
/tmp/data/.vgw_meta.bucket1
/tmp/data/.vgw_meta.bucket1/acl
/tmp/data/bucket1
/tmp/data/bucket1/.vgw_meta.file
/tmp/data/bucket1/.vgw_meta.file/etag
/tmp/data/bucket1/.vgw_meta.file/X-Amz-Meta.Mtime
/tmp/data/bucket1/file
/tmp/data/bucket1/.sgwtmp

Those metadata files are scattered around. I would not do that. I recommend concentrating them in a single folder. Consider this: versitygw has no idea, what else is happening with file system. Data synchronization, sharing with other applications, ... And since this could be anything it is hard to say how those other applications and systems would react to these "littered" files, to do whatever they are supposed to do without accidentally processing vgw's metadata files. It seems easier to handle such potential issues, by having just one single location to worry about, or at the very least: as little as possible (there seems to be a .sgwtmp folder per bucket as well ... I think that's ok though ... but I would not recommend to got any further in terms of number of hidden files somewhere).

I think it might also be a good idea to have a list in the documentation with all hidden (dot) files/folders which are being used by versitygw, to maximize transparency and therefore compatibility with other systems. Maybe even with a suggestion if those files should be included in backups or not (I think .sgwtmp might be a candidate to be excluded from backups).

But back to topic: Aside from the details: It seems to work good. In my case /tmp is a ramdisk without support for extended attributes. Everything works fine. No errors so far.

[benmcclelland]

Ah yes, I like the top level folder better. I can change that. Good idea about the metadata option too, that will help to constrain the number of option flags for these.

[benmcclelland]

@static-moonlight branch updated with changes requested. This adds a top level dir to store metadata separately:

/tmp/gw
/tmp/gw/.vgw_meta
/tmp/gw/.vgw_meta/testbucket
/tmp/gw/.vgw_meta/testbucket/.meta
/tmp/gw/.vgw_meta/testbucket/.meta/acl
/tmp/gw/.vgw_meta/testbucket/dir1
/tmp/gw/.vgw_meta/testbucket/dir1/dir2
/tmp/gw/.vgw_meta/testbucket/dir1/dir2/test
/tmp/gw/.vgw_meta/testbucket/dir1/dir2/test/.meta
/tmp/gw/.vgw_meta/testbucket/dir1/dir2/test/.meta/X-Amz-Meta.s3cmd-attrs
/tmp/gw/.vgw_meta/testbucket/dir1/dir2/test/.meta/etag
/tmp/gw/testbucket
/tmp/gw/testbucket/.sgwtmp
/tmp/gw/testbucket/dir1
/tmp/gw/testbucket/dir1/dir2
/tmp/gw/testbucket/dir1/dir2/test

The option is changed to --metadata sidecar now too.

[benmcclelland]

Thats what I was worried about too. This is a general sidecar file problem if files/directories get moved/deleted outside of the application, then the sidecars get orphaned. I'm not really excited about having something that has to keep an eye out for orphaned metadata here. Maybe there could be an explicit repair option to invoke scanning and cleanup. It's not really scalable to have something constantly scanning the namespace in the background.

I think it is definitely expected that files will get accessed/created/removed/moved outside of versitygw. Thats sort of the whole point to mapping the object names into the posix namespace for the compatibility with other posix utilities.

[static-moonlight]

A few more thoughts on that:

1. I noticed that the metadata storage path could be simpler: <data-dir>/.vgw_meta/<bucket-name>/<object-name>/ instead of <data-dir>/.vgw_meta/<bucket-name>/<object-name>/.meta/ (right now there is one hidden .meta folder inside the already hidden .vgw_meta folder ... that seems a bit unnecessary)
2. When it come to house keeping: you don't need to cleanup what's not there in the first place. I was looking at the typical metadata per object that I saw in my tests. I don't know if there are potentially more metadata attributes, but when it comes to those 2, I think they might be unnecessary, maybe even a potential source for problems (this even applies to the metadata that would have been stored in the xattr).
   1. X-Amz-Meta.Mtime: Is this any different from the actual mtime of the file in the file system? What happens, when an outside application changes this file? I mean you said it yourself: things like that are considered fair game. No matter where this is stored (sidecar metadata file or xattr), it's unrealistic to expect, that an outside application is updating those meta information. So instead of storing the mtime as meta attribute with a potentially inconsistent value, how about using the actual mtime from the file system?
   2. etag: This is used for transfer integrity, right? As far as I know, it's basically a checksum, correct? So for smaller files, it could be generated on the fly. And the same problem applies here as well: if an outside application changes that file, the etag might be rendered invalid?!
3. These metadata attributes are being created by vgw, right? What if a file was initially created by an outside application. Would vgw create additional metadata for that file at some point, or wouldn't there any vgw metadata for that file ever? And if there aren't any, what's the worst thing that can happen?

I not suggestion to completely avoid metadata. But if there is a way to reduce them without sacrificing functionality, this might result in significantly less metadata sidecar files. Which means, a potentially existing watchdog process doesn't need to do that much, and doesn't need to run that often and the impact of orphaned metadata files is generally smaller.

[benmcclelland]

I think X-Amz-Meta.Mtime is being set by the s3 client. The gateway doesn't add any internal metadata other than etag. And you are correct that etag is a checksum that might get out of sync if the file is modified outside of the gateway. I'll have to think about this. There is an optimization to not have to recalculate the checksum on GET if it is stored. This can be a significant performance improvement for larger files. This brings up another concern too that technically all object metadata should be removed if the file is deleted or modified. In object semantics, replacing an object also clears out all of the previous object's metadata. Maybe the sidecar metadata needs to store the file timestamp so that all metadata is invalidated if the file is modified.

[static-moonlight]

Maybe the sidecar metadata needs to store the file timestamp so that all metadata is invalidated if the file is modified.

1. It's not just the sidecar metadata, it's also the xattr. As far as I know, it depends on the application and how a file was changed, for the xattr to get removed or preserved. In the worst case you have to assume that both, xattr and sidecar metadata, are potentially invalid. Which is bad.
2. inotify come to mind as well. it might be an option to keep the gateway updated about filesystem changes.
3. And when it comes to actually having/collecting metadata, maybe that can be left to whoever operates the gateway. You could let the user decide if they want the metadata or not (meaning: put in a switch to either ignore the clients metadata or to persist them).

To keep things simple you could push back the sidecar metadata files for now. There seem to be a few generic issues to be solved first, and it might be easier to just need to worry about one metadata storage mechanism for the moment (xattr). When there is a strategy in place, I am confident it can be applied to other mechanisms as well.

[benmcclelland]

Yes, you are right. I think we do need a more generic strategy for detecting stale metadata. Thanks for the input on this!