Mapping to unix accounts when using POSIX backend #428
-
|
Hello all, First off this is a super cool project! I've been looking for a stateless S3 gateway for some time now and this is by far the most promising project I've come across. I have a rather unique use case in mind that I would like to discuss the feasibility of. After reading through the docs I'm unsure if it fits within the vision of this project. Essentially I would like to use this gateway with IAM, specifically LDAP, alongside the POSIX backend and have the permissions that are enforced come from the actual POSIX permissions. And I would like to have this apply to all items within the "bucket", not just the root level. If I understand correctly, this would entail that:
In my use case it is correct to assume that any user within my LDAP directory is also a local unix user. And I have no need to modify the POSIX permissions or extended ACLs via the S3 API. Really I just need the basic object operations (put, delete, get, list) as well as uploads. Is this something that is even possible? And if so does it align with the goals/scope of this project? If so I would love to contribute in any way I can! Thanks for everyone's time. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
|
@kyleaupton Thanks for checking out the project! The difficulty with the posix permissions is that ideally we would want the filesystem to enforce the permissions/ACLs. And in order to do that we would need to set the effective userid/groupid to that user for any filesystem access. Normally this would be done with seteuid()/setegid() to set the effective uid/gid for that thread. But with Go, the goroutines are not 1:1 with OS threads, but instead lightweight routines scheduled across available threads. There are ways to lock a single goroutine to an os thread, but then this starts to have implications on the scalability of the service if each incoming request needs to fork a new thread. So for a service that needs to handle many simultaneous requests from multiple users, the seteuid() route is not really the best way. So an alternative would be to have some sort of middleware authorization layer that looks at the posix permissions and ACLs to determine if the requesting account should have access. This would work, but can get complicated trying to implement all the possible ACL rules out there. The posix ACLs are implemented in the kernel, and I'm not aware of any Go project that has attempted to parse and enforce posix ACLs. It might be easier to start off with just plain file permissions since the enforcement rules are much more simple. If you want to try something with no code changes, you can run the gateway as a user. Then you would get filesystem level permission enforcement without needing setuid(). In this scenario you would probably have a gateway for each user running on different ports instead of a single gateway handling multiuser. The downside here would be managing all the gateways running for all users and making sure they get run as that user. The gateway accounts (through all IAM services) have the ability to store uid/gid/projectid per user. And there is a near future feature upcoming to optionally chown/chgrp the files as they are created to the user account uid/gid. This doesnt really help with permissions enforcement, but might be a step in the right direction by at least getting the correct ownership in place. We are certainly open to ideas on how we could make file permissions work. So far it has been complicated enough that it hasnt made it past the rough ideas being sketched out phases. I think a good design/implementation of posix permission enforcement would certainly be in scope for this project. |
Beta Was this translation helpful? Give feedback.
-
|
@kyleaupton I found this old repo that hasn't been updated in a while, but appears to have the pieces needed to retrieve and parse posix ACLs: https://github.com/joshlf/go-acl. The missing piece here would be functionality to test access enforcement of the parsed ACL based on the IAM UID/GID. In case you are still interested in pursuing this. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the the library suggestion! I've looked into implementing this functionality a good bit, and I think this is absolutely a path that we could take. Like you already mentioned, from an abstract viewpoint we really just need to 1) parse POSIX permissions + extended ACLs for a given path, and 2) run some sort of access check algorithm against that. For step 2, there is a defined access check algorithm outlined in the POSIX 1003.1e family of "standards". However, the term "standards" is used loosely here. See https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/gruenbacher/gruenbacher_html/main.html That being said, I still believe that the access check algorithm defined in the above link should mostly work across unix-like systems. I started work on this approach. Specifically I shell out to What I'm currently getting stuck on are my lack of golang skills and making the required architecture changes to the project. From my understanding any bit of permission data gathered from a backend is shimmed into a AWS style ACL, or comes directly as the AWS style, and that is what is used to determine access. And only on the "bucket" level as well. I would need to essentially introduce a new type of permission, or continue to do the shim. Also I would need to implement a new function on the backend interface that can verify permissions for a specific file. And that means changing existing middleware layers, other backends, and much more. There's also the new concept of default permissions, those determine what permission get applied to newly created files. Not to mention changes to the IAM service so accounts can have a list of groups. All of that to say, I believe there would be substantial changes needed. My company is on a tight deadline to deliver a first iteration of S3 functionality in our product. Because of that, actual implementation from my side will likely not be anytime soon. We're just going to have S3 access be disjoint from our existing file access/permissions for the time being. We eventually would like to work towards contributing these changes, however the time frame is unknown. Thanks for your time! |
Beta Was this translation helpful? Give feedback.
-
|
Is the goal to try to allow clients to set the posix ACLs through the set bucket/object ACL interface? Or are posix ACLs set ahead of time in the filesystem, and you just want the gateway read access to honor these ACLs? |
Beta Was this translation helpful? Give feedback.
-
|
Yep they're set ahead of time, just need to honor them. In my use case I have no need to create/delete buckets or ACLs through the gateway. |
Beta Was this translation helpful? Give feedback.
-
|
This is completely untested, but curious if this is sort of the direction you were thinking: |
Beta Was this translation helpful? Give feedback.
@kyleaupton Thanks for checking out the project!
The difficulty with the posix permissions is that ideally we would want the filesystem to enforce the permissions/ACLs. And in order to do that we would need to set the effective userid/groupid to that user for any filesystem access. Normally this would be done with seteuid()/setegid() to set the effective uid/gid for that thread. But with Go, the goroutines are not 1:1 with OS threads, but instead lightweight routines scheduled across available threads. There are ways to lock a single goroutine to an os thread, but then this starts to have implications on the scalability of the service if each incoming request needs to fork a new thread. S…