-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fs.open
returns ResourceNotFound
(404) even though fs.exists
finds the path
#70
Comments
I was able to reproduce, seems like if some other user wrote the file this occurs. If I comment out line 439 it works as expected. |
I ran in to this issue and investigated a bit. TL;DR: This behavior is intended by It turns out that it's related to how S3 handles the directories that represent the path to the file. S3 is not really a filesystem as it doesn't create the directory structure to the file, but only stores the full path as a key. So if you upload to s3 something like
As a last note, I would suggest to add |
The issue I mention only happens when a different user uploads the file whether it be via s3fs or aws cli even when all permitions are ok. If user who is trying to read it via s3fs is the one that did the upload, via either s3fs or aws cli, it works fine. So your explanation doesn't hold since it does in fact work, but only sometimes, depending on who did the upload with aws cli. I'm pretty sure this is not intended behavior on s3fs part as sometimes it works sometimes it doesn't depending on who uploaded the file even when the permitions are ok. |
@poudro - The explanation does hold. Let me explain it differently.... In a POSIX filesystem, the full file path always exists. So you can 'stat' every part of it. So with "/temp/file.ext" as an example; In Object storage (such as S3) it is possible for All that said - I have the same issue (and gripe) with S3FS, because not all S3 clients create the I wish there was a way to tell S3FS to not 'stat' dirs for specific mounts.... |
In that case, why does it work as expected sometimes and doesn't work other times? To be clear, I'm not talking about the semantics of POSIX vs Object storage here (which I am familiar with), I'm talking about the fact that when a program "sometimes works and sometimes doesn't" it heavily smells of a bug somewhere. My solution works for me, it's maybe not the best approach, if someone has a better way around it that's fine. So to summarize again:
To me this is not proper behavior and has nothing to do with POSIX vs Object storage |
@poudro Are you saying that files uploaded by 2 users, with the same key prefix, result in an error? e.g. /some/place/user1.txt vs /some/place/user2.txt Or are the key prefixes different? e.g. /user1/file.ext vs /user2/file.ext Are the users using the same software to upload/create the objects? Or different software? I'd like to try and recreate your issue, so some specifics would help me to do that. |
@geoffjukes here's the more detailed scenario:
In this scenario U1 can read file via s3fs, but U2 will raise a |
@poudro When U1 creates the object at Note that "seeing" a "directory" in the web interface, does not imply an object exists with that key. Edit: Re-reading - Just confirming - U1 can read the object with S3FS, U2 cannot, both using the exact same code, but different accounts? Then maybe the permission issue is on the object at |
The permissions in my experiment were actually set at the bucket level, seems unlikely to me then that it could be linked to permissions at the path level. |
It happens. I've experienced it myself many times. EDIT: https://docs.aws.amazon.com/AmazonS3/latest/userguide/managing-acls.html
|
Been having a strange issue with
fs.open
on various files.The code is basically
Sometimes
fs.exists
will say that a file exists in s3 but thefs.open
step will return aResourceNotFound
error.I tracked the error in
fs.open
to this section of the code https://github.com/PyFilesystem/s3fs/blob/master/fs_s3fs/_s3fs.py#L433-L441, but I'm a bit at a loss as to why an exception should be triggered here but not for thefs.exists
.Finally, if I delete the file via other means (awscli or in s3 browser), and recreate the file (by copying a local copy to s3 via awscli), it will then work without triggering an error.
Any help on what I'm doing wrong would be greatly appreciated.
Because of this last element it's a bit hard to reproduce, but if you need more details please let me know.
The text was updated successfully, but these errors were encountered: