when using readbytes(path), get ResourceNotFound

[jordaniac89]

with open_fs('s3://' + self.bucket + '/') as s3fs:
                print(s3fs.exists(path)) ## true
                print(s3fs.isfile(path)) ## false
                return s3fs.readbytes(path) ## ResourceNotFound

where the path is the full path (minus the bucket) to the file, i.e. a/b/c.txt

I've made sure that my credentials in ~/.aws/credentials are correct, but I'm not sure if there's something else I'm doing wrong, or if readbytes is supported.

[jordaniac89]

after further debugging, I believe I found the issue in _s3fs.py:438

obj = self.s3.Object(self._bucket_name, _dir_key)

at this point, the _dir_key is the path minus the filename, i.e. a/b/ when the full path is a/b/c.txt, which s3 will not be able to find. But your ResourceNotFound throws the error with the full path

raise errors.ResourceNotFound(path)

which makes it look like the full path couldn't be found. When I replaced the line above with path
obj = self.s3.Object(self._bucket_name, path)

it returned my file correctly. Also, let me know if the way that I'm passing in the path may be what's causing this.

[willmcgugan]

Did you create the file with S3FS?

Please see limitations in the docs.

[jordaniac89]

I did see that, and that was my other thought, but all files have been created either through the S3 GUI or through AWS APIs/SDKs. Interestingly, I just tested a totally new directory by creating it with your interface

with open_fs('s3://' + self.bucket + '/') as s3fs:
                if not s3fs.exists(path):
                    s3fs.makedirs(path)

                full_filepath = path + '/' + 'test2.txt'
                s3fs.writebytes(full_filepath, cbytes)

and then reading it with the method readbytes(), and it worked. Most of our S3 contents were created through the Java S3 SDK, so I'm wondering if it doesn't create the directory object. Anyway, I'll look into that. Thanks for the info!

[zimeon]

I got caught out by the need for objects that simulate directories too. This issue was very helpful, I feel the limitations description isn't very clear.

I would like to be able to write code that reads from S3 without needing objects to simulate directories. It seems that for at least some operations (such as readbytes()) a solution is creating the S3FS filesystem object with strict=False. For example, where:

>>> with open_fs('s3://' + bucket + path) as s3fs:
...   print(len(s3fs.readtext('v1/inventory.json')))
... 
Traceback (most recent call last):
[snip]
  File "/Users/simeon/.python_venv/py38/lib/python3.8/site-packages/fs_s3fs/_s3fs.py", line 582, in openbin
    info = self.getinfo(path)
  File "/Users/simeon/.python_venv/py38/lib/python3.8/site-packages/fs_s3fs/_s3fs.py", line 441, in getinfo
    raise errors.ResourceNotFound(path)
fs.errors.ResourceNotFound: resource 'v1/inventory.json' not found

the operation works with strict=False which avoids the getinfo(...) call:

>>> with S3FS(bucket, path, strict=False) as s3fs:
...   print(len(s3fs.readtext('v1/inventory.json')))
... 
2026

would be nice to be able to do this with the open_fs(url) call.