-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Be less strict in non-strict mode #60
base: master
Are you sure you want to change the base?
Conversation
4986e8f
to
5581a9e
Compare
@willmcgugan could you take a look, please? |
Also, because we don't care about directory markers in non-strict mode I think about replacing |
@mrk-its I don't see the benefit in that, apart from avoiding the extra work of creating directories. Unless there is some major bottleneck there, I would prefer if directories created in non-strict mode where still there when opened in strict mode. |
@willmcgugan On my production S3 buckets I simply do not have these directory markers at all (Instead I see a lot of empty files with suffix What about changes in this PR? |
I also have this issue, I am working with a shared bucket where creating extra meta information objects in the bucket would be an unfortunate complication, and one certainly not to be followed by other folks accessing it via the CLI tools. |
@willmcgugan However, you don't always control the S3 bucket that you connect to. What is the down side to get one of this PR's in #60 or #51 ? |
@nivm I have a backlog of PRs and issues to look through, but fundamentally the problem is satisfying everyone's use case. It may not even be possible, given how S3 isn't quite a real filesystem. |
Motivation:
S3FS backend do not work well if bucket contains file structure without proper directory markers, even in non-strict mode. This patch skips few directory checks for
strict=False
mode (for default, strict mode old behaviour should be preserved) and makes my integration tests happy.It does the same as #51 but in few more places.
It should fix #55, #52 and #57 in non-strict mode