Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[to#127]Fix ossfile upload_chunk location bug #129

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

idling11
Copy link
Collaborator

@idling11 idling11 commented Jun 1, 2023

This PR is a bug fix for ossfile location bug in upload_chunk function.

@idling11 idling11 requested a review from karajan1001 June 1, 2023 01:30
@idling11 idling11 self-assigned this Jun 1, 2023
@idling11 idling11 added the bug Something isn't working label Jun 1, 2023
@idling11
Copy link
Collaborator Author

idling11 commented Jun 1, 2023

UT faild. I will debug it in local env

@idling11
Copy link
Collaborator Author

idling11 commented Jun 1, 2023

@karajan1001 I try to mock a env to run your test_file.py UT by using my own oss. If I have no get_bucket_info auth to this bucket, result is PositionNotEqualToLength error as well. Pls check your oss permission in env. Aprreciated!

@karajan1001
Copy link
Collaborator

karajan1001 commented Jun 4, 2023

@karajan1001 I try to mock a env to run your test_file.py UT by using my own oss. If I have no get_bucket_info auth to this bucket, result is PositionNotEqualToLength error as well. Pls check your oss permission in env. Aprreciated!

Hello, First thank you to do the research. I also looked into the code and found that the problem comes from the difference between self.loc and self.offset . The error all comes from when we want to append something to an existing object. In this case self.loc will point the end of the file while self.offset always point to 0. So this is where the PositionNotEqualToLength comes from.
I'm also confused with the usage of self.offset and had already created a ticket in fsspec/filesystem_spec#1285.

@zigmengineer
Copy link

@idling11 @karajan1001 Can this PR be merged please?

@karajan1001
Copy link
Collaborator

@dependabot rebase

@Wangsongming
Copy link

I also have this problem, it seems that large files are uploaded using sharding when transferring OSS, but it is directly ended without waiting for the return, resulting in the situation of large files that have been unable to pass up, I hope to solve it as soon as possible

Collecting |2.00 [00:00, 250entry/s]
Pushing D:\python\lib\site-packages\ossfs\async_oss.py:388: RuntimeWarning: coroutine 'resumable_upload' was never awaitedile/s]
await self._call_oss(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Pushing
2 files pushed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants