Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Object_store: Create an upload method that handles concurrency #6832

Open
midnattsol opened this issue Dec 3, 2024 · 1 comment
Open

Object_store: Create an upload method that handles concurrency #6832

midnattsol opened this issue Dec 3, 2024 · 1 comment
Assignees
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@midnattsol
Copy link

This issue is related with #5277. Basically it is created to split the task of creating a download and upload methods in two different tasks with two different PR for easy review.

The original issue was

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As a user, I would like to utilize the object_store to put and retrieve files. Currently, I can employ the put and get methods for this purpose. However, I am responsible for handling any issues that arise. For instance, when uploading a file to the store, I must verify its size to determine whether multipart upload is necessary. Similarly, when retrieving a large file, I cannot download it in its entirety before writing it to the local file system, as this would cause excessive memory usage. Instead, I may need to use get_range and write specific portions of the file locally. Moreover, downloading different ranges could potentially be executed in parallel.

Regards

@midnattsol midnattsol added the enhancement Any new improvement worthy of a entry in the changelog label Dec 3, 2024
@midnattsol
Copy link
Author

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

No branches or pull requests

1 participant