Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add move file functionality between storage buckets #56

Open
ChameleonTartu opened this issue Mar 30, 2022 · 1 comment
Open

Add move file functionality between storage buckets #56

ChameleonTartu opened this issue Mar 30, 2022 · 1 comment
Assignees

Comments

@ChameleonTartu
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
I figured out that I need to use gsutil mv gs://bucketA/fileA gs://bucketB/fileB

Describe the solution you'd like
It includes several stages:

  1. Move inside the same bucket
  2. Move between different buckets
  3. Move from the local dir to the bucket
  4. Move from the bucket to the local dir

Possible solution is to have 4 different functions or have the same function for all 4 scenarios. Both solutions seem okay the 2nd one sounds be a bit more universal while the first may make code cleaner and implementation more straightforward.

@thomasborgen

@thomasborgen
Copy link
Owner

@ChameleonTartu Great that you can work on this.

I like 1 and 2. I think it could be implemented by creating a copy and a copy_to_bucket function that takes a keyword argument delete_source: bool. Then we could have a move and move_to_bucket helper functions which basically just calls the copy functions with delete_source=True

But to be sure the above is a good idea we should have a look into the gcp storage bucket library. If theres anything we can call directly there to move something we don't need to have the move function call some copy function, it can juts call gcp storage buckets move directly.

For 3 and 4 I think our current upload_file and download_file could handle those with a delete_source argument.

I also think we have an issue with our function naming download_file reads the data and returns bytes, it doesn't write to a file which the function name says. upload_file takes bytes as input and not a file which one would actually expect from the function name. I'll make another issue on this.

@ChameleonTartu ChameleonTartu self-assigned this May 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants