Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can gsutil_cp use the project id for requester pays buckets? #82

Closed
smgogarten opened this issue Aug 16, 2023 · 2 comments
Closed

can gsutil_cp use the project id for requester pays buckets? #82

smgogarten opened this issue Aug 16, 2023 · 2 comments
Assignees

Comments

@smgogarten
Copy link

gsutil_ls clearly uses the current value of gcloud_project() when operating on requester pays buckets. Can gsutil_cp do the same?

> gcloud_project()
[1] "terra-7b87e585"
> gsutil_requesterpays(bucket)
gs://fc-secure-5c77ebef-0e78-495e-b5a2-9a2cdbe074db 
                                               TRUE 
> gsutil_ls(bucket)
[1] "gs://fc-secure-5c77ebef-0e78-495e-b5a2-9a2cdbe074db/data_tables/"
[2] "gs://fc-secure-5c77ebef-0e78-495e-b5a2-9a2cdbe074db/notebooks/"  
[3] "gs://fc-secure-5c77ebef-0e78-495e-b5a2-9a2cdbe074db/submissions/"
> gsutil_cp("hello_world.txt", bucket)
Error: 'gsutil -m cp '/home/rstudio/gregor-data-tracking/hello_world.txt' 'gs://fc-secure-5c77ebef-0e78-495e-b5a2-9a2cdbe074db'' failed:
  Copying file:///home/rstudio/gregor-data-tracking/hello_world.txt [Content-Type=text/plain]...
BadRequestException: 400 Bucket is a requester pays bucket but no user project provided.
    CommandException: 1 file/object could not be transferred.
  exit status: 1

In a terminal:

rstudio@9688223a5f27:~/$ gsutil -u terra-7b87e585 cp hello_world.txt gs://fc-secure-5c77ebef-0e78-495e-b5a2-9a2cdbe074db/
Copying file://hello_world.txt [Content-Type=text/plain]...
/ [1 files][    6.0 B/    6.0 B]                                                
Operation completed over 1 objects/6.0 B.  
@LiNk-NY LiNk-NY self-assigned this Aug 17, 2023
@LiNk-NY
Copy link
Contributor

LiNk-NY commented Aug 17, 2023

Hi @smgogarten
It looks like in the code the requester pays flag -u project_id is only triggered when the source is a .gsutil_is_uri. It should also work when the destination is a gsutil uri. I will look into a fix.

cc: @mtmorgan

@LiNk-NY
Copy link
Contributor

LiNk-NY commented Aug 22, 2023

Hi Stephanie, @smgogarten
This should be fixed in devel version 1.13.7. Let us know if it works for you.
Thanks!
Best,
Marcel

@LiNk-NY LiNk-NY closed this as completed Aug 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants