[catalystproject-af, bhki] Enable object storage

[jnywong]

Context

The bhki community would benefit from the recommended data transfer workflow proposed in #4213 to get started with analysing ~GBs of .fastq files for RNA sequencing.

Proposal

Set up a persistent storage bucket so that they can upload genomics data from a local server.

Updates and actions

No response

[sgibson91]

I'm looking at this next.

Since we haven't got the full "LEAP" style setup in AWS, we will also need to figure out the "magic incantation" users will need to get their credentials to use to push to the bucket locally.

[sgibson91]

There is now a persistent bucket set up for the BHKI community

[sgibson91]

"Magic incantation" is here: https://github.com/scottyhq/jupyter-cloud-scoped-creds/blob/f9e30953f20bc3fb6971077f847c7021b88c728a/jupyter_cloud_scoped_creds/__init__.py#L16

I will find an appropriate hub to test this on and create a minimal example of pushing a file from local to the hub.

@jnywong For this to work, the AWS CLI needs to be installed in the image and it currently isn't as this community are using the Jupyter SciPy image. The Pangeo image ships with the AWS CLI installed, so we should communicate that that is the better image choice for this particular workflow.

[sgibson91]

I recorded a video and shared directly with @jnywong on how to use the magic incantation to upload data from a local machine to an AWS S3 bucket and then seen the data from the hub. Hopefully that is enough to create some documentation for the community, but I am happy to answer further questions.

[jnywong]

Amazing 😍 I am looking forward to sharing a Catalyst demo with you soon because of your work 👏