Skip to content

Latest commit

 

History

History
64 lines (40 loc) · 2 KB

hosting_source_datasets_on_ewc.md

File metadata and controls

64 lines (40 loc) · 2 KB
Raw

How to host a zarr-based source data on EWC (European Weather Cloud)

First you will neet to get an account. You will have to choose either a EUMETSAT or ECMWF tenancy (drawbacks/benefits between the two?). You will receive S3 credentials (I, Leif, did anyway).

1. Create bucket

Using the command line

lcd@wrk1:/dmidata/projects/cloudphysics/danra/data$ s3cmd mb s3://danra
Bucket 's3://danra/' created
lcd@wrk1:/dmidata/projects/cloudphysics/danra/data$ s3cmd info s3://danra
s3://danra/ (bucket):
   Location:  default
   Payer:     BucketOwner
   Ownership: none
   Versioning:none
   Expiration rule: none
   Block Public Access: none
   Policy:    none
   CORS:      none
   ACL:       *anon*: READ
   ACL:       cci1-ewcloud-ecmwf-ml-pilot: FULL_CONTROL
   URL:       http://object-store.os-api.cci1.ecmwf.int/danra/

Using the web interface

  1. Log in to the Morpheus webportal using the tenancy you have been given (ECMWF or EUMETSAT)

  2. Create a bucket in the S3 object store of the tenancy you have been given.

2. Upload files

To both of the commands below you can add the flag --acl-public to make the files publically readable.

Single file:

(base) lcd@wrk1:/dmidata/projects/cloudphysics/danra/data$ s3cmd put CHANGELOG.md s3://danra/
upload: 'CHANGELOG.md' -> 's3://danra/CHANGELOG.md'  [1 of 1]
 1060 of 1060   100% in    0s    12.35 KB/s  done

Directory:

(base) lcd@wrk1:/dmidata/projects/cloudphysics/danra/data$ s3cmd sync v0.2.2/ s3://danra/v0.2.2/ --debug

Note: trailing slashes are required, s3cmd does at lot of checks before uploading which is why I include --debug to see what is happening