Support for Google Cloud Storage URLs (like WeatherBench2 data)

[yoninachmany]

What I am changing

• Adding read support for GCS URLs like gs://weatherbench2/datasets/era5/1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr (see https://weatherbench2.readthedocs.io/en/latest/data-guide.html)

How I did it

• Using gcsfs to open URLs like gs://... in reader.py
• Pulling in changes from remove morecantile deprecated method and set upper limit for starlette #49

How you can test it

• Running map.html with this Zarr store URL (gs://weatherbench2/datasets/era5/1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr) and choosing a variable with a time dimension from https://weatherbench2.readthedocs.io/en/latest/data-guide.html
• For example, visualizing 2m_temperature data with http://127.0.0.1:8000/map?url=gs%3A%2F%2Fweatherbench2%2Fdatasets%2Fera5%2F1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr&variable=2m_temperature&consolidated=true&rescale=208%2C310&colormap_name=coolwarm should produce

Related Issues

• Integrated fixes from remove morecantile deprecated method and set upper limit for starlette #49 rather than branching off that patch branch
  • I'm not sure if this project is still being maintained, these were all the changes I needed to visualize WeatherBench2 data
  • I'm working on an interactive way to explore ERA5 data like https://maps.demo.carbonplan.org/ and https://prototype-maps.demo.carbonplan.org/

[vincentsarago]

👋 @yoninachmany

Thanks for you PR, I'm a bit worry about adding another dependency. Could we make gcsfs an optional dependency?

[yoninachmany]

Yes!

P.S. I realized I can use the public HTTPS URL, which is already supported: https://storage.googleapis.com/weatherbench2/datasets/era5/1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr/

[thijsvanwinden]

@vincentsarago I kind of need this for a project. Anything I can do to get this merged?

[vincentsarago]

@thijsvanwinden the branch is outated so we would need to update it and also make sure the test/lint passes

I also feel we should have tests added

Sadly I do not have bandwidth to take care of this right now

[abarciauskas-bgse]

@yoninachmany thank you for making these changes - we have decided this repo is going to be a dedicated repo for the VEDA titiler-xarray API, which doesn't require google storage access at this time. And as an FYI, the xarray module has been pulled into the core titiler-repo: https://github.com/developmentseed/titiler/tree/main/src/titiler/xarray/titiler/xarray. If you are still using titiler's xarray technology it would be cool to see a new API repo using GCS.