Pangeo + PyVista-Xarray

[banesullivan]

A place to discuss getting PyVista-Xarray to work with the Pangeo stack

cc @jbusecke

[bjlittle]

@banesullivan I think setting expectations on the limits of what can be rendered in a scene perhaps needs some discussion and/or clarification for the community.

I get the feeling that once people get past the initial excitement of being able to use pyvista-xarray they're (pangeo community especially) are going to want to render at scale. I'm saying this on the back of a lovely conversation with @TomNicholas at SciPy 2023.

My initial experience on geovista and playing with pyvista is that we're going to hit the limits quite easily, so I'd welcome any thoughts on a strategy and way forward here. I'd love to know if you've got anything up your sleeve? 😉

My uses case was attempting to render an 11k x 11k image from the new EUMETSAT MTG FCI satellite which is coming into service. I think currently anything that's close to 100M points is a problem.

e.g., here's a down-sampled MTG example:

[banesullivan]

Excellent request!! We need to document how to handle large datasets like this.

I think the key here is to use subsampling. In scenarios where you are zoomed out to show the full extent of the dataset as you show above, you probably don't need to load the full resolution of the data to have the same visual effect. However, once you zoom into specific areas, you may want/need the full resolution, thus running into this problem rather quickly.

@annehaley is working to create some examples using the larger datasets on Pangeo-Forge to showcase some ways to work around these limitations per #50.

Also, I placed a proof of concept example of here: https://github.com/pyvista/pyvista-xarray/blob/main/examples/level_of_detail.py

And there is a GeoVista version of it here: https://github.com/pyvista/pyvista-xarray/blob/main/examples/level_of_detail_geovista.py (however this example needs algorithm support from GeoVista per one of my previous draft PRs)

[bjlittle]

@banesullivan I was wondering whether it was worth exploring leveraging the datashader pipeline for this type of adaptive subsampling. I would have hoped that their API would cater for this type of thing as an abstract and generic plugin. Dunno? Just throwing out some ideas 🤔

Also, have you had any experience with the vtkAdaptiveSubdivisionFilter? Would this be appropriate here? Again, I've not played with this but was wondering whether there's a VTK filter that is suitable for this adaptive mesh rendering based on zoom level?

[TomNicholas]

Amazing to see this discussion going on!

If you want an aspirational pangeo-style dataset to try to visualize with geovista, the boss-level challenge would be the LLC4320 ocean model data. This dataset is:

• Very high spatial and temporal resolution (1/48th degree globally x hourly over a year)
• Global so requires map projections
• Large variables (8TB for each surface field...), but chunked into chunksizes similar to most pangeo-type Zarr data (75MB chunks)
• Available as Zarr in the cloud (in a requester-pays GCP bucket)
• On a cubed-sphere lat-lon-cap (llc) grid topology
• On a staggered grid
• Similarly-gridded datasets are available at lower resolution for testing (anything LLC or ECCO)
• Looks dope

What's currently uploaded is just the surface fields from that simulation, but with all the depths included the entire dataset would be ~5PB 😬

[jbusecke]

Thanks for this discussion here and generally big props for the work on the pyvista stack!
I actually tried to visualize the Dataset @TomNicholas mentioned here on the LEAP-Pangeo Hub, which has credentials already setup for this dataset (and thus avoids the requester pays issue).
Unfortunately we ran into some gnarly dependecy issues. I think the solution for that is to create a custom docker image which can then be exposed on the hub. I am already working on that and will report back once we make some progress.

I think this dataset is an ambitious but worthwile full stack challenge, spanning the areas of visualization, storage (think zarr chunking/sharding), and array compute (dask etc).

I get the feeling that once people get past the initial excitement of being able to use pyvista-xarray they're (pangeo community especially) are going to want to render at scale.

That certainly sounds very true 🤪, but maybe to lend some motivation to solving this issue in a general way (via subsampling? as @banesullivan suggested), I truly believe this could be complete game changer in the way many scientists will be able to investigate/absorb their data. I am convinced that this is an issue worth pursuing that will actually improve/accellerate science, in addition to the maybe more common usecase of using these tools for a polished presentation at the end of a research project.

I would be very happy to support this effort in any way I can. If you think it would be useful to have access to the LEAP cloud resources (or anything that would help), please let me know.