Support for unstructured grid with a vertical dimension

[dennissergeev]

✨ Feature Request

Firstly, thanks very much for creating this library, I'm excited to use it in my research!

Secondly, would it be possible to generalise geovista.Transform.from_unstructured() for it to take in arrays with more than 2 dimensions, i.e. vertical levels (and possibly time)?

Motivation

I am trying to visualise LFRic output (happy to eventually contribute to the gallery by the way), and while the from_unstructured() method works great for 2D arrays, I would like to plot something w.r.t. model height. Currently there's no obvious way to do this as that function accepts only longitudes and latitudes, throwing an error if I pass a full 3D array as data.

This is probably related to the "isosurfaces support" to-do item.

[bjlittle]

Hey @dennissergeev,

Stoked to hear that you're excited about geovista... I am too!

Prior to geovista, I was playing with animating model time-series and also providing a slider so the user can manually control the render of spatial data with another single dimension (such as time). To be honest, that worked reasonably well. Here's a proof-of-concept I'm currently working on for geovista... it's showcasing an unstructured ocean model with variable resolution quad-faces:

Apologies for the poor quality GIF, it doesn't really do the colormap justice... but I think you get the idea of the direction I'm headed in with that kind of capability.

So time-series support is definitely in the pipeline, perhaps along with ipywidget integration for jupyter... but the overriding goal here is for geovista to easily integrate into the existing scientific python ecosystem e.g., scripting, notebooks, https://panel.holoviz.org/ et al... and this should be a level playing field for the community, with regards to whether you're coming to geovista from iris, xarray, cupy, numpy or just where ever i.e., geovista is specifically agnostic of iris.

There are some fundamentals that I'm keen to bank first for geovista such as CRS, planar projection and texture mapping support, but later isosurfaces, isolines, streamplots, winds visualization, mesh warping for ocean bathymetry and surface altitudes, web services tiling etc are all in play 👍

With regards to the vertical I've not quite thought that far ahead to be honest (geovista is just born, so I'm focusing on a solid core first and building outwards). Nevertheless, what kinda viz are you particularly interested in?

I was mulling over geodesic ribbon trajectory slices through unstructured meshes to show vertical levels in the planar, but I've not ventured too far into this space other than daydreaming about it... however I am motivated by concrete use cases though 😉

So if you can be more specific and give me some examples that would really help 👍

[dennissergeev]

Thanks for the speedy response @bjlittle!

Here's a proof-of-concept I'm currently working on for geovista...

Looks awesome! I would love to use this once you're happy with the prototype.

the overriding goal here is for geovista to easily integrate into the existing scientific python ecosystem

I agree that's the way to go. I'm mostly using iris because I mostly use the UM in my research (and now starting to use LFRic).

There are some fundamentals that I'm keen to bank first for geovista

Fair enough. I can try to help with some "good first-time issues"...

Nevertheless, what kinda viz are you particularly interested in?

I can of course just plot some slices of the vertical coordinate, but eventually I would like to create plots with the vertical dimension, i.e. planet's radius, such as vertical cross-sections, isosurfaces, wind vectors. As an example, here's what I made with PyVista a while ago: https://dennissergeev.github.io/exoconvection-apj-2020/

If this kind of visualisation can be combined with the time animation as in your gif above, that would be even more awesome.

[bjlittle]

@dennissergeev Awesome, I've come across that visualization before, super cool, congrats!

Okay, I can see what you're aiming for, and that all makes sense, thanks. I'm guessing your Trappist-1e 3D viz could also be projected e.g., say the sphere to Plate Carree, but still have the 3D vertical convection layers on top, thus taking advantage of a 2D planar but viewed in 3D, if you get my meaning 🤔

I think that your original 3D viz is perhaps currently possible in ParaView. I know that @tinyendian (Wolfgang Hayek @ NIWA) might be able to comment or point you to what's possible there too - if you're interested?

[dennissergeev]

Thanks @bjlittle!

I'm guessing your Trappist-1e 3D viz could also be projected e.g., say the sphere to Plate Carree,

Yes, certainly. Visualising it both in a planar projection or as a 3D sphere would be great! I guess that's why you're envisaging CRS support - to have an API similar to cartopy?

I think that you're original 3D viz is perhaps currently possible in ParaView.

That would be useful to know, so any tips from Wolfgang would be appreciated.
At the same time, this is not urgent and I'm happy to wait/contribute for geovista to mature enough to support this.

P.S. Great logo by the way!

[tinyendian]

Hi @dennissergeev and @bjlittle, visualisations such as Trappist-1e (which looks great btw!!) can definitely be done with ParaView, including time-dependence of the model data, spherical projection, moving cameras, interactive web publication, ...

I haven't used PyVista before, but, looking at the examples, it seems to use a similar philosophy as ParaView, providing a simplifying layer on top of VTK. ParaView comes with a few perks, such as a powerful GUI, and a client-server mode, which is useful for very large models that require an HPC backend to handle. Its Python scripting capabilities are comprehensive and integrate nicely with the GUI, but the price to pay is that the scripts look somewhat obscure compared to PyVista, the latter seems a bit more Pythonic.

Very happy to help, if you'd like to give ParaView a try (it can read CF-netCDF and LFRic output), or if you have any questions.

Btw, I worked in the exoplanet group in Exeter 10 years ago as a postdoc, great to see the work that you guys are doing there! And say hi to Nathan for me please 🙂.

[dennissergeev]

hi @tinyendian,

Thanks! Yes, the exoplanet group is growing stronger, so expect more cool 3D visualisations of exoplanet simulations 😄!

I actually used ParaView a little bit in the past, though only as a GUI. Now I prefer using PyVista because of its pythonic API and integration with Jupyter, but in the meantime if I wanted to give ParaView another go and visualise LFRic output, where should I start?

[tinyendian]

Hi @dennissergeev, the easiest way to start with ParaView is a conda installation of the LFRic reader plugin, which will install the latest ParaView GUI as a dependency,

conda install -c conda-forge lfric_reader

If you have a ParaView build on your system already (along with the required build tools and dependencies), you can also easily build the reader yourself, see the instructions on https://github.com/niwa/lfric_reader.

Note that Jupyter integration was recently added to ParaView, too, https://blog.kitware.com/paraview-jupyter-notebook/, but I haven't tried this out yet.

With regards to Python scripting, I agree that PyVista coding looks a lot nicer and straightforward. ParaView really has its strengths when it comes to interactive visualisation (have a look at, e.g., linked cameras for comparing two models), and handling very large datasets (hundreds of millions of cells) efficiently. Maybe there will be ways to integrate Iris/PyVista and ParaView a bit in the future, e.g., via automatically generated ParaView scripts (ParaView can be fully controlled by Python scripting).

Looking forward to seeing more exoplanet simulations 😍!

[dennissergeev]

Thanks very much @tinyendian! I'll give it a go.

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 180 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[github-actions]

This stale issue has been automatically closed due to a lack of community activity.

If you still care about this issue, then please either:

• Re-open this issue, if you have sufficient permissions, or
• Add a comment pinging @bjlittle who will re-open on your behalf.

[bjlittle]

Hey @dennissergeev,

Just wanted to let you know that I want to get my ducks-in-a-row to schedule time for supporting this issue. The first step in this journey will be to extend the capability of the geovista.bridge to ingest mesh data with a vertical dimension.

I'm guessing that you've got a tonne of archived data that you'd be keen for geovista to visualise out-the-box... is it possible for you to share it with me somehow?

Cheers 😄

[dennissergeev]

Hey @bjlittle!

This is exciting to hear! Happy to help test / develop things, just ping me.
And I'm happy to supply some sample data, of course. Here's an output file from one of my recent simulations:
https://we.tl/t-gt1W6B4mkl (about 100 Mb)

The mesh is C48 and it has 38 vertical levels. Currently, the output netCDF file does not store the level height values (only level numbers), but in this simulation I used the um_L38_29t_9s_40km level set with the following eta spacing:

     (/                                                                                                                                 &   
        0.0000000_r_def,  0.0005095_r_def,  0.0020380_r_def,  0.0045854_r_def, &
        0.0081519_r_def,  0.0127373_r_def,  0.0183417_r_def,  0.0249651_r_def, &
        0.0326074_r_def,  0.0412688_r_def,  0.0509491_r_def,  0.0616485_r_def, &
        0.0733668_r_def,  0.0861040_r_def,  0.0998603_r_def,  0.1146356_r_def, &
        0.1304298_r_def,  0.1472430_r_def,  0.1650752_r_def,  0.1839264_r_def, &
        0.2037966_r_def,  0.2246857_r_def,  0.2465938_r_def,  0.2695209_r_def, &
        0.2934670_r_def,  0.3184321_r_def,  0.3444162_r_def,  0.3714396_r_def, &
        0.3998142_r_def,  0.4298913_r_def,  0.4620737_r_def,  0.4968308_r_def, &
        0.5347160_r_def,  0.5763897_r_def,  0.6230643_r_def,  0.6772068_r_def, &
        0.7443435_r_def,  0.8383348_r_def,  1.0000000_r_def /)

Let me know if you need more info!

[bjlittle]

@dennissergeev Starting to play in this space, so I just thought that I'd give you a minor update...

I've been working on a use case with an oceanographer at the Met Office, where they have 50 depth levels (bathymetry) of ORCA ocean model data (on tri-polar curvilinear grid), which has been pre-processed through a canny edge filter to detect potential temperature gradients. Initially, they want to visualise the data as a point cloud, which is an ideal first use case.

I was able to easily extend geovista to create the geo-referenced point cloud, which kinda looks like this:

• I've rendered the points with their levels values
• Used the cmocean deep colormap, which is super lush
• Added a texture map base layer with opacity (to confirm geo-refrencing)
• Added 10m Natural Earth coastlines (to confirm geo-refrencing)

votemper.mp4

Next steps are to:

• make it easy to reproject the point cloud (say to Plate Carree), which might make interactive interrogation a wee bit more intuitive
• extend to support non-polydata, and integrate into the geovista.bridge API

In hindsight, I should have rendered this example with culling=back to make the things slightly less confusing, but this is just a PoC at the moment.

Baby steps.

[bjlittle]

votemper-coastlines.mp4

[bjlittle]

votemper-coastlines-widget.mp4

[dennissergeev]

Sorry for my slow response, I was on holiday :)

This looks amazing, many thanks for sharing! Is this functionality available in the latest release? (No rush, just wondering.)

And since you've shown that point cloud rendering is possible, I guess extending this to vectors/arrows to show the flow velocity in 3D should be quite straightforward, right?

[allcontributors]

@bjlittle

I've put up a pull request to add @dennissergeev! 🎉

[bjlittle]

@all-contributors please add @mgrover1 for ideas and promotion

[allcontributors]

@bjlittle

I've put up a pull request to add @mgrover1! 🎉

[edmundhenley-mo]

Quick question ... when we get to the point where we have a proof-of-concept for the volume rendering in a feature branch (I'm going to figure out when I can schedule this work), would you guys be happy to kick the tyres to test/play and feedback?

@bjlittle - just found this one - subscribed immediately!
TLDR: once you've got your proof-of-concept done, shout iff you want to extend onwards-and-upwards for some space weather use-cases! Happy to help with some data wrangling & tyre kicking if so - no worries if you've got hands full with use-cases from lower down!

Click for context, and potential good 1st target = WACCM-X

Context

This sort of volume / isosurface rendering could be very interesting for space weather use-cases in Earth's ionosphere, thermosphere and below (where we have more-easily supportable geographic lat/lon coords, as well as more bonkers ones too!). These regions can be highly dynamic, with multi-scale spatiotemporal effects which ~require seeing ~4D representations to understand full picture - risk losing insights if you collapse data straight to 2D or lower.
Geovista's interactivity - ability to move round from dayside to nightside, or look down on the poles - could be super-useful for enabling workflows which start with exploration of the full-fat datasets.

• i.e. basically exactly the same points @dennissergeev and Cristina made in lightning talks on Geovista!

There's some very nice videos here illustrating this sort of thing - look at all of those ripples as the solar storm hits on 17 Mar 2015 - the 4D synoptic view here invaluable - a lower-dimensional representation couldn't cut it!

• Thermospheric temperature
• Ionospheric electron number density - volumetric equivalent of the total electron content in iris examples
• More details on what's being plotted in Figure 1 of Palmroth+ 2021

From dim recollection I believe @eelcodoornbos generated these with Blender (?). In any case, using output from NCAR's WACCM-X model.

WACCM-X: a good first target?

Iff this is of interest, WACCM-X might be a good target for dipping geovista's toes in the space weather water (if you've not already?):

• it's a state-of-the-art community model, coupling into lower atmosphere (Liu+ 2017)
• well-behaved: nice netCDF output, on sane geographic grid, vertical on pressure levels - and data volumes not too big by terrestrial standards, so good for exploration (characteristics in slide 5 of WACCM-X tutorial
• so good place to start!
• Note NCAR's TIE-GCM another good contender: similar characteristics, and very widely used

Later can see if Geovista's ugrid support might be an easy way around trickier things like geomagnetic coordinates in other models, ...

There's WACCM-X output available from the NASA Community Coordinated Modeling Center (CCMC). E.g. for this "St Patrick's day storm", there's run "WACCMX-Weimer-01_2015-03-TP-01_102523_IT_1", and from there:

• Data download
• Vis API

Happy to put some effort into giving this a whirl if of interest!

[bjlittle]

@all-contributors please add @edmundhenley-mo for ideas and userTesting

[allcontributors]

@bjlittle

I've put up a pull request to add @edmundhenley-mo! 🎉

[bjlittle]

@edmundhenley-mo Great! 🚀

Good news! I've finally managed to secure a deployment into a science team to work on this ... it's the only way I'm going to get the time to do it!

I'll come knocking at your door afterwards for sure! 👍

[bjlittle]

Finally got some cycles to make some progress ... 😉

[dennissergeev]

Amazing 🤩

[mgrover1]

This is INCREDIBLE! Great work here @bjlittle 👍

[bjlittle]

This is all proof-of-concept, but it's looking quite encouraging.

I'm currently working with some amazing Atmospheric Dispersion scientists with some gorgeous datasets, and I'll be looking to consolidate and bank this capability within geovista soon 👍

I'll update the geovista YouTube channel with some animations to showcase this further, but I think I'm at the stage where things are coming together ... and finally it would be pretty cool to blow the dust off your datasets and play with them too 😄

[dennissergeev]

Great animations, thanks for uploading them!

Looking forward to using this for my exoplanet simulations!

[bjlittle]

Same dataset but with isosurfaces and an opacity linear transfer function for plume-like rendering ...

The widgets won't be part of geovista, they just make it convenient to quickly alter what and how geometries are being rendered 👍

Also see YouTube animation ...

[dennissergeev]

Wow this is amazing!!! Isosurfaces are the main thing I'm waiting for in geovista, so thank you so much for working on this!

[mo-benjaminevans]

This looks great @bjlittle. It will help with expert judgment on plume dispersion, engagament and model validation etc. In regards this issue and #16 it would be great to be able to overlay 3D vectors on top of the 3D plume. Is the plan that both issues/PRs should work together?

[bjlittle]

@mo-benjaminevans Yes, indeed.

Both #15 and #16 completely complement each other. Being able to visualize how winds influence plume behaviour is a critical part of your science. So just to confirm, you will be able to overlay 3D vectors on top of a 3D plume 👍

You'll be glad to hear that the plan is for both features to be available in the next release of geovista 0.6.0.

[pp-mo]

This is quite exciting.
I purposely omitted 3d from my "UGRID conformance rules" proposal (now included in published conventions here) because I couldn't see how to make the existing statements hang together.
It has even been proposed that we might pause the whole 3D thing for UGRID since it had so far seen little interest,
But perhaps practical efforts like this can show the way to making it a real thing.

[jmb64]

Hi,
I must start by saluting the work of @bjlittle, @banesullivan, and all the developers
and contributors to the pyvista/geovista endeavour !
I've been using this software for a little while now and read this issue 15 (and others)
with great interest. Posts by @jbusecke, @denissergeev really resonate with my own experience.
I have been involved in many different projects in my career (land surface processes, atmospheric
data assimilation based on adjoint methods, development of our global coupled atmosphere-ocean
operational NWP forecast model here at ECCC, ... All along the way I always considered the need for
validation, diagnostics and visualization tools to pivotal role in the success of the scientific
journey. I was amazed a couple of years ago with the advent of panel/holoviz and the possibility
to build small interactive visualization application instead of relying on the generation of zillions
of static images. Now I feel that 3D isosurfaces, real-time 3D direct volume rendering are
tools that are within reach of scientific like me with no solid expertise in the field. Using
pyvista gave me the opportunity to discover VTK (even though it's been there for over 30 years !)
and ParaView. I wish this happened earlier in my career but ... From my neophyte's perspective it
seems that there is a convergence of factors happening (ex.: hardware:NVIDIA's RTX cores,
software:ANARI standard API for 3D rendering, rise of AI and gaming engines, a multiplication of
rendering backends and a consciousness that this technology has been hidden most of the time
in software frameworks of (academic) experts (with not bad intention but a lack of time and resources).
A last thing I would like to add, in searching for for volume rendering application one particular
feature (or lack of) is the georeferencing information (coastlines, lat, lon, height, depth, pressure,...).
I have documented a few initiatives over the years (ParaView) but none seem to have stuck. In geophysics
( I deal with ocean and atmospheric datasets) this missing piece is really annoying. I was really please
to discover that geovista had this in mind (lat, lon graticules).
Well, I guess this is how I wanted to add my support to this work with both hands !

Oh, I almost forgot. I was wondering if @jbusecke the python code for the Pacific OMZ isosurfaces
(https://vimeo.com/783450053) was public somewhere ? :-)

Cheers,
Jean-Marc

[banesullivan]

Oh, I almost forgot. I was wondering if @jbusecke the python code for the Pacific OMZ isosurfaces
(https://vimeo.com/783450053) was public somewhere ? :-)

Reposting this last bit so that it pings Julius: is this code open by chance? 😄

[jmb64]

Thanks @banesullivan.
I figured the first ping would do it but then ... maybe not ! :-)

[banesullivan]

Oh it totally did 🤦🏻‍♂️ I didn't see the first @ tag at the top of the message. I'm just extra eager to see if the code is available 😄