How to integrate custom exploration interfaces into a VEDA app

[j08lue]

Description

Evidence is growing that VEDA instances need to provide data exploration interfaces that are tailored to a specific dataset, data type, or user workflow, i.e. not matching the generic multi-layer raster timeseries exploration of the "Exploration & Analysis" interface.

The U.S. GHG Center will soon have four of such custom interfaces: https://github.com/US-GHG-Center/custom-interfaces/

• https://earth.gov/ghgcenter/custom-interfaces/emit-ch4plume-v1
• https://earth.gov/ghgcenter/custom-interfaces/noaa-gggrn-ghg-concentrations

And we have before heard about the upcoming need for 3D data exploration or custom analytics functions such as GHG flux totals calculation.

We are working on a simplified way to see animated data on a 3D globe for EIC and are seeing new possibilities of performant GPU-enabled geospatial data exploration with Lonboard.

In user stories

1. As a user without a lot of domain expertise, I want to be guided to explore specific features of the data, so I can visualize what the subject matter expert wants to show me.
2. As a scientist exploring a specific phenomenon, I want to have the best specialized tools available to me, such that I can work efficiently.

Prior discussions

• https://github.com/NASA-IMPACT/veda-architecture/issues/290 (see summary)
• [Design] Make embedded visualization interfaces feel more official US-GHG-Center/veda-config-ghg#206

How to build

Configurable generic interface vs building from scratch

It is comparatively simple to build these interfaces from scratch, compared to the complexity of building a fully configurable web-GIS.

Unless heavily limited in options for a specific use case, generic GIS interfaces are overwhelming to less technical users.

Python-based e.g. JupyterLite, Streamlit, etc.

We have experimented with JupyterLite, but were unable to get the performance and look and feel that would make them nicely blend into a VEDA dashboard.

• https://github.com/NASA-IMPACT/veda-architecture/issues/308

details on JupyterLite

The main incentives for JupyterLite were:

1. As a user of the VEDA Dashboard, I want to understand which datasets to choose and how to analyze them, so I can answer a contextual question about the state of the environment, prediction, or prescription.
2. As a scientist, I want to guide users with limited technical and scientific understanding, to take steps in the data analysis, so they do not need me in the loop to tweak parameters of the analysis for their use case.
3. As a developer, I need a lot of knowledge transfer from users and scientists to build a web environment that retraces the scientific analysis and is user-friendly.

Notebooks compiled in the browser seemed like a nice shortcut - scientists could funnel their domain expertise into a notebook that they could easily develop and change when needed and we had a short time to publication for our Dashboard users.

However, performance as well as UX were challenging to get right: Some notebooks took minutes to load, others still 10x longer than the same thing coded up in JS, like here:

• Pyodide: https://nasa-impact.github.io/veda-interactive-emission-plumes/voici/render/plumes.html
• JS: https://nasa-impact.github.io/interactive-emission-plumes/

So we abandoned the notebook/pyodide/wasm concept.

Integration

Currently, our approach is to build these interfaces externally and embed them into a page in a VEDA Dashboard using iframe/embed (e.g. GHG Center methane plume exploration). This has obviously a lot of caveats over building these interfaces as part of the main app.

1. bad, because: risk of loosing of credibility due to half-baked and unofficial appearance ([Design] Make embedded visualization interfaces feel more official US-GHG-Center/veda-config-ghg#206)
2. bad, because: hard to get look and feel to match main app
3. bad, because: limited viewport
4. good, because: simple to do
5. good, because: custom app can add special dependencies without affecting the main app

Acceptance criteria

• Indicated medium and longer-term ways to better fulfil the need for custom exploration interfaces integrated in VEDA apps

[aboydnw]

@brianmfreitag @j08lue The approach we discussed last week was to expose certain “building blocks” for instances to use, such as the map block, date picker, etc. that are not quite “features” of VEDA, but would allow instances to create custom features that have a similar look and feel to VEDA features. We did something like this for the urban dashboard. This is our current technical approach.

Then, we would monitor custom features across apps (like we are doing with the urban dashboard) and assess whether we should bring back into core VEDA to expose to all the other instances. This is our roadmap approach.

Even though we might actually do the custom feature (like the ESRI prototype) we are making decisions on a case-by-case basis as to whether it is a custom feature or a core VEDA feature, juggling things like time pressure, common requirements, and other roadmap priorities. We have been refining a list of features on this ticket and plan to post that in a more public location in the future.

Is there something more here that we need to solve?

[j08lue]

Awesome, great to pull these discussions together.

To me, there are two more points to sort out:

1. Design approach - can we provide suggestions / a sample implementation on how these custom apps (built with VEDA UI building blocks or not) could integrate into an instance?
2. Project management approach - what support does our team provide? Can we do some hands-on custom UI development ourselves / dogfooding to try out whether our single-use-to-building-block design and decision process works?

Do you have positions on these two? They are basically about how much of these custom interfaces is "our business".