Split and optimize the dataflow rather than the Vega spec

[domoritz]

Vega visualization specs compile to the Vega dataflow spec, which then gets instantiated by the Vega runtime. I believe that the lower-level dataflow makes it easier to do complete optimizations. For example, Vega specs have axes and legends as separate components but in the dataflow, they get compiled to marks. By using the dataflow, Vega Fusion could unlock more optimizations.

[jonmmease]

Yeah, this might be really helpful. One reason I didn't do this was that I wasn't sure what all of the low-level transforms do. e.g. sieve and repeat show up in the dataflow viewer a lot and I wasn't sure of what they do.

The other thing is that VegaFusion works in a batch mode on top of the column-oriented Arrow format (rather than streaming rows like the Vega Dataflow does), so I wasn't sure if all of these operators would apply in that context. What do you think?

[domoritz]

https://idl.cs.washington.edu/papers/reactive-vega-architecture/ explains the architecture of Vega a bit more and can help you understand the operators. I think the architecture can support columnar execution as well (or at least I think it doesn't support it less well than the Vega spec format).

[jonmmease]

Here are a couple more thoughts on what would make it easier for VegaFusion to switch to working at the DataFlow graph level. Some of these may already exist and I'm just not familiar with where to look.

• It would be helpful to have a declarative JSON specification of the dataflow graph. In particular, this representation wouldn't include custom JavaScript objects and would be round-trip serializable. Ideally, the representation would have a JSON schema like the Vega and Vega-Lite specifications do.
• It would be important to be able to construct a Vega View instance from this specification without a corresponding Vega specification.
• It would be helpful to have transform documentation like https://vega.github.io/vega/docs/transforms/ for the "internal" transforms listed in the README at https://github.com/vega/vega/tree/master/packages/vega-transforms. Hopefully this documentation could be at a level that doesn't dictate whether the implementation is streaming or batch oriented.

[jonmmease]

Very nice! Thanks for the example of how round-trip JSON serialization is already possible. So the main limitation, then, is probably just that I don't understand what all of the dataflow graph nodes do 🙃.

I've read the How Vega Works notebook and Reactive Vega paper, but I'm still foggy on how I'd implement some of the details in the context of VegaFusion. Here are a couple that come to mind:

• The dataflow graph has the concept of pulse edges and parameter edges. My understanding is that datasets are streamed through the pulse edges and signals are passed along the parameter edges. In VegaFusion's columner batch context, would it be possible to ignore the distinction and pass entire datasets along the pulse edges? In VegaFusion, nodes values have to be a function of their parent values and they can't be stateful.
• I don't understand what kind of data flows along pulse edges in the context of rendering. What is the output of datajoin in https://observablehq.com/@vega/how-vega-works? Are mark items special objects in a dataset? We probably wouldn't want these to end up on the server, but I'm not sure how to reason about them.

[domoritz]

Paging @jheer and @arvind who have deeper insights into the semantics of the dataflow.

[jheer]

Hi!

Regarding edges:

• "pulse" edges indeed involve streaming data batches. The pulse object contains add/rem/mod sets for added, removed, or modified tuples, respectively. Pulses can also enable reference to a data "source" for operations that need to "reflow" by operating over all tuples rather than just delta sets.
• "parameter" edges are just dependencies between node values. Operator nodes in the Vega dataflow are stateful and expose a "value" property. Changes to a node's value can trigger reactive updates for all downstream dependents.

At a high-level, the Vega JS runtime combines two ideas: functional reactive programming (via the node values and parameter edges, plus evaluation functions on each node) and streaming data processing (via pulses for the flow of data). In addition, the dependency graph is analyzed to assign each node a rank value, which is used within a priority heap to handle evaluation order upon updates.

For rendering, each scenegraph item is also a valid Vega "tuple". The output of the data join is a scenegraph item (that also has a "data" property referring to the backing data tuple). These items are then subject to subsequent transforms (e.g., layouts or bounding box calculations) before being passed to a "render" transform that helps the actual renderer keep track of which items require (re)draw.

[jheer]

Regarding the point of this issue, the generated vega dataflow specification is designed for the specifics of the JS reactive runtime. Alternative runtimes may make different decisions, and so I would not expect a one-to-one correspondence between the dataflow JSON spec and the architecture of an alternative runtime. So while I agree with @domoritz that the dataflow exposes additional information that may aid optimization or more complete server-side processing, I'd expect some non-trivial analysis + translation logic would be needed to map this to VegaFusion's architecture.

[jonmmease]

Thanks, Jeff! That's very helpful

[jonmmease]

Here's a future optimization usecase I've been thinking about.

I'd like to eventually add selective server-side rendering support to VegaFusion. By "selective" I mean I'd like to find a way to render some marks on the server and some on the client. The motivating example is the large scatter plot where it would be nice to pre-render the symbol mark on the server, but allow the client to still render the axis rules and text.

The way I'm currently picturing doing this is to add custom transforms to VegaFusion that generate images from mark specifications. There are details to work out, but a render_symbol transform would basically accept all of the same options as the symbol mark specification, and it would produce a dataset where one column would contain the rendered image. This dataset-with-image-column would be passed to the client and to the Vega JavaScript library to be rendered using an image mark type.

By planning at the Vega-specification level, I can imagine a planning transformation that would automatically replace a symbol mark with an image mark on the client and a render_symbol transform on the server. My intuition is that this kind of transformation would be harder to reason about if the Planner only had access to the dataflow graph.