Offsite Followup

[jhellerstein]

Thanks all for a successful offsite on the 15th! Let's assemble a list of Followup plans and stakeholders. We should schedule initial meetings among the stakeholders in the coming week or two.

1. Program Properties for Optimization: In order to optimize programs, we need to identify the programs properties that feed into the optimization. This seems to span definitions of streams, ticks and iteration (as proposed by @shadaj), temporal persistence and operator monotonicity (as used by @davidchuyaya) and stream properties in the Hydroflow implementation like blocking/streaming/monotonicity (as implemented by @MingweiSamuel and @jhellerstein). Stakeholders: @shadaj, @davidchuyaya, @MingweiSamuel, @jhellerstein.
2. Canonical Definitions: We should develop uniform terminology for use across the team. @nacrooks specifically asked for a canonical definition of monotonicity. Currently we have a mix of uses across monotone functions, monotonic logic, monotone streams and monotonicity over time. Let's see clarify/contextualize/taxonomize. Are there other terms that require clarification or contextualization? Stakeholders: @jhellerstein @davidchuyaya @shadaj @MingweiSamuel
3. Engineering Practices: We should shift some engineering practices. Some suggestions included:

• 1-pager design docs for non-trivial projects
• git discipline: exploratory code in branches rather than main
• repo organization: explicit "nursery" for projects that are nearing relevance.
• improved doc integration: we've got some of our docs generated from docstrings. How do we improve documentation?
• inclusion of a full-time programmer in PR approval to main
• weekly demo sessions
  stakeholders: @rohitkulshreshtha @MingweiSamuel @jhellerstein @shadaj @conor-23 @davidchuyaya

[davidchuyaya]

For monotonicity, what my rewrites care about is whether node behaves monotone across time (if the node is treated as a black box over inputs and outputs), but not necessarily that it contains monotonic functions or that facts within the node are monotone across time.

That might be important to distinguish too, because this allows for a node to be "monotonic" because it uses a threshold at the end.
On the other end of a spectrum, a 2P-Set contains monotonic logic but its output is not monotone across time.

[jhellerstein]

Hey @davidchuyaya -- I feel like you need both monotonic ops and persistence over time. Let's consider an example with a non-monotone but static operator:

c1 = source_iter([1,2,3,4,5,6,7,8,9]) -> map(|i| i + 1);
c2 = c1 -> fold::<'static>(median) -> for_each(|i| println!("Median is {}", i));

As a single component, this produces exactly one value -- in the first tick. If we decouple c1 from c2, the batching is different, so we are likely to see multiple printed messages, one per tick of c2 until it gets all the data from c1. Depending on how we implement median, c2 should get the right answer eventually, but it will be apparent to an observer that it's been decoupled.

I still believe what we said in the paper:

A simple sufficient condition for monotonicity is to ensure that (a) 𝐶2’s input relations are persisted, and (b) 𝐶2’s rules do not contain negation or aggregation

The operator monotonicity is in part (b).

[davidchuyaya]

What I meant was that not all functions in the node needed to be monotonic if we had a threshold at the end. Otherwise yes, non-monotonic functions with only persist would require state machine (or partial independent) decoupling, where batches and ordering is maintained.

[jhellerstein]

The idea of "downstream threshold removes the need for upstream associativity" is interesting! We should add this to the discussion of all the algebraic optimizations we can consider.

More evidence that we may do well to break down monotonicity (for which all the ACI properties are a sufficient condition) into individual properties that we want to check.

[jhellerstein]

That said, the discussion in the paper is conservative and correct -- monotonicity of logic over time guarantees decoupling is safe.

We do need to change your hypothesis that kicked off the discussion -- persistence over time is not enough, but perhaps "downstream observer associativity over time" is sufficient to tolerate decoupling.

[shadaj]

I think even with thresholds though the algebraic properties are somewhat orthogonal. All you need is a partial order and a guarantee that a fold in your pipeline spits out values monotone in that partial order when given larger prefixes of the input. And the same for the boolean value that is eventually used as a condition.

[jhellerstein]

Flagging this scenario as one to explore further when we have the optimizer conversation.

…

On Mon, May 20, 2024 at 12:05 PM Shadaj Laddad ***@***.***> wrote: I think even with thresholds though the algebraic properties are somewhat orthogonal. All you need is a partial order and a guarantee that a fold in your pipeline spits out values monotone in that partial order when given larger prefixes of the input. And the same for the boolean value that is eventually used as a condition. — Reply to this email directly, view it on GitHub <#1203 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC54QLV2N6NOWBW2WMIXSLZDJCRFAVCNFSM6AAAAABH4ZBPH6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIOJZHE2TS> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>