Easier handling of scalar variables

[jhellerstein]

Suppose I want to normalize my data -- i.e. compute the sum of items in a flow, and then use that scalar value in a subsequent flow as the denominator. I can do this now via cross_join:

   data = source_iter(vec!(1,2,3)) -> tee();
   count = data -> fold(0, |accum, i| accum + i);
   attach = cross_join();
   data -> [0]attach;
   count -> [1]attach;
   attach -> map(|(val, cnt)| (val, (val as f32)/(cnt as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n));

It would be nice to be able to just reference count directly:

   data = source_iter(vec!(1,2,3)) -> tee();
   count = data -> fold(0, |accum, i| accum + i);
   data -> map(|val| (val, (val as f32)/(count as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n));

Or perhaps more verbosely, an operator that puts count into a global set of vars:

   data = source_iter(vec!(1,2,3)) -> tee();
   count = data -> fold(0, |accum, i| accum + i) -> vars();
   data -> map(|val| (val, (val as f32)/(count as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n));

Or even more verbosely have the vars operator populate a global HashMap variable vars:

   data = source_iter(vec!(1,2,3)) -> tee();
   count = data -> fold(0, |accum, i| accum + i) -> vars(|v| (`count, v));
   data -> map(|val| (val, (val as f32)/(vars[`count] as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n));

[shadaj]

I'd definitely prefer the non-HashMap scalar flow design, having a hashmap introduces a strange typing model where every key has a different value type that feels unintuitive.

…

On Fri, Jan 20, 2023, 12:27 AM Joe Hellerstein ***@***.***> wrote: Suppose I want to normalize my data -- i.e. compute the sum of items in a flow, and then use that scalar value in a subsequent flow as the denominator. I can do this now via cross_join: data = source_iter(vec!(1,2,3)) -> tee(); count = data -> fold(0, |accum, i| accum + i); attach = cross_join(); data -> [0]attach; count -> [1]attach; attach -> map(|(val, cnt)| (val, (val as f32)/(cnt as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n)); It would be nice to be able to just reference count directly: data = source_iter(vec!(1,2,3)) -> tee(); count = data -> fold(0, |accum, i| accum + i); data -> map(|val| (val, (val as f32)/(count as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n)); Or perhaps more verbosely, an operator that puts count into a global set of vars: data = source_iter(vec!(1,2,3)) -> tee(); count = data -> fold(0, |accum, i| accum + i) -> vars(); data -> map(|val| (val, (val as f32)/(count as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n)); Or even more verbosely have the vars operator populate a global HashMap variable vars: data = source_iter(vec!(1,2,3)) -> tee(); count = data -> fold(0, |accum, i| accum + i) -> vars(|v| (`count, v)); data -> map(|val| (val, (val as f32)/(vars[`count] as f32))) -> for_each(|(v, n)| println!("val: {}, normalized: {}", v, n)); — Reply to this email directly, view it on GitHub <#355>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEESTY4O7NYUA33OKCA52DWTGFA3ANCNFSM6AAAAAAUAWC7ZQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[zzlk]

The issue of what to do with state also came up again when trying to implement topolotree. When implementing things, programmers tend to want to start from state/data structures and then think about how to operate on them later. But in hydroflow you kind of have to start from the operation and then figure out how to store state in that operation, it's tricky to model problems in this way.

[MingweiSamuel]

Implemented #797 #1065 #1091 #1133 #1134 etc