Merge batcher generic over containers (#474)

* Split default merge batcher This change splits the default merge batcher implementation into a type that maintains the outer part of its algorithm, specifically knows how to maintain chains, and an inner part that knows how to maintain the individual batches in chains. The benefit is that the outer part does not need to know about the contents of the containers it holds on to because that's encapsulated in the inner trait's implementation. Signed-off-by: Moritz Hoffmann <[email protected]> * Undo some changes, rip out old columnated batcher Signed-off-by: Moritz Hoffmann <[email protected]> * Address feedback Signed-off-by: Moritz Hoffmann <[email protected]> * Formatting and renaming Signed-off-by: Moritz Hoffmann <[email protected]> * Address feedback Signed-off-by: Moritz Hoffmann <[email protected]> --------- Signed-off-by: Moritz Hoffmann <[email protected]>
TimelyDataflow · Apr 26, 2024 · b281e50 · b281e50
1 parent d18497c
commit b281e50
Show file tree

Hide file tree

Showing 9 changed files with 630 additions and 574 deletions.
diff --git a/src/operators/arrange/arrangement.rs b/src/operators/arrange/arrangement.rs
@@ -464,7 +464,7 @@ where
 
             input.for_each(|cap, data| {
                 capabilities.insert(cap.retain());
-                batcher.push_batch(data);
+                batcher.push_container(data);
             });
 
             // The frontier may have advanced by multiple elements, which is an issue because

diff --git a/src/operators/reduce.rs b/src/operators/reduce.rs
@@ -240,7 +240,7 @@ pub trait ReduceCore<G: Scope, K: ToOwned + ?Sized, V: Data, R: Semigroup> where
     ///          .map(|x| (x, x))
     ///          .reduce_abelian::<_,_,ValSpine<_,_,_,_>>(
     ///             "Example",
-    ///              Clone::clone, 
+    ///              Clone::clone,
     ///              move |_key, src, dst| dst.push((*src[0].0, 1))
     ///          )
     ///          .trace;