[CORE] Basic runnable version of ACBO (Advanced CBO) #5058

zhztheplayer · 2024-03-21T01:14:51Z

See proposal #5057

This is the first runnable version of ACBO with TPCH SF 1.0 and TPCDS 10.0 passed.

After this patch, one could set spark.gluten.sql.advanced.cbo.enabled=true to enable ACBO. It's by default disabled.

Issues:

It's now only replacing TransformPreOverrides() with a rough cost model to do fallback;
It's not tested with CH backend yet. @baibaichen @zzcclp If you would like to evaluate it for CH; Otherwise I could disable it temporarily for CH in next patch;
It may generate slow plan since some operators like filter / aggregation are not yet considered in first version of ACBO. Some refactors required and the side-effect of the issue will be amplified in performance test;
The first version would only be runnable. Further integration works are required to make it ready for production.

The following improvements are on the way:

Enable constraint propagation (for ops like AQEShuffleReadExec which could propagate children's convention to parent);
Enable pattern based rule-matching;

The required facilities of the above were already added but not enabled yet. Will enable and test them in PRs respectively.

github-actions · 2024-03-21T01:17:47Z

Run Gluten Clickhouse CI

github-actions · 2024-03-21T01:30:42Z

Run Gluten Clickhouse CI

github-actions · 2024-03-21T02:26:23Z

Run Gluten Clickhouse CI

github-actions · 2024-03-21T03:49:44Z

Run Gluten Clickhouse CI

github-actions · 2024-03-21T04:45:39Z

Run Gluten Clickhouse CI

github-actions · 2024-03-21T05:32:47Z

Run Gluten Clickhouse CI

github-actions · 2024-03-21T05:47:45Z

Run Gluten Clickhouse CI

zhztheplayer

Will keep filling code's document during further development.

And I'll make some notes on the PR first.

zhztheplayer · 2024-03-21T05:50:44Z

.github/workflows/velox_be.yml

+      - name: TPC-H SF1.0 && TPC-DS SF10.0 Parquet local spark3.2 with advanced CBO
+        run: |
+          $PATH_TO_GLUTEN_TE/$OS_IMAGE_NAME/gha/gha-checkout/exec.sh 'cd /opt/gluten/tools/gluten-it && \
+          mvn clean install -Pspark-3.2 \
+          && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
+            --local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
+            --extra-conf=spark.gluten.sql.advanced.cbo.enabled=true \
+          && GLUTEN_IT_JVM_ARGS=-Xmx20G sbin/gluten-it.sh queries-compare \
+            --local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=40g -s=10.0 --threads=32 --iterations=1 \
+            --extra-conf=spark.gluten.sql.advanced.cbo.enabled=true'


CI Job for ACBO + Velox + TPC-H SF1 + TPC-DS SF10

zhztheplayer · 2024-03-21T05:51:48Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala

+  }
+}
+
+class Cbo[T <: AnyRef] private (


Cbo is a stateless optimization context consisting of configs and utilities.

zhztheplayer · 2024-03-21T05:52:47Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala

+    assert(!notThrew, message)
+  }
+
+  private def validateModels(): Unit = {


Do validation on user's API implementations.

zhztheplayer · 2024-03-21T05:55:18Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala

+  }
+}
+
+trait CboCluster[T <: AnyRef] {


CboCluster is a set of nodes sharing the same context (the so-called "logical properties") in the original input plan. One cluster can derive its own set of CboGroups. Nodes in one CboGroup share the same ("physical") properties.

Add comments?

Will raise a independent PR to add code doc. The code is under frequent modification as of now.

zhztheplayer · 2024-03-21T05:58:12Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboConfig.scala

+case class CboConfig(
+    plannerType: PlannerType = PlannerType.Dp
+)
+
+object CboConfig {
+  sealed trait PlannerType
+  object PlannerType {
+    case object Exhaustive extends PlannerType
+    case object Dp extends PlannerType
+  }


Dp is the default planner implementation while Exhaustive is currently only used for testing. It's expected that we can implement parallelized optimization on exhaustive planner comparatively easier than on dp planner in future.

zhztheplayer · 2024-03-21T05:59:29Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboGroup.scala

+
+import io.glutenproject.cbo.memo.MemoStore
+
+trait CboGroup[T <: AnyRef] {


A set of nodes that share the same property set in the same cluster.

zhztheplayer · 2024-03-21T06:00:07Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala

+  }
+}
+
+trait CanonicalNode[T <: AnyRef] extends CboNode[T] {


Canonical node is a node with all children replaced by resident groups.

zhztheplayer · 2024-03-21T06:00:35Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala

+    extends CanonicalNode[T]
+}
+
+trait GroupNode[T <: AnyRef] extends CboNode[T] {


A node that exactly represents a group.

zhztheplayer · 2024-03-21T06:01:25Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala

+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {


A single immutable node wrapper that hides the tree structure from it.

For representing tree structure, use CboPath.

zhztheplayer · 2024-03-21T06:02:30Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboPlanner.scala

+trait Best[T <: AnyRef] {
+  import Best._
+  def rootGroupId(): Int
+  def bestNodes(): Set[InGroupNode[T]]
+  def winnerNodes(): Set[InGroupNode[T]]
+  def costs(): InGroupNode[T] => Option[Cost]
+  def path(allGroups: Int => CboGroup[T]): KnownCostPath[T]
+}


Best is basically the output of one shot of planning.

zhztheplayer · 2024-03-21T06:03:11Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboPlanner.scala

+trait PlannerState[T <: AnyRef] {
+  def cbo(): Cbo[T]
+  def memoState(): MemoState[T]
+  def rootGroupId(): Int
+  def best(): Best[T]
+}


The immutable dump of the planner.

zhztheplayer · 2024-03-21T06:04:19Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/ForwardMemoTable.scala

+
+import scala.collection.mutable
+
+class ForwardMemoTable[T <: AnyRef] private (override val cbo: Cbo[T])


The memo table that handles cluster merging / forwarding internally.

zhztheplayer · 2024-03-21T06:05:37Z

gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/Memo.scala

+  def defineEquiv(node: CanonicalNode[T], newNode: T): Unit
+}
+
+trait Memo[T <: AnyRef] extends Closure[T] {


Memo is the basic structure that stores the whole search space of planner. All the nodes stored in it are canonized.

github-actions · 2024-03-25T13:51:09Z

Run Gluten Clickhouse CI

github-actions · 2024-03-26T02:27:25Z

Run Gluten Clickhouse CI

github-actions · 2024-03-26T04:25:47Z

Run Gluten Clickhouse CI

github-actions · 2024-03-26T04:26:42Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T00:28:55Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T05:06:28Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T06:05:00Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T07:11:18Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T07:15:39Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T08:14:06Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T08:15:57Z

Run Gluten Clickhouse CI

This reverts commit 8fce2b9.

This reverts commit f23e095.

github-actions · 2024-03-27T08:22:19Z

Run Gluten Clickhouse CI

github-actions · 2024-03-27T08:31:23Z

Run Gluten Clickhouse CI

This comment was marked as abuse.

Sign in to view

apache deleted a comment from github-actions bot Mar 21, 2024

This comment was marked as abuse.

Sign in to view

zhztheplayer commented Mar 21, 2024

View reviewed changes

zhztheplayer force-pushed the wip-cbo branch from b0d9e58 to 6a0eee5 Compare March 26, 2024 02:26

zhztheplayer force-pushed the wip-cbo branch from e54b11d to 4e72ef7 Compare March 26, 2024 04:26

zhztheplayer force-pushed the wip-cbo branch from 4e72ef7 to 8a05370 Compare March 27, 2024 00:28

This was referenced Mar 27, 2024

[VL] Vanilla Spark broadcast exchange + R2C is slow sometimes #5136

Closed

[GLUTEN-5136][VL] Duplicated output from Spark-to-Velox broadcast relation conversion #5141

Merged

cbo

0abd657

zhztheplayer force-pushed the wip-cbo branch from 8a05370 to 0abd657 Compare March 27, 2024 05:05

fixup

f0af95a

zhztheplayer added 2 commits March 27, 2024 14:57

fixup

79721ce

fixup

32d0bf5

zhztheplayer added 4 commits March 27, 2024 16:02

debug

f23e095

fixup

4567db2

debug

8fce2b9

fixup

bf398d0

zhztheplayer added 2 commits March 27, 2024 16:21

Revert "debug"

2eb76a1

This reverts commit 8fce2b9.

Revert "debug"

0014a45

This reverts commit f23e095.

zhztheplayer merged commit c1e1cca into apache:main Mar 27, 2024
31 of 32 checks passed


		import io.glutenproject.cbo.memo.MemoStore

		trait CboGroup[T <: AnyRef] {


		import scala.collection.mutable

		class ForwardMemoTable[T <: AnyRef] private (override val cbo: Cbo[T])

[CORE] Basic runnable version of ACBO (Advanced CBO) #5058

[CORE] Basic runnable version of ACBO (Advanced CBO) #5058

Conversation

zhztheplayer commented Mar 21, 2024 • edited Loading

This comment was marked as abuse.

github-actions bot commented Mar 21, 2024

This comment was marked as abuse.

github-actions bot commented Mar 21, 2024

This comment was marked as abuse.

github-actions bot commented Mar 21, 2024

This comment was marked as abuse.

This comment was marked as abuse.

github-actions bot commented Mar 21, 2024

This comment was marked as abuse.

github-actions bot commented Mar 21, 2024

This comment was marked as abuse.

This comment was marked as abuse.

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

zhztheplayer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

zhanglistar Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

zhztheplayer Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Mar 25, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

zhztheplayer commented Mar 21, 2024 •

edited

Loading

zhztheplayer Mar 21, 2024 •

edited

Loading

zhztheplayer Mar 21, 2024 •

edited

Loading

zhanglistar Mar 25, 2024 •

edited

Loading

zhztheplayer Mar 21, 2024 •

edited

Loading

zhztheplayer Mar 21, 2024 •

edited

Loading

zhztheplayer Mar 21, 2024 •

edited

Loading

zhztheplayer Mar 21, 2024 •

edited

Loading