[SPARK-52283][CONNECT] Declarative Pipelines DataflowGraph creation and resolution
#51003
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sryza
changed the title
DataflowGraph creation and resolutionDataflowGraph creation and resolution
sryza
reviewed
May 23, 2025
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/GraphIdentifierManager.scala
Outdated
Show resolved
Hide resolved
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelinesErrors.scala
Outdated
Show resolved
Hide resolved
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala
Outdated
Show resolved
Hide resolved
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala
Outdated
Show resolved
Hide resolved
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/GraphIdentifierManager.scala
Outdated
Show resolved
Hide resolved
...pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/GraphRegistrationContext.scala
Show resolved
Hide resolved
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
Show resolved
Hide resolved
sryza
reviewed
May 23, 2025
project/SparkBuild.scala
Outdated
| ) | ||
| }, | ||
|
|
||
| (assembly / test) := { }, |
There was a problem hiding this comment.
Do we need this assembly stuff?
sryza
reviewed
May 23, 2025
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DataflowGraph.scala
Outdated
Show resolved
Hide resolved
sryza
reviewed
May 23, 2025
| import org.apache.spark.sql.util.CaseInsensitiveStringMap | ||
|
|
||
| /** | ||
| * Test suite for converting a [[PipelineDefinition]]s into a connected [[DataflowGraph]]. These |
There was a problem hiding this comment.
Suggested change
| * Test suite for converting a [[PipelineDefinition]]s into a connected [[DataflowGraph]]. These | |
| * Test suite resolving the flows in a [[DataflowGraph]]. These |
sryza
reviewed
May 23, 2025
| import org.apache.spark.sql.types.{IntegerType, StructType} | ||
|
|
||
| /** | ||
| * Test suite for converting one or more [[Pipeline]]s into a connected [[DataflowGraph]]. These |
There was a problem hiding this comment.
Suggested change
| * Test suite for converting one or more [[Pipeline]]s into a connected [[DataflowGraph]]. These | |
| * Test suite for resolving the flows in a [[DataflowGraph]]. These |
jonmio
reviewed
May 23, 2025
| @@ -2025,6 +2031,18 @@ | |||
| ], | |||
| "sqlState" : "42613" | |||
| }, | |||
| "INCOMPATIBLE_BATCH_VIEW_READ": { | |||
| "message": [ | |||
| "View <datasetIdentifier> is not a streaming view and must be referenced using read. This check can be disabled by setting Spark conf pipelines.incompatibleViewCheck.enabled = false." | |||
There was a problem hiding this comment.
What is the purpose of this conf and do we really need it?
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.spark.sql.pipelines |
There was a problem hiding this comment.
Nit: Should this be in some other package/directory?
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.spark.sql.pipelines |
| } | ||
|
|
||
| /** | ||
| * Core processor that is responsible for analyzing each flow and sort the nodes in |
|
|
||
| /** | ||
| * Processes the node of the graph, re-arranging them if they are not topologically sorted. | ||
| * Takes care of resolving the flows and virtualization if needed for the nodes. |
There was a problem hiding this comment.
Can you elaborate on what virtualization entails?
| * @param upstreamNodes Upstream nodes for the node | ||
| * @return | ||
| */ | ||
| def processNode(node: GraphElement, upstreamNodes: Seq[GraphElement]): Seq[GraphElement] = { |
There was a problem hiding this comment.
Nit: Document return. I'm especially curious why this is a Seq and when processNode would return more than one element
Comment on lines
+89
to
+92
| // Table will be virtual in either of the following scenarios: | ||
| // 1. If table is present in context.fullRefreshTables | ||
| // 2. If table has any virtual inputs (flows or tables) | ||
| // 3. If the table pre-existing metadata is different from current metadata |
There was a problem hiding this comment.
Not sure I follow this comment - it seems like resolvedInputs will always contain a pointer to a VirtualTableInput for any table being resolved?
| val result = | ||
| flowFunctionResult match { | ||
| case f if f.dataFrame.isSuccess => | ||
| // Merge the flow's inputs' confs into confs for this flow, throwing if any conflict |
There was a problem hiding this comment.
Can you provide an example in the docs of why conflicting confs can't be supported?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR introduces the
DataflowGraph, a container for Declarative Pipelines datasets and flows, as described in the Declarative Pipelines SPIP. It also adds functionality forGraphRegistrationContext)It also introduces various secondary changes:
SparkBuildto support declarative pipelines.pom.xmlfor the module.Why are the changes needed?
In order to implement Declarative Pipelines.
Does this PR introduce any user-facing change?
No changes to existing behavior.
How was this patch tested?
New test suites:
ConnectValidPipelineSuite– test cases where the graph can be successfully resolvedConnectValidPipelineSuite– test cases where the graph fails to be resolvedWas this patch authored or co-authored using generative AI tooling?
No