Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweak mermaid dataflow arrow heuristics for downstream pipelines #57

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ryannedolan
Copy link
Collaborator

Summary

Tweaked the heuristics used for rendering the dataflow arrows in mermaid diagrams via the !mermaid command.

Details

Previously, any downstream resources were fully connected to an upstream resource, even when downstream resources were themselves connected in series.

With this change, downstream resources are only connected to the upstream resource if they aren't already connected to something upstream.

As this is a heuristic, it's likely to break for some topologies, but it seems to work for existing use-cases. Keep in mind that this is only for visually inspecting pipelines -- the arrows are non-functional.

Testing

This was tested with an internal (non-public) adapter, with results along these lines:

flowchart
  subgraph SqlJob
  R-299574215["pipeline.sql: CREATE DATABASE I..."]
  end
  subgraph EtlJob
  R769688330["catalog: redacted
database: tracking
kafkaPipeline: tracking
kafkaTopic: REDACTED1
table: REDACTED1"]
  end
  subgraph KafkaTopic
  R-1619492592["topicName: REDACTED1"]
  end
  
  subgraph ReverseEtlJob
  R1874360940["kafkaPipeline: tracking
kafkaTopic: MyPinotTable
table: MyPinotTable"]
  end
  subgraph BrooklinDatastream
  R-1748512199["brooklinCluster: https://brooklin....
connectorType: BrooklinFunction
name: brooklinsampledb1...
sourceDatabase: BrooklinSampleDb1
sourceTable: SampleTable1"]
  end
  subgraph IcebergTable
  R1232766216["
database: tracking
table: REDACTED1"]
  end
  subgraph REDACTED
  R1207860856["
database: tracking
table: REDACTED1"]
  R-1812063814["catalog: pinot
database: tracking
table: MyPinotTable"]
  end
  subgraph PinotTable
  R1460909616["table: MyPinotTable"]
  end
  subgraph EspressoTable
  R-809211083["database: BrooklinSampleDb1
name: sampletable1
table: SampleTable1"]
  end
  
  subgraph SqlJob
    R1207860856 --> R-299574215
    R-1748512199 --> R-299574215
  end
  subgraph EtlJob
    R-1619492592 --> R769688330
  end
  subgraph KafkaTopic
  end
  
  subgraph ReverseEtlJob
    R-1812063814 --> R1874360940
  end
  subgraph BrooklinDatastream
    R-809211083 --> R-1748512199
  end
  subgraph IcebergTable
    R769688330 --> R1232766216
  end
  subgraph REDACTED
    R1232766216 --> R1207860856
    R-299574215 --> R-1812063814
  end
  subgraph PinotTable
    R1874360940 --> R1460909616
  end
  subgraph EspressoTable
  end
  
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant