feat(ingestion/sql-common): add column level lineage for external tables

[acrylJonny]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[sgomezvillamor]

Not sure if fetching the upstream schema from the DataHub graph is an existing pattern in the connectors, so I'm asking:

• what if the upstream schema isn't published yet? should we report a warning?
• Isn't the connector itself producing the upstream schema? if so, may we skip fetching it from the DataHub graph?
• may we overloading the DataHub graph with too many requests, if we call it hundreds/thousands of times?

Let me know your thoughts when you get a chance!

[hsheth2]

Agree that we should avoid making graph calls like this if we can avoid it.

@acrylJonny what was the reason we need this lookup?

[acrylJonny]

Essentially it was to ensure that the upstream dataset has the same column as the downstream. We could choose to skip this validation though and simply link the upstream with the downstream blindly. This would remove the need to obtain the schema of the upstream.

[hsheth2]

Yep let's just do the blind linking for now - it's what we do for most other sources that generate external lineage

The main risk there is actually column casing mismatches, but for those we'd need to use a fuzzy matcher - we recently added this to looker, but imo it's probably not necessary here

[hsheth2]

The SqlParsingAggregator has a add_known_lineage_mapping that generates CLL based on the schema of the downstream. Ideally we'd centralize on using that as the external lineage mechanism.

Long term, I want to move sql_common.py to use the SqlParsingAggregator instead of the older SqlParsingBuilder. Internal ticket tracking that - https://linear.app/acryl-data/issue/ING-779/refactor-move-sql-common-to-use-sqlparsingaggregator

In the short term, I'm ok with having this CLL generation logic, although all the complexity of the simplify_field_path logic worries me a bit on this PR.

[hsheth2]

Now that #12220 has been merged, we can make this implementation be a bit cleaner.