feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

acrylJonny · 2025-01-21T15:20:10Z

Adding functionality to the existing Grafana connector. The existing connector supports Dashboard identification only Changed implement the following:

Charts
- Input datasets
- Input Columns
Datasets
- Dataset Schema
- Subtyping with definition where SQL
Lineage:
- between upstream sources and datasets
- between datasets and charts
- between charts and workbooks
- Supports SQL lineage reconciliation using datahub.sql_parsing.sqlglot_lineage falling back to sqlparse
Tag extraction with propagation back from dashboard through to chart and dataset
Owner extraction

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

codecov · 2025-01-21T15:58:54Z

Codecov Report

Attention: Patch coverage is 97.04142% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...datahub/ingestion/source/grafana/grafana_source.py	94.17%	6 Missing ⚠️
...on/src/datahub/ingestion/source/grafana/lineage.py	91.04%	6 Missing ⚠️
...rc/datahub/ingestion/source/grafana/grafana_api.py	97.10%	2 Missing ⚠️
...rc/datahub/ingestion/source/grafana/field_utils.py	98.93%	1 Missing ⚠️

Files with missing lines	Coverage Δ
...b-react/src/app/ingest/source/builder/constants.ts	`100.00% <100.00%> (ø)`
...hub/ingestion/source/grafana/entity_mcp_builder.py	`100.00% <100.00%> (ø)`
...datahub/ingestion/source/grafana/grafana_config.py	`100.00% <100.00%> (ø)`
...ion/src/datahub/ingestion/source/grafana/models.py	`100.00% <100.00%> (ø)`
...ion/src/datahub/ingestion/source/grafana/report.py	`100.00% <100.00%> (ø)`
...tion/src/datahub/ingestion/source/grafana/types.py	`100.00% <100.00%> (ø)`
...rc/datahub/ingestion/source/grafana/field_utils.py	`98.93% <98.93%> (ø)`
...rc/datahub/ingestion/source/grafana/grafana_api.py	`97.10% <97.10%> (ø)`
...datahub/ingestion/source/grafana/grafana_source.py	`95.04% <94.17%> (ø)`
...on/src/datahub/ingestion/source/grafana/lineage.py	`91.04% <91.04%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e18fa0...ac2e4cd. Read the comment docs.

metadata-ingestion/src/datahub/ingestion/source/grafana/models.py

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

mayurinehate

Looking good. I'd much appreciate a "Concept Mapping" section in docs.

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

mayurinehate · 2025-01-28T07:49:46Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

+        )
+
+        # Generate dashboard container first
+        yield from gen_containers(


If I understand correctly this is new addition over only dashboard entity that was present earlier. so one container corresponding to each dashboard - to encompass panels (chart) and datasource (dataset) entity ?

If so wondering if datasources are dashboard scoped or global in grafana.

That is correct. For data sources (the database name, type and the database) are global but the SQL itself for each visual is set in each chart.

mayurinehate · 2025-01-28T07:53:17Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

-                platform_instance=self.source_config.platform_instance,
+
+            yield from add_dataset_to_container(
+                container_key=folder_key,


You can set parent_container_key = folder_key in gen_containers for dashboard instead

@mayurinehate, would you be able to suggest the appropriate changes as this is not super clear what the proposal is?

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

mayurinehate · 2025-01-28T09:30:51Z

metadata-ingestion/src/datahub/ingestion/source/grafana/lineage.py

+            return None
+
+        ds_type, ds_uid = self._extract_datasource_info(panel.datasource)
+        raw_sql = self._extract_raw_sql(panel.targets)


Just curious if we ingest raw query used in panel anywhere in model. I believe the language of query depends on the source it integrates with.

mayurinehate · 2025-01-28T09:31:41Z

metadata-ingestion/src/datahub/ingestion/source/grafana/lineage.py

+            return None
+
+        ds_type, ds_uid = self._extract_datasource_info(panel.datasource)
+        raw_sql = self._extract_raw_sql(panel.targets)


Just curious if we ingest raw query used in panel anywhere in datahub model? I believe the language of query depends on the source it integrates with.

Would you be able to expand upon this?

metadata-ingestion/src/datahub/ingestion/source/grafana/lineage.py

metadata-ingestion/src/datahub/ingestion/source/grafana/types.py

mayurinehate · 2025-01-28T09:35:33Z

metadata-ingestion/tests/integration/grafana/test_grafana.py

    service_account = grafana_client.create_service_account(
-        name="example-service-account", role="Viewer"
+        name="example-service-account", role="Admin"


Any specific reason for changing role here ? Do we require elevated permissions for changes in this PR?

You need the admin permission to obtain access to the data source. A lot of the information used for lineage required admin access because otherwise you can't get compile the lineage.

sgomezvillamor · 2025-01-29T12:27:20Z

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

+        from_start = sql.index("from")
+        select_part = sql[select_start:from_start].strip()
+
+        columns = [col.strip().split()[-1].strip() for col in select_part.split(",")]


Assuming that split by , will actually split by columns may be wrong assumption, eg: SELECT CONCAT(first_name, ' ', last_name) AS full_name FROM users
Not sure if we need to address this complexity though. WDYT?

I've added some more logic that targets this type of SQL

acrylJonny · 2025-02-14T11:23:05Z

Looking good. I'd much appreciate a "Concept Mapping" section in docs.

Concept map has now been added

…datahub into grafana-improvements

acrylJonny added 2 commits January 21, 2025 15:17

initial commit

d4be2de

Delete grafana2

660daf8

acrylJonny marked this pull request as draft January 21, 2025 15:20

Merge branch 'master' into grafana-improvements

6cfbf5f

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Jan 21, 2025

vercel bot deployed to Preview January 21, 2025 15:55 View deployment

acrylJonny and others added 3 commits January 21, 2025 21:44

second commit

23e4eca

updating tests and adding tags and ownership options

5468909

Merge branch 'master' into grafana-improvements

e684eed

acrylJonny marked this pull request as ready for review January 21, 2025 22:21

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Jan 21, 2025

acrylJonny added 2 commits January 21, 2025 22:39

error updates and better formatting of docs

eda52c4

Update grafana_api.py

507c811

vercel bot deployed to Preview January 21, 2025 23:39 View deployment

hsheth2 reviewed Jan 22, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/grafana/models.py Outdated Show resolved Hide resolved

hsheth2 requested review from sgomezvillamor and mayurinehate January 22, 2025 00:04

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 22, 2025

acrylJonny and others added 3 commits January 22, 2025 11:08

updating to use basemodel. Cleaning up code

0e9f0dc

test updates

c832e91

Merge branch 'master' into grafana-improvements

0fe68d2

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 22, 2025

reorder libraries to be alphabetical

c0e63d7

vercel bot had a problem deploying to Preview January 22, 2025 11:54 Failure

sgomezvillamor reviewed Jan 22, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py Outdated Show resolved Hide resolved

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 22, 2025

mayurinehate reviewed Jan 28, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 28, 2025

sgomezvillamor reviewed Jan 29, 2025

View reviewed changes

acrylJonny and others added 2 commits February 14, 2025 09:27

addressing comments

7646b89

Merge branch 'master' into grafana-improvements

6bd09ed

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Feb 14, 2025

vercel bot deployed to Preview February 14, 2025 10:09 View deployment

addressing comments

51310f9

vercel bot had a problem deploying to Preview February 14, 2025 11:44 Failure

acrylJonny and others added 2 commits February 17, 2025 09:39

Improving overall experience

54d142f

Merge branch 'master' into grafana-improvements

b4b7409

vercel bot had a problem deploying to Preview February 17, 2025 10:01 Failure

acrylJonny added 2 commits February 17, 2025 10:10

Update constants.ts

b4da306

Merge branch 'grafana-improvements' of https://github.com/acrylJonny/…

40b75b8

…datahub into grafana-improvements

vercel bot had a problem deploying to Preview February 17, 2025 10:32 Failure

Merge branch 'master' into grafana-improvements

9bd5e7d

vercel bot had a problem deploying to Preview February 18, 2025 18:48 Failure

Update grafana_pre.md

1dbe6f9

vercel bot deployed to Preview February 18, 2025 19:09 View deployment

acrylJonny and others added 2 commits February 18, 2025 23:41

test file updates

8e150cd

Merge branch 'master' into grafana-improvements

addfd15

vercel bot had a problem deploying to Preview February 19, 2025 00:20 Failure

Merge branch 'master' into grafana-improvements

40c2ff9

mayurinehate self-requested a review February 19, 2025 09:41

vercel bot had a problem deploying to Preview February 19, 2025 09:57 Failure

Merge branch 'master' into grafana-improvements

ac2e4cd

vercel bot deployed to Preview February 20, 2025 17:34 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

acrylJonny commented Jan 21, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

mayurinehate left a comment

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

acrylJonny Feb 14, 2025

mayurinehate Jan 28, 2025

acrylJonny Feb 14, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

acrylJonny Feb 14, 2025

mayurinehate Jan 28, 2025

acrylJonny Feb 14, 2025

sgomezvillamor Jan 29, 2025

acrylJonny Feb 14, 2025

acrylJonny commented Feb 14, 2025

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

Are you sure you want to change the base?

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

Conversation

acrylJonny commented Jan 21, 2025 • edited Loading

Checklist

codecov bot commented Jan 21, 2025 • edited Loading

Codecov Report

mayurinehate left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acrylJonny commented Feb 14, 2025

acrylJonny commented Jan 21, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading