Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: JDBC SQL lineage reports is missing schema support #33794

Closed
1 of 17 tasks
stankiewicz opened this issue Jan 29, 2025 · 2 comments · Fixed by #33795
Closed
1 of 17 tasks

[Bug]: JDBC SQL lineage reports is missing schema support #33794

stankiewicz opened this issue Jan 29, 2025 · 2 comments · Fixed by #33795

Comments

@stankiewicz
Copy link
Contributor

stankiewicz commented Jan 29, 2025

What happened?

when querying JDBCIO:

.withQuery("select id, first_name, last_name from public.users")

reported lineage is reporting table segment as public.users.

expected output should be two segments: public, users as per Dataplex FQN spec:
cloudsql_{scheme}:{projectId}.{location}.{instanceId}.{databaseId}.{schemaId}.{tableId}

if sql query/inputed table is missing schema then beam reports:
cloudsql_{scheme}:{projectId}.{location}.{instanceId}.{databaseId}.{tableId}

which dataplex interprets as
cloudsql_{scheme}:{projectId}.{location}.{instanceId}.{databaseId}.{schemaId}

Issue Priority

Priority: 3 (minor)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@stankiewicz
Copy link
Contributor Author

@Abacn , fyi

@stankiewicz stankiewicz changed the title [Bug]: Cloud SQL lineage reports schema and table as one segment [Bug]: JDBC SQL lineage reports schema and table as one segment Jan 29, 2025
@stankiewicz
Copy link
Contributor Author

stankiewicz commented Jan 29, 2025

I'm also thinking if we are missing quoted identifiers e.g.

select * from `select`

https://dev.mysql.com/doc/refman/8.4/en/identifiers.html

@stankiewicz stankiewicz changed the title [Bug]: JDBC SQL lineage reports schema and table as one segment [Bug]: JDBC SQL lineage reports is missing schema support Jan 29, 2025
@github-actions github-actions bot added this to the 2.63.0 Release milestone Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant