Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-35745] add documentation for flink lineage #25762

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[FLINK-35745] add documentation for flink lineage #25762

wants to merge 2 commits into from
+226 −1

Conversation

HuangZhenQiu
Copy link
Contributor

What is the purpose of the change

Add documentation for native lineage support in Flink. Mainly for connector developers.

Brief change log

  • Add data_lineage.md under docs/internals for both English and Chinese
  • Improve the existing content in job_status_listerner.md

Verifying this change

  • Build the docs locally and verified end to end

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (o)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (docs)

@HuangZhenQiu
Copy link
Contributor Author

Screenshot 2024-12-07 at 10 18 35 PM

@HuangZhenQiu HuangZhenQiu mentioned this pull request Dec 8, 2024
Open
@flinkbot
Copy link
Collaborator

flinkbot commented Dec 8, 2024 •
edited
Loading

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

davidradl
davidradl reviewed Dec 9, 2024
View reviewed changes
docs/content/docs/internals/data_lineage.md
contains the Lineage Graph metadata will be sent to Job Status Listeners.

# Lineage Data Model
Flink native lineage interfaces are defined in two layers. The first layer is the generic interface for all Flink jobs and connector, and the second layer defines
Copy link
Contributor

@davidradl davidradl Dec 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest a picture showing the layers at a component level.

@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-35745-lineage-documentation branch from 16cbcaa to 8fd10b9 Compare December 13, 2024 02:35
@HuangZhenQiu
Copy link
Contributor Author

@davidradl
Thanks for the kind review and suggestions for wording. Please take a look again. Yes, the PR should be merged after JDBC diff.

@davidradl
Copy link
Contributor

Reviewed by Chi on 12/12/24 Group to test and/or review outside of the meeting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davidradl davidradl davidradl left review comments

Assignees
No one assigned
Labels
component=Documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@HuangZhenQiu @flinkbot @davidradl