[FLINK-36069][runtime/rest] Extending job detail rest API to expose json stream graph

[yuchen-ecnu]

What is the purpose of the change

• Extending job detail rest API to adapt to Incremental JobGraph Generation in batch mode.
• Modify web ui to support job graph auto-refresh for Incremental JobGraph Generation.

Brief change log

• Modify JobDetailHandler to return the incrementally updated job graphs.
• Modify the web ui to display the incremental job graph and the unconverted stream node.
• Add a new checkbox in the webui to select whether to display the unconverted stream node or not.

Verifying this change

This change added tests and can be verified as follows:

• Added test that validates that JsonPlanGenerator generates JsonStreamGraph works well
• Added test that validates that JobDetailHandler returns Incremental JobGraph as expected

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? not applicable

[yuchen-ecnu]

Hi @JunRuiLee, can you help to review this PR?

[flinkbot]

CI report:

• 5ff6712 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[JunRuiLee]

Thanks @yuchen-ecnu for contributing, I've left two comments, PTAL!

[JunRuiLee]

I prefer to retrieve the JSON of the stream graph from ExecutionPlanSchedulingContext. For example, we could add a method getJsonStreamGraph in ExecutionPlanSchedulingContext. This context will return the JSON held by AdaptiveGraphManager. When the stream graph is updated by the DefaultStreamGraphContext, the AdaptiveGraphManager should update this JSON.

With this update, we could avoid manually passing this JSON to the execution graph and keep this change simple.

[yuchen-ecnu]

Great tips! ExecutionPlanSchedulingContext really helps a lot. I have remove the unexpected setJsonStreamGraph methods from ExecutionGraph in the revised version.

[JunRuiLee]

How about handle pending-operators in the JobDetailsHandler#createJobDetailsInfo. This way, we wouldn't need to introduce any meaningless statuses to a common enum class.

[yuchen-ecnu]

Agree with you. Since it’s not a valid status for stream jobs, how about including it in the serialized JSON stream graph (JsonPlanGenerator#generateJsonStreamGraph)? This would help avoid adding interfaces like getPendingOperatorCounts() to the ExecutionGraph and ArchivedExecutionGraph for processing in JobDetailsHandler#createJobDetailsInfo. And it will be processed in frontend code.