[FLINK-34565] Enhance flink kubernetes configMap to accommodate additional configuration files

[surendralilhore]

What is the purpose of the change

The purpose of this change is to empower users to effortlessly add additional files to the Flink ConfigMap, essential for running jobs in app mode on Kubernetes. Currently, the Flink Kubernetes client supports only a fixed set of files (flink-conf.yaml, logback-console.xml, log4j-console.properties) in the ConfigMap for JobManager and TaskManager Pods. However, in certain scenarios, users may need to include extra configuration files, such as user-defined job properties. Without this change, users are forced to employ workarounds, like manually creating and mounting ConfigMaps or Secrets, to incorporate dynamic configuration files into the JobManager and TaskManager Pods.

In my case, I aim to include log4j-cli.properties (for running specific Flink commands on each pod) and hive-site.xml (for metastore). A similar use case was discussed on the user mailing list: link to the mailing list thread

Brief change log

The method FlinkConfMountDecorator#getLocalLogConfFiles() provides a fixed set of files that need to be included in the ConfigMap. I have modified this method to incorporate an additional set of files, which is configured through the new property 'kubernetes.flink.configmap.additional.resources'. This property allows users to specify a semicolon-separated list of local files in the client Flink config directory that should be included in the Flink ConfigMap.

Verifying this change

This change added tests and can be verified as follows:

• Added FlinkConfMountDecoratorTest#testConfigMapAdditionalResource() which will create two dummy extra files and configure the same files in 'kubernetes.flink.configmap.additional.resources'.
• The test will verify if flinkConfMountDecorator is able to get the additional files as part of the ConfigMap or not.
• The case when this property is not configured is already covered by an existing test case.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 5b15d0b Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[surendralilhore]

@gyfora , can you please review .

[gyfora]

@surendralilhore , could you please fill out the pull-request template properly and provide a practical example that would highlight the need for such a feature?

And also show how this feature would help with that practical problem.

[surendralilhore]

Thanks @gyfora.

I have updated PR template. Added detailed description.

[surendralilhore]

Hi @zhuzhurk,

Please can you review this PR.