[K8S][HELM] Add Hadoop configuration support

[dnskr]

🔍 Description

Issue References 🔗

The PR adds basic support for Hadoop configuration files to be used by Apache Kyuubi, Apache Spark etc.

Describe Your Solution 🔧

The PR adds:

• Apache Hadoop related ConfigMap that has to be mounted to Kyuubi pods as files.
• HADOOP_CONF_DIR environment variable to Kyuubi pods.

Notes:

• The ConfigMap change restarts Kyuubi pods!
• Only core-site.xml and hdfs-site.xml files are added. Not sure yarn-site.xml, mapred-site.xml or other files should be added. Please, let me know if so.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Install the chart with configured Spark and the following values:

# Hadoop configuration files
hadoopConf:
  # The value (templated string) is used for core-site.xml file
  # See https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml and Hadoop documentation for more details
  coreSite: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>test.core.site</name>
        <value>test-core-site</value>
      </property>
    </configuration>

  # The value (templated string) is used for hdfs-site.xml file
  # See https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml and Hadoop documentation for more details
  hdfsSite: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>test.hdfs.site</name>
        <value>test-hdfs-site</value>
      </property>
    </configuration>

Run beeline from Kyuubi pod:

./bin/beeline -u 'jdbc:hive2://kyuubi-thrift-binary:10009' -n test-user

Check specified property values:

0: jdbc:hive2://kyuubi-thrift-binary:10009> set test.core.site;
+-----------------+-----------------+
|       key       |      value      |
+-----------------+-----------------+
| test.core.site  | test-core-site  |
+-----------------+-----------------+


0: jdbc:hive2://kyuubi-thrift-binary:10009> set test.hdfs.site;
+-----------------+-----------------+
|       key       |      value      |
+-----------------+-----------------+
| test.hdfs.site  | test-hdfs-site  |
+-----------------+-----------------+

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f67140e) 61.03% compared to head (1d66b3e) 60.99%.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6046      +/-   ##
============================================
- Coverage     61.03%   60.99%   -0.05%     
  Complexity       23       23              
============================================
  Files           623      623              
  Lines         37162    37162              
  Branches       5035     5035              
============================================
- Hits          22682    22666      -16     
- Misses        12029    12033       +4     
- Partials       2451     2463      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

Actually, it's likely to put additional configuration files under HADOOP_CONF_DIR in practice, for example, hive-site.xml, hbase-site.xml, so we may want to allow the user to inject custom files.

Another concern is that embeding large-XML into YAML may be not user-friendly, it's easy to mess up the indent.

[chgl]

Definitely agree. The approach I often see is providing both options: either provide a (templated) inline config - which as you said can get unwieldy fast - or provide the name of an existing config map. The latter allows creating that config map from a file or some other way, without relying on functionality in the chart.

[pan3793]

@chgl are there examples of some projects that have similar cases?

[chgl]

The ones that I am aware of:

• https://github.com/bitnami/charts/blob/main/bitnami/zookeeper/values.yaml#L203-L214, - https://github.com/bitnami/charts/blob/main/bitnami/postgresql/values.yaml#L293-L309.
• Superset: https://github.com/apache/superset/blob/master/helm/superset/values.yaml#L135 (fairly extreme since entire python snippets are inlined) with https://github.com/apache/superset/blob/master/helm/superset/values.yaml#L50 as an option to reference an existing secret.
• akhq: https://github.com/tchiotludo/akhq/blob/dev/helm/akhq/values.yaml#L36-L46

[dnskr]

Closed in favor to:

• [Improvement] Define a general Helm chart configuration approach #6123
• [K8S][HELM] Implement new configuration approach #6521