Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[K8S][HELM] Add Hadoop configuration support #6046

Closed
wants to merge 1 commit into from
Closed

[K8S][HELM] Add Hadoop configuration support #6046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions charts/kyuubi/templates/kyuubi-configmap-hadoop.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{{/*
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/}}

apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-hadoop
labels:
{{- include "kyuubi.labels" . | nindent 4 }}
data:
{{- with .Values.hadoopConf.coreSite }}
core-site.xml: |
{{- tpl . $ | nindent 4 }}
{{- end }}
{{- with .Values.hadoopConf.hdfsSite }}
hdfs-site.xml: |
Copy link
Member

@pan3793 pan3793 Feb 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, it's likely to put additional configuration files under HADOOP_CONF_DIR in practice, for example, hive-site.xml, hbase-site.xml, so we may want to allow the user to inject custom files.

Another concern is that embeding large-XML into YAML may be not user-friendly, it's easy to mess up the indent.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely agree. The approach I often see is providing both options: either provide a (templated) inline config - which as you said can get unwieldy fast - or provide the name of an existing config map. The latter allows creating that config map from a file or some other way, without relying on functionality in the chart.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chgl are there examples of some projects that have similar cases?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ones that I am aware of:

{{- tpl . $ | nindent 4 }}
{{- end }}
0 ...ubi/templates/kyuubi-spark-configmap.yaml → ...ubi/templates/kyuubi-configmap-spark.yaml
View file
Open in desktop
File renamed without changes.
8 changes: 8 additions & 0 deletions charts/kyuubi/templates/kyuubi-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ spec:
{{- include "kyuubi.selectorLabels" . | nindent 8 }}
annotations:
checksum/conf: {{ include (print $.Template.BasePath "/kyuubi-configmap.yaml") . | sha256sum }}
checksum/conf-hadoop: {{ include (print $.Template.BasePath "/kyuubi-configmap-hadoop.yaml") . | sha256sum }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets: {{- toYaml . | nindent 8 }}
Expand All @@ -65,6 +66,8 @@ spec:
env:
- name: KYUUBI_CONF_DIR
value: {{ .Values.kyuubiConfDir }}
- name: HADOOP_CONF_DIR
value: {{ .Values.hadoopConfDir }}
- name: SPARK_CONF_DIR
value: {{ .Values.sparkConfDir }}
{{- with .Values.env }}
Expand Down Expand Up @@ -110,6 +113,8 @@ spec:
volumeMounts:
- name: conf
mountPath: {{ .Values.kyuubiConfDir }}
- name: conf-hadoop
mountPath: {{ .Values.hadoopConfDir }}
- name: conf-spark
mountPath: {{ .Values.sparkConfDir }}
{{- with .Values.volumeMounts }}
Expand All @@ -122,6 +127,9 @@ spec:
- name: conf
configMap:
name: {{ .Release.Name }}
- name: conf-hadoop
configMap:
name: {{ .Release.Name }}-hadoop
- name: conf-spark
configMap:
name: {{ .Release.Name }}-spark
Expand Down
21 changes: 21 additions & 0 deletions charts/kyuubi/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,27 @@ kyuubiConf:
# See example at conf/log4j2.xml.template https://kyuubi.readthedocs.io/en/master/configuration/settings.html#logging for more details
log4j2: ~

# $HADOOP_CONF_DIR directory
hadoopConfDir: /opt/hadoop/conf
# Hadoop configuration files
hadoopConf:
# The value (templated string) is used for core-site.xml file
# See https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml and Hadoop documentation for more details
coreSite: ~
# coreSite: |
# <?xml version="1.0"?>
# <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
# <configuration>
# <property>
# <name>hadoop.tmp.dir</name>
# <value>/tmp/hadoop-${user.name}</value>
# </property>
# </configuration>

# The value (templated string) is used for hdfs-site.xml file
# See https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml and Hadoop documentation for more details
hdfsSite: ~

# $SPARK_CONF_DIR directory
sparkConfDir: /opt/spark/conf
# Spark configuration files
Expand Down
Loading