Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Helm charts for deployment. #35

Open
wants to merge 7 commits into
base: developing
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions helm/esdk/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
30 changes: 30 additions & 0 deletions helm/esdk/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
apiVersion: v2
name: esdk
description: Huawei Container Storage Interface (CSI) Driver

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# compatible Kubernetes versions, helm installation fails fail if the cluster runs an unsupported Kubernetes version
kubeVersion: ">= 1.13.0 < 1.19.3"
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "2.2.13"

home: https://github.com/Huawei/eSDK_K8S_Plugin
sources:
- https://github.com/Huawei/eSDK_K8S_Plugin
10 changes: 10 additions & 0 deletions helm/esdk/templates/huawei-csi-configmap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
kind: ConfigMap
apiVersion: v1
metadata:
name: huawei-csi-configmap
namespace: kube-system
data:
csi.json: |
{{ $length := len .Values.backends }} {{ if gt $length 0 }} { {{ end }}
"backends": {{ .Values.backends | toPrettyJson | nindent 8 }}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In our chart we decided to just dump the YAML structure from values.yaml directly into csi.json: https://github.com/adfinis-sygroup/helm-charts/blob/c23bb7d8380df256e454cc143510de60511ae2e2/charts/huawei-csi-plugin/templates/configmap.yaml#L9

Might be easier then constructing a JSON and more flexible if you would decide to add more then just the backends section in the future.

{{ $length := len .Values.backends }} {{ if gt $length 0 }} } {{ end }}
165 changes: 165 additions & 0 deletions helm/esdk/templates/huawei-csi-controller.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: huawei-csi-controller
namespace: kube-system
---
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to Helm best practices each resource definition should be defined in a separate template.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: huawei-csi-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["get", "list", "watch", "create", "delete", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots/status"]
verbs: ["update"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: huawei-csi-provisioner-role
subjects:
- kind: ServiceAccount
name: huawei-csi-controller
namespace: kube-system
roleRef:
kind: ClusterRole
name: huawei-csi-provisioner-runner
apiGroup: rbac.authorization.k8s.io

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: huawei-csi-attacher-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: huawei-csi-attacher-role
subjects:
- kind: ServiceAccount
name: huawei-csi-controller
namespace: kube-system
roleRef:
kind: ClusterRole
name: huawei-csi-attacher-runner
apiGroup: rbac.authorization.k8s.io

---
kind: Deployment
apiVersion: apps/v1
metadata:
name: huawei-csi-controller
namespace: kube-system
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoding a namespace is not recommended in Helm charts. We should discuss how to best approach the issue with the kube-system namespace being hardcoded in secretGenerate and secretUpdate.

spec:
replicas: 1
selector:
matchLabels:
app: huawei-csi-controller
template:
metadata:
labels:
app: huawei-csi-controller
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a the default template function for labels created by helm create ... would allow you to be more consistent regarding your labels and simplify the configuration of labels. See ... for how we're using the .labels and .selectorLabels from _helpers.tpl in our Helm chart: https://github.com/adfinis-sygroup/helm-charts/blob/c23bb7d8380df256e454cc143510de60511ae2e2/charts/huawei-csi-plugin/templates/daemonset-node.yaml#L7

spec:
serviceAccount: huawei-csi-controller
hostNetwork: true
containers:
- name: csi-provisioner
image: {{ .Values.images.sidecar.provisioner }}
args:
- "--csi-address=$(ADDRESS)"
- "--timeout=6h"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
imagePullPolicy: {{ .Values.imagePullPolicy }}
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/

- name: csi-attacher
image: {{ .Values.images.sidecar.attacher }}
args:
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
imagePullPolicy: {{ .Values.imagePullPolicy }}
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/
- name: huawei-csi-driver
image: {{ required "Must provide the CSI controller service container image." .Values.images.huaweiCsiControllerService }}
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--controller"
- "--containerized"
- "--backend-update-interval=120"
- "--driver-name=csi.huawei.com"
env:
- name: CSI_ENDPOINT
value: /var/lib/csi/sockets/pluginproxy/csi.sock
imagePullPolicy: {{ .Values.imagePullPolicy }}
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/
- name: log
mountPath: /var/log
- name: config-map
mountPath: /etc/huawei
- name: secret
mountPath: /etc/huawei/secret
volumes:
- name: socket-dir
emptyDir:
- name: log
hostPath:
path: /var/log/
type: Directory
- name: config-map
configMap:
name: huawei-csi-configmap
- name: secret
secret:
secretName: huawei-csi-secret

132 changes: 132 additions & 0 deletions helm/esdk/templates/huawei-csi-node.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: huawei-csi-node
namespace: kube-system

---
kind: ClusterRole
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to Helm best practices each resource definition should be defined in a separate template.

apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: huawei-csi-driver-registrar-runner
namespace: kube-system
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: huawei-csi-driver-registrar-role
subjects:
- kind: ServiceAccount
name: huawei-csi-node
namespace: kube-system
roleRef:
kind: ClusterRole
name: huawei-csi-driver-registrar-runner
apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: huawei-csi-node
namespace: kube-system
spec:
selector:
matchLabels:
app: huawei-csi-node
template:
metadata:
labels:
app: huawei-csi-node
spec:
serviceAccountName: huawei-csi-node
hostPID: true
hostNetwork: true
containers:
- name: csi-node-driver-registrar
image: {{ .Values.images.sidecar.registrar }}
args:
- "--csi-address=/csi/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/plugins/csi.huawei.com/csi.sock"
imagePullPolicy: {{ .Values.imagePullPolicy }}
volumeMounts:
- name: plugin-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: huawei-csi-driver
image: {{ required "Must provide the CSI node service node image." .Values.images.huaweiCsiNodeService }}
args:
- "--endpoint=/csi/csi.sock"
- "--containerized"
- "--driver-name=csi.huawei.com"
- "--volume-use-multipath={{ .Values.multipath.enable }}"
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
imagePullPolicy: {{ .Values.imagePullPolicy }}
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -f /csi/csi.sock"]
volumeMounts:
- name: plugin-dir
mountPath: /csi
- name: pods-dir
mountPath: /var/lib/kubelet
mountPropagation: "Bidirectional"
- name: etc-dir
mountPath: /etc
- name: log-dir
mountPath: /var/log
- name: dev-dir
mountPath: /dev
mountPropagation: "HostToContainer"
- name: iscsi-dir
mountPath: /var/lib/iscsi
- name: config-map
mountPath: /etc/huawei
- name: secret
mountPath: /etc/huawei/secret
volumes:
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins/csi.huawei.com
type: DirectoryOrCreate
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
- name: pods-dir
hostPath:
path: /var/lib/kubelet
type: Directory
- name: etc-dir
hostPath:
path: /etc
type: Directory
- name: dev-dir
hostPath:
path: /dev
type: Directory
- name: iscsi-dir
hostPath:
path: /var/lib/iscsi
- name: log-dir
hostPath:
path: /var/log/
type: Directory
- name: config-map
configMap:
name: huawei-csi-configmap
- name: secret
secret:
secretName: huawei-csi-secret

27 changes: 27 additions & 0 deletions helm/esdk/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# An array of storages with the access info
backends:
- storage: "fusionstorage-san"
name: "test-esdk-deployment"
urls:
- "https://1.2.3.4:28443"
pools:
- "StoragePool001"
parameters:
protocol: "iscsi"
portals:
- "1.2.3.4"
images:
# The image name and tag for the attacher, provisioner and registrar sidecars. These must match the appropriate Kubernetes version.
sidecar:
attacher: quay.io/k8scsi/csi-attacher:v1.2.1
provisioner: quay.io/k8scsi/csi-provisioner:v1.6.0
registrar: quay.io/k8scsi/csi-node-driver-registrar:v2.0.1
# The image name and tag for the Huawei CSI controller service container
huaweiCsiControllerService: huawei-csi:test
# The image name and tag for the Huawei CSI node service container
huaweiCsiNodeService: huawei-csi:test
# Default image pull policy for container images
imagePullPolicy: "IfNotPresent"
Copy link

@AmitRoushan AmitRoushan Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comments on each attribute for easy of use

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

multipath:
enable: false