Add Slurm autoscaling

.github/workflows/publish-helm-chart.yml

@@ -1,37 +1,26 @@
-name: Release Charts
-
-on:
-  push:
-    branches:
-      - main
-
+name: Publish charts
+# Run the tasks on every push
+on: push
 jobs:
-  release:
-    # depending on default permission settings for your org (contents being read-only or read-write for workloads), you will have to add permissions
-    # see: https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token
-    permissions:
-      contents: write
+  publish_charts:
+    name: Build and push Helm charts
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout
-        uses: actions/checkout@v3
+      - name: Check out the repository
+        uses: actions/checkout@v2
         with:
+          # This is important for the semver action to work correctly
+          # when determining the number of commits since the last tag
           fetch-depth: 0
+          submodules: true
 
-      - name: Configure Git
-        run: |
-          git config user.name "$GITHUB_ACTOR"
-          git config user.email "[email protected]"
-
-      - name: Install Helm
-        uses: azure/setup-helm@v3
-        env:
-          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
+      - name: Get SemVer version for current commit
+        id: semver
+        uses: stackhpc/github-actions/semver@master
 
-      - name: Run chart-releaser
-        uses: helm/[email protected]
+      - name: Publish Helm charts
+        uses: stackhpc/github-actions/helm-publish@master
         with:
-          charts_dir: .
-        env:
-          CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
-
+          token: ${{ secrets.GITHUB_TOKEN }}
+          version: ${{ steps.semver.outputs.version }}
+          app-version: ${{ steps.semver.outputs.short-sha }}

.gitignore

@@ -0,0 +1,3 @@
+# Build artifacts from local helm install
+slurm-cluster-chart/Chart.lock
+slurm-cluster-chart/charts/

README.md

@@ -1,8 +1,7 @@
 # Slurm Docker Cluster
 
-This is a multi-container Slurm cluster using Kubernetes.  The Helm chart
-creates a named volume for persistent storage of MySQL data files as well as
-an NFS volume for shared storage.
+This is a multi-container Slurm cluster using Kubernetes. The Slurm cluster Helm chart creates a named volume for persistent storage of MySQL data files. By default, it also installs the
+RookNFS Helm chart (also in this repo) to provide shared storage across the Slurm cluster nodes.
 
 ## Dependencies
 
@@ -27,51 +26,59 @@ The Helm chart will create the following named volumes:
 
 * var_lib_mysql     ( -> /var/lib/mysql )
 
-A named ReadWriteMany (RWX) volume mounted to `/home` is also expected, this can be external or can be deployed using the scripts in the `/nfs` directory (See "Deploying the Cluster")
+A named ReadWriteMany (RWX) volume mounted to `/home` is also expected, this can be external or can be deployed using the provided `rooknfs` chart directory (See "Deploying the Cluster").
 
 ## Configuring the Cluster
 
-All config files in `slurm-cluster-chart/files` will be mounted into the container to configure their respective services on startup. Note that changes to these files will not all be propagated to existing deployments (see "Reconfiguring the Cluster").
-Additional parameters can be found in the `values.yaml` file, which will be applied on a Helm chart deployment. Note that some of these values will also not propagate until the cluster is restarted (see "Reconfiguring the Cluster").
+All config files in `slurm-cluster-chart/files` will be mounted into the container to configure their respective services on startup. Note that changes to these files will not all be propagated to existing deployments (see "Reconfiguring the Cluster"). Additional parameters can be found in the `values.yaml` file for the Helm chart. Note that some of these values will also not propagate until the cluster is restarted (see "Reconfiguring the Cluster").
 
 ## Deploying the Cluster
 
 ### Generating Cluster Secrets
 
 On initial deployment ONLY, run
 ```console
-./generate-secrets.sh
+./generate-secrets.sh [<target-namespace>]
 ```
-This generates a set of secrets. If these need to be regenerated, see "Reconfiguring the Cluster"
+This generates a set of secrets in the target namespace to be used by the Slurm cluster. If these need to be regenerated, see "Reconfiguring the Cluster"
+
+Be sure to take note of the Open Ondemand credentials, you will need them to access the cluster through a browser
 
 ### Connecting RWX Volume
 
-A ReadWriteMany (RWX) volume is required, if a named volume exists, set `nfs.claimName` in the `values.yaml` file to its name. If not, manifests to deploy a Rook NFS volume are provided in the `/nfs` directory. You can deploy this by running
-```console
-./nfs/deploy-nfs.sh
-```
-and leaving `nfs.claimName` as the provided value.
+A ReadWriteMany (RWX) volume is required for shared storage across cluster nodes. By default, the Rook NFS Helm chart is installed as a dependency of the Slurm cluster chart in order to provide a RWX capable Storage Class for the required shared volume. If the target Kubernetes cluster has an existing storage class which should be used instead, then `storageClass` in `values.yaml` should be set to the name of this existing class and the RookNFS dependency should be disabled by setting `rooknfs.enabled = false`. In either case, the storage capacity of the provisioned RWX volume can be configured by setting the value of `storage.capacity`.
+
+See the separate RookNFS chart [values.yaml](./rooknfs/values.yaml) for further configuration options when using the RookNFS to provide the shared storage volume.
 
 ### Supplying Public Keys
 
 To access the cluster via `ssh`, you will need to make your public keys available. All your public keys from localhost can be added by running
 
 ```console
-./publish-keys.sh
+./publish-keys.sh [<target-namespace>]
 ```
+where `<target-namespace>` is the namespace in which the Slurm cluster chart will be deployed (i.e. using `helm install -n <target-namespace> ...`). This will create a Kubernetes Secret in the appropriate namespace for the Slurm cluster to use. Omitting the namespace arg will install the secrets in the default namespace.
 
 ### Deploying with Helm
 
 After configuring `kubectl` with the appropriate `kubeconfig` file, deploy the cluster using the Helm chart:
 ```console
 helm install <deployment-name> slurm-cluster-chart
 ```
+
+NOTE: If using the RookNFS dependency, then the following must be run before installing the Slurm cluster chart
+```console
+helm dependency update slurm-cluster-chart
+```
+
 Subsequent releases can be deployed using:
 
 ```console
 helm upgrade <deployment-name> slurm-cluster-chart
 ```
 
+Note: When updating the cluster with `helm upgrade`, a pre-upgrade hook will prevent upgrades if there are running jobs in the Slurm queue. Attempting to upgrade will set all Slurm nodes to `DRAINED` state. If an upgrade fails due to running jobs, you can undrain the nodes either by waiting for running jobs to complete and then retrying the upgrade or by manually undraining them by accessing the cluster as a privileged user. Alternatively you can bypass the hook by running `helm upgrade` with the `--no-hooks` flag (may result in running jobs being lost)
+
 ## Accessing the Cluster
 
 Retrieve the external IP address of the login node using:
@@ -128,6 +135,7 @@ srun singularity exec docker://ghcr.io/stackhpc/mpitests-container:${MPI_CONTAIN
 ```
 
 Note: The mpirun script assumes you are running as user 'rocky'. If you are running as root, you will need to include the --allow-run-as-root argument
+
 ## Reconfiguring the Cluster
 
 ### Changes to config files
@@ -171,3 +179,5 @@ and then restart the other dependent deployments to propagate changes:
 ```console
 kubectl rollout restart deployment slurmd slurmctld login slurmdbd
 ```
+
+# Known Issues

image/Dockerfile

@@ -9,12 +9,17 @@ LABEL org.opencontainers.image.source="https://github.com/stackhpc/slurm-docker-
 ARG SLURM_TAG=slurm-23.02
 ARG GOSU_VERSION=1.11
 
+COPY kubernetes.repo /etc/yum.repos.d/kubernetes.repo
+
 RUN set -ex \
     && yum makecache \
     && yum -y update \
     && yum -y install dnf-plugins-core epel-release \
     && yum -y install dnf-plugins-core \
     && yum config-manager --set-enabled powertools \
+    && yum -y module enable ruby:2.7 nodejs:14 \
+    && yum -y install https://yum.osc.edu/ondemand/2.0/ondemand-release-web-2.0-1.noarch.rpm \
+    && yum -y module install ruby nodejs \
     && yum -y install \
        wget \
        bzip2 \
@@ -42,6 +47,8 @@ RUN set -ex \
        hwloc-devel \
        openssh-server \
        apptainer \
+       ondemand \
+       kubectl \
     && yum clean all \
     && rm -rf /var/cache/yum
 
@@ -93,6 +100,7 @@ RUN mkdir /etc/sysconfig/slurm \
 
 VOLUME /etc/slurm
 COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
+COPY --chown=slurm:slurm --chmod=744 k8s-slurmd-* /usr/local/bin/
 ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
 
 CMD ["slurmdbd"]

image/docker-entrypoint.sh

@@ -48,7 +48,7 @@ then
     done
     echo "-- slurmdbd is now active ..."
 
-    echo "---> Setting permissions for state directory ..."
+    echo "---> Setting ownership for state directory ..."
     chown slurm:slurm /var/spool/slurmctld
 
     echo "---> Starting the Slurm Controller Daemon (slurmctld) ..."
@@ -86,6 +86,8 @@ then
     chown root:root /home
     chmod 755 /home
 
+    echo "---> Setting up ssh for user"
+
     mkdir -p /home/rocky/.ssh
     cp /tmp/authorized_keys /home/rocky/.ssh/authorized_keys
 
@@ -99,25 +101,75 @@ then
     done
     popd > /dev/null
 
+    echo "---> Complete"
     echo "---> Starting sshd"
-    ssh-keygen -A
+    cp /tempmounts/etc/ssh/* /etc/ssh/
+    chmod 600 /etc/ssh/ssh_host_dsa_key
+    chmod 600 /etc/ssh/ssh_host_ecdsa_key
+    chmod 600 /etc/ssh/ssh_host_ed25519_key
+    chmod 600 /etc/ssh/ssh_host_rsa_key
     /usr/sbin/sshd
 
-    start_munge --foreground
+    start_munge
+
+    echo "---> Setting up self ssh capabilities for OOD"
+
+    if [ -f /home/rocky/.ssh/id_rsa.pub ]; then
+        echo "ssh keys already found"
+    else
+            ssh-keygen -t rsa -f /home/rocky/.ssh/id_rsa -N ""
+            chown rocky:rocky /home/rocky/.ssh/id_rsa /home/rocky/.ssh/id_rsa.pub
+    fi
+
+    ssh-keyscan localhost > /etc/ssh/ssh_known_hosts
+    echo "" >> /home/rocky/.ssh/authorized_keys #Adding newline to avoid breaking authorized_keys file
+    cat /home/rocky/.ssh/id_rsa.pub >> /home/rocky/.ssh/authorized_keys
+
+    echo "---> Starting Apache Server"
+
+    # mkdir --parents /etc/ood/config/apps/shell
+    # env > /etc/ood/config/apps/shell/env
+
+    /usr/libexec/httpd-ssl-gencerts
+    /opt/ood/ood-portal-generator/sbin/update_ood_portal
+    mkdir --parents /opt/rh/httpd24/root/etc/httpd/
+
+    /usr/bin/htdbm -cb /opt/rh/httpd24/root/etc/httpd/.htpasswd.dbm rocky $ROCKY_OOD_PASS
+    /usr/sbin/httpd -k start -X -e debug
 
 elif [ "$1" = "check-queue-hook" ]
 then
     start_munge
 
+    scontrol update NodeName=all State=DRAIN Reason="Preventing new jobs running before upgrade"
+
     RUNNING_JOBS=$(squeue --states=RUNNING,COMPLETING,CONFIGURING,RESIZING,SIGNALING,STAGE_OUT,STOPPED,SUSPENDED --noheader --array | wc --lines)
 
     if [[ $RUNNING_JOBS -eq 0 ]]
     then
-            exit 0
+        exit 0
     else
-            exit 1
+        exit 1
     fi
 
+elif [ "$1" = "undrain-nodes-hook" ]
+then
+    start_munge
+    scontrol update NodeName=all State=UNDRAIN
+    exit 0
+
+elif [ "$1" = "generate-keys-hook" ]
+then
+    mkdir -p ./temphostkeys/etc/ssh
+    ssh-keygen -A -f ./temphostkeys
+    kubectl create secret generic host-keys-secret \
+    --dry-run=client \
+    --from-file=./temphostkeys/etc/ssh \
+    -o yaml | \
+    kubectl apply -f -
+
+    exit 0
+
 elif [ "$1" = "debug" ]
 then
     start_munge --foreground

image/k8s-slurmd-create

@@ -0,0 +1,16 @@
+#!/usr/bin/bash
+
+echo "$(date) Resume invoked $0 $*" &>> /var/log/slurm/power_save.log
+
+APISERVER=https://kubernetes.default.svc
+SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
+NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
+TOKEN=$(cat ${SERVICEACCOUNT}/token)
+CACERT=${SERVICEACCOUNT}/ca.crt
+
+hosts=$(scontrol show hostnames $1) # this is purely a textual expansion, doens't depend on defined nodes
+for host in $hosts
+do
+   ( sed s/SLURMD_NODENAME/$host/ /etc/slurm/slurmd-pod-template.yml | \
+   kubectl --server $APISERVER --token $TOKEN --certificate-authority $CACERT create -f - )
+done

image/k8s-slurmd-delete

@@ -0,0 +1,15 @@
+#!/usr/bin/bash
+
+echo "$(date) Suspend invoked $0 $*" >> /var/log/slurm/power_save.log
+
+APISERVER=https://kubernetes.default.svc
+SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
+NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
+TOKEN=$(cat ${SERVICEACCOUNT}/token)
+CACERT=${SERVICEACCOUNT}/ca.crt
+
+hosts=$(scontrol show hostnames $1) # this is purely a textual expansion, doens't depend on defined nodes
+for host in $hosts
+do
+   kubectl --server $APISERVER --token $TOKEN --certificate-authority $CACERT delete pod $host
+done

image/kubernetes.repo

@@ -0,0 +1,6 @@
+[kubernetes]
+name=Kubernetes
+baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
+enabled=1
+gpgcheck=1
+gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

publish-keys.sh

@@ -1,3 +1,8 @@
-kubectl create configmap authorized-keys-configmap \
+NAMESPACE="$1"
+if [[ -z $1 ]]; then
+    NAMESPACE=default
+fi
+echo Installing in namespace $NAMESPACE
+kubectl -n $NAMESPACE create configmap authorized-keys-configmap \
 "--from-literal=authorized_keys=$(cat ~/.ssh/*.pub)" --dry-run=client -o yaml | \
-kubectl apply -f -
+kubectl  -n $NAMESPACE apply -f -