High-Performance Kubernetes for NERSC

This is a fork from CARV-ICS-FORTH/HPK. The repo has been adapted to run with podman-hpc and Slurm, which are in use at NERSC.

The commands to install and run remain the same except for make run-kubemaster which has been replaced by make run-podman.

The Kubernetes control plane is now a new container adapted from the original.

High-Performance Kubernetes

This modified version of High-Performance Kubernetes (HPK), allows HPC users to run their own private "mini Clouds" on a typical HPC cluster, which uses a single container to run the Kubernetes control plane and a Virtual Kubelet Provider implementation to translate container lifecycle management commands from Kubernetes-native to Slurm/Podman-hpc.

In contrast to a typical Kubernetes installation at the Cloud:

• HPK uses a pass-through scheduler, which assigns all pods to the single hpk-kubelet that represents the cluster. In practice, this means that all scheduling is delegated to Slurm.
• All Kubernetes services are converted to headless. This avoids the need for internal, virtual cluster IPs that would need special handling at the network level. As a side effect, HPK services that map to multiple pods are load-balanced at the DNS level if clients support it.

HPK is a continuation of the KNoC project, a Virtual Kubelet Provider implementation that can be used to bridge Kubernetes and HPC environments.

Trying it out

Compile the hpk-kubelet using make.

make build

Then you need to start the Kubernetes Master and hpk-kubelet seperately.

To run the Kubernetes Master:

make run-podman

Once the master is up and running, you can start the hpk-kubelet:

make run-kubelet

Now you can configure and use kubectl:

export KUBE_PATH=~/.k8sfs/kubernetes/
export KUBECONFIG=${KUBE_PATH}/admin.conf
kubectl get nodes

In case that you experience DNS issues, you should retry starting the Kubernetes Master with:

export EXTERNAL_DNS=<your dns server>
make run-kubemaster

The above command will set CoreDNS to forward requests for external names to your DNS server.