Using MIT's Satori cluster

[glwagner]

This top-level post is a WIP.

Here's a description of MIT's Satori cluster: https://mit-satori.github.io/

Example snippet from a slurm script:

#!/bin/bash
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=16
#SBATCH --mem=100GB
#SBATCH --time 24:00:00

# Upload modules
module purge all
module add spack
module add cuda/11.4
module load openmpi/3.1.6-cuda-pmi-ucx-slurm-jhklron

# MPI specific exports
export OMPI_MCA_pml=^ucx
export OMPI_MCA_osc=^ucx
export OMPI_MCA_btl_openib_allow_ib=true

# Julia specific enviromental variables
export COMMON="/nobackup/users/ssilvest/perlmutter-test"
export JULIA_DEPOT_PATH=":${COMMON}/depot"
export JULIA_LOAD_PATH="${JULIA_LOAD_PATH}:$(pwd)/satori"
export JULIA_CUDA_MEMORY_POOL=none
export JULIA=julia

# Profile specific variable
export JULIA_NVTX_CALLBACKS=gc

# Number of threads in SLURM mode
export JULIA_NUM_THREADS=${SLURM_CPUS_PER_TASK:=1}

cat > launch.sh << EoF_s
#! /bin/sh
export CUDA_VISIBLE_DEVICES=0,1,2,3
exec \$*
EoF_s
chmod +x launch.sh

if $PROFILE; then
   NSYS="nsys profile --trace=nvtx,cuda,mpi --output=${COMMON}/report_N${SLURM_JOB_NUM_NODES}_R${RESOLUTION}_${PRECISION}"
fi

$NSYS srun --mpi=pmi2 ./launch.sh $JULIA --check-bounds=no --project experiments/run.jl ${RESOLUTION:=3}

[ali-ramadhan]

Is it worth mentioning that Satori is a PowerPC cluster and that

CUDA.jl v5.3 is the last version with support for PowerPC (removed in v5.4)

according to the CUDA.jl README?

I haven't run Julia or Oceananigans.jl on Satori in a while but I remember seeing some related warnings, although my code did run. So maybe it'll all be fine as long as long as Oceananigans.jl is compatible with CUDA.jl v5.3?

[glwagner]

Everything is worth mentioning! That's helpful. I'm not sure what "support" means either. I report back if I find more info either way.