[Manual WIP] Devito on Jade2

Needs update

Installing Devito on Jade2

Before you start, important readings:
https://docs.jade.ac.uk/en/latest/

# After completing the registration
# Do `ssh` to your login node (private key needed)
ssh -l your_username jade2.hartree.stfc.ac.uk

# Create a python3 virtual env and activate it
module load python/3.8.6

Load env nvidia-hpc sdk

module load use.dev
# This one works best
module load nvidia-hpc-sdk/22.11-cuda11.8

#-bash-4.2$ module load nvidia-hpc-sdk/21.9-cuda11.0
#	GCC 5.4.0 environment now loaded
#
#	NVIDIA-HPC-SDK 21.9 with CUDA-11.0, OpenMPI-4.0.5 and GCC-5.4.0 loaded
# [ggb92-axf03@dgk328 ~]$ module load nvidia-hpc-sdk/22.11-cuda11.8 
#	NVIDIA-HPC-SDK 21.9 with CUDA-11.0 and OpenMPI-4.0.5 loaded
# [ggb92-axf03@dgk328 ~]$ 


module load gcc

#(python38-env) -bash-4.2$ module load gcc
#	GCC 9.1.0 environment now loaded
#The following have been reloaded with a version change:
#  1) gcc/5.4.0 => gcc/9.1.0

# pip3 install --ignore-installed --no-cache-dir -r requirements-mpi.txt

# If devito is not cloned:
git clone https://github.com/devitocodes/devito
pip3 install --upgrade -e .
# Install mpi4py with HPC SDK MPI, takes a few minutes possibly
CC=nvc CFLAGS="-noswitcherror -tp=px" pip install --ignore-installed --no-cache-dir mpi4py

# ch-dir to work filesystem
# may look sth like: /jmain02/home/J2AD015/axfxx/xxgxxx-axfxx/devito

Then you should be able to run a single-node kernel:

Interactive:

salloc --partition=devel --gres=gpu:1
srun --partition=devel --gres=gpu:1 --pty /bin/bash

Let's go single node multi-gpu:

salloc --partition=devel --gres=gpu:2
srun --partition=devel --gres=gpu:2 --pty /bin/bash

Trying:

DEVITO_MPI=1 DEVITO_ARCH=nvc++ DEVITO_LOGGING=DEBUG DEVITO_PLATFORM=nvidiaX DEVITO_LANGUAGE=openacc mpirun -n 2 python examples/seismic/acoustic/acoustic_example.py -d 124 124 124 --tn 100

srun --job-name "test-interactive" --gres=gpu:1 --pty bash -i

We cannot find stddef.h

module load llvm