Are the dependency constraints on Pytorch still applicable?

[bastonero]

Dear Developers,
Thanks first of all for the amazing work, it is really useful and easy to use.

I am trying to install Nequip on LUMI-G, where they suggest to install Python packages via Singularity containers. Unfortunately, the ROCM version of those is already greater than what is compatible for Pytorch 1.10.0 < version < 1.12.0. It would be great to know if these constaints are still applicable, or you already did some tests (e.g. using Pytorch 2.x).

Thanks again for any help.

[Linux-cpp-lisp]

Hi @bastonero ,

Thank you, glad to hear!

I think it has mostly been established that newer PyTorch works fine for AMD GPUs, for which some relevant threads are:

• Training slowdown #311
• Allegro LAMMPS on AMD GPUs pair_allegro#23
• ERROR: CUDA_tensor_histogram failed to get free global memory when using `nequip-train` #402

The newest versions may also be fine on CUDA, but that has not been completely confirmed yet. 1.* versions that are not 1.11 should be avoided. If you encounter this first issue, or find anything that works (or doesn't work) unexpectedly, please post that info here!

[henriasv]

Hi, on LUMI-G, there is already a container available for nequip and pair_allegro. The procedure is supposed to be the following:

wget https://462000008.lumidata.eu/easyconfigs/LAMMPS-stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303.eb
Then, for example (or wherever it is convenient to store builds:
export EBU_USER_PREFIX=/project/{your_project}/EasyBuild

Then:

module load LUMI/23.09
module load partition/container
module load EasyBuild-user

eb LAMMPS-stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303.eb

Now you can do (typically before nequip-train or lmp -in ... in your job script):

module purge
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

With these modules loaded, commands nequip-train, equip-evaluate, nequip-deploy, nequip-benchmark and lmp (lammps with pair allegro) are available.

Typical job script using lammps:

#!/bin/bash
#SBATCH --job-name=lammps
#SBATCH --time=0-04:00:00
#SBATCH --mem-per-cpu=8000
#SBATCH --cpus-per-task=4
#SBATCH --ntasks=1
#SBATCH --account={your_project}
#SBATCH --gpus=1
#SBATCH --partition=small-g

# Load modules
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

# Setup cpu-binding
# This part is only relevant when running with multiple GPUS. Make sure to ask for nodes, not just many gpus
CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"

# Run program
srun lmp -in myrun.inp > stdout.out 2> stderr.out

Please let me know whether this works, and whether you are experiencing any problems. Need to write up this procedure for my team as well, so bonus if you can check whether it works.

[ozakary]

Hello @bastonero and @henriasv, apologies for the late question on this thread. I attempted to run LAMMPS with pair_allegro on LUMI-G following the procedure provided above by @henriasv (the container available for nequip and pair_allegro) and got the following error in the stderr.out file:

lmp: error while loading shared libraries: libamdhip64.so.6: cannot open shared object file: No such file or directory srun: error: nid005002: task 0: Exited with exit code 127 srun: Terminating StepId=8585979.0

The command lines used are:

export EBU_USER_PREFIX=/projappl/project_<project_ID>/EasyBuild

module load LUMI/23.09
module load partition/container
module load EasyBuild-user

eb LAMMPS-stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303.eb

module purge
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

I hope I didn't miss any command.

Your help in this issue will be greatly appreciated.

[henriasv]

Hi, the container installation has changed after the maintenance stop in September. I'll get back to you with updated instructions

[henriasv]

Hi, I have put a container recipe on GitHub: https://github.com/henriasv/allegro-nequip-installation-lumi
This version should work on the current software stack on LUMI.

[ozakary]

Thank you for providing this container recipe; it works perfectly both on a single GPU and across multiple GPUs.

Does this recipe need to be updated for future changes in the LUMI software stack?

[henriasv]

Hi, in general yes, so a small update of the recipe each time rocm is updated on LUMI, typically once a year I think. The update was very small, but not super easy to make..