Compiling and running NekRS on TACC's Frontera.

[Avino774]

Hi neks,

I am new to nekRS. I recently installed nekRS on TACC's Frontera using the intel mpi compiler as described in the CMakeOutput.log file:
C compiler : /opt/intel/oneapi/mpi/2021.9.0/bin/mpicc
C++ compiler : /opt/intel/oneapi/mpi/2021.9.0/bin/mpicxx
Fortran compiler : /opt/intel/oneapi/mpi/2021.9.0/bin/mpif77
The Fortran compiler identification is IntelLLVM.

I have run into some issues when using NekRS on Frontera, and I try to explain the problem as clearly as I understand it.

=========== nrspre and nrsmpi ===================

After successful install, I try to run turbChannel from one of the examples. I replaced 'mpirun' in the nrspre and nrsmpi script in ~/.local/nekrs/bin to the following ( mpirun is not recommended on TACC ):

#For Frontera only, nrspre script:
ibrun -np 1 $NEKRS_HOME/bin/nekrs --setup $1 --build-only $2

#Frontera only, nrsmpi script:
ibrun -np $2 $NEKRS_HOME/bin/nekrs --setup $1 ${@:3}

It would be great if I could have NekRS generate an executable file(.x) as in nek5000. However, nrspre does not work and throws out an error.

=================== Using nrsmpi + OCCA information ===================

For turbChannel examples, I run the following command inside a bash slurm script:
nrsmpi turbChannel 1

Using nrsmpi gives success when using one/two tasks yields successful run (serial backend). But increasing the no. of mpitasks gives a Segmentaion fault error similar to above. I find that the OCCA backend may be the problem, since the run crashes after printing out 'Initializng device'.
There are 3 modes to OCCA available on Frontera($ occa mode). I have tried to manually set the OCCA backend in the par file for each run:

• dpcpp ( Segmentation fault error )
• OpenMP ( have not tried )
• Serial ( nrsmpi works, with mpi tasks = 1/2; 3 or more : Segmentation fault error )

This is the output of $occa env:
Basic:
- OCCA_DIR : [NOT SET]
- OCCA_CACHE_DIR : [NOT SET]
- OCCA_VERBOSE : [NOT SET]
- OCCA_UNSAFE : 0
Makefile:
- CXX : [NOT SET]
- CXXFLAGS : [NOT SET]
- CC : [NOT SET]
- CFLAGS : [NOT SET]
- FC : [NOT SET]
- FCFLAGS : [NOT SET]
- LDFLAGS : [NOT SET]
Backend Support:
- OCCA_OPENMP_ENABLED : 1
- OCCA_CUDA_ENABLED : 0
- OCCA_HIP_ENABLED : 0
- OCCA_OPENCL_ENABLED : 0
- OCCA_DPCPP_ENABLED : 1
- OCCA_METAL_ENABLED : 0
Run-Time Options:
- OCCA_CXX : [NOT SET]
- OCCA_CXXFLAGS : [NOT SET]
- OCCA_CC : [NOT SET]
- OCCA_CFLAGS : [NOT SET]
- OCCA_LDFLAGS : [NOT SET]
- OCCA_COMPILER_SHARED_FLAGS : [NOT SET]
- OCCA_INCLUDE_PATH : [NOT SET]
- OCCA_LIBRARY_PATH : [NOT SET]
- OCCA_KERNEL_PATH : [NOT SET]
- OCCA_OPENCL_COMPILER_FLAGS : [NOT SET]
- OCCA_DPCPP_COMPILER : [NOT SET]
- OCCA_DPCPP_COMPILER_FLAGS : [NOT SET]
- OCCA_CUDA_COMPILER : [NOT SET]
- OCCA_CUDA_COMPILER_FLAGS : [NOT SET]
- OCCA_HIP_COMPILER : [NOT SET]
- OCCA_HIP_COMPILER_FLAGS : [NOT SET]

This is the output of $occa info :

========+======================+=================================================
 CPU(s) | Processor Name       | Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz    
        | Memory               | 187.22 GB                                       
        | Clock Frequency      | 4 GHz                                           
        | SIMD Instruction Set | SSE2                                            
        | SIMD Width           | 128 bits                                        
        | L1d Cache Size       |    32 KB                                        
        | L1i Cache Size       |    32 KB                                        
        | L2 Cache Size        |     1 MB                                        
        | L3 Cache Size        | 38.50 MB                                        
========+======================+=================================================
 dpcpp  | Platform 0           | Intel(R) FPGA Emulation Platform for OpenCL(TM) 
        |----------------------+-------------------------------------------------
        | Device 0             | Intel(R) FPGA Emulation Device                  
        | Device Type          | accelerator                                     
        | Compute Cores        | 112                                             
        | Global Memory        | 187.22 GB                                       
        | Local Memory         | 256 KB                                          
        |----------------------+-------------------------------------------------
        | Platform 1           | Intel(R) OpenCL                                 
        |----------------------+-------------------------------------------------
        | Device 0             | Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz    
        | Device Type          | cpu                                             
        | Compute Cores        | 112                                             
        | Global Memory        | 187.22 GB                                       
        | Local Memory         | 32 KB                                           

==============================================

This problem persists when running other examples as well. I read in one of the discussion pages (#517) that LLVM is not presently supported. Is this the problem here too?

[stgeke]

I suggest to use the serial backend and gcc compilers.

[Avino774]

Hi stgeke,

Thanks for your reply.

1. Using g++ compiler, serial backend instead of intel and 'ibrun' instead of mpirun in the nrsmpi script. Using Frontera's gcc module, there seems to be shared librabries nekRS cannot find.

error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory

Loading other available gcc modules on Frontera such as gcc/9.1.0 also gives the same error.

=============

2. In a fresh environment, I performed a new install of nekRS using gcc module on Frontera:

CC=gcc CXX=g++ FC=gfortran ./nrsconfig -DCMAKE_INSTALL_PREFIX=$HOME/.local/nekrs

This is a summary of the build:

----------------- Summary -----------------
Installation directory: /home1/08172/avinodh/.local/nekrs
C compiler: /opt/apps/gcc/12.2.0/bin/gcc
C++ compiler: /opt/apps/gcc/12.2.0/bin/g++
Fortran compiler: /opt/apps/gcc/12.2.0/bin/gfortran
Default backend : DPCPP
CPU backend compiler: /opt/apps/gcc/12.2.0/bin/g++ (flags: -w -O3 -g -march=native -mtune=native -ffast-math)
DPCPP backend enabled (flags: -w -O3 -fsycl -fdebug-info-for-profiling -gline-tables-only -ffp-contract=fast)
GPU aware MPI support: ON
---------------------------------------------

However, there seems to be an issue install with MPI_fortran, (in CMakeError.log) and the installation breaks:
The MPI test test_mpi for Fortran in mode F90_MODULE failed to compile.

[stgeke]

You have to pass the correct MPI wrappers (using gcc) to nrsconfig