hit example in the latest commit on the next branch

[makrandak]

Hello,

I am trying to run the hit example in the latest commit on the next branch. I am getting an error in setup of pMG

estimating maxEigenvalue ... Aborting in /home/mani/khanwale/projects/nekrs_next/src/elliptic/MG/ellipticMultiGridLevelSetup.cpp:eigenValue
invalid matrix entries!

Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

I did not change the example at all. So should be reproducible from the files in the examples folder.

I am not sure if it is a compilation issue or not, the code does not run on multiple gpus either. UCX module is loaded.

Thank you very much for your help and time.

With sincere regards,
Makrand

[yslan]

Hi,

Can you show what's the kernel version the code uses for Ax?

grep "^Ax: " logfile

[makrandak]

This is the output of the grep

Ax: N=7 wordSize=64 GDOF/s=6.46739 GB/s=617.853 GFLOPS/s=1081.24 constCoeff=1 poisson=1 kernelVer=8
Ax: N=7 wordSize=32 GDOF/s=10.8674 GB/s=519.099 GFLOPS/s=1816.85 constCoeff=1 poisson=1 kernelVer=8
Ax: N=3 wordSize=32 GDOF/s=6.11646 GB/s=463.945 GFLOPS/s=927.889 constCoeff=1 poisson=1 kernelVer=8
Ax: Ndim=3 N=7 wordSize=64 GDOF/s=9.69398 GB/s=578.813 GFLOPS/s=1693.03 constCoeff=0 poisson=0 kernelVer=2

Thanks for your help and time.

With sincere regards,
Makrand

[yslan]

Thanks!
kernelVer = 8 is the new kernel we just added last week. Suspecting a bug. We will look into this.

As a workaround, you could change this to 8 to avoid it.

nekRS/src/bench/axHelm/benchmarkAx.cpp

Line 141 in 4f87e0e

const int Nkernels = 8;

[stgeke]

I disabled the recently added Ax kernels. Just checkout latest next.

[makrandak]

It works now for a single GPU. I still get the following crash for multiple GPU nodes. UCX module is loaded, and multi-node GPU runs work for the master branch which I have modified with the same UCX module.

Error with multiple GPUs

default FP precision: 64
MPI tasks: 2

Initializing device

---[ Error ]--------------------------------------------------------------------
    File     : /home/mani/khanwale/projects/nekrs_next/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 31
    Function : device
    Message  : Device: Creating Device
               CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE
    Stack
      12 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      11 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      10 occa::cuda::device::device(occa::json const&)
       9 occa::cuda::cudaMode::newDevice(occa::json const&)
       8 occa::device::setup(occa::json const&)
       7 occa::device::setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
       6 device_t::device_t(setupAide&, comm_t&)
       5 platform_t::platform_t(setupAide&, int, int)
       4 nekrs::setup(int, int, int, int, int, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int)
       3 /home/mani/khanwale/.local/nekrs_next/bin/nekrs(main+0x10fa)
       2 /lib64/libc.so.6(__libc_start_main+0xe5)
       1 /home/mani/khanwale/.local/nekrs_next/bin/nekrs(_start+0x2e)
================================================================================

Abort(1) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1

Thank you very much for looking into this.

With sincere regards,
Makrand

[yslan]

Hi Makrand,

Can you post how you submit the job for 2 nodes?

It could also be unlucky event that one of the node has issues.
Given that you only submit 2 nodes, I'd suggest to run nvidia-smi from submit script to check the device is indeed active and visible.

The new next branch has new submit scripts.
However, we didn't change the way nekrs runs the jobs. It should be a minimal effort to port script to a new version.

[makrandak]

I am in an interactive job, so I am just submitting the following command with the understanding that each mpi process binds to 1 GPU. The command I use is
mpirun -np 2 ~/.local/nekrs_next/bin/nekrs --setup hit.par > log.txt 2>&1 &

for 2 nodes (each containing 1 gpu).

Let me know if I am mistaken.

Thank you very much for your help.

With sincere regards,
Makrand

[stgeke]

You need to ensure that each MPI process binds to its own GPU using the logical ID 0. One way to achieve this is by using the environment variable CUDA_VISIBLE_DEVICES.