OpenMP task parallelization not activated on TGCC Irene Rome

[hwlr0]

Hello!

I noticed that when running Smilei on TGCC's Irene cluster and Rome partition (I haven't tried on other partitions of this cluster) (machine=joliot_curie_rome for compilation), in the Initializing MPI part of Smilei's log, this prints:

OpenMP task parallelization not activated

My colleagues at CELIA laboratory running Smilei on the same partition have the same print. I tried recompiling Smilei differently by removing config=no_mpi_tm from the make command. This enables MPI threads but OpenMP task parallelization remains not activated. Forcing a compiler option by setting export MPICXX='mpicxx -fopenmp' had no effect. Setting export OMP_PROC_BIND=true and export OMP_PLACES=cores had no effect either.

Thus, I have three questions for you please:

Question 1: I don't know how are MPI threads related to OpenMP. The machine file recommends not enabling the former for some reason. But I could compile the code successfully when removing config=no_mpi_tm. The instruction to add this option being 3 years old, maybe it is just no longer necessary? Is it even possible for OpenMP tasks to be effective when MPI threads are not enabled?

Question 2: At CELIA, we are also wondering how things work when we have multiple threads per mpi process (64 in my case) but OpenMP is not activated. Indeed, using several MPI threads still seems to run the code faster compared to using only one, despite the non-activation of OpenMP task parallelization.

Question 3: Last but not least, do you have any hint how to solve this issue and enable OpenMP please?

Additionnal infos:

• Smilei version : 5.1-34-g60be16288-master

• I have no idea of what version of OpenMP is installed or even where to look for this information.

• Compilation environment informations (obtained by doing make env):

make: python: Command not found
/bin/sh: mpicxx: command not found
make: python: Command not found
make: python: Command not found
VERSION : 
SMILEICXX : mpicxx
OPENMP_FLAG : -fopenmp -D_OMP
HDF5_ROOT_DIR : 
FFTW3_LIB_DIR : 
/bin/sh: python: command not found
/bin/sh: python: command not found
SITEDIR : 
PYTHONEXE : python
PY_CXXFLAGS : 
PY_LDFLAGS : 
CXXFLAGS : -D__VERSION=\"\" -DOMPI_SKIP_MPICXX -std=c++14  -Isrc -Isrc/Pusher -Isrc/Python -Isrc/Radiation -Isrc/Field -Isrc/ParticleBC -Isrc/ParticleInjector -Isrc/Checkpoint -Isrc/Collisions -Isrc/MultiphotonBreitWheeler -Isrc/Ionization -Isrc/Merging -Isrc/ElectroMagn -Isrc/Interpolator -Isrc/ElectroMagnBC -Isrc/SmileiMPI -Isrc/Species -Isrc/ElectroMagnSolver -Isrc/Patch -Isrc/DomainDecomposition -Isrc/Profiles -Isrc/Projector -Isrc/Particles -Isrc/picsar_interface -Isrc/Params -Isrc/PartCompTime -Isrc/Diagnostic -Isrc/MovWindow -Isrc/Tools -Ibuild/src/Python  -O3 -g -fopenmp -D_OMP
LDFLAGS : -lhdf5  -lm -fopenmp -D_OMP
GPU_COMPILER : 
GPU_COMPILER_FLAGS : 
COMPILER_INFO : 

• Running environment informations (job log):

+ SCRIPT_PID=174017
+ /bin/bash -x /tmp/tmp.THao4k0i2U
+ set +x
+ unset _mlshdbg
+ '[' 1 = 1 ']'
+ case "$-" in
+ set +x
+ unset _mlshdbg
+ module purge
+ local _mlredir=0
+ '[' -n '' ']'
+ case " $@ " in
+ '[' 0 -eq 0 ']'
+ _module_raw purge
+ unset _mlshdbg
+ '[' 1 = 1 ']'
+ case "$-" in
+ set +x
module dfldatadir/own (Data Directory) cannot be unloaded

Unloading datadir/own <aL>
  ERROR: Dependent dfldatadir/own is loaded

Unloading datadir/celia <aL>
  ERROR: Dependent datadir/own and dfldatadir/own are loaded

Unloading ccc/1.0
  ERROR: Dependent datadir/celia and datadir/own and dfldatadir/own are loaded
+ unset _mlshdbg
+ return 1
+ module load mpi
+ local _mlredir=0
+ '[' -n '' ']'
+ case " $@ " in
+ '[' 0 -eq 0 ']'
+ _module_raw load mpi
+ unset _mlshdbg
+ '[' 1 = 1 ']'
+ case "$-" in
+ set +x
load module flavor/buildmpi/openmpi/4
load module feature/mkl/single_node
load module flavor/buildcompiler/intel/20
load module feature/openmpi/io/standard
load module feature/openmpi/mpi_compiler/intel
load module feature/openmpi/net/auto
load module flavor/ucx/standard
load module flavor/cuda/nvhpc-222
load module cuda/11.6
load module flavor/libccc_user/hwloc2
load module hwloc/2.5.0
load module pmix/4.2.2
load module licsrv/intel
load module c++/intel/20.0.0
load module c/intel/20.0.0
load module fortran/intel/20.0.0
load module feature/hcoll/multicast/enable
load module sharp/2.4.2
load module hcoll/4.7.3191
load module ucx/1.12.1
load module feature/mkl/lp64
load module feature/mkl/sequential
load module feature/mkl/vector/amd
load module mkl/20.0.0
load module intel/20.0.0
load module mpi/openmpi/4.1.4
+ unset _mlshdbg
+ return 0
+ module load hdf5
+ local _mlredir=0
+ '[' -n '' ']'
+ case " $@ " in
+ '[' 0 -eq 0 ']'
+ _module_raw load hdf5
+ unset _mlshdbg
+ '[' 1 = 1 ']'
+ case "$-" in
+ set +x
load module flavor/hdf5/serial
load module hdf5/1.12.0
+ unset _mlshdbg
+ return 0
+ module switch flavor/hdf5/serial flavor/hdf5/parallel
+ local _mlredir=0
+ '[' -n '' ']'
+ case " $@ " in
+ '[' 0 -eq 0 ']'
+ _module_raw switch flavor/hdf5/serial flavor/hdf5/parallel
+ unset _mlshdbg
+ '[' 1 = 1 ']'
+ case "$-" in
+ set +x
unload module hdf5/1.12.0
unload module flavor/hdf5/serial
load module flavor/hdf5/parallel
�[36mWARNING: the loaded flavor/hdf5/parallel does not exists, so flavor/hdf5/parallel intel--20.0.2__openmpi--4.0.1/parallel is used instead�[0m
load module hdf5/1.12.0
+ unset _mlshdbg
+ return 0
+ module load python3/3.8.10
+ local _mlredir=0
+ '[' -n '' ']'
+ case " $@ " in
+ '[' 0 -eq 0 ']'
+ _module_raw load python3/3.8.10
+ unset _mlshdbg
+ '[' 1 = 1 ']'
+ case "$-" in
+ set +x
load module python3/3.8.10
+ unset _mlshdbg
+ return 0
+ export HDF5_ROOT_DIR=/ccc/products/hdf5-1.12.0/intel--20.0.2__openmpi--4.0.1/parallel
+ HDF5_ROOT_DIR=/ccc/products/hdf5-1.12.0/intel--20.0.2__openmpi--4.0.1/parallel
+ export PYTHONEXE=python3
+ PYTHONEXE=python3
+ export PYTHONHOME=/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default
+ PYTHONHOME=/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default
+ export PATH=/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/bin:/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/bin:/ccc/products/hdf5-1.12.0/intel--20.0.2__openmpi--4.0.1/parallel/bin:/ccc/products/openmpi-4.1.4/intel--20.0.0/default/bin:/ccc/products/ucx-1.12.1/system/default/bin:/ccc/products/icc-20.0.0/system/default/20.0.0/bin/intel64:/ccc/products/pmix-4.2.2/system/default/bin:/ccc/products/hwloc-2.5.0/system/default/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/ccc/products/ccc_users_env/bin:/ccc/products/ccc_crontab/bin:/ccc/products/ccc_prod/bin:/ccc/products/ccc_visu/bin:/ccc/products/ccc_users_tools/bin:/ccc/cont003/home/celia/lerreurh/bin:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/bin:/ccc/products/ifort-20.0.0/system/default/20.0.0/bin/intel64:/ccc/products/sharp-2.4.2/system/default/bin:/ccc/products/hcoll-4.7.3191/system/default/bin
+ PATH=/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/bin:/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/bin:/ccc/products/hdf5-1.12.0/intel--20.0.2__openmpi--4.0.1/parallel/bin:/ccc/products/openmpi-4.1.4/intel--20.0.0/default/bin:/ccc/products/ucx-1.12.1/system/default/bin:/ccc/products/icc-20.0.0/system/default/20.0.0/bin/intel64:/ccc/products/pmix-4.2.2/system/default/bin:/ccc/products/hwloc-2.5.0/system/default/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/ccc/products/ccc_users_env/bin:/ccc/products/ccc_crontab/bin:/ccc/products/ccc_prod/bin:/ccc/products/ccc_visu/bin:/ccc/products/ccc_users_tools/bin:/ccc/cont003/home/celia/lerreurh/bin:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/bin:/ccc/products/ifort-20.0.0/system/default/20.0.0/bin/intel64:/ccc/products/sharp-2.4.2/system/default/bin:/ccc/products/hcoll-4.7.3191/system/default/bin
+ export LD_LIBRARY_PATH=/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/lib:/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/lib:/ccc/products/hdf5-1.12.0/intel--20.0.2__openmpi--4.0.1/parallel/lib:/ccc/products/openmpi-4.1.4/intel--20.0.0/default/lib/openmpi:/ccc/products/openmpi-4.1.4/intel--20.0.0/default/lib:/ccc/products/mkl-20.0.0/intel--20.0.0__openmpi--4.0.1/default/lib/intel64:/ccc/products/mkl-20.0.0/intel--20.0.0__openmpi--4.0.1/default/20.0.0/mkl/lib/intel64:/ccc/products/ucx-1.12.1/system/default/lib:/ccc/products/ifort-20.0.0/system/default/20.0.0/lib/intel64:/ccc/products/icc-20.0.0/system/default/20.0.0/lib/intel64:/ccc/products/pmix-4.2.2/system/default/lib:/ccc/products/hwloc-2.5.0/system/default/lib:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/math_libs/lib64:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/extras/CUPTI/lib64:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/lib64:/ccc/products/sharp-2.4.2/system/default/lib::/ccc/products/hcoll-4.7.3191/system/default/lib
+ LD_LIBRARY_PATH=/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/lib:/ccc/products/python3-3.8.10/intel--20.0.0__openmpi--4.0.1/default/lib:/ccc/products/hdf5-1.12.0/intel--20.0.2__openmpi--4.0.1/parallel/lib:/ccc/products/openmpi-4.1.4/intel--20.0.0/default/lib/openmpi:/ccc/products/openmpi-4.1.4/intel--20.0.0/default/lib:/ccc/products/mkl-20.0.0/intel--20.0.0__openmpi--4.0.1/default/lib/intel64:/ccc/products/mkl-20.0.0/intel--20.0.0__openmpi--4.0.1/default/20.0.0/mkl/lib/intel64:/ccc/products/ucx-1.12.1/system/default/lib:/ccc/products/ifort-20.0.0/system/default/20.0.0/lib/intel64:/ccc/products/icc-20.0.0/system/default/20.0.0/lib/intel64:/ccc/products/pmix-4.2.2/system/default/lib:/ccc/products/hwloc-2.5.0/system/default/lib:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/math_libs/lib64:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/extras/CUPTI/lib64:/ccc/products/nvhpc-22.2/system/default/Linux_x86_64/22.2/cuda/11.6/lib64:/ccc/products/sharp-2.4.2/system/default/lib::/ccc/products/hcoll-4.7.3191/system/default/lib
+ export OMPI_MCA_btl_portals4_use_rdma=0
+ OMPI_MCA_btl_portals4_use_rdma=0
+ export HDF5_USE_FILE_LOCKING=FALSE
+ HDF5_USE_FILE_LOCKING=FALSE
+ export OMP_SCHEDULE=dynamic
+ OMP_SCHEDULE=dynamic
+ export OMP_NUM_THREADS=64
+ OMP_NUM_THREADS=64
+ ccc_mprun ../smilei walls2D.py
+ exit 0

Thanks for helping!
Howel

[charlesprouveur]

I moved this to the discussions. Please do not ask for help in the issues.

Regarding your questions:

Openmp task is not available (it never was released), this is different from MPI + openmp.
On CPU you should be able to execute in parallel using the number of MPI ranks and the number of threads per mpi rank (as you can see, you are compiling with mpicxx using the option -fopenmp -D_OMP)
The number of threads per mpi rank is case dependent.
More details can be found in the doc. You can start here for instance: https://smileipic.github.io/Smilei/Understand/parallelization.html and here https://smileipic.github.io/Smilei/Use/run.html

Best regards,
Charles

[hwlr0]

Thank you very much for your answer and sorry for posting at the wrong place!

Best regards,
Howel