Request to add toolchain for NEC #53
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -60,29 +60,6 @@ Balthasar Reuter ([email protected]) | |
| move parameter structures to constant memory. To enable this variant, | ||
| a suitable CUDA installation is required and the `--with-cuda` flag | ||
| needs to be passed at the build stage. | ||
|
There was a problem hiding this comment. I think this here was deleted by accident? Or was this intended? There was a problem hiding this comment. It'd also be great, if you could add a small section (or bullet point) to the README to tell users about the availability of the NEC variant. Probably a small paragraph? There was a problem hiding this comment. Sorry, the deleted part was not intentional. I thought I already added how to compile on NEC in the README file. I'll check that again. |
||
|
|
||
| ## Download and Installation | ||
|
|
||
|
|
@@ -231,6 +208,17 @@ cd build | |
| ./bin/dwarf-cloudsc-fortran 4 16384 32 # The cleaned-up Fortran | ||
| ./bin/dwarf-cloudsc-c 4 16384 32 # The standalone C version | ||
| ``` | ||
| ### Building on NEC SX-AURORA TSUBAS | ||
|
There was a problem hiding this comment. Is the VE not called TSUBAS A ? Applies also to the line below. There was a problem hiding this comment. No, VE is vector engine and NEC SX-AURORA TSUBAS is typically used as the architecture. |
||
| To build on NEC SX-AURORA TSUBAS system, run the following commands | ||
|
|
||
| ```sh | ||
| ./cloudsc-bundle create | ||
| HDF5_ROOT=HDF5-installation-PATH ./cloudsc-bundle build --arch arch/ecmwf/aurora/nec/4.0.0/ [--single-precision] [--with-mpi] --hdf5 ON --cloudsc-fortran ON --cloudsc-prototype1 OFF --verbose --log DEBUG | ||
| ``` | ||
|
|
||
| Currently available `NEC ompiler/version` selections are: | ||
|
There was a problem hiding this comment. "compiler" is missing a "c" |
||
|
|
||
| * `nec/4.0.0 (nfort, ncc, nc++)` | ||
|
There was a problem hiding this comment. Could you add also an example invocation and approximate performance that can be expected? |
||
|
|
||
| ### Running on ECMWF's Atos BullSequana XH2000 | ||
|
|
||
|
|
@@ -272,27 +260,6 @@ srun bash -c "CUDA_VISIBLE_DEVICES=\$SLURM_LOCALID bin/dwarf-cloudsc-gpu-scc-hoi | |
|
|
||
| In principle, the same should work for multi-node execution (`-N 2`, `-N 4` etc.) once interconnect issues are resolved. | ||
|
|
||
|
There was a problem hiding this comment. Please re-add this paragraph |
||
| ## Loki transformations for CLOUDSC | ||
|
|
||
| [Loki](https://github.com/ecmwf-ifs/loki) is an in-house developed | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # Source me to get the correct configure/build/run environment | ||
|
|
||
| # Store tracing and disable (module is *way* too verbose) | ||
| { tracing_=${-//[^x]/}; set +x; } 2>/dev/null | ||
|
|
||
| module_load() { | ||
| echo "+ module load $1" | ||
| module load $1 | ||
| } | ||
| module_unload() { | ||
| echo "+ module unload $1" | ||
| module unload $1 | ||
| } | ||
|
|
||
| export FC=nfort | ||
| export CC=ncc | ||
| export CXX=nc++ | ||
|
|
||
| set -x | ||
|
|
||
| # Increase stack size to maximum | ||
| ulimit -S -s unlimited | ||
|
|
||
| # Enable floating point error trapping at run time | ||
| export VE_FPE_ENABLE=DIV,INV,FOF,FUF,INE | ||
|
|
||
|
|
||
| export PATH="/local/hdd/nabr/openmpi/nvhpc-nompi/20.9/bin:$PATH" | ||
|
There was a problem hiding this comment. I don't think this absolute path setting should be necessary - also, it looks like it is carried over from volta? |
||
|
|
||
| # Restore tracing to stored setting | ||
| if [[ -n "$tracing_" ]]; then set -x; else set +x; fi | ||
|
|
||
| export ECBUILD_TOOLCHAIN="./toolchain.cmake" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
|
|
||
| #################################################################### | ||
| # COMPILER | ||
| #################################################################### | ||
|
|
||
| include( /opt/nec/ve/share/cmake/toolchainVE.cmake ) | ||
|
There was a problem hiding this comment. Traditionally, the low-level toolchains (per compiler version) are symlink up to the |
||
|
|
||
|
|
||
| set( ECBUILD_FIND_MPI ON ) | ||
|
|
||
| #################################################################### | ||
| # Enviroment Variables | ||
| #################################################################### | ||
| set(NMPI_ROOT /opt/nec/ve/mpi/2.23.0) | ||
|
There was a problem hiding this comment. Is this the same on all NEC VE machines, or should this maybe be exported as an environment variable in env.sh and then picked up in the toolchain file? There was a problem hiding this comment. No, it's not. I think environment variable is the better idea for it. |
||
|
|
||
| #################################################################### | ||
| # OpenMP FLAGS | ||
| #################################################################### | ||
|
|
||
| set( OpenMP_C_FLAGS "-fopenmp " ) | ||
| set( OpenMP_CXX_FLAGS "-fopenmp " ) | ||
| set( OpenMP_Fortran_FLAGS "-fopenmp " ) | ||
|
|
||
| #################################################################### | ||
| # OpenAcc FLAGS | ||
| #################################################################### | ||
|
|
||
| set( OpenACC_Fortran_FLAGS "-acc -ta=tesla:lineinfo,deepcopy,maxregcount:100,fastmath" ) | ||
|
There was a problem hiding this comment. These look suspiciously like Nvidia GPU flags; does NEC really support OpenACC, or are these development/debug leftovers? There was a problem hiding this comment. Sorry, they are leftovers that I forgot to remove them. There was a problem hiding this comment. These look suspiciously like Nvidia GPU flags; does NEC really support OpenACC, or are these development/debug leftovers? There was a problem hiding this comment. No, it doesn't. Leftovers as well! |
||
| set( OpenACC_Fortran_FLAGS "${OpenACC_Fortran_FLAGS} -Mvect=levels:6" ) | ||
| set( OpenACC_Fortran_FLAGS "${OpenACC_Fortran_FLAGS} -Mconcur=levels:6" ) | ||
| set( OpenACC_Fortran_FLAGS "${OpenACC_Fortran_FLAGS} -Minfo" ) | ||
|
|
||
| #################################################################### | ||
| # NEC MPI Compiler | ||
| #################################################################### | ||
|
|
||
| set(MPI_C_COMPILER ${NMPI_ROOT}/bin/mpincc CACHE FILEPATH "") | ||
| set(MPI_C_INCLUDE_PATH ${NMPI_ROOT}/include CACHE FILEPATH "") | ||
| set(MPI_C_LIBRARIES ${NMPI_ROOT}/lib64/ve/libmpi.a CACHE FILEPATH "") | ||
| set(MPI_C_COMPILE_FLAGS "-D_MPIPP_INCLUDE" CACHE STRING "") | ||
|
|
||
| set(MPI_CXX_COMPILER ${NMPI_ROOT}/bin/mpinc++ CACHE FILEPATH "") | ||
| set(MPI_CXX_INCLUDE_PATH ${NMPI_ROOT}/include CACHE FILEPATH "") | ||
| set(MPI_CXX_LIBRARIES ${NMPI_ROOT}/lib64/ve/libmpi++.a CACHE FILEPATH "") | ||
|
|
||
| set(MPI_Fortran_COMPILER ${NMPI_ROOT}/bin/mpifort CACHE FILEPATH "") | ||
| set(MPI_Fortran_INCLUDE_PATH ${NMPI_ROOT}/include CACHE FILEPATH "") | ||
| set(MPI_Fortran_ADDITIONAL_INCLUDE_DIR ${NMPI_ROOT}/lib/ve/module CACHE FILEPATH "") | ||
| set(MPI_Fortran_LIBRARIES ${NMPI_ROOT}/lib64/ve/libmpi.a CACHE FILEPATH "") | ||
| set(MPI_Fortran_COMPILE_FLAGS "-D_MPIPP_INCLUDE" CACHE STRING "") | ||
| #################################################################### | ||
| # COMMON FLAGS | ||
| #################################################################### | ||
|
|
||
| set(ECBUILD_Fortran_FLAGS "-fpic") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mstack-arrays") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fdiag-vector=3") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fcse-after-vectorization") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-collapse ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-fusion ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-interchange ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-unroll-complete=200 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -ftrace") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fmove-loop-invariants-if ") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -freplace-loop-equation ") | ||
|
There was a problem hiding this comment. I don't mind having commented out options like this lying around, but could we maybe group them, and or annotate them with comments that provide a bit of context as to why they are enabled/disabled? There was a problem hiding this comment. Yes, we can comment them because -ftrace enables profiling feature and -floop-unroll-complete=200 are used for optimisation purpose here. |
||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -msched-interblock ") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-floating-divide-instruction ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-power-to-explog ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-sqrt-instruction ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-threshold=3 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -finline-functions ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -finline-max-depth=5 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -finline-max-function-size=200 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-merge-conditional ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fivdep ") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-strip-mine ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -muse-mmap ") | ||
| ##set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-packed") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -report-all") | ||
|
|
||
| set( ECBUILD_Fortran_FLAGS_BIT "-O4 -mvector-fma" ) | ||
|
|
||
| set( ECBUILD_C_FLAGS "-O2 " ) | ||
|
|
||
| set( ECBUILD_CXX_FLAGS "-O2" ) | ||
|
|
||
| # Fix for C++ template headers needed for Serialbox | ||
| set( GNU_HEADER_INCLUDE "-I/usr/local/apps/gcc/7.3.0/lib/gcc/x86_64-linux-gnu/7.3.0/include-fixed" ) | ||
| set( ECBUILD_CXX_FLAGS "${ECBUILD_CXX_FLAGS} ${GNU_HEADER_INCLUDE}" ) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
|
|
||
| #################################################################### | ||
| # COMPILER | ||
| #################################################################### | ||
|
|
||
| include( /opt/nec/ve/share/cmake/toolchainVE.cmake ) | ||
|
|
||
|
|
||
| set( ECBUILD_FIND_MPI ON ) | ||
|
|
||
| #################################################################### | ||
| # Enviroment Variables | ||
| #################################################################### | ||
| set(NMPI_ROOT /opt/nec/ve/mpi/2.23.0) | ||
|
|
||
| #################################################################### | ||
| # OpenMP FLAGS | ||
| #################################################################### | ||
|
|
||
| set( OpenMP_C_FLAGS "-fopenmp " ) | ||
| set( OpenMP_CXX_FLAGS "-fopenmp " ) | ||
| set( OpenMP_Fortran_FLAGS "-fopenmp " ) | ||
|
|
||
| #################################################################### | ||
| # OpenAcc FLAGS | ||
| #################################################################### | ||
|
|
||
| set( OpenACC_Fortran_FLAGS "-acc -ta=tesla:lineinfo,deepcopy,maxregcount:100,fastmath" ) | ||
| set( OpenACC_Fortran_FLAGS "${OpenACC_Fortran_FLAGS} -Mvect=levels:6" ) | ||
| set( OpenACC_Fortran_FLAGS "${OpenACC_Fortran_FLAGS} -Mconcur=levels:6" ) | ||
| set( OpenACC_Fortran_FLAGS "${OpenACC_Fortran_FLAGS} -Minfo" ) | ||
|
|
||
| #################################################################### | ||
| # NEC MPI Compiler | ||
| #################################################################### | ||
|
|
||
| set(MPI_C_COMPILER ${NMPI_ROOT}/bin/mpincc CACHE FILEPATH "") | ||
| set(MPI_C_INCLUDE_PATH ${NMPI_ROOT}/include CACHE FILEPATH "") | ||
| set(MPI_C_LIBRARIES ${NMPI_ROOT}/lib64/ve/libmpi.a CACHE FILEPATH "") | ||
| set(MPI_C_COMPILE_FLAGS "-D_MPIPP_INCLUDE" CACHE STRING "") | ||
|
|
||
| set(MPI_CXX_COMPILER ${NMPI_ROOT}/bin/mpinc++ CACHE FILEPATH "") | ||
| set(MPI_CXX_INCLUDE_PATH ${NMPI_ROOT}/include CACHE FILEPATH "") | ||
| set(MPI_CXX_LIBRARIES ${NMPI_ROOT}/lib64/ve/libmpi++.a CACHE FILEPATH "") | ||
|
|
||
| set(MPI_Fortran_COMPILER ${NMPI_ROOT}/bin/mpifort CACHE FILEPATH "") | ||
| set(MPI_Fortran_INCLUDE_PATH ${NMPI_ROOT}/include CACHE FILEPATH "") | ||
| set(MPI_Fortran_ADDITIONAL_INCLUDE_DIR ${NMPI_ROOT}/lib/ve/module CACHE FILEPATH "") | ||
| set(MPI_Fortran_LIBRARIES ${NMPI_ROOT}/lib64/ve/libmpi.a CACHE FILEPATH "") | ||
| set(MPI_Fortran_COMPILE_FLAGS "-D_MPIPP_INCLUDE" CACHE STRING "") | ||
| #################################################################### | ||
| # COMMON FLAGS | ||
| #################################################################### | ||
|
|
||
| set(ECBUILD_Fortran_FLAGS "-fpic") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mstack-arrays") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fdiag-vector=3") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fcse-after-vectorization") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-collapse ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-fusion ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-interchange ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-unroll-complete=200 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -ftrace") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fmove-loop-invariants-if ") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -freplace-loop-equation ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -msched-interblock ") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-floating-divide-instruction ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-power-to-explog ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-sqrt-instruction ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-threshold=3 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -finline-functions ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -finline-max-depth=5 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -finline-max-function-size=200 ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-merge-conditional ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -fivdep ") | ||
| ###set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -floop-strip-mine ") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -muse-mmap ") | ||
| ##set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -mvector-packed") | ||
| set(ECBUILD_Fortran_FLAGS "${ECBUILD_Fortran_FLAGS} -report-all") | ||
|
|
||
| set( ECBUILD_Fortran_FLAGS_BIT "-O4 -mvector-fma" ) | ||
|
|
||
| set( ECBUILD_C_FLAGS "-O2 " ) | ||
|
|
||
| set( ECBUILD_CXX_FLAGS "-O2" ) | ||
|
|
||
| # Fix for C++ template headers needed for Serialbox | ||
| set( GNU_HEADER_INCLUDE "-I/usr/local/apps/gcc/7.3.0/lib/gcc/x86_64-linux-gnu/7.3.0/include-fixed" ) | ||
| set( ECBUILD_CXX_FLAGS "${ECBUILD_CXX_FLAGS} ${GNU_HEADER_INCLUDE}" ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should only affect the GNU compiler, does this really have to be removed for NEC? If it makes problems with the NEC compiler, could this be suitably guarded so it still takes effect for GNU?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it makes compiling error on NEC.