GEOSgcm + GNU does not layout regress

[mathomp4]

This is interesting. The latest GEOSgcm v11 no longer passes layout regression with GNU. It still passes start-stop, but not layout. The annoying thing is, layout regression is the harder of the two. We know start-stop is usually "look at restarts" and that was the solution to the last issue about this (see #380).

Usually layout regression failures are compiler flag changes or MPI changes, but the GNU compiler flags and MPI stack did not change with v11. The Intel ones changed, but that doesn't affect GNU!

This might require some thinking so I'll ping @tclune @bena-nasa @atrayano @aoloso

[tclune]

In a perfect world, we'd now have component-level layout regression tests ...

I guess the next step is to ask which components had the big changes in this version.

[mathomp4]

I guess the next step is to ask which components had the big changes in this version.

The answer of "all of them" is not hyperbole:

#537 (comment)

[mathomp4]

Tests run today (per the request of @amdasilva and @tclune) with the latest Chem and GOCART (using #529), show that GNU still has this issue. It start-stop regresses, but fails layout.

[mathomp4]

I am currently attempting to do a run with GNU where I've added -finit-local-zero -finit-derived which per the man page:

   -finit-local-zero
   -finit-derived
   -finit-integer=n
   -finit-real=<zero|inf|-inf|nan|snan>
   -finit-logical=<true|false>
   -finit-character=n
       The -finit-local-zero option instructs the compiler to initialize local "INTEGER", "REAL", and "COMPLEX" variables to zero, "LOGICAL"
       variables to false, and "CHARACTER" variables to a string of null bytes.  Finer-grained initialization options are provided by the
       -finit-integer=n, -finit-real=<zero|inf|-inf|nan|snan> (which also initializes the real and imaginary parts of local "COMPLEX" variables),
       -finit-logical=<true|false>, and -finit-character=n (where n is an ASCII character value) options.

       With -finit-derived, components of derived type variables will be initialized according to these flags.  Components whose type is not
       covered by an explicit -finit-* flag will be treated as described above with -finit-local-zero.

[mathomp4]

Okay. Can't do that everywhere it seems:

pe=00083 FAIL at line=01149    MAPL_LatLonGridFactory.F90               <inconsistent min for lat_range>

I'll try doing it just in GOCART...

[mathomp4]

Alas. Just using those flags in GOCART does not help. I still get a layout failure.