libm and reproducibility

[ngs333]

I) Intro
Some of the parameters that can affect the performance (run time) and reproducibility of NCTools C language based apps includes the hardware, the compiler and its settings, and the libraries used. The latter, in particular the libm (libm.a and libm.so) are considered here.

libm can affect run time and reproducibility because :
a) the implementation of libm by gnu is not fixed. GNU libm is a component of the libc library collection (https://www.gnu.org/software/libc/) and as one can see by checking the release notes, there has been active
development changes to the mathematics function. For example, a recent (2021, version 2.34) release notes indicates that :

The following bugs are resolved with this release:
  [5781] math: Slow dbl-64 sin/cos/sincos for special values
  [14469] math: Inaccurate j0f function
  [14470] math: Inaccurate j1f function
  [14471] math: Inaccurate y0f function
  [14472] math: Inaccurate y1f function

b) The Intel compiler can use the gnu libm for all libm functions, or it can also use a different Intel provided library (libmif) for a subset of the gnu libm functionality. Like the gnu linray, the Intel library also undergoes development and improvement and there is no guarantee that the mathematics functions are unchanged and/or reproduce completely with different Intel compiler version (that may bring in different versions of libmif) and different hardware.

II) Run time performance differences.
All else being equal, an NCTools battery of unit and functionality tests was seen to run about six percent faster with the Intel libraries than with the gnu libraries using the Intel compiler 18 on recent AVX enabled Analysis hardware. Several targeted tests were done with grid generation and remapping applications and the run time difference were seen to slower up to ten percent with the gnu only libraries (and the same compiler flags -O2 -xSSE2)

Complete reproducibility can be achieved across all Analysis hosts (some that have AVX instruction set(s),
and some that don't) if the Intel compilers do not use the Intel libraries. If the Intel library is used, then a difference in the tenth decimal place was seen in some applications across Analysis host that differ on whether of not they have AVX.

III) Reproducibility : Intel 18 vs 11

IV) Reproducibility : Gaea vs Analysis

V) Reproducibility with Fortran applications.

VI) Things to do
a) A unit test of key functions used in climate modeling can be maintained. These functions can be checked at their "tricky" points, boundaries, domain points where the slope is max or zero, etc. This may be important because seeing reproduction by any one NCTools app run is not a guarantee that with different arguemnts to the app that reproducibility will be the case. E.g. A different grid size or grid origin may evaluate a key trig function at a "tricky" point that wast being encoutered before.