incorrect results for OpenFOAM v10 + v11 when built with GCC 11.3.0 or 12.3.0 and -ftree-vectorize

[boegel]

We got a report that installations of OpenFOAM (OpenFOAM-10-foss-2022a.eb, OpenFOAM-10-foss-2023a.eb, OpenFOAM-11-foss-2023a.eb) produce incorrect results, see also https://bugs.openfoam.org/view.php?id=4076 .

After a lot of digging, we figured out that this problem is caused by a compiler bug: if these easyconfigs are changed to avoid the use of the -ftree-vectorize compiler option, the problem is resolved:

toolchainopts = {'vectorize': False}

[nicolasdlss]

The incorrect results when using -ftree-vectorize are noticeable when using strict solver tolerances.
On the one hand, with -ftree-vectorize the results are (slightly) different and also different compared to older versions (8 and 9).
On the other hand and more importantly, when using -ftree-vectorize, the results depend on the partitioning. For some partitionings a large error remained after convergence, which is several order of magnitude bigger than the used solver tolerances. See also https://bugs.openfoam.org/view.php?id=4076.

[bartoldeman]

Is the issue specific to 11.3.0 and 12.3.0 only or does it also happen with earlier/later versions of GCC?

[joris13]

When testing on a lid driven cavity with icoFoam, with or without -ftree-vectorize does not result in a strong performance difference, so this option can be avoided without heavy penalty.

[boegel]

Is the issue specific to 11.3.0 and 12.3.0 only or does it also happen with earlier/later versions of GCC?

Problem doesn't appear with OpenFOAM easyconfigs using a toolchain older than 2022a, seems like we're hitting something that was introduced in auto-vectorizer of GCC 11.x and more recent.

[Micket]

Related, same(?) or at least very similar tree-vectorizer issue: #15495
In that case, -O3 also "solved" the issue there.

The actual issue here is that OpenFOAM doesn't have a robust test suite that is desperately needs.
I would also really like to test this with ASAN enabled -fsanitize=address. If someone could post a jobscript for running the reproducing test case I would appreciate it.

[boegel]

@nicolasdlss Can you provide some clear instructions on how to reproduce the problem, and what to look out for?

@Micket Since we already disable -ftree-vectorize for the other variant of OpenFOAM (ever since #15495), I think it makes sense to also do that for openfoam.org variant, as proposed in #20958, especially since the performance impact seems very minimal...

[nicolasdlss]

This is a fast-running test (order of 1 minute) with an numerical outcome that can be easily used for comparison.
Instructions on how to use the test and interpret the results are given in the README.
GitHub_reproducer.tar.gz

[boegel]

Re-opening this, I would really like to have some kind of sanity check in place for this, so we don't re-introduce this problem again in the future.

@nicolasdlss How feasible would it be to leverage a tutorial case that is included with OpenFOAM for this?

We could also look into integrating your minimal test case as a sanity check of course, but that's a bit messier since it involves external input files, etc.