FESOM2 model bias with different mesh partitions

[PengyangSong]

Dear FESOM2 developers,
I tested FESOM2.1 with different mesh partitions (CORE-II dist_432 and dist_288). The tests are integrated for 62 years (JRA55, 1958-2019) and the last 10 years mean is analyzed. You can find the temperature and salinity differences in the below figures.

I am not an expert in computational science. Also I didn't check the MPI part of FESOM2 code. I am just curious if this bias is acceptable, it is prominent in the North Atlantic and the Arctic.

Run folder:
432_dist: /work/ollie/psong/FESOM2_ICE/fesom2_CTRL
288_dist: /work/ollie/psong/FESOM2_ICE/fesom2_CTRL2

Visualization code:
/home/oliie/psong/post_fesom2_ice/ipynb_code/hydro_cyl4_bias.ipynb

[JanStreffing]

Can you scale this but the SST and SSS variability? Some of this (in the North Atlantic) could be down to not being bit-reproducible and then amplifying the natural variability. I'm not so sure this can explain the salt signals in the Arctic ocean though.

[PengyangSong]

Here it is.

[PengyangSong]

Dear FESOM2 developers, I tested FESOM2.1 with different mesh partitions (CORE-II dist_432 and dist_288). The tests are integrated for 62 years (JRA55, 1958-2019) and the last 10 years mean is analyzed. You can find the temperature and salinity differences in the below figures.

I am not an expert in computational science. Also I didn't check the MPI part of FESOM2 code. I am just curious if this bias is acceptable, it is prominent in the North Atlantic and the Arctic.

Run folder: 432_dist: /work/ollie/psong/FESOM2_ICE/fesom2_CTRL 288_dist: /work/ollie/psong/FESOM2_ICE/fesom2_CTRL2

Visualization code: /home/oliie/psong/post_fesom2_ice/ipynb_code/hydro_cyl4_bias.ipynb

A small mistake, these are 288 CPUs minus 432 CPUs.

[patrickscholz]

Hi Penyang
Since you integrate over 62 year, i think this error at the end is ok between two different partitions. You should keep in mind that different partitions means also different halo exchange so different numbers at summed, multiplied ... So i think it is to expect that some numerical bias arises that will of course accumulate over time. The meandering structure of the golf stream is especially sensitive to these small numerical differences that is why there the highest biases occur.

There would be a problem if these differences would occur between two simulations using the same partition. We had this problem some time ago for ollie, but solved that by implying some proper compiler flag (see #145 ).

[pgierz]

Unrelated to the actual problem, but I am gathering plotting notebooks that can then be hooked into esm-tools via papermill. The idea is to define it in such a way that any analysis (CDO or whatever else), plotting, reading the mesh, or whatever else can be grabbed from the top level of your experiment. Could you modify your notebooks to accommodate that? We can also have a brief video call and I can explain if you want.

[JanStreffing]

Dont we already have this in: https://github.com/FESOM/fdiag/blob/master/fdiag/templates/difference.ipynb ?
Maybe we can make that more beautiful based on Pengyangs notebooks.

[pgierz]

@JanStreffing, yes, sort-of. But it is rather specific to FESOM and I would like to make it more versatile for everything. I haven't really had the change to look into detail at fdiag yet, but Pengyang and I would sit together and I would explain my ideas. Would you like to join in that meeting?

[JanStreffing]

I would be interested. I'm using fdiag for both FESOM2 and OpenIFS, and Nikolay was open to having templates that are non fesom2 in there. I agree the core structure is a bit more fesom2 specific that you would probably want for esm_tools. I did recently work on templates for fdiag that use cdo + dask for loading. (e.g. FESOM/fdiag#10)

[pgierz]

OK, tomorrow at 15:00 if you have time. I will send you and Pengyang a webex link shortly before (and hopefully have some time to look at your linked issue)

[pgierz]

Sorry, I meant: Friday at 15:00