zg500 (AERday): different members have identical values

[lisesg]

Describe the bug
I just got an email for a PAMIP collaborator analyzing the zg500 data that there are some strange features in the CMOR-ized zg500 field. The NorESM2 has noisy behaviour when sinuosity (waviness) is plotted as a function of time. One thing that has been identified is that several members have identical values. I have confirmed this, and checked that this is not the case with the raw model output.

To Reproduce
To see that values are identical for different members in the CMOR-ized output, compare values for members 2 and 3 from CMOR-ized pdSST-pdSIC:

• ncdump -v zg500 /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-pdSIC/r2i1p1f1/AERday/zg500/gn/latest/zg500_AERday_NorESM2-LM_pdSST-pdSIC_r2i1p1f1_gn_20000601-20010531.nc | less
• ncdump -v zg500 /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-pdSIC/r3i1p1f1/AERday/zg500/gn/latest/zg500_AERday_NorESM2-LM_pdSST-pdSIC_r3i1p1f1_gn_20000601-20010531.nc | less

To see that values are non-identical for two members in the raw model output, compare values for members 2 and 3 from raw model output from pdSST-pdSIC:

• ncdump -v Z500 /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819/atm/hist/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819.cam_0002.h1.2000-06-13-00000.nc | less
• ncdump -v Z500 /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819/atm/hist/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819.cam_0003.h1.2000-06-13-00000.nc | less

[YanchunHe]

I will look at this right now.

[lisesg]

Another thing is that there seems to be differences between for instance CMOR-ized zg500 for pdSST-pdSIC member 2 and the raw Z500 for member 2. Should member 2 in the raw data correspond to member 2 in the CMOR-ized data? To reproduce (make sure to look as the same date):

• ncview /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-pdSIC/r2i1p1f1/AERday/zg500/gn/latest/zg500_AERday_NorESM2-LM_pdSST-pdSIC_r2i1p1f1_gn_20000601-20010531.nc &
• ncview /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819/atm/hist/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819.cam_0002.h1.2000-06-13-00000.nc &

[YanchunHe]

Do other monthly fields have such similar problems in addition to the daily zg500?

[YanchunHe]

A quick check that other monthly fields, e.g., ts in Amon are different for member 2 and 3.

[lisesg]

I have the same impression, that is, I have not been able to find the same problem for other fields yet.

[YanchunHe]

I think I found where the problem is. I updated the format of name list files slightly for your last several experiments, but not the first two experiments in the beginning. So that my script to submit the cmorization jobs does not set correct member tags for the v20191009 version (in which AERday is included).

So, for the six PAMIP experiments:

• pdSST-futAntSIC
• pdSST-pdSIC
• pdSST-piArcSIC
• pdSST-futArcSIC
• pdSST-piAntSIC
• piSST-pdSIC

Should only the earliest first two cmorized experiments were affected:

• pdSST-futArcSIC
• pdSST-pdSIC

I will redo the cmorization.

---

Could @monsieuralok @IngoBethke retracted fields from ESGF, or we just publish a newer version?

Effected experiments:

• pdSST-futArcSIC
• pdSST-pdSIC

Effected fields:

• AERday/zg500/gn/v20191009/zg500_AERday_NorESM2-LM_pdSST-pdSIC_r42i1p1f1_gn_20000601-20010531.nc
• Eday/ta850/gn/v20191009/ta850_Eday_NorESM2-LM_pdSST-pdSIC_r42i1p1f1_gn_20000601-20010531.nc

Including all the 100 ensemble members

[YanchunHe]

Fixed in the v20191018 version.
put in the queue to publish to ESGF.
refer #28

[YanchunHe]

fix to the namelist files are committed to: 02e6773

[lisesg]

Great news! Thanks a lot! :)

[lisesg]

It seems like the corrected data was never published on ESGF? I just got an email from a user who rediscovered this bug, and when I do a search on ESGF, I can only find the old version (v20191009).

The updated version (v20191018) exists on NIRD.

[lisesg]

Another issue is that members 101-200 appear to be missing in the updated version: #182 and #183

[YanchunHe]

It seems like the corrected data was never published on ESGF? I just got an email from a user who rediscovered this bug, and when I do a search on ESGF, I can only find the old version (v20191009).

The updated version (v20191018) exists on NIRD.

Yes, this is strange but indeed it was not published.

I updated some metadata, so that it should be able publish them right now.

I will update their individual experiment (refer to #28).

[YanchunHe]

Another issue is that members 101-200 appear to be missing in the updated version: #182 and #183

I will check this too.

[YanchunHe]

Another issue is that members 101-200 appear to be missing in the updated version: #182 and #183
@lisesg

I don't see there are daily output in the extended experiments.

Could you please check that?

[lisesg]

Thanks for looking into this @YanchunHe! :)

The daily output is in the atmospheric h1 files. These seem to exist for all the experiments when I do a simple ls. For instance:
ls /projects/NS2345K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem176-200_f19_mg17_20200313/atm/hist/*h1*

[YanchunHe]

Thanks for looking into this @YanchunHe! :)

The daily output is in the atmospheric h1 files. These seem to exist for all the experiments when I do a simple ls. For instance: ls /projects/NS2345K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem176-200_f19_mg17_20200313/atm/hist/*h1*

OK, thanks! I looked into wrong place. Now I find it.

I will update the #182 and #183 issues.