Converting to DEPHY format

[dustinswales]

As part of the CCPP v7.0.0 release, all SCM cases included in the repository are in the DEPHY format. Provided with the codebase are script(s) to convert cases in the "old" SCM format to the DEPHY format.
However, this script is not documented, so it's not clear how to use it.

@grantfirl @ligiabernardet

[grantfirl]

Here is the broad-stroke documentation for using it:

The script is in ccpp-scm/scm/etc/scripts/dephy_converter.py. It only takes one argument: -n name_of_case (where name_of_case corresponds to an existing case data file found in the ccpp-scm/scm/data/processed_case_data directory without the .nc extension).

The script reads in the old file in ccpp-scm/scm/data/processed_case_data together with the associated case configuration namelist in ccpp-scm/scm/etc/case_config and outputs a new DEPHY-formatted case file in ccpp-scm/scm/data/processed_case_data named "name_of_case_SCM_driver.nc". It also modifies that case configuration namelist for the case.

Before trying to use the script, I would create a backup of the case data file and case configuration namelist. Then, you can check that the conversion worked by running the original case and the DEPHY version and comparing the output. It may not be bit-for-bit, but the output should look VERY similar if plotted.

[dustinswales]

Thanks @grantfirl!
Let's see if this is all that I-Kuan needs, and if so we can a) Add this to the documentation and/or b) Create a discussion thread with your response as the "Answer" for others to see?
The latter seems adequate to me until we can update the docs.

[ihursmas]

Thank you @dustinswales and @grantfirl!

So far in the CCPP SCM v7.0.0, I've converted the required forcings from the conventional format to DEPHY format without any issues, and I've run all seven 33-hour periods of the ATOMIC test case using the SCM forcings in both the conventional and DEPHY formats and using three physics suites (GFS_v16, GFS_v17_p8, and RRFS_v1beta). For all the periods and physics suites tested, the outputs of advective tendencies - i.e. “T_force_tend”, “qv_force_tend”, “u_force_tend”, and “v_force_tend” - are identical between the runs driven by the conventional- and DEPHY-format forcings. The boundary forcings are identical between the forcing files (under data/processed_case_input/) in the conventional and DEPHY formats as well. In general, the runs driven by different forcing formats are similar, regardless of choices of tested period and physics suite. However, one or two specific combinations of period and physics suite show noticeable differences, particularly in cloud fraction and/or surface fluxes, between the runs driven by different forcing formats. Would this relatively significant sensitivity (of certain variables, under certain run cases) to forcing format be a concern, or is it actually not so uncommon?

[dustinswales]

Thank you @dustinswales and @grantfirl!

So far in the CCPP SCM v7.0.0, I've converted the required forcings from the conventional format to DEPHY format without any issues, and I've run all seven 33-hour periods of the ATOMIC test case using the SCM forcings in both the conventional and DEPHY formats and using three physics suites (GFS_v16, GFS_v17_p8, and RRFS_v1beta). For all the periods and physics suites tested, the outputs of advective tendencies - i.e. “T_force_tend”, “qv_force_tend”, “u_force_tend”, and “v_force_tend” - are identical between the runs driven by the conventional- and DEPHY-format forcings. The boundary forcings are identical between the forcing files (under data/processed_case_input/) in the conventional and DEPHY formats as well. In general, the runs driven by different forcing formats are similar, regardless of choices of tested period and physics suite. However, one or two specific combinations of period and physics suite show noticeable differences, particularly in cloud fraction and/or surface fluxes, between the runs driven by different forcing formats. Would this relatively significant sensitivity (of certain variables, under certain run cases) to forcing format be a concern, or is it actually not so uncommon?

@ihursmas
Great to hear that things are working, well kinda working at least.
"Significant sensitivity" worries me, so I would like to know more about your configuration. Can you provide some details? (e.g. your SCM configuration and possibly the case file(s))

[ihursmas]

@dustinswales
Sure thing. When using the GFS_v16 physics suite and the period from 22 UTC on Jan. 16 to 06 UTC Jan. 18, 2020, the differences between the runs driven by forcings in the conventional vs. DEPHY formats are most significant.

The corresponding files can be found here:

• case files: atomic_ERA5_Jan16T22Jan18T06_dephy.nml and atomic_ERA5_Jan16T22Jan18T06.nml under /scratch2/BMC/mcwi/I-kuan.Hu/CCPPSCM.v7/v7.0.0/scm/etc/case_config;
• forcing files: atomic_ERA5_Jan16T22Jan18T06_dephy_SCM_driver.nc and atomic_ERA5_Jan16T22Jan18T06.nc under /scratch2/BMC/mcwi/I-kuan.Hu/CCPPSCM.v7/v7.0.0/scm/data/processed_case_input.

The runs are
atomic_ERA5_Jan16T22Jan18T06_dephy_SCM_GFS_v16 and atomic_ERA5_Jan16T22Jan18T06_SCM_GFS_v16 under /scratch2/BMC/mcwi/I-kuan.Hu/CCPPSCM.v7/v7.0.0/scm/output
Based on the diagnostics using NCVIEW, I think the differences stem from a 2D variable rad_cloud_fraction, which affects the 1D variables such as max_cloud_fraction, tprcp_rate_inst, and all the variables associated with surface turbulent and radiative heat fluxes. However, pwat still looks very similar between the runs driven by different formats of forcings.

The same period conducted using GFS_v17_p8 or RRFS_v1beta shows much smaller discrepancies between the runs driven by different formats of forcings, but I think those differences are still relatively noticeable compared to their counterparts for other periods.

Let me know if you don't have assess to all these files listed above (I will then try to deliver them here). Thank you!

[dustinswales]

@ihursmas Sorry, I was out yesterday. I'm taking a look at this now.

[dustinswales]

@ihursmas @grantfirl
I created some plots for all the fields in output.nc files, and there are noticeable differences in most of the fields.

cd scm/test
./cmp_scmout.py -fbl /scratch2/BMC/mcwi/I-kuan.Hu/CCPPSCM.v7/v7.0.0/scm/output/atomic_ERA5_Jan16T22Jan18T06_SCM_GFS_v16/output.nc -frt /scratch2/BMC/mcwi/I-kuan.Hu/CCPPSCM.v7/v7.0.0/scm/output/atomic_ERA5_Jan16T22Jan18T06_dephy_SCM_GFS_v16/output.nc
NOTE I-Kuan if you want to use this script, you will need to modify this plotting script to point to a local directory.

T_force_tend:

qv_force_tend:

qv:

Temperature:

So the forcing is nearly identical between DEPHY and non-DEPHY runs, there are some bitwise differences, but nothing that stands out to me as egregious . But the state in the SCM evolves quite differently?

[ihursmas]

@dustinswales
Thank you for taking a further analysis!
Yes these results you showed look consistent with what I saw. Fortunately the different evolutions between the DEPHY and non-DEPHY driven runs are only noticeable for this specific period (i.e. Jan16T22Jan18T06), and the differences seem to be less when using the physics suites other than GFS_v16 (like GFS_v17_p8 and RRFS_v1beta). I honestly don't know where the cause of these differences could be, given almost identical forcings in the DEPHY and non-DEPHY formats...

[dustinswales]

@ihursmas No worries. @hertneky is looking into this closer to get to the root cause of these differences. We don't expect differences between runs with DEPHY vs. non-DEPHY, so we need to understand what's going on.

[ihursmas]

@dustinswales Thanks! I quickly checked the sensitivity to column_area (I set it to 1.69E8, and I think the default is 2E9): the T and qv differences are smaller yet the qc (which would lead to cloud fraction) difference is larger. I'm also wondering if the momentum nudging (by setting mom_forcing_type = 3 and relax_time = 3600.0) in the conventional format were genuinely reflected in the DEPHY format, as I saw relatively large difference in u_force_tend and v_force_tend (compared to T_force_tend and qv_force_tend which are several orders smaller than their magnitudes of individual run) between the DEPHY and non-DEPHY runs.

u_force_tend:

v_force_tend:

[hertneky]

@ihursmas I've compared the configs for both and I don't see any discrepancies there. I am going to look for anything that might cause diffs in the code, specifically the u_/v_force_tend.

[ihursmas]

Hi @hertneky and @dustinswales, do we have any updates? Thanks!

[hertneky]

Hi @ihursmas I have not found anything that is specifically causing those differences we see. I replicated your runs and looked at various variables, many of which seemed to have rounding/precision errors that grow with time. In the scm_input.F90, I did notice precision diffs between orig/dephy for initializing some of the variables (double vs single precision). I decided to build/run with 32-bit to see if the diffs reduced, but came across an issue with running the atomic case in 32-bit for the dephy format. Noting that a supplied case in dephy format, 'twpice', did not have the issue, so it may be unique to the atomic dephy case. Again, this is just for single precision as everything ran fine otherwise. @scrasmussen will help look at the single precision issue.

[lisa-bengtsson]

@hertneky @dustinswales @ihursmas @grantfirl Hi! I'm just checking if anyone has any updates on this work? It seems that it is not related to precision, but something else in the tendency files?

[hertneky]

@ihursmas @dustinswales @grantfirl

First, did you run the dephy_converter without any modifications? I may need to test mods and want to assure I am using the same starting point as you.

There was an issue with how column area was applied for the DEPHY format. Fixing this issue had no affect on your latest results, since you had column_area set the same in both nmls.

• It was not properly assigning the area from the DEPHY forcing file, so it was taking whatever was in the nml - the column_area in nml for DEPHY is intended to be used if doing scalability tests and otherwise should use what is set in the forcing file; however, it was not working correctly

There have been issues opened wrt w_ls/omega not being present in the DEPHY forcing.

• For non-DEPHY format, your omega is read in and assigned to scm_state%omega
• For DEPHY format, omega is not present in the forcing file, since forc_wap = 0, and is therefor not read in/assigned. It looks like omega is used by the physics, but I am no expert with the physics. @grantfirl @dustinswales Can you comment on this?

[ihursmas]

@hertneky Thank you for getting back to this.
I didn't modify dephy_converter.py; All I did is ./dephy_converter -n atomic_ERA5_[a certain period].
And a good point about the lack of w_ls. I can generate a set of forcing files with the presence of w_ls for sensitivity tests. My guess is that although w_ls does not impact the large-scale forcings (since I directly used the large-scale advective tendencies as forcings, i.e. thermo_forcing_type = 1 for the non-DEPHY format), it might play a role in the closure in the PBL scheme.