Updates to GDASapp to account for new JCB policies

[danholdaway]

• Renaming of atmosphere YAMLS and keys.
• Add marine YAML files to JCB
• Add creation of 3DVarFGAT YAML for marine using JCB (not used yet but YAML matched exactly).

Once approved I will merge the JCB PRs and then we can merge this after updating the hashes.

A G-W PR will follow the approval and merge of this PR.

G-W branch required for testing: https://github.com/danholdaway/global-workflow/tree/feature/rename_atm

[RussTreadon-NOAA]

Orion test
Install feature/rename_atm at 2ec1767 inside g-w devleop at c44d0ac8. Initial run of test_gdasapp yielded the following ctest failures

64% tests passed, 17 tests failed out of 47

Label Time Summary:
gdas-utils    =   4.10 sec*proc (9 tests)
script        =   4.10 sec*proc (9 tests)

Total Test time (real) = 1370.19 sec

The following tests FAILED:
        1756 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1757 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1758 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1761 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1762 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1763 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
        1764 - test_gdasapp_soca_socahybridweights (Failed)
        1765 - test_gdasapp_soca_incr_handler (Failed)
        1766 - test_gdasapp_soca_ens_handler (Failed)
        1775 - test_gdasapp_atm_jjob_var_init (Failed)
        1776 - test_gdasapp_atm_jjob_var_run (Failed)
        1777 - test_gdasapp_atm_jjob_var_inc (Failed)
        1778 - test_gdasapp_atm_jjob_var_final (Failed)
        1779 - test_gdasapp_atm_jjob_ens_init (Failed)
        1780 - test_gdasapp_atm_jjob_ens_run (Failed)
        1781 - test_gdasapp_atm_jjob_ens_inc (Failed)
        1782 - test_gdasapp_atm_jjob_ens_final (Failed)

Need to update g-w sorc/jcb to be consistent with jcb-algorithms and jcb-gdas used in sorc/gdas.cd/parm. Update working copy of sorc/jcb to jcb branch feature/rename_atm at d167c6c. Rerun test_gdasapp. More ctests pass. Failures are limited to soca tests

81% tests passed, 9 tests failed out of 47

Label Time Summary:
gdas-utils    =   4.66 sec*proc (9 tests)
script        =   4.66 sec*proc (9 tests)

Total Test time (real) = 1787.51 sec

The following tests FAILED:
        1756 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1757 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1758 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1761 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1762 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1763 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
        1764 - test_gdasapp_soca_socahybridweights (Failed)
        1765 - test_gdasapp_soca_incr_handler (Failed)
        1766 - test_gdasapp_soca_ens_handler (Failed)

Some of these failures may flip to Passed if the initial test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP failure is resolved.

Took a peek at /work2/noaa/da/rtreadon/git/global-workflow/rename_atm/sorc/gdas.cd/build/gdas/test/soca/gw/testrun/JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP.out.

The traceback flagged gdas.t12z.insitu_surface_trkob.2018041512.nc4 as a missing file. This, however, may not be the cause of the failure. The traceback ends with

  File "/work2/noaa/da/rtreadon/git/global-workflow/rename_atm/parm/gdas/jcb-algorithms/3dfgat.yaml.j2", line 45, in top-level template code
    {% include observation_from_jcb + '.yaml.j2' %}
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/site-packages/jinja2/loaders.py", line 218, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: insitu_profile_bathy.yaml.j2
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP 1717527206 1

[danholdaway]

Thanks for testing @RussTreadon-NOAA. Passing tests require https://github.com/danholdaway/global-workflow/tree/feature/rename_atm branch of global-workflow. You should be able to switch branch and run the tests again without a rebuild since it's just a change to jcb and config that matters.

[danholdaway]

Ah I see you did that. Perhaps the failure is because of more observations added to obs_list without those YAMLs going to JCB as well. Let me check.

[danholdaway]

@RussTreadon-NOAA I fixed that failure by adding the insitu YAML files to JCB. There should be zero downstream impact on the other tests, which should all pass again.

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Thu Jun  6 13:20:48 CDT 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Jun  6 14:10:32 CDT 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu Jun  6 14:25:32 CDT 2024
Tests: 64% tests passed, 17 tests failed out of 47
	1842 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1843 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1844 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1847 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1848 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1849 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
	1850 - test_gdasapp_soca_socahybridweights (Failed)
	1851 - test_gdasapp_soca_incr_handler (Failed)
	1852 - test_gdasapp_soca_ens_handler (Failed)
	1861 - test_gdasapp_atm_jjob_var_init (Failed)
	1862 - test_gdasapp_atm_jjob_var_run (Failed)
	1863 - test_gdasapp_atm_jjob_var_inc (Failed)
	1864 - test_gdasapp_atm_jjob_var_final (Failed)
	1865 - test_gdasapp_atm_jjob_ens_init (Failed)
	1866 - test_gdasapp_atm_jjob_ens_run (Failed)
	1867 - test_gdasapp_atm_jjob_ens_inc (Failed)
	1868 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/1144/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Tue Jun 18 22:05:37 UTC 2024 on hfe04
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jun 18 22:48:47 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Tue Jun 18 23:09:55 UTC 2024
Tests: 67% tests passed, 16 tests failed out of 48
	1843 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1844 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1845 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1848 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1849 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1850 - test_gdasapp_soca_socahybridweights (Failed)
	1851 - test_gdasapp_soca_incr_handler (Failed)
	1852 - test_gdasapp_soca_ens_handler (Failed)
	1861 - test_gdasapp_atm_jjob_var_init (Failed)
	1862 - test_gdasapp_atm_jjob_var_run (Failed)
	1863 - test_gdasapp_atm_jjob_var_inc (Failed)
	1864 - test_gdasapp_atm_jjob_var_final (Failed)
	1865 - test_gdasapp_atm_jjob_ens_init (Failed)
	1866 - test_gdasapp_atm_jjob_ens_run (Failed)
	1867 - test_gdasapp_atm_jjob_ens_inc (Failed)
	1868 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1144/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Jun 19 00:38:56 UTC 2024 on hfe04
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Jun 19 01:24:03 UTC 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Wed Jun 19 01:49:26 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 48

[RussTreadon-NOAA]

Both GDASApp ctests and g-w C96C48_ufs_hybatmDA CI run on various platforms. Results are acceptable.

Approve.

[RussTreadon-NOAA]

@CoryMartin-NOAA, @guillaumevernieres, and @DavidNew-NOAA - the changes in this PR have been tested via g-w PR #2700 with acceptable results. I'd like to merge this PR in GDASApp develop. Any objections?

@danholdaway has three JCB PRs related to GDASApp PR #1144 (this PR):

• jcb #7
• jcb-algoritms #2
• jcb-gdas #15

Each of the jcb PRs have been approved by Cory, David, and Russ. These jcb PRs should also be merged into their respective develop. Are we OK with doing so?

[DavidNew-NOAA]

@RussTreadon-NOAA No objection here