Cycle experiment on orion and hercules

[apchoiCMD]

Create the necessary yamls to run the gw-ci on Orion and Hercules. Leverage what was done in PR #1130 to facilitate this work.

Notes:

• The marine obs for cycling on Orion and Hercules were copied here: /work/noaa/da/marineda/gfs-marine/data
• The soca fix files are here: /work/noaa/da/marineda/gfs-marine/data/static
• The initial conditiions are here: /work2/noaa/marine/jmeixner/cycling/IC_2021063000_V2

[guillaumevernieres]

Trying to run the low-res g-w ctest:

• the mediator restart is not correct and needs to be removed
• no obs in glopara or /work/noaa/da/marineda/gfs-marine/data/obs
• on Orion and Heracules, iteration is needed to change under 100 at var.yaml, otherwise, issue about out of memery
• Account issue (da-cpu)
• ...

[apchoiCMD]

Until today, results are below and we need to add obs at least 20210324 to 20210327 from Hera Machine

(gdasapp) Orion-login-4[46] mindoc$ ctest -I 1,1
Test project /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci                                                                                            d/build/gdas/test/gw-ci
    Start 1: WCDA-3DVAR-C48mx500
1/1 Test #1: WCDA-3DVAR-C48mx500 ..............   Passed    5.65 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
manual    =   5.65 sec*proc (1 test)

Total Test time (real) =   5.66 sec
(gdasapp) Orion-login-4[48] mindoc$ ctest -I 2,2
Test project /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci
    Start 2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200
1/1 Test #2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200 ...   Passed  405.19 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
manual    = 405.19 sec*proc (1 test)

Total Test time (real) = 405.20 sec
(gdasapp) Orion-login-4[49] mindoc$ ctest -I 3,3
Test project /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci
    Start 3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
1/1 Test #3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 ...   Passed  626.01 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
manual    = 626.01 sec*proc (1 test)

Total Test time (real) = 626.01 sec
(gdasapp) Orion-login-4[50] mindoc$ ctest -I 4,4
Test project /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci
    Start 4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800
1/1 Test #4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800 ...   Passed   74.76 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
manual    =  74.76 sec*proc (1 test)

Total Test time (real) =  74.77 sec
(gdasapp) Orion-login-4[51] mindoc$ ctest -I 5,5
Test project /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci
    Start 5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800
1/1 Test #5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800 ...   Passed  115.45 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
manual    = 115.45 sec*proc (1 test)

Total Test time (real) = 115.46 sec
(gdasapp) Orion-login-4[52] mindoc$ ctest -I 6,6
Test project /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci
    Start 6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800
1/1 Test #6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800 ...***Failed  147.37 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
manual    = 147.37 sec*proc (1 test)

Total Test time (real) = 147.37 sec

The following tests FAILED:
          6 - WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800 (Failed)
Errors while running CTest
Output from these tests are in: /work2/noaa/da/mchoi3/workflow_06042024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[apchoiCMD]

Obs for gdas.20210324 to gdas.20210327 are staged at /work/noaa/da/marineda/gfs-marine/data/obs.

[guillaumevernieres]

Obs for gdas.20210324 to gdas.20210327 are staged at /work/noaa/da/marineda/gfs-marine/data/obs.

Thanks @apchoiCMD . Can you restart the DA ctests starting from WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800?

[apchoiCMD]

Obs for gdas.20210324 to gdas.20210327 are staged at /work/noaa/da/marineda/gfs-marine/data/obs.

Thanks @apchoiCMD . Can you restart the DA ctests starting from WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800?

Obs for gdas.20210324 to gdas.20210327 are staged at /work/noaa/da/marineda/gfs-marine/data/obs.

Thanks @apchoiCMD . Can you restart the DA ctests starting from WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800?

Working on it-

[guillaumevernieres]

@apchoiCMD , lets remind each other tomorrow to submit issues in the global-workflow to stage the fix files and gfs obs to glopara.

[apchoiCMD]

@apchoiCMD , let remind each other tomorrow to submit issues in the global-workflow to stage the fix files and gfs obs to glopara.

@guillaumevernieres yes, I will definitely remind you tomorrow for sure-