Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to run gw ctests using feature/submodule #828

Conversation

RussTreadon-NOAA
Copy link
Contributor

This PR includes two updates:

  1. bring develop at a34773a into feature/submodule
  2. update feature/submodule so that gw ctests run to completion.

One gw ctest, test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY, still fails after the changes in 2.

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Dec 31, 2023
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA:feature/submodule at 0a604a1 was installed on Hera and Orion inside gw develop at a3c5099. test_gdasapp, including gw tests, were run on both machines with the following results.

Hera ctests

(gdasapp) Hera(hfe03):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/build$ ctest --output-on-failure -R test_gdasapp > stdout_test_gdasapp.txt 2>&1
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/build
      Start 1339: test_gdasapp_util_coding_norms
 1/55 Test #1339: test_gdasapp_util_coding_norms ........................   Passed    1.58 sec
      Start 1340: test_gdasapp_util_ioda_example
 2/55 Test #1340: test_gdasapp_util_ioda_example ........................   Passed   13.45 sec
      Start 1341: test_gdasapp_util_prepdata
 3/55 Test #1341: test_gdasapp_util_prepdata ............................   Passed    3.80 sec
      Start 1342: test_gdasapp_util_rads2ioda
 4/55 Test #1342: test_gdasapp_util_rads2ioda ...........................   Passed    1.74 sec
      Start 1343: test_gdasapp_util_ghrsst2ioda
 5/55 Test #1343: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.41 sec
      Start 1344: test_gdasapp_util_smap2ioda
 6/55 Test #1344: test_gdasapp_util_smap2ioda ...........................   Passed    0.21 sec
      Start 1345: test_gdasapp_util_smos2ioda
 7/55 Test #1345: test_gdasapp_util_smos2ioda ...........................   Passed    0.28 sec
      Start 1346: test_gdasapp_util_viirsaod2ioda
 8/55 Test #1346: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.33 sec
      Start 1347: test_gdasapp_util_icecamsr2ioda
 9/55 Test #1347: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.19 sec
      Start 1679: test_gdasapp_check_python_norms
10/55 Test #1679: test_gdasapp_check_python_norms .......................   Passed    1.67 sec
      Start 1680: test_gdasapp_check_yaml_keys
11/55 Test #1680: test_gdasapp_check_yaml_keys ..........................   Passed    0.40 sec
      Start 1681: test_gdasapp_jedi_increment_to_fv3
12/55 Test #1681: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    5.67 sec
      Start 1682: test_gdasapp_convert_ewok_yaml
13/55 Test #1682: test_gdasapp_convert_ewok_yaml ........................   Passed    0.81 sec
      Start 1683: test_gdasapp_setup_cycled_exp
14/55 Test #1683: test_gdasapp_setup_cycled_exp .........................   Passed    2.10 sec
      Start 1684: test_gdasapp_convert_bufr_temp_dbuoy
15/55 Test #1684: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    4.13 sec
      Start 1685: test_gdasapp_convert_bufr_salt_dbuoy
16/55 Test #1685: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    1.21 sec
      Start 1686: test_gdasapp_convert_bufr_temp_mbuoyb
17/55 Test #1686: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.84 sec
      Start 1687: test_gdasapp_convert_bufr_salt_mbuoyb
18/55 Test #1687: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.23 sec
      Start 1688: test_gdasapp_convert_bufr_tesacprof
19/55 Test #1688: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.34 sec
      Start 1689: test_gdasapp_convert_bufr_trkobprof
20/55 Test #1689: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.21 sec
      Start 1690: test_gdasapp_convert_bufr_sfcships
21/55 Test #1690: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.30 sec
      Start 1691: test_gdasapp_convert_bufr_sfcshipsu
22/55 Test #1691: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.67 sec
      Start 1692: test_gdasapp_soca_obsdb
23/55 Test #1692: test_gdasapp_soca_obsdb ...............................   Passed    7.53 sec
      Start 1693: test_gdasapp_soca_nsst_increment_to_mom6
24/55 Test #1693: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   23.46 sec
      Start 1694: test_gdasapp_soca_prep
25/55 Test #1694: test_gdasapp_soca_prep ................................   Passed   10.30 sec
      Start 1695: test_gdasapp_soca_concatioda
26/55 Test #1695: test_gdasapp_soca_concatioda ..........................   Passed   21.88 sec
      Start 1696: test_gdasapp_soca_run_clean
27/55 Test #1696: test_gdasapp_soca_run_clean ...........................   Passed    0.16 sec
      Start 1697: test_gdasapp_soca_setup_obsproc
28/55 Test #1697: test_gdasapp_soca_setup_obsproc .......................   Passed    1.37 sec
      Start 1698: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
29/55 Test #1698: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   79.09 sec
      Start 1699: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
30/55 Test #1699: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   74.96 sec
      Start 1700: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
31/55 Test #1700: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed  1034.91 sec
      Start 1701: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
32/55 Test #1701: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed  331.18 sec
      Start 1702: test_gdasapp_soca_copy_scratch
33/55 Test #1702: test_gdasapp_soca_copy_scratch ........................   Passed    1.49 sec
      Start 1703: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
34/55 Test #1703: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.65 sec
      Start 1704: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
35/55 Test #1704: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   10.68 sec
      Start 1705: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
36/55 Test #1705: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed   42.26 sec
{'machine': 'hera', 'resdet': 48, 'resens': 48, 'jjobs': ['JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY'], 'gw environement': {'experiment identifier': {'PSLOT': 'gdas_test', 'HOMEgfs': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../..', 'COMPONENT': 'ocean', 'RUN': 'gdas', 'CDUMP': 'gdas', 'COMROOT': 'dummy_COMROOT', 'jobid': 'dummy_jobid'}, 'cycle info': {'PDY': 20180415, 'cyc': 12, 'assym_freq': 6}, 'backgrounds': {'COM_SRC': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/COM'}, 'working directories': {'ROTDIRS': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs/ROTDIRS', 'EXPDIRS': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs/experiments', 'STMP': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs'}, 'jedi': {'OOPS_TRACE': 1, 'OOPS_DEBUG': 1, 'OMP_NUM_THREADS': 1}, 'run scripts': {'GDASPREPOCNOBSPY': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exglobal_prep_ocean_obs.py'}}, 'setup_expt config': {'base': {'DO_JEDIATMVAR': 'NO', 'DO_JEDIATMENS': 'NO', 'DO_JEDIOCNVAR': 'YES', 'DO_JEDILANDDA': 'NO', 'DO_MERGENSST': 'NO', 'NMEM_ENS': '4', 'DOHYBVAR': 'YES'}, 'ocnanal': {'SOCA_INPUT_FIX_DIR': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/soca_static', 'CASE_ANL': 'C48', 'SOCA_OBS_LIST': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/obs/obs_list_small.yaml', 'COMIN_OBS': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/obs/r2d2-shared', 'SOCA_NINNER': 1, 'R2D2_OBS_SRC': 'gdasapp', 'R2D2_OBS_DUMP': 'soca', 'SABER_BLOCKS_YAML': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/berror/saber_blocks.yaml', 'NICAS_RESOL': 1, 'NICAS_GRID_SIZE': 150}, 'prepoceanobs': {'SOCA_OBS_LIST': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/obsproc/obs_list_obsproc.yaml', 'OBSPROC_CONFIG': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/obsproc/obsproc_config_test.yaml', 'DMPDIR': '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/test/soca/gw/obsproc'}}, 'job options': {'account': 'da-cpu', 'qos': 'batch', 'output': 'JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.out', 'nodes': 1, 'ntasks': 16, 'partition': 'hera', 'time': '00:10:00'}}
machine is hera
gPDY: 20180415
gcyc: 06
assim_freq: 6
RUN: gdas
running sbatch --wait run_jjobs.sh ...
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush/soca/run_jjobs.py", line 302, in <module>
    main()
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush/soca/run_jjobs.py", line 298, in main
    run_card.execute(submit=True)
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush/soca/run_jjobs.py", line 253, in execute
    subprocess.check_output(["sbatch", "--wait", self.name])
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['sbatch', '--wait', 'run_jjobs.sh']' returned non-zero exit status 1.

      Start 1706: test_gdasapp_soca_socahybridweights
37/55 Test #1706: test_gdasapp_soca_socahybridweights ...................   Passed   74.63 sec
      Start 1707: test_gdasapp_soca_incr_handler
38/55 Test #1707: test_gdasapp_soca_incr_handler ........................   Passed  299.33 sec
      Start 1708: test_gdasapp_soca_ens_handler
39/55 Test #1708: test_gdasapp_soca_ens_handler .........................   Passed   42.84 sec
      Start 1709: test_gdasapp_land_create_ens
40/55 Test #1709: test_gdasapp_land_create_ens ..........................   Passed    5.15 sec
      Start 1710: test_gdasapp_land_imsproc
41/55 Test #1710: test_gdasapp_land_imsproc .............................   Passed   12.70 sec
      Start 1711: test_gdasapp_land_apply_jediincr
42/55 Test #1711: test_gdasapp_land_apply_jediincr ......................   Passed    3.73 sec
      Start 1712: test_gdasapp_land_letkfoi_snowda
43/55 Test #1712: test_gdasapp_land_letkfoi_snowda ......................   Passed   42.26 sec
      Start 1713: test_gdasapp_convert_bufr_adpsfc_snow
44/55 Test #1713: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    6.23 sec
      Start 1714: test_gdasapp_convert_bufr_adpsfc
45/55 Test #1714: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.55 sec
      Start 1715: test_gdasapp_convert_gsi_satbias
46/55 Test #1715: test_gdasapp_convert_gsi_satbias ......................   Passed    1.67 sec
      Start 1716: test_gdasapp_store_gsi_satbias
47/55 Test #1716: test_gdasapp_store_gsi_satbias ........................   Passed    4.10 sec
      Start 1717: test_gdasapp_setup_atm_cycled_exp
48/55 Test #1717: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.23 sec
      Start 1718: test_gdasapp_atm_jjob_var_init
49/55 Test #1718: test_gdasapp_atm_jjob_var_init ........................   Passed   43.90 sec
      Start 1719: test_gdasapp_atm_jjob_var_run
50/55 Test #1719: test_gdasapp_atm_jjob_var_run .........................   Passed  106.13 sec
      Start 1720: test_gdasapp_atm_jjob_var_final
51/55 Test #1720: test_gdasapp_atm_jjob_var_final .......................   Passed   42.17 sec
      Start 1721: test_gdasapp_atm_jjob_ens_init
52/55 Test #1721: test_gdasapp_atm_jjob_ens_init ........................   Passed   43.80 sec
      Start 1722: test_gdasapp_atm_jjob_ens_run
53/55 Test #1722: test_gdasapp_atm_jjob_ens_run .........................   Passed  330.19 sec
      Start 1723: test_gdasapp_atm_jjob_ens_final
54/55 Test #1723: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.15 sec
      Start 1724: test_gdasapp_aero_gen_3dvar_yaml
55/55 Test #1724: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.74 sec

98% tests passed, 1 tests failed out of 55

Label Time Summary:
gdas-utils    =  21.99 sec*proc (9 tests)
script        =  21.99 sec*proc (9 tests)

Total Test time (real) = 2832.31 sec

The following tests FAILED:
        1705 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest
(gdasapp) Hera(hfe03):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/build$

Orion ctests

(gdasapp) Orion-login-2:/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/build$ ctest --output-on-failure -R test_gdasapp
Test project /work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/build
      Start 1339: test_gdasapp_util_coding_norms
 1/55 Test #1339: test_gdasapp_util_coding_norms ........................   Passed    2.97 sec
      Start 1340: test_gdasapp_util_ioda_example
 2/55 Test #1340: test_gdasapp_util_ioda_example ........................   Passed   11.41 sec
      Start 1341: test_gdasapp_util_prepdata
 3/55 Test #1341: test_gdasapp_util_prepdata ............................   Passed    3.60 sec
      Start 1342: test_gdasapp_util_rads2ioda
 4/55 Test #1342: test_gdasapp_util_rads2ioda ...........................   Passed    1.56 sec
      Start 1343: test_gdasapp_util_ghrsst2ioda
 5/55 Test #1343: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.13 sec
      Start 1344: test_gdasapp_util_smap2ioda
 6/55 Test #1344: test_gdasapp_util_smap2ioda ...........................   Passed    0.13 sec
      Start 1345: test_gdasapp_util_smos2ioda
 7/55 Test #1345: test_gdasapp_util_smos2ioda ...........................   Passed    0.42 sec
      Start 1346: test_gdasapp_util_viirsaod2ioda
 8/55 Test #1346: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.15 sec
      Start 1347: test_gdasapp_util_icecamsr2ioda
 9/55 Test #1347: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.14 sec
      Start 1679: test_gdasapp_check_python_norms
10/55 Test #1679: test_gdasapp_check_python_norms .......................   Passed    1.89 sec
      Start 1680: test_gdasapp_check_yaml_keys
11/55 Test #1680: test_gdasapp_check_yaml_keys ..........................   Passed    1.36 sec
      Start 1681: test_gdasapp_jedi_increment_to_fv3
12/55 Test #1681: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    8.68 sec
      Start 1682: test_gdasapp_convert_ewok_yaml
13/55 Test #1682: test_gdasapp_convert_ewok_yaml ........................   Passed    0.18 sec
      Start 1683: test_gdasapp_setup_cycled_exp
14/55 Test #1683: test_gdasapp_setup_cycled_exp .........................   Passed    2.22 sec
      Start 1684: test_gdasapp_convert_bufr_temp_dbuoy
15/55 Test #1684: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    2.87 sec
      Start 1685: test_gdasapp_convert_bufr_salt_dbuoy
16/55 Test #1685: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.21 sec
      Start 1686: test_gdasapp_convert_bufr_temp_mbuoyb
17/55 Test #1686: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.20 sec
      Start 1687: test_gdasapp_convert_bufr_salt_mbuoyb
18/55 Test #1687: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.20 sec
      Start 1688: test_gdasapp_convert_bufr_tesacprof
19/55 Test #1688: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.21 sec
      Start 1689: test_gdasapp_convert_bufr_trkobprof
20/55 Test #1689: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.20 sec
      Start 1690: test_gdasapp_convert_bufr_sfcships
21/55 Test #1690: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.20 sec
      Start 1691: test_gdasapp_convert_bufr_sfcshipsu
22/55 Test #1691: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.20 sec
      Start 1692: test_gdasapp_soca_obsdb
23/55 Test #1692: test_gdasapp_soca_obsdb ...............................   Passed   10.02 sec
      Start 1693: test_gdasapp_soca_nsst_increment_to_mom6
24/55 Test #1693: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   30.80 sec
      Start 1694: test_gdasapp_soca_prep
25/55 Test #1694: test_gdasapp_soca_prep ................................   Passed    4.87 sec
      Start 1695: test_gdasapp_soca_concatioda
26/55 Test #1695: test_gdasapp_soca_concatioda ..........................   Passed    2.36 sec
      Start 1696: test_gdasapp_soca_run_clean
27/55 Test #1696: test_gdasapp_soca_run_clean ...........................   Passed    0.09 sec
      Start 1697: test_gdasapp_soca_setup_obsproc
28/55 Test #1697: test_gdasapp_soca_setup_obsproc .......................   Passed    0.53 sec
      Start 1698: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
29/55 Test #1698: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.75 sec
      Start 1699: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
30/55 Test #1699: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   10.21 sec
      Start 1700: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
31/55 Test #1700: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   42.18 sec
      Start 1701: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
32/55 Test #1701: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.19 sec
      Start 1702: test_gdasapp_soca_copy_scratch
33/55 Test #1702: test_gdasapp_soca_copy_scratch ........................   Passed    1.34 sec
      Start 1703: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
34/55 Test #1703: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.30 sec
      Start 1704: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
35/55 Test #1704: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   10.19 sec
      Start 1705: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
36/55 Test #1705: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed   42.22 sec
{'machine': 'orion', 'resdet': 48, 'resens': 48, 'jjobs': ['JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY'], 'gw environement': {'experiment identifier': {'PSLOT': 'gdas_test', 'HOMEgfs': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../..', 'COMPONENT': 'ocean', 'RUN': 'gdas', 'CDUMP': 'gdas', 'COMROOT': 'dummy_COMROOT', 'jobid': 'dummy_jobid'}, 'cycle info': {'PDY': 20180415, 'cyc': 12, 'assym_freq': 6}, 'backgrounds': {'COM_SRC': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/COM'}, 'working directories': {'ROTDIRS': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs/ROTDIRS', 'EXPDIRS': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs/experiments', 'STMP': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs'}, 'jedi': {'OOPS_TRACE': 1, 'OOPS_DEBUG': 1, 'OMP_NUM_THREADS': 1}, 'run scripts': {'GDASPREPOCNOBSPY': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exglobal_prep_ocean_obs.py'}}, 'setup_expt config': {'base': {'DO_JEDIATMVAR': 'NO', 'DO_JEDIATMENS': 'NO', 'DO_JEDIOCNVAR': 'YES', 'DO_JEDILANDDA': 'NO', 'DO_MERGENSST': 'NO', 'NMEM_ENS': '4', 'DOHYBVAR': 'YES'}, 'ocnanal': {'SOCA_INPUT_FIX_DIR': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/soca_static', 'CASE_ANL': 'C48', 'SOCA_OBS_LIST': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/obs/obs_list_small.yaml', 'COMIN_OBS': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/obs/r2d2-shared', 'SOCA_NINNER': 1, 'R2D2_OBS_SRC': 'gdasapp', 'R2D2_OBS_DUMP': 'soca', 'SABER_BLOCKS_YAML': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/berror/saber_blocks.yaml', 'NICAS_RESOL': 1, 'NICAS_GRID_SIZE': 150}, 'prepoceanobs': {'SOCA_OBS_LIST': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/obsproc/obs_list_obsproc.yaml', 'OBSPROC_CONFIG': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/parm/soca/obsproc/obsproc_config_test.yaml', 'DMPDIR': '/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/test/soca/gw/obsproc'}}, 'job options': {'account': 'da-cpu', 'qos': 'batch', 'output': 'JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.out', 'nodes': 1, 'ntasks': 16, 'partition': 'orion', 'time': '00:10:00'}}
machine is orion
gPDY: 20180415
gcyc: 06
assim_freq: 6
RUN: gdas
running sbatch --wait run_jjobs.sh ...
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush/soca/run_jjobs.py", line 302, in <module>
    main()
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush/soca/run_jjobs.py", line 298, in main
    run_card.execute(submit=True)
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush/soca/run_jjobs.py", line 253, in execute
    subprocess.check_output(["sbatch", "--wait", self.name])
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['sbatch', '--wait', 'run_jjobs.sh']' returned non-zero exit status 1.

      Start 1706: test_gdasapp_soca_socahybridweights
37/55 Test #1706: test_gdasapp_soca_socahybridweights ...................   Passed   42.21 sec
      Start 1707: test_gdasapp_soca_incr_handler
38/55 Test #1707: test_gdasapp_soca_incr_handler ........................   Passed   10.16 sec
      Start 1708: test_gdasapp_soca_ens_handler
39/55 Test #1708: test_gdasapp_soca_ens_handler .........................   Passed   10.16 sec
      Start 1709: test_gdasapp_land_create_ens
40/55 Test #1709: test_gdasapp_land_create_ens ..........................   Passed    1.42 sec
      Start 1710: test_gdasapp_land_imsproc
41/55 Test #1710: test_gdasapp_land_imsproc .............................   Passed    4.02 sec
      Start 1711: test_gdasapp_land_apply_jediincr
42/55 Test #1711: test_gdasapp_land_apply_jediincr ......................   Passed    3.03 sec
      Start 1712: test_gdasapp_land_letkfoi_snowda
43/55 Test #1712: test_gdasapp_land_letkfoi_snowda ......................   Passed   11.09 sec
      Start 1713: test_gdasapp_convert_bufr_adpsfc_snow
44/55 Test #1713: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    3.73 sec
      Start 1714: test_gdasapp_convert_bufr_adpsfc
45/55 Test #1714: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.18 sec
      Start 1715: test_gdasapp_convert_gsi_satbias
46/55 Test #1715: test_gdasapp_convert_gsi_satbias ......................   Passed    1.47 sec
      Start 1716: test_gdasapp_store_gsi_satbias
47/55 Test #1716: test_gdasapp_store_gsi_satbias ........................   Passed    0.60 sec
      Start 1717: test_gdasapp_setup_atm_cycled_exp
48/55 Test #1717: test_gdasapp_setup_atm_cycled_exp .....................   Passed    0.61 sec
      Start 1718: test_gdasapp_atm_jjob_var_init
49/55 Test #1718: test_gdasapp_atm_jjob_var_init ........................   Passed   44.75 sec
      Start 1719: test_gdasapp_atm_jjob_var_run
50/55 Test #1719: test_gdasapp_atm_jjob_var_run .........................   Passed  106.22 sec
      Start 1720: test_gdasapp_atm_jjob_var_final
51/55 Test #1720: test_gdasapp_atm_jjob_var_final .......................   Passed   42.17 sec
      Start 1721: test_gdasapp_atm_jjob_ens_init
52/55 Test #1721: test_gdasapp_atm_jjob_ens_init ........................   Passed   44.52 sec
      Start 1722: test_gdasapp_atm_jjob_ens_run
53/55 Test #1722: test_gdasapp_atm_jjob_ens_run .........................   Passed  298.21 sec
      Start 1723: test_gdasapp_atm_jjob_ens_final
54/55 Test #1723: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.17 sec
      Start 1724: test_gdasapp_aero_gen_3dvar_yaml
55/55 Test #1724: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.25 sec

98% tests passed, 1 tests failed out of 55

Label Time Summary:
gdas-utils    =  20.50 sec*proc (9 tests)
script        =  20.50 sec*proc (9 tests)

Total Test time (real) = 992.83 sec

The following tests FAILED:
        1705 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest
(gdasapp) Orion-login-2:/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/build$

@RussTreadon-NOAA
Copy link
Contributor Author

test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY failure

test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY failed for the same reason on Hera and Orion. The initial failure is due to exgdas_global_marine_analysis_vrfy.py being unable to local a python module

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[36]: /work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 22, in <module>
    import gen_eva_obs_yaml
ModuleNotFoundError: No module named 'gen_eva_obs_yaml'

This is puzzling since PYTHONPATH includes /sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/eva. This is the directory in which gen_eva_obs_yaml.py resides. A check of online document suggested that sorc/gdas/../../sorc/gdas.cd was problematic. gdas is a soft link pointing up one directory: gdas -> ../ The real path to the eva directory inside sorc/gdas.cd is sorc/gdas.cd/ush/eva.

ctest submits test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY via run_jjobs.sh from build/gdas/test/soca/gw/testrun. run_jjobs.sh sets PYTHONPATH. Manually edit PYTHONPATH in local copy of run_jjobs.sh to include the real path to the eva and soca ush directories. Submit run_jjobs.sh. exgdas_global_marine_analysis_vrfy.py got further but failed for a different reason: file gdas.t09z.ocngrid.nc could not be found. The path to the file was incorrect. Make the following change in exgdas_global_marine_analysis_vrfy.py

-comout = os.getenv('COM_OCEAN_ANALYSIS')
-com_ice_history = os.getenv('COM_ICE_HISTORY_PREV')
-com_ocean_history = os.getenv('COM_OCEAN_HISTORY_PREV')
+comout = os.path.realpath(os.getenv('COM_OCEAN_ANALYSIS'))
+com_ice_history = os.path.realpath(os.getenv('COM_ICE_HISTORY_PREV'))
+com_ocean_history = os.path.realpath(os.getenv('COM_OCEAN_HISTORY_PREV'))

Resubmit. Ran further but still failed as follows (log from Hera)

INFO DataCollections:   hofxQC::seaIceFraction       (float32) | Min=+0.0000e+00, Max=+9.9999e-01, RMS=+6.6021e-01
INFO DataCollections: --------------------------------------------------------------------------------
INFO EvaSetup: Running figure driver
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/eva/bin/eva", line 8, in <module>
    sys.exit(main())
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py"\
, line 120, in main
    eva(config_file)
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py"\
, line 85, in eva
    figure_driver(eva_dict, data_collections, timing, logger)
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/plotting/batch\
/base/plot_tools/figure_driver.py", line 43, in figure_driver
    graphics = graphics_section.get('figure_list')
AttributeError: 'list' object has no attribute 'get'
running eva on evayamls/eva_icec_emc_2018041512.yaml
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_\
vrfy.py", line 230, in <module>
    subprocess.run(['eva', infile], check=True)
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['eva', 'evayamls/eva_icec_emc_2018041512.yaml']' returned non-zero exit status 1.

Stop investigation here since we may need updates to /scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/eva. Similar error observed on Orion.

@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review December 31, 2023 21:30
@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Dec 31, 2023
@emcbot emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed orion-GW-RT Queue for automated testing with global-workflow on Orion hera-GW-RT Queue for automated testing with global-workflow on Hera labels Dec 31, 2023
@emcbot
Copy link

emcbot commented Dec 31, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Sun Dec 31 22:01:05 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Dec 31 22:53:12 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Sun Dec 31 23:53:06 UTC 2023
Tests: 98% tests passed, 1 tests failed out of 55
	1705 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/828/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Dec 31, 2023
@RussTreadon-NOAA
Copy link
Contributor Author

orion-GW-RT-running

The automated Orion global-workflow test is still marked as running. A check of /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/828/global-workflow/sorc/log.build shows

Creating logs folder
Creating ../exec folder
./build_all.sh: line 83: gfs_utils.fd/ush/detect_machine.sh: No such file or directory
./build_all.sh: line 84: gfs_utils.fd/ush/module-setup.sh: No such file or directory
FATAL: Unable to determine target machine

du -k in the sorc directory returns

Orion-login-2:/work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/828/global-workflow/sorc$ du -k
4       ./ufs_model.fd
4       ./gsi_enkf.fd
4       ./gsi_monitor.fd
4       ./gdas.cd
4       ./wxflow
4       ./verif-global.fd
4       ./gsi_utils.fd
4       ./logs
4       ./ufs_utils.fd
4       ./gfs_utils.fd
116     .

This is odd. RussTreadon-NOAA:feature/submodule ci/gw_driver.sh includes the --recursive option when cloning global-workflow

workflow_url="https://github.com/NOAA-EMC/global-workflow.git"
# loop through all open PRs
for pr in $open_pr_list; do
  gh pr edit $pr --remove-label $CI_LABEL --add-label ${CI_LABEL}-Running
  echo "Processing Pull Request #${pr}"
  mkdir -p $GDAS_CI_ROOT/workflow/PR/$pr
  cd $GDAS_CI_ROOT/workflow/PR/$pr

  # clone global workflow develop branch
  git clone --recursive $workflow_url

The same script is executed on Hera where the GW-RT clone, build, and execute ran.

@RussTreadon-NOAA
Copy link
Contributor Author

@CoryMartin-NOAA and @guillaumevernieres , what is the role of the gdas -> ../ link in sorc?

Using Hera as the example, we wind up with PROJECT_SOURCE_DIR=/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas which actually points at /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd

PROJECT_SOURCE_DIR is used throughout test/soca/gw/CMakeLists.txt The ../ appears problematic in some spots.

test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP example

When this test executes exgdas_global_marine_analysis_prep.py, the script fails with the following

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 199 in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/site-packages/xarray/backends/lru_cache.py", line 53, in__getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas.cd/build/gdas/tst/soca/gw/testrun/testjjobs/ROTDIRS/gdas_test/gdas.20180415/06/model_data/ice/history/gdas.t06z.icef003.nc',), 'r', (('clobber', True), ('diskles', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysisprep.py", line 432, in <module>
    yaml_name='bkg_list.yaml')
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysisprep.py", line 174, in gen_bkg_list
    os.path.join(out_path, agg_ice_filename))
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysisprep.py", line 98, in cice_hist2fms
    ds = xr.open_dataset(input_filename)

In this failure variable input_filename is /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/build/gdas/test/soca/gw/tesrun/testjjobs/ROTDIRS/gdas_test/gdas.20180415/06//model_data/ice/history/gdas.t06z.icef003.nc

If os.path.realpath is applied toinput_filename and this result is passed xr.open_dataset as shown below

    input_filename_real = os.path.realpath(input_filename)

    # open the CICE history file
    ds = xr.open_dataset(input_filename_real)

the script runs to completion.

os.path.realpath converts input_filename to /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/build/gdas/test/soca/gw/testrun/testjjobs/ROTDIRS/gdas_test/gdas.20180415/06/model_data/ice/history/gdas.t06z.icef003.nc. The sorc/gdas/../../sorc/gdas.cd part of input_filename is removed.

test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY example

As mentioned above, the test initially fails because an eva module is not found

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[36]: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 22, in <module>
    import gen_eva_obs_yaml
ModuleNotFoundError: No module named 'gen_eva_obs_yaml'

What's odd is that PYTHONPATH includes eva modules. The log file shows

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[30]: PYTHONPATH=
/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/soca:
/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/:
/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/eva:
/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../ush/python/wxflow/src:
/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/ush:
/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/build/gdas/../lib/python3.7:
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.1/envs/unified-env/install/intel/2021.5.0/py-pybind11-2.8.1-qbii5yd/lib/python3.10/site-packages:
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.1/envs/unified-env/install/intel/2021.5.0/bufr-12.0.1-ud2j3kr/lib64/python3.10/site-packages:
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.1/envs/unified-env/install/intel/2021.5.0/bufr-12.0.1-ud2j3kr/lib/python3.10/site-packages

If sorc/gdas/../../sorc/gdas.cd/ is removed from the paths to ush, ush/soca, and ush/eva, then exgdas_global_marine_analysis_vrfy.py gets past the initial module imports but fails later with the previously mentioned error.

What's a good solution to the above issues?

@CoryMartin-NOAA
Copy link
Contributor

@RussTreadon-NOAA the role of gdas -> ../ was to move the ecbuild bundle into sorc/ to make things less messy when the repositories are cloned, but allow for the rest of GDASApp to build with cmake.

I think those ../ spots need modified to ../.. now that things are one directory down from where they were previously.

My guess is that the $PYTHONPATH is also no longer correct somehow. @guillaumevernieres any comments on what the best way to modify these SOCA tests are?

Is putting everything in sorc/ going to be worth the hassle?

@guillaumevernieres
Copy link
Contributor

@RussTreadon-NOAA the role of gdas -> ../ was to move the ecbuild bundle into sorc/ to make things less messy when the repositories are cloned, but allow for the rest of GDASApp to build with cmake.

I think those ../ spots need modified to ../.. now that things are one directory down from where they were previously.

My guess is that the $PYTHONPATH is also no longer correct somehow. @guillaumevernieres any comments on what the best way to modify these SOCA tests are?

Is putting everything in sorc/ going to be worth the hassle?

I think it's worth the effort @CoryMartin-NOAA . If the only issue is the ocean verify test, we can commented out for now and fix it later. I'll try to find some time to look into it today, but we've got an other blocking issue that I need to fix first.

@RussTreadon-NOAA
Copy link
Contributor Author

Is placing jedi repos in sorc/ an EE2 requirement? Would the following work: keep jedi repos at the same level as they are in develop but replace them with submodules? I say this in ignorance since I don't understand the inner workings of cmake and ecbuild.

If we want to stick with submodules in sorc/, RussTreadon-NOAA:feature/submodule at 0a604a1 builds on both Hera and Orion. All GW test-gdasapp Pass except test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.

@CoryMartin-NOAA
Copy link
Contributor

My main concern (perhaps it is being too OCD) is that the root directory of GDASApp then becomes littered with 10+ submodules instead of just a few subdirectories like sorc/ ush/ test/

I think there are probably a few cmake tricks we just need to learn to make this whole process easier. But I just need to figure out which breadcrumb to follow first.

@RussTreadon-NOAA
Copy link
Contributor Author

I prefer the cleaner look of sorc/ but getting there is proving to be much more time consuming than I anticipated.

Is there a way to define HOMEgfs in test/soca/gw/CMakeLists.txt without /../..?. Currently we have

  # substitute a few variables in the test yaml
  message("ctest for ${jjob}")
  set(HOMEgfs ${PROJECT_SOURCE_DIR}/../..)

The ATM gw ctests set HOMEgfs at script level via

# Set g-w HOMEgfs
topdir=$(cd "$(dirname "$(readlink -f -n "${bindir}" )" )/../../.." && pwd -P)
export HOMEgfs=$topdir

Might Rahul have useful cmake insights for this and other questions we have?

@CoryMartin-NOAA
Copy link
Contributor

I think we want (maybe?)
set(HOMEgfs ${PROJECT_SOURCE_DIR}/../..)
to become
set(HOMEgfs ${CMAKE_SOURCE_DIR}/)
or something like that?

@RussTreadon-NOAA
Copy link
Contributor Author

Added prints to cmake. For my Hera installation CMAKE_SOURCE_DIR is /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/sorc. With this, we need set(HOMEgfs ${CMAKE_SOURCE_DIR}/../../..)

Using CMAKE_SOURCE_DIR avoids the soft link sorc/gdas -> ../. Sidestepping the soft link may be beneficial. I don't know without testing.

@RussTreadon-NOAA RussTreadon-NOAA added orion-GW-RT Queue for automated testing with global-workflow on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 2, 2024
@emcbot emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jan 2, 2024
@CoryMartin-NOAA
Copy link
Contributor

Looking more closely at this, the EVA test will fail because the EVA version is updated but the YAML for this ctest was not updated. @guillaumevernieres suggested yesterday to comment out this ctest and it would get fixed at a later date.

@CoryMartin-NOAA CoryMartin-NOAA removed the orion-GW-RT-Running Automated testing with global-workflow running on Orion label Jan 3, 2024
@RussTreadon-NOAA
Copy link
Contributor Author

I'm fine with punting the test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY fix to a new PR.

That said, do we know

  1. which eva yaml(s) need to be changed
  2. what needs to be changed in the eva yaml(s)?

If we know the answer to these questions, I can make the eva yaml(s) updates in this PR.

@CoryMartin-NOAA
Copy link
Contributor

it might be as simple as adding the following:

  plotting_backend: Emcpy
  figure_list:

to the graphics section of the YAML, see: https://github.com/JCSDA-internal/eva/pull/158/files
Not sure how many files in GDASApp are affected, perhaps only https://github.com/NOAA-EMC/GDASApp/blob/develop/ush/eva/marine_gdas_plots.yaml for the ctest (but others in that directory probably also need updated)

@RussTreadon-NOAA
Copy link
Contributor Author

Great advice @CoryMartin-NOAA. I simply added

 graphics:

+  plotting_backend: Emcpy
+
   # ---------- Map Plots ----------

to ush/eva/marine_gdas_plots.yaml. A rerun of test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY got further. The traceback now points at processing of preevayamls/eva_icec_emc_2018041512.yaml.

2024-01-03 16:11:44:INFO:Writing replaced template to preevayamls/eva_icec_emc_2018041512.yaml
2024-01-03 16:11:44:ERROR:Error occurred when attempting to load: preevayamls/eva_icec_emc_2018041512.yaml, error: while parsing a block mapping
  in "preevayamls/eva_icec_emc_2018041512.yaml", line 64, column 3
expected <block end>, but found '-'
  in "preevayamls/eva_icec_emc_2018041512.yaml", line 70, column 3
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 224, in <module>
    marine_eva_post.marine_eva_post(infile, 'evayamls', diagdir)
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/ush/eva/marine_eva_post.py", line 26, in marine_eva_post
    for dataset in input_yaml_dict['datasets']:
UnboundLocalError: local variable 'input_yaml_dict' referenced before assignment
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1704297986 1

Any suggestions @guillaumevernieres or @AndrewEichmann-NOAA ?

The above is copied from /work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/build/gdas/test/soca/gw/testrun/JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.o16133803

@guillaumevernieres
Copy link
Contributor

Great advice @CoryMartin-NOAA. I simply added

 graphics:

+  plotting_backend: Emcpy
+
   # ---------- Map Plots ----------

to ush/eva/marine_gdas_plots.yaml. A rerun of test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY got further. The traceback now points at processing of preevayamls/eva_icec_emc_2018041512.yaml.

2024-01-03 16:11:44:INFO:Writing replaced template to preevayamls/eva_icec_emc_2018041512.yaml
2024-01-03 16:11:44:ERROR:Error occurred when attempting to load: preevayamls/eva_icec_emc_2018041512.yaml, error: while parsing a block mapping
  in "preevayamls/eva_icec_emc_2018041512.yaml", line 64, column 3
expected <block end>, but found '-'
  in "preevayamls/eva_icec_emc_2018041512.yaml", line 70, column 3
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 224, in <module>
    marine_eva_post.marine_eva_post(infile, 'evayamls', diagdir)
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/ush/eva/marine_eva_post.py", line 26, in marine_eva_post
    for dataset in input_yaml_dict['datasets']:
UnboundLocalError: local variable 'input_yaml_dict' referenced before assignment
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1704297986 1

Any suggestions @guillaumevernieres or @AndrewEichmann-NOAA ?

The above is copied from /work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/build/gdas/test/soca/gw/testrun/JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.o16133803

My suggestion is to comment out and ignore for now @RussTreadon-NOAA . We're all tied up doing other things right now but we should be able to get to it before the end of the week. Thanks for trying though!

Copy link
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RussTreadon-NOAA and others added 4 commits January 4, 2024 16:22
@emilyhcliu found that permission restrictions on
`bufr2ioda_subpfl_argo_profiles.py` prevent it from being executed.
[`feature/chmod`](https://github.com/NOAA-EMC/GDASApp/tree/feature/chmod)
was created and the file permissions changed so that
`bufr2ioda_subpfl_argo_profiles.py` is executable.

Fixes NOAA-EMC#788
- The UFO filters are rearranged by using the explicit filter ordering
with `obs pre filters`, `obs prior filters` and `obs post filters`
options.
- Add `Temporal Thinning filter` into the `obs post filters` section and
and arrange it after the `Background Check filter` and before the `Buddy
check filter`.
- Rename the IMS IODA file name and copy it into the `rundir/obs`
directory for the late use in the snow analysis in the global workflow
(NOAA-EMC/global-workflow#2033).

Co-authored-by: Cory Martin <[email protected]>
@RussTreadon-NOAA RussTreadon-NOAA deleted the branch NOAA-EMC:feature/submodule January 10, 2024 15:41
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/submodule branch January 10, 2024 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hera-GW-RT-Failed Automated testing with global-workflow failed on Hera
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants