Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename mtiasi to iasi #1292

Merged
merged 2 commits into from
Nov 6, 2024
Merged

Rename mtiasi to iasi #1292

merged 2 commits into from
Nov 6, 2024

Conversation

emilyhcliu
Copy link
Collaborator

@emilyhcliu emilyhcliu commented Sep 23, 2024 •
edited
Loading

  1. Rename mtiasi to iasi in GDASApp
  2. There is a companion PR #29 in jcb-gdas
  • rename mtiasi to iasi
  • add reduce obs space
  • remove obsolete QC-flag related variables in filters

This partly resolves Issue #1291

@emilyhcliu emilyhcliu changed the title Rename mtiasi to iasi in observationn filter and converter Rename mtiasi to iasi Sep 24, 2024
@emilyhcliu emilyhcliu mentioned this pull request Sep 24, 2024
Merged
RussTreadon-NOAA
RussTreadon-NOAA reviewed Sep 24, 2024
View reviewed changes
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes in this PR are necessary but not sufficient.

Build feature/iasi inside g-w PR #2985. test_gdasapp works apart from know failures due to g-w PR #2920 not yet being in develop.

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/radbcor_iasi/sorc/gdas.cd/build
      Start 1531: test_gdasapp_util_coding_norms
 1/46 Test #1531: test_gdasapp_util_coding_norms .....................................   Passed    2.50 sec
      Start 1532: test_gdasapp_util_ioda_example
 2/46 Test #1532: test_gdasapp_util_ioda_example .....................................   Passed    2.93 sec
      Start 1533: test_gdasapp_util_prepdata
 3/46 Test #1533: test_gdasapp_util_prepdata .........................................   Passed    1.24 sec

...

      Start 1912: test_gdasapp_atm_jjob_ens_final
45/46 Test #1912: test_gdasapp_atm_jjob_ens_final ....................................   Passed   42.36 sec
      Start 1913: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1913: test_gdasapp_aero_gen_3dvar_yaml ...................................   Passed    0.80 sec

87% tests passed, 6 tests failed out of 46

Label Time Summary:
gdas-utils    =  10.06 sec*proc (13 tests)
manual        = 1433.11 sec*proc (9 tests)
script        =  10.06 sec*proc (13 tests)

Total Test time (real) = 2666.47 sec

The following tests FAILED:
        1893 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 (Failed)
        1894 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 (Failed)
        1895 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 (Subprocess killed)
        1896 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 (Subprocess killed)
        1897 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 (Subprocess killed)
        1898 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 (Subprocess killed)

Tests with g-w CI adding metop-b iasi fail. Changes are needed in

  • parm/jcb-gdas/observation_chronicle
  • parm/jcb-gdas/observations/atmosphere-lgetkf
  • parm/jcb-gdas/observations/atmosphere

These directories contain files named mtiasi*

observation_chronicle/atmosphere/mtiasi_metop-a.yaml
observation_chronicle/atmosphere/mtiasi_metop-b.yaml
observations/atmosphere-lgetkf/mtiasi_metop-a.yaml.j2
observations/atmosphere-lgetkf/mtiasi_metop-b.yaml.j2
observations/atmosphere/mtiasi_metop-a.yaml.j2
observations/atmosphere/mtiasi_metop-b.yaml.j2

These files need to be git moved to

observation_chronicle/atmosphere/iasi_metop-a.yaml
observation_chronicle/atmosphere/iasi_metop-b.yaml
observations/atmosphere-lgetkf/iasi_metop-a.yaml.j2
observations/atmosphere-lgetkf/iasi_metop-b.yaml.j2
observations/atmosphere/iasi_metop-a.yaml.j2
observations/atmosphere/iasi_metop-b.yaml.j2

@RussTreadon-NOAA
Copy link
Contributor

Attempt to run g-w C96C48_ufs_hybatmDA CI job gdasatmanlvar with Metop-B IASI on Hera. Job aborted because IASI bias correction files were not present. Add them to COMROOT and reran. Job failed with OOM error. Job was running 96 tasks on 8 nodes (12 tasks per node). Hera compute nodes have 96 Gb. Increase node count to 48, 2 tasks per node, and rerun. Job still fails with OOM errors.

@emilyhcliu , you mentioned reduced obs space. How is this configured in the yaml?

@emilyhcliu
Copy link
Collaborator Author

emilyhcliu commented Oct 9, 2024 •
edited
Loading

@RussTreadon-NOAA I thought no one had reviewed my PR yet.
Thanks so much for reviewing.

The reduce obs space is added to the thinning filter as the following:
https://github.com/NOAA-EMC/jcb-gdas/blob/4f60151866386c3a61dc53cabad9652b5ebda84c/observations/atmosphere/iasi_metop-b.yaml.j2#L169-L182

It is just adding one line under action in the data thinning filter:

action:
      name: reject
      name: reduce obs space

@emilyhcliu
Copy link
Collaborator Author

emilyhcliu commented Oct 9, 2024 •
edited
Loading

@RussTreadon-NOAA This PR has a companion PR #29 in jcb-gdas
@danholdaway had approved the PR but not merged yet.

@emilyhcliu
Copy link
Collaborator Author

emilyhcliu commented Oct 9, 2024 •
edited
Loading

This changes in this PR are necessary but not sufficient.

Build feature/iasi inside g-w PR #2985. test_gdasapp works apart from know failures due to g-w PR #2920 not yet being in develop.

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/radbcor_iasi/sorc/gdas.cd/build
      Start 1531: test_gdasapp_util_coding_norms
 1/46 Test #1531: test_gdasapp_util_coding_norms .....................................   Passed    2.50 sec
      Start 1532: test_gdasapp_util_ioda_example
 2/46 Test #1532: test_gdasapp_util_ioda_example .....................................   Passed    2.93 sec
      Start 1533: test_gdasapp_util_prepdata
 3/46 Test #1533: test_gdasapp_util_prepdata .........................................   Passed    1.24 sec

...

      Start 1912: test_gdasapp_atm_jjob_ens_final
45/46 Test #1912: test_gdasapp_atm_jjob_ens_final ....................................   Passed   42.36 sec
      Start 1913: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1913: test_gdasapp_aero_gen_3dvar_yaml ...................................   Passed    0.80 sec

87% tests passed, 6 tests failed out of 46

Label Time Summary:
gdas-utils    =  10.06 sec*proc (13 tests)
manual        = 1433.11 sec*proc (9 tests)
script        =  10.06 sec*proc (13 tests)

Total Test time (real) = 2666.47 sec

The following tests FAILED:
        1893 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 (Failed)
        1894 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 (Failed)
        1895 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 (Subprocess killed)
        1896 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 (Subprocess killed)
        1897 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 (Subprocess killed)
        1898 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 (Subprocess killed)

Tests with g-w CI adding metop-b iasi fail. Changes are needed in

  • parm/jcb-gdas/observation_chronicle
  • parm/jcb-gdas/observations/atmosphere-lgetkf
  • parm/jcb-gdas/observations/atmosphere

These directories contain files named mtiasi*

observation_chronicle/atmosphere/mtiasi_metop-a.yaml
observation_chronicle/atmosphere/mtiasi_metop-b.yaml
observations/atmosphere-lgetkf/mtiasi_metop-a.yaml.j2
observations/atmosphere-lgetkf/mtiasi_metop-b.yaml.j2
observations/atmosphere/mtiasi_metop-a.yaml.j2
observations/atmosphere/mtiasi_metop-b.yaml.j2

These files need to be git moved to

observation_chronicle/atmosphere/iasi_metop-a.yaml
observation_chronicle/atmosphere/iasi_metop-b.yaml
observations/atmosphere-lgetkf/iasi_metop-a.yaml.j2
observations/atmosphere-lgetkf/iasi_metop-b.yaml.j2
observations/atmosphere/iasi_metop-a.yaml.j2
observations/atmosphere/iasi_metop-b.yaml.j2

The PR #29 in jcb-gdas should resolve issues you mentioned.
I issued this PR and the above-mentioned jcb-gdas PR at the same time.
The jcb PR has been approved but has not been merged yet.

danholdaway added a commit to NOAA-EMC/jcb-gdas that referenced this pull request Oct 23, 2024
@emilyhcliu @danholdaway
Remove obselete entries and add "reduce obs space" option iasi filter (…
4bce236 
…#29)

This PR proposes the following changes for IASI:
1. Remove obsolete entries related to QC flags in the observation filter
2. Rename mtiasi to iasi
3. Add `reduce obs space

There is a companion [GDASApp PR
#1292](NOAA-EMC/GDASApp#1292)
This partly resolves Issue
NOAA-EMC/GDASApp#1291

Co-authored-by: Emily Liu <[email protected]>
Co-authored-by: Dan Holdaway <[email protected]>
@emilyhcliu
Copy link
Collaborator Author

@ADCollard Thanks for reviewing.

@CoryMartin-NOAA CoryMartin-NOAA merged commit 5bb63e7 into develop Nov 6, 2024
5 checks passed
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/iasi branch November 6, 2024 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RussTreadon-NOAA RussTreadon-NOAA RussTreadon-NOAA left review comments

@ADCollard ADCollard ADCollard approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@emilyhcliu @RussTreadon-NOAA @ADCollard @CoryMartin-NOAA