Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ODAS updates to allow runing reanalysis on Milan #6

Merged
merged 3 commits into from
Oct 9, 2024

Conversation

vrx-
Copy link
Contributor

@vrx- vrx- commented Oct 9, 2024

Fixes the following issues:

  1. Pagepool memory limitation in SLES17 to read observation files.

  2. Users without priority stuck in queue with observer jobs.

  3. Model BOMBING when ADT sigo variance is below a threshold. This is expected, but the user needs to either correct altimeter observations or remove the problematic altimeter for the entire cycle.

  4. Hardcoded path dependencies for utilities like ncap2 and mpi4py requires code modification every time a software change is implemented.

Modifications:

  1. Xarray parallelized lazy operations were implemented to replace numpy array operations:

UMD_utils/ocean_obs.py replaces old code stored as
UMD_utils/ocean_obs.legacy.py

Legacy is conserved because the new xarray readers were not implemented for some legacy observer readers.
New readers might be implemented as needed.

  1. The IN_LINE environmental variable set in ocean_das_config, when set to True, allows for observers to
    be executed within the same job. This option is slower than spawning multiple observer jobs, but in the long run, it minimizes queue wait times for users.
    NOTE: IN_LINE=False (old configuration) should not be used on Milan nodes due to cpu per node quota.

scripts/oda_run.j replaces old code stored as
scripts/oda_run.legacy.j

scripts/ocean_observer.py replaces old code stored as
scripts/ocean_observer.legacy.py

  1. An option BOMB_ON_SIGO was implemented in ocean_das_config so when set to false, it will skip the altimeter data for
    the single observer, while conserving the ADT obs for the rest of the cycle. BOMB_ON_SIGO default is set to True, to be kept for observation people to correct observation files.

  2. Hard coded paths where removed and new modules were aded to g5_modules.

@vrx- vrx- requested a review from a team as a code owner October 9, 2024 18:06
@amolod amolod added the Skip Changelog Skips the Changelog Enforcerx label Oct 9, 2024
@amolod amolod merged commit dabb6aa into main Oct 9, 2024
1 of 2 checks passed
@amolod amolod deleted the feature/vruizxom/odas_milan branch October 9, 2024 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Skip Changelog Skips the Changelog Enforcerx
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants