Fix wave restart for cold start and add ic version file #3112
Conversation
The stage job was incorrectly putting wave restarts into the gfs directory. The forecast job looks for them in the gdas directory, so this is updated. Additionally, the restarts were also not being copied from the staged directory to `$DATA`, so now they are. The process is identical to that of non-RERUN warm starts, so the code is re- factored a bit to avoid duplication. Resolves NOAA-EMC#3109
WalterKolczynski-NOAA
requested review from
JessicaMeixner-NOAA,
KateFriedman-NOAA and
DavidHuber-NOAA
|
This won't work yet because I need to move all the wave restarts in ICSDIR. |
I have a directory of just 1 IC test so I can get it moved and tested from there pretty easily. Quick question, I should be staging in the previous cycle gdas, correct? |
Yes. Should be the same as we have now, except gfs ➡ gdas |
|
@WalterKolczynski-NOAA - I have test running. It's not "clean" in the sense that I merged your changes into my branch - but the stage-IC job succeeded and forecast is in the queue. I'll report in the morning and thank you so much for a quick fix to this problem! |
|
@WalterKolczynski-NOAA - My test from last night has a wave IC!!!! Thank you again for this quick update. Even just having this branch means we can move forward with getting some runs comparing different physics options for the wave model now. |
Adds a new vesion file for IC directories. Unlike other version files, this one uses an associative array instead of different variables. With the version file in place, the versions are updated on most of the directories to switch to the relocated wave restarts. Refs: NOAA-EMC#3109
|
New IC versions have been created for the relocated wave restarts. This required adding an IC versions file. Note: the high-res cases (C768/C1152) likely will still not work. They use the wave grid name as the restart suffix, but for single-grid waves, ending in |
Adds a new vesion file for IC directories. Unlike other version files, this one uses an associative array instead of different variables. With the version file in place, the versions are updated on most of the directories to switch to the relocated wave restarts. Refs: NOAA-EMC#3109
I have access to add additional links to files owned by climate group or can coordinate this being found. Can you let me know which directories? |
WalterKolczynski-NOAA
added
CI-Wcoss2-Building
There was a problem hiding this comment.
Looks good, thanks for the IC updates/fixes @WalterKolczynski-NOAA ! Approve pending successful completion of CI testing.
There was a problem hiding this comment.
I've tested the wave parts of these changes and approve those changes. This PR however touches a lot more than just wave ICs with the IC version change, so maybe posting the output of the CI for more people to look at before giving the official approval might be a good idea?
|
@WalterKolczynski-NOAA - Can you remind me again of which machine which ICs I need to update for the wave model? I wanted to say you said an HR3? I can also work with @jiandewang to get this updated on WCOSS2 and run an HR4 test there as part of the review as well. |
@WalterKolczynski-NOAA i just looked on wcoss2, this looks update? So can I run an HR4 like test there now? In the meantime, I'm not sure what you need changed otherwise so I'll wait for your response on testing and what else needs changed. |
@JessicaMeixner-NOAA Needs to be repeated on all machines. (These changes are related to a previous PR fixing wave restarts #3009.) |
emcbot
added
CI-Hera-Building
|
Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 with error logs: Follow link here to view the contents of the above file(s): (link) |
emcbot
added
CI-Hera-Failed
|
Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 in |
|
Experiment C48_S2SW FAILED on Hera in Build# 3 with error logs: Follow link here to view the contents of the above file(s): (link) |
|
Experiment C48_S2SW FAILED on Hera in Build# 3 in |
emcbot
added
CI-Hera-Failed
|
CI Failed on Hera in Build# 3 |
WalterKolczynski-NOAA
added
CI-Hera-Ready
emcbot
added
CI-Hera-Building
emcbot
added
CI-Hera-Passed
|
CI Passed on Hera in Build# 1 |
WalterKolczynski-NOAA
merged commit 64bc179
into
NOAA-EMC:develop
|
@WalterKolczynski-NOAA @JessicaMeixner-NOAA might this change slightly alter the forecast fields when cycling? |
I have not cycled w/waves yet --- i'm not sure if we have all the proper things in place or not. I am over-due to look at this. Are you cycling w/waves and seeing issues? |
|
@danholdaway - To my knowledge, this PR should only change answers with tests that have waves included as it would have included a wave IC and therefor change answers. To my knowledge, there are no CI tests that cycle with waves, so these tests should not have changed answers from this PR based on my understanding. @WalterKolczynski-NOAA would know more. |
|
Thanks @JessicaMeixner-NOAA, we should be OK with what we're testing actually. Looking back through PRs to see if there's anything obvious that might have changed forecast output. |
* develop: Remove WAFS files and references from `develop` (NOAA-EMC#3263) fix intel stack version number on c5 (NOAA-EMC#3258) Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252) Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255) Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235) Only run METplus in the 3Dvar tests (NOAA-EMC#3245) Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106) Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246) Add domain level to wave gridded COM path (NOAA-EMC#3137) CI JJOB Tests using CMake (NOAA-EMC#3214) Make assorted updates to waves (NOAA-EMC#3190) Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236) Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211) Add additional GEFS variables needed for AI/ML applications (NOAA-EMC#3221) Add bmat task dependency to marine LETKF task (NOAA-EMC#3224) Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229) Reinstate product groups (NOAA-EMC#3208) Additional fixes for downstream jobs (NOAA-EMC#3187) Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215) Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220) Update upload-artifact to v4 (NOAA-EMC#3216) Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217) Update g-w to cycle with C1152 ATM (NOAA-EMC#3206) Separate use of initial increment/perturbation file from REPLAY/+03 ICs (NOAA-EMC#3119) Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207) Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212) Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146) Remove multi-grid wave support (NOAA-EMC#3188) Add echgres as a dependency for earc (NOAA-EMC#3202) Ensure OCNRES and ICERES have 3 digits in the archive script (NOAA-EMC#3199) Set runtime shell requirements within Jenkins Pipeline (NOAA-EMC#3171) Add efcs and epos to ufs_hybatm xml (NOAA-EMC#3192) (NOAA-EMC#3193) Fix GEFS and SFS compile flags in build_all.sh (NOAA-EMC#3197) Remove early-cycle EnKF forecast (NOAA-EMC#3185) Fix mod_icec bug in atmos_prod (NOAA-EMC#3167) Create compute build option (NOAA-EMC#3186) Support global-workflow using Rocky 8 on CSPs (NOAA-EMC#2998) Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175) Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163) Update compression options for GEFS history files (NOAA-EMC#3184) Update compression options for high res history files (NOAA-EMC#3178) Turn DO_TEST_MODE off (NOAA-EMC#3177) Hotfix for gdas_arch div/0 (NOAA-EMC#3169) Allow building of the ufs-weather-model, WW3 pre/post execs for GFS, GEFS, SFS in the same clone of global-workflow (NOAA-EMC#3098) Switch Aerosol DA to use JCB and Jedi class (NOAA-EMC#3125) Update ufs-weather-model to 2024-12-06 commit (NOAA-EMC#3145) Enable traditional threading as an option (NOAA-EMC#3149) Update HPC_ACCOUNT on Hercules to fv3-cpu (NOAA-EMC#3164) Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test (NOAA-EMC#3120) Update GSI analysis jobs to use COMIN/COMOUT (NOAA-EMC#3092) Update HPC Tier Definitions (NOAA-EMC#3138) Add marine hybrid envar (NOAA-EMC#3041) Archive the experiment directory along with git status/diff output (NOAA-EMC#3105) Use stochastic restart patterns on rerun (NOAA-EMC#3077) Point Jenkinsfile back to CI/ (NOAA-EMC#3139) Fix wave restart for cold start and add ic version file (NOAA-EMC#3112) Allow users to override the default account at setup time (NOAA-EMC#3127) Refactor gridded wave post (NOAA-EMC#3014) Update docs related to NOAA CSPs (NOAA-EMC#3043) Allow APP to differ between RUNs (NOAA-EMC#2943) Run one executable for soca2cice (instead of two) (NOAA-EMC#3118) Speed up GSI analysis jobs in CI testing (NOAA-EMC#3115) Make aerosol output frequency variable (NOAA-EMC#2982) Add new stations to GFS BUFR sounding products (NOAA-EMC#3107) JCB-based obs+bias staging, Jedi class updates, and marine B-matrix refactoring (NOAA-EMC#2992) Enable tapering of atm ens perts at the model top (NOAA-EMC#3097) Update JGDAS ENKF POST job (NOAA-EMC#3090) SFS Runs at C96mx100 (NOAA-EMC#2960) Move machine-based options from config.base to host files (NOAA-EMC#3053) Remove RUNDIRS before running CI cases to cover re-run events (NOAA-EMC#3076) CI GitHub pipeline (hotfix) update for fetching repo name (NOAA-EMC#3084) Update JGDAS ENKF ECEN job (NOAA-EMC#3050) Update snow obs processing job (NOAA-EMC#3055) Update to action workflow pipeline in default repo for development (NOAA-EMC#3062) Update to action workflow pipeline in default repo for development (NOAA-EMC#3061) Update workflow pipeline (NOAA-EMC#3060) PW CI pipeline update5 ready for review so it can be merged and tested (NOAA-EMC#3059) Revert "GitHub CI Pipeline update for debugging forked PR support" (NOAA-EMC#3057) GitHub CI Pipeline update for debugging forked PR support (NOAA-EMC#3056) Add more ocean variables for post-processing in GEFS (NOAA-EMC#2995) Auto provisioning of PW clusters from GitHub CI added (NOAA-EMC#3051) Fix the name of the TC tracker filenames in archive.py (NOAA-EMC#3030) Make wxflow links static instead of from link_workflow (NOAA-EMC#3008) Update global jdas enkf diag job with COMIN/COMOUT for COM prefix (NOAA-EMC#2959) Add run and finalize methods to marine LETKF task (NOAA-EMC#2944) Fix wave restarts and GEFS FHOUT/FHMAX (NOAA-EMC#3009) Disabling hyper-threading (NOAA-EMC#2965) GitHub Actions Pipeline Updates for Self-Hosted Runners on PW (NOAA-EMC#3018) CI jekninsfile update hotfix (NOAA-EMC#3038) Update gdas.cd (NOAA-EMC#2978) Add ability to add tag to pslots with generate_workflows (NOAA-EMC#3036) CI update to shell environment with HOMEgfs to HOME_GFS for systems that need the path (NOAA-EMC#3013) Quick updated to Jenkins (health check) launch script (NOAA-EMC#3033) Document the generate_workflows.sh script (NOAA-EMC#3028) Replace gfs_cyc with an interval (NOAA-EMC#2928) Hotfix: Fix generate_workflows.sh optional build flags (NOAA-EMC#3024) Add a tool to run multiple YAML cases locally (NOAA-EMC#3004) Hotfix: Correctly set overwrite option when specified (NOAA-EMC#3021)
Description
The stage job was incorrectly putting wave restarts into the gfs directory. The forecast job looks for them in the gdas directory, so this is updated.
Additionally, the restarts were also not being copied from the staged directory to
$DATA, so now they are. The process is identical to that of non-RERUN warm starts, so the code is refactored a bit to avoid duplication.As part of updating the ICs with the new restart location, an IC version file is added to support different version numbers for different IC directories. Unlike other version files, this one uses an associative array rather than individual variables.
Resolves #3109
Type of change
Change characteristics
How has this been tested?
Checklist