Remove pslot dir from local archive folder on CI case completion #2961

TerrenceMcGuinness-NOAA · 2024-09-25T20:49:02Z

Description

This Pr removes pslot dir (with hash) from local archive folder on CI case completion and also adds cleanup_experiment function in the BASH utilities to clean up after a CI case runs. This change consolidates cleanup actions and adds functionality to clean the local archive folders.

Key changes:

Modified ci/Jenkinsfile to include cleanup steps for the DATAROOT directory before new experiments are ran incase the pipeline is being ran for a second time.
Updated $HOMEgfs/ci/scripts/check_ci.sh to use the new cleanup_experiment function under the BASH CI system.
Added cleanup_experiment function in $HOMEgfs/ci/scripts/utils/ci_utils.sh to handle various cleanups including DATAROOT, ARCDIR, and ATADIR directories.
Added get config var from EXPDIR utility get_config_var.py for getting config values in BASH

Resolves #2954, #3066

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

This PR was self tested with the CI system.
# Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
I have made corresponding changes to the system documentation if necessary

…rive files

…A/global-workflow into ci_clean_loc_arch

ci/scripts/utils/ci_utils.sh

emcbot · 2024-09-26T09:53:53Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_b5113675

emcbot · 2024-09-26T11:56:01Z

Experiment C48_S2SW FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C48_S2SW_b5113675

emcbot · 2024-09-26T12:01:32Z

Experiment C48_ATM FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C48_ATM_b5113675

emcbot · 2024-09-26T12:01:35Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C48_S2SWA_gefs_b5113675

emcbot · 2024-09-26T13:15:15Z

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C96_atm3DVar_b5113675

emcbot · 2024-09-26T13:39:54Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C96C48_hybatmDA_b5113675

emcbot · 2024-09-26T13:46:32Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_b5113675

emcbot · 2024-09-26T14:29:05Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2961/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_b5113675

emcbot · 2024-09-26T14:29:14Z

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/2961


Experiment C48mx500_3DVarAOWCDA_b5113675 Completed 2 Cycles: *SUCCESS* at Thu Sep 26 09:53:40 UTC 2024
Experiment C48_S2SW_b5113675 Completed 1 Cycles: *SUCCESS* at Thu Sep 26 11:55:53 UTC 2024
Experiment C48_ATM_b5113675 Completed 1 Cycles: *SUCCESS* at Thu Sep 26 12:01:25 UTC 2024
Experiment C48_S2SWA_gefs_b5113675 Completed 1 Cycles: *SUCCESS* at Thu Sep 26 12:01:26 UTC 2024
Experiment C96_atm3DVar_b5113675 Completed 3 Cycles: *SUCCESS* at Thu Sep 26 13:15:02 UTC 2024
Experiment C96C48_hybatmDA_b5113675 Completed 3 Cycles: *SUCCESS* at Thu Sep 26 13:39:45 UTC 2024
Experiment C96C48_ufs_hybatmDA_b5113675 Completed 2 Cycles: *SUCCESS* at Thu Sep 26 13:46:23 UTC 2024
Experiment C96C48_hybatmaerosnowDA_b5113675 Completed 3 Cycles: *SUCCESS* at Thu Sep 26 14:29:00 UTC 2024

…Jenkins file to cleanup experiment

ci/scripts/utils/ci_utils.sh

emcbot · 2024-11-08T17:52:08Z

Experiment C48_S2SWA_gefs FAILED on Orion in Build# 5 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C48_S2SWA_gefs_788dd8fb

emcbot · 2024-11-08T17:53:49Z

Experiment C48_S2SW FAILED on Orion in Build# 5 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C48_S2SW_788dd8fb

emcbot · 2024-11-08T17:54:06Z

CI Failed on Orion in Build# 5
Built and ran in directory /work2/noaa/stmp/CI/ORION/2961


Experiment C48_S2SWA_gefs_788dd8fb Terminated with  tasks failed and  dead at Fri Nov  8 11:52:02 AM CST 2024
Experiment C48_S2SWA_gefs_788dd8fb Terminated: **
Experiment C48_S2SW_788dd8fb Terminated with  tasks failed and  dead at Fri Nov  8 11:53:42 AM CST 2024
Experiment C48_S2SW_788dd8fb Terminated: **

emcbot · 2024-11-08T19:16:52Z

Experiment C48_ATM FAILED on Orion in Build# 7 with error logs:

/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C48_ATM_7d3b8bfe/logs/2021032312/gfs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-08T19:17:00Z

Experiment C48_ATM FAILED on Orion in Build# 7 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C48_ATM_7d3b8bfe

emcbot · 2024-11-08T19:17:01Z

Experiment C48_S2SW FAILED on Orion in Build# 7 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C48_S2SW_7d3b8bfe

emcbot · 2024-11-08T19:17:09Z

Experiment C96_atm3DVar FAILED on Orion in Build# 7 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C96_atm3DVar_7d3b8bfe

emcbot · 2024-11-08T19:17:09Z

Experiment C96_S2SWA_gefs_replay_ics FAILED on Orion in Build# 7 with error logs:

/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_7d3b8bfe/logs/2020110100/gefs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-08T19:17:13Z

Experiment C96C48_hybatmDA FAILED on Orion in Build# 7 with error logs:

/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96C48_hybatmDA_7d3b8bfe/logs/2021122018/enkfgdas_stage_ic.log
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96C48_hybatmDA_7d3b8bfe/logs/2021122018/gdas_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-08T19:17:17Z

Experiment C96_S2SWA_gefs_replay_ics FAILED on Orion in Build# 7 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_7d3b8bfe

emcbot · 2024-11-08T19:17:18Z

Experiment C48_S2SWA_gefs FAILED on Orion in Build# 7 with error logs:

/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C48_S2SWA_gefs_7d3b8bfe/logs/2021032312/gefs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-08T19:17:22Z

Experiment C96C48_hybatmDA FAILED on Orion in Build# 7 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C96C48_hybatmDA_7d3b8bfe

emcbot · 2024-11-08T19:17:26Z

Experiment C48_S2SWA_gefs FAILED on Orion in Build# 7 in
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/EXPDIR/C48_S2SWA_gefs_7d3b8bfe

emcbot · 2024-11-08T19:17:35Z

CI Failed on Orion in Build# 7
Built and ran in directory /work2/noaa/stmp/CI/ORION/2961


**CI Failed** on Orion in Build# 5<br>Built and ran in directory `/work2/noaa/stmp/CI/ORION/2961`

Experiment C48_S2SWA_gefs_788dd8fb Terminated with tasks failed and dead at Fri Nov 8 11:52:02 AM CST 2024
Experiment C48_S2SWA_gefs_788dd8fb Terminated: **
Experiment C48_S2SW_788dd8fb Terminated with tasks failed and dead at Fri Nov 8 11:53:42 AM CST 2024
Experiment C48_S2SW_788dd8fb Terminated: **
Experiment C48_ATM_7d3b8bfe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Nov 8 01:16:41 PM CST 2024
Experiment C48_ATM_7d3b8bfe Terminated: FAIL
Error logs:
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C48_ATM_7d3b8bfe/logs/2021032312/gfs_stage_ic.log
Experiment C48_S2SW_7d3b8bfe Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Fri Nov 8 01:16:48 PM CST 2024
Experiment C48_S2SW_7d3b8bfe Terminated: FAIL
Error logs:
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C48_S2SW_7d3b8bfe/logs/2021032312/gfs_stage_ic.log
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C48_S2SW_7d3b8bfe/logs/2021032312/gfs_waveinit.log
Experiment C96_atm3DVar_7d3b8bfe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Nov 8 01:16:54 PM CST 2024
Experiment C96_atm3DVar_7d3b8bfe Terminated: FAIL
Experiment C96C48_hybatmDA_7d3b8bfe Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Fri Nov 8 01:16:54 PM CST 2024
Experiment C96C48_hybatmDA_7d3b8bfe Terminated: FAIL
Experiment C96_S2SWA_gefs_replay_ics_7d3b8bfe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Nov 8 01:16:55 PM CST 2024
Experiment C96_S2SWA_gefs_replay_ics_7d3b8bfe Terminated: FAIL
Error logs:
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96_atm3DVar_7d3b8bfe/logs/2021122018/gdas_stage_ic.log
Error logs:
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96C48_hybatmDA_7d3b8bfe/logs/2021122018/enkfgdas_stage_ic.log
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96C48_hybatmDA_7d3b8bfe/logs/2021122018/gdas_stage_ic.log
Error logs:
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_7d3b8bfe/logs/2020110100/gefs_stage_ic.log
Experiment C48_S2SWA_gefs_7d3b8bfe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Nov 8 01:17:03 PM CST 2024
Experiment C48_S2SWA_gefs_7d3b8bfe Terminated: FAIL
Error logs:
/work2/noaa/stmp/CI/ORION/2961/RUNTESTS/COMROOT/C48_S2SWA_gefs_7d3b8bfe/logs/2021032312/gefs_stage_ic.log

TerrenceMcGuinness-NOAA · 2024-11-09T12:55:40Z

Changes moved to PR 3076

TerryMcGuinness-NOAA and others added 4 commits September 24, 2024 20:28

added a cleanup_experment funciton to ci_utils that cleans local arch…

1c159d8

…rive files

Merge branch 'NOAA-EMC:develop' into ci_clean_loc_arch

e7aa3ba

moved rm for EXPDIR into cleanup_expermiment function

463655a

Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…

b511367

…A/global-workflow into ci_clean_loc_arch

TerrenceMcGuinness-NOAA added the CI/CD Issue related to CI/CD label Sep 25, 2024

TerrenceMcGuinness-NOAA requested a review from WalterKolczynski-NOAA September 25, 2024 20:49

TerrenceMcGuinness-NOAA self-assigned this Sep 25, 2024

TerrenceMcGuinness-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Sep 25, 2024

github-advanced-security bot found potential problems Sep 25, 2024

View reviewed changes

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Sep 26, 2024

fixed bracket sytax error in ci_utils and missnamed function call in …

40c96b2

…Jenkins file to cleanup experiment

github-advanced-security bot found potential problems Sep 26, 2024

View reviewed changes

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

fixed SC2155 in bash utils for cleanup

54ce216

github-advanced-security bot found potential problems Sep 26, 2024

View reviewed changes

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

ci/scripts/utils/ci_utils.sh Fixed Show fixed Hide fixed

emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Nov 8, 2024

emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Nov 8, 2024

emcbot added the CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress label Nov 8, 2024

emcbot removed the CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress label Nov 8, 2024

emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Nov 8, 2024

TerrenceMcGuinness-NOAA closed this Nov 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove pslot dir from local archive folder on CI case completion #2961

Remove pslot dir from local archive folder on CI case completion #2961

TerrenceMcGuinness-NOAA commented Sep 25, 2024 •

edited

Loading

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

TerrenceMcGuinness-NOAA commented Nov 9, 2024

Remove pslot dir from local archive folder on CI case completion #2961

Remove pslot dir from local archive folder on CI case completion #2961

Conversation

TerrenceMcGuinness-NOAA commented Sep 25, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Sep 26, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

emcbot commented Nov 8, 2024

TerrenceMcGuinness-NOAA commented Nov 9, 2024

TerrenceMcGuinness-NOAA commented Sep 25, 2024 •

edited

Loading