Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove pslot dir from local archive folder on CI case completion #2961

Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
1c159d8
added a cleanup_experment funciton to ci_utils that cleans local arch…
TerryMcGuinness-NOAA Sep 24, 2024
e7aa3ba
Merge branch 'NOAA-EMC:develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Sep 25, 2024
463655a
moved rm for EXPDIR into cleanup_expermiment function
TerryMcGuinness-NOAA Sep 25, 2024
b511367
Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…
TerryMcGuinness-NOAA Sep 25, 2024
40c96b2
fixed bracket sytax error in ci_utils and missnamed function call in …
TerryMcGuinness-NOAA Sep 26, 2024
54ce216
fixed SC2155 in bash utils for cleanup
TerryMcGuinness-NOAA Sep 26, 2024
cacde8c
fixed SC2115 for var expation in path for clean experiment util
TerryMcGuinness-NOAA Sep 26, 2024
c8a10a9
needed the OR truch for var assigment via actions in bash
TerryMcGuinness-NOAA Sep 26, 2024
7c93f4b
fixed the evaluation of HOMDIR so that internal strification of env v…
TerryMcGuinness-NOAA Sep 27, 2024
bb92a39
Merge branch 'develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Sep 27, 2024
bfcbba5
still needed archive tagged back onto HOMEDIR
TerryMcGuinness-NOAA Sep 27, 2024
69fc51b
Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…
TerryMcGuinness-NOAA Sep 27, 2024
49b4489
new linee at aend of ci_utils
TerryMcGuinness-NOAA Sep 27, 2024
b776a59
Merge branch 'NOAA-EMC:develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Oct 23, 2024
d8af194
removed eval in cleanup_experiment and added envsubst at end of pipe …
TerryMcGuinness-NOAA Oct 23, 2024
d54bd8d
updated archival directory to ARCDIR from config.base in cleanup_expe…
TerryMcGuinness-NOAA Oct 24, 2024
85f6369
forgot to clean up EXPDIR replacement in cleanup_experment
TerryMcGuinness-NOAA Oct 24, 2024
6a09b36
forgot to clean up EXPDIR replacement in cleanup_experment still
TerryMcGuinness-NOAA Oct 24, 2024
32b8cd0
Update ci/scripts/utils/ci_utils.sh
TerrenceMcGuinness-NOAA Oct 25, 2024
e01d9d3
added get_config_var util to simplify cleanup_experment
TerryMcGuinness-NOAA Nov 5, 2024
3d5e851
removed debug outputs in cleanup_experiment
TerryMcGuinness-NOAA Nov 5, 2024
162474a
added next step to use get_config_var to remove previous DATAROOT dat…
TerryMcGuinness-NOAA Nov 5, 2024
7da3721
fix conficts and merged develop back in
TerryMcGuinness-NOAA Nov 5, 2024
c6ad4ec
updated Jenkinsfile to use clean experiment and to also clean DATADIR…
TerryMcGuinness-NOAA Nov 5, 2024
bef32e7
updating correct ATARDIR variable in cleaup
TerryMcGuinness-NOAA Nov 5, 2024
02642fc
added true for shell norms
TerryMcGuinness-NOAA Nov 5, 2024
62ec3f6
Update ci/scripts/utils/ci_utils.sh
TerrenceMcGuinness-NOAA Nov 6, 2024
7e5b4fb
Update ci/scripts/utils/ci_utils.sh
TerrenceMcGuinness-NOAA Nov 6, 2024
bf3b80f
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
cd4e27b
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
5ea8d3d
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
99b9322
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
f56be33
Update ci/scripts/utils/get_config_var.py
TerrenceMcGuinness-NOAA Nov 6, 2024
9a381b4
fixed name of get_config_var in pipeline (left off the py extention)
TerryMcGuinness-NOAA Nov 6, 2024
788dd8f
DATAROOT is not a config.base value but is construced from STMP in ta…
TerryMcGuinness-NOAA Nov 6, 2024
7d3b8bf
updated Jenkins file with correct RUNDIRS string (had RUNDIR as a bug)
TerryMcGuinness-NOAA Nov 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ pipeline {
sh(script: " rm -f ${error_file}")
try {
sh(script: "${HOMEgfs}/ci/scripts/run-check_ci.sh ${CUSTOM_WORKSPACE} ${pslot} ${system}")
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cleanup_experiment ${CUSTOM_WORKSPACE}/RUNTESTS/EXPDIR/${pslot}")
} catch (Exception error_experment) {
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cancel_batch_jobs ${pslot}")
ws(CUSTOM_WORKSPACE) {
Expand Down
3 changes: 1 addition & 2 deletions ci/scripts/check_ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,7 @@ for pr in ${pr_list}; do
fi
if [[ "${rocoto_state}" == "DONE" ]]; then
#Remove Experment cases that completed successfully
rm -Rf "${pslot_dir}"
rm -Rf "${pr_dir}/RUNTESTS/COMROOT/${pslot}"
"${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh" cleanup_experiment "${pslot_dir}"
rm -f "${output_ci_single}"
# echo "\`\`\`" > "${output_ci_single}"
DATE=$(date +'%D %r')
Expand Down
17 changes: 17 additions & 0 deletions ci/scripts/utils/ci_utils.sh
DavidHuber-NOAA marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,20 @@
fi
echo "${URL}"
}

function cleanup_experiment() {
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
# cleanup_experiment function
# This function takes a directory path of an EXPDIR and
# gets HOMEDIR value from config.base to
# clean up the archived files on disk.

local PSLOT_PATH="$1"
local pslot=$(basename "${PSLOT_PATH}")
Fixed Show fixed Hide fixed
local HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]')
Fixed Show fixed Hide fixed
local ARCHIVEDIR="${HOMEDIR}/archive"

rm -rf "${ARCHIVEDIR}/${pslot}"
Fixed Show fixed Hide fixed
rm -Rf "${PSLOT_PATH}"
rm -Rf "${PSLOT_PATH}/../COMROOT/${pslot}"
}

Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
Loading