New data container jenkinsfile #9
Conversation
|
@baacha123 is aws-s3.sh ready? |
Yes Jong its is ready and in PR |
|
Please for the aw-s3 shell script. i have leave instructions to update file path in script |
| @@ -1,103 +1,140 @@ | |||
| pipeline { | |||
| agent { | |||
There was a problem hiding this comment.
Can you undo all of the whitespace changes? It's hard to see what's changed. I can help you on this.
| stage('RUN DATA SCRIPT') { | ||
| steps { | ||
| echo "Pulling input-data-20220414, Creating Docker image & Pushing it to DockerHub" | ||
| sh 'PASSWORD= ${DockerHub_PASSWORD} /path/to/your/aws-s3.sh' |
There was a problem hiding this comment.
You don't need the PASSWORD= ${DockerHub_PASSWORD} part, you'll set an environment variable and import the credential from the Jenkins credential store. Check out the Handling credentials section in https://www.jenkins.io/doc/book/pipeline/jenkinsfile/. I can help you with this too.
Also, use "${WORKSPACE}/aws-s3.sh" for the path. Although, I'm not sure if the script should be checked in at the root of the repo, maybe in a scripts folder? @jkbk2004
| declare -a subfolders=("CICE_FIX/100" "CICE_IC/100" "CPL_FIX/aC96o100" "FV3_fix/fix_co2_proj" "FV3_fix_tiled/C96" "FV3_input_data/INPUT" "FV3_input_data/INPUT_L127" "FV3_input_data/INPUT_L127_mx100" "FV3_input_data/ORO_FLAKE" "FV3_input_data/RESTART" "FV3_input_data_gsd/FV3_input_data_C96_with_aerosols" "FV3_input_data_gsd/drag_suite" "FV3_input_data_INCCN_aeroclim/mg2_IN_CCN" "FV3_input_data_INCCN_aeroclim/MERRA2" "FV3_input_data_INCCN_aeroclim/aer_data" "FV3_input_data_INCCN_aeroclim/aer_data/LUTS" "FV3_input_frac/C96_L64.mx100_frac" "FV3_input_frac/C96_L127.mx100_frac" "MOM6_IC/100" "MOM6_IC/100/2011100100" "MOM6_FIX/100" "fv3_regional_control/INPUT" "fv3_regional_control/RESTART" "GOCART/p7/ExtData/dust" "GOCART/p7/ExtData/MERRA2/sfc" "GOCART/p7/ExtData/QFED/2013/04" "GOCART/p7/ExtData/QFED/2013/save03" "GOCART/p7/ExtData/CEDS/v2019/2013" "GOCART/p7/ExtData/monochromatic" "GOCART/p7/ExtData/optics" "GOCART/p7/ExtData/MEGAN_OFFLINE_BVOC/v2019-10/2013" "GOCART/p7/ExtData/PIESA/L127" "GOCART/p7/ExtData/PIESA/sfc/HTAP/v2.2" "GOCART/p7/ExtData/volcanic" "GOCART/p7/rc" "GOCART/p8/ExtData/dust" "GOCART/p8/ExtData/MERRA2/sfc" "GOCART/p8/ExtData/QFED/2021/03" "GOCART/p8/ExtData/QFED/2013/04" "GOCART/p8/ExtData/CEDS/v2019/2021" "GOCART/p8/ExtData/CEDS/v2019/2013" "GOCART/p8/ExtData/monochromatic" "GOCART/p8/ExtData/optics" "GOCART/p8/ExtData/MEGAN_OFFLINE_BVOC/v2019-10/2021" "GOCART/p8/ExtData/MEGAN_OFFLINE_BVOC/v2019-10/2013" "GOCART/p8/ExtData/PIESA/L127" "GOCART/p8/ExtData/PIESA/sfc/HTAP/v2.2" "GOCART/p8/ExtData/volcanic" "GOCART/p8/rc" "GOCART/ExtData/dust" "GOCART/ExtData/monochromatic" "GOCART/ExtData/optics" "GOCART/ExtData/PIESA/sfc/HTAP/v2.2" "GOCART/ExtData/PIESA/sfc/QFED/NRT/v2.5" "GOCART/ExtData/volcanic" "GOCART/rc") | ||
| PASSWORD=${PASSWORD:-'abc'} | ||
|
|
||
| #update line number 6, 13, and 15 with path where script will be running |
There was a problem hiding this comment.
You can use the variable "${WORKSPACE}" for where the repo will be cloned.
This is the new jenkins file that will cp files from s3, create and image and push to dockerhub and use that image to run pipeline.
NOTE: The script path has to be updated once confirmed. Also i did not want to hard code dockerhub password in code since it is a bad practice, I have added a variable $PASSWORD which will reference the password from jenkins credentials.