Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the repair job for the reforecast #2838

Merged
merged 63 commits into from
Oct 11, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
8181536
Add the repair job scripts
HongGuan-NOAA Aug 15, 2024
08f4f81
Merge remote-tracking branch 'EMC/develop' into repair
HongGuan-NOAA Aug 15, 2024
fbe4688
Correct warning errors
HongGuan-NOAA Aug 22, 2024
e8d7fa0
Correct warning errors.
HongGuan-NOAA Aug 22, 2024
225eab1
Correct warning errors.
HongGuan-NOAA Aug 22, 2024
e7b5e7f
Fix Python error
HongGuan-NOAA Aug 22, 2024
0afdfbe
Correct the errors
HongGuan-NOAA Aug 23, 2024
e7c3f9b
Correct error
HongGuan-NOAA Aug 23, 2024
c969dda
Correct the error
HongGuan-NOAA Aug 23, 2024
5e25361
Correct the error
HongGuan-NOAA Aug 23, 2024
3be3217
Correct the error
HongGuan-NOAA Aug 23, 2024
e447511
Add the Python code
HongGuan-NOAA Aug 23, 2024
092a787
Correct the Python code
HongGuan-NOAA Aug 23, 2024
8d03a14
Revert "Add the Python code"
HongGuan-NOAA Aug 23, 2024
9624baa
Correct the Python code
HongGuan-NOAA Aug 23, 2024
faf1c84
Add some corrections
EricSinsky-NOAA Aug 23, 2024
22792e3
Correct the config file
HongGuan-NOAA Aug 23, 2024
02d1cf5
Remove repair job in gefs_tasks.py
EricSinsky-NOAA Aug 23, 2024
bae7d72
Correct the Python code
HongGuan-NOAA Aug 23, 2024
5f307ca
Correct the scripts based on the suggestions of the workfloow team
HongGuan-NOAA Sep 10, 2024
8719f90
Make the corrections based on the suggestion of the workflow tearm
HongGuan-NOAA Sep 10, 2024
33391a9
Make the corrections based on the suggestions of the workflow team
HongGuan-NOAA Sep 10, 2024
3bde14c
Make the corrections based on the suggestions of the workflow team
HongGuan-NOAA Sep 10, 2024
d366fcd
Make the corrections based on the suggestion of the workflow team
HongGuan-NOAA Sep 10, 2024
ed47548
Make the corrections based on the suggestions of the workflow team
HongGuan-NOAA Sep 10, 2024
416e0bc
Make the corrections based on the suggestions of teh workflow team
HongGuan-NOAA Sep 10, 2024
e7f0141
Correct the Python code
HongGuan-NOAA Sep 10, 2024
9d89ef9
Make the correction for the warning errors
HongGuan-NOAA Sep 10, 2024
bb436b7
Make the xorrections for the errors
HongGuan-NOAA Sep 10, 2024
c119f81
Delete the old file
HongGuan-NOAA Sep 10, 2024
251410f
Delete more files
HongGuan-NOAA Sep 10, 2024
528a2e0
Add the parmlist which are in the PSL f03 reanalysis
HongGuan-NOAA Sep 11, 2024
9dda05a
Add the varlist for ave/acc/min/max variables in f06 master file
HongGuan-NOAA Sep 11, 2024
57d24c4
Change excutable name from gefs_6h_ave_1mem to gefs_6h_ave_1mem.x
HongGuan-NOAA Sep 11, 2024
b4387f9
Revert "Remove repair job in gefs_tasks.py"
EricSinsky-NOAA Sep 25, 2024
0ceaca9
Revert "Add some corrections"
EricSinsky-NOAA Sep 25, 2024
a43ab9b
Merge remote-tracking branch 'HG/repair' into repair_pr
EricSinsky-NOAA Sep 25, 2024
92bee3d
Update gfs_utils hash
EricSinsky-NOAA Sep 25, 2024
cfa75ec
Link new gfs_utils executable for repair task
EricSinsky-NOAA Sep 25, 2024
3ccacc9
Merge pull request #2 from EricSinsky-NOAA/repair_pr
HongGuan-NOAA Sep 25, 2024
7669667
Update the task name
HongGuan-NOAA Sep 25, 2024
d30fdc1
update the file
HongGuan-NOAA Sep 25, 2024
c8c5391
Update the defaults.yaml
HongGuan-NOAA Sep 25, 2024
7b416e0
Remove the two parm files here
HongGuan-NOAA Sep 26, 2024
dcc04e7
Modify the Python code
HongGuan-NOAA Sep 26, 2024
fef9dd6
Fix the python code error
HongGuan-NOAA Sep 26, 2024
f1cb9e7
Correct the Python code
HongGuan-NOAA Oct 3, 2024
732fb9c
Correct the parameters
HongGuan-NOAA Oct 3, 2024
bf9b792
Add a parameter file
HongGuan-NOAA Oct 4, 2024
b9f8024
Add a parameter file definition
HongGuan-NOAA Oct 4, 2024
574ecfe
Correct Python script
HongGuan-NOAA Oct 8, 2024
b523fdb
Make correction
HongGuan-NOAA Oct 8, 2024
913c4dd
Correct the script
HongGuan-NOAA Oct 8, 2024
d88ea83
Make the correction
HongGuan-NOAA Oct 8, 2024
126d854
Correct the script
HongGuan-NOAA Oct 8, 2024
02d144e
Correct the scripts
HongGuan-NOAA Oct 8, 2024
5bfe5e3
correct the script
HongGuan-NOAA Oct 8, 2024
611cabd
Correct the script
HongGuan-NOAA Oct 9, 2024
6d7d8e6
Correct the script
HongGuan-NOAA Oct 9, 2024
9b03433
Make correction
HongGuan-NOAA Oct 9, 2024
e759f89
Make the correction 1
HongGuan-NOAA Oct 9, 2024
1bb8f13
Correct the script 1
HongGuan-NOAA Oct 9, 2024
00b7a30
Correct script 2
HongGuan-NOAA Oct 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions jobs/JGEFS_ATMOS_ACC
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
#! /usr/bin/env bash
#
#
#####################################################################
# 8/14/2024, Hong Guan Scripts to repair F0306 for GEFSv13 reforecasts
####################################################################
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version control records the history of the file, it does not need to be done here.

Suggested change
# 8/14/2024, Hong Guan Scripts to repair F0306 for GEFSv13 reforecasts
####################################################################

source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "repairf0306" -c "base repairf0306"

####################################
# obtain unique process id (pid) and make temp directory
####################################

export pid=$$
export date=${CDATE}
#yyyymmdd=`echo $date | cut -c1-8`
#echo `date` $0 `date -u` begin

####################################
# File To Log Msgs
####################################
export jlogfile=${jlogfile:-${COMROOT}/logs/jlogfiles/jlogfile.${job}.${pid}}
#echo $jlogfile

####################################
# Determine Job Output Name on System
####################################
export outid="LL${job}"
export jobid="${outid}.o${pid}"
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile
#echo $outid,$jobid,$pgmout,$pgmerr
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of this is necessary. The jjob header takes care of what is needed, and most of the rest we don't use at all.

Suggested change
####################################
# obtain unique process id (pid) and make temp directory
####################################
export pid=$$
export date=${CDATE}
#yyyymmdd=`echo $date | cut -c1-8`
#echo `date` $0 `date -u` begin
####################################
# File To Log Msgs
####################################
export jlogfile=${jlogfile:-${COMROOT}/logs/jlogfiles/jlogfile.${job}.${pid}}
#echo $jlogfile
####################################
# Determine Job Output Name on System
####################################
export outid="LL${job}"
export jobid="${outid}.o${pid}"
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile
#echo $outid,$jobid,$pgmout,$pgmerr


####################################
####################################
# Specify Execution Areas
####################################
export HOMEgefs=${HOMEgfs}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please just use HOMEgfs instead of defining HOMEgefs.

export EXECacc=${EXECacc:-${HOMEgefs}/exec}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have EXECgfs for this.

export SORCacc=${SORCacc:-${HOMEgefs}/sorc/gefs_postacc.fd}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary

export SCRIPTSens_acc=${HOMEgefs}/scripts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use SCRgfs

export FIXens_acc=${FIXacc:-${HOMEgefs}/fix}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIXgfs


export PARMgefs=${HOMEgefs}/rocoto/parm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a valid path.

echo "${PARMgefs}"

# Construct COM variables from templates
#YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_HISTORY COM_ATMOS_MASTER
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_MASTER
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_MASTER
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL



##############################
# Run setpdy and initialize PDY variables
##############################
#
#export COMIN_master=${COMIN_master:-$COMROOT/$PSLOT/gefs.$PDY/00/mem001/model_data/atmos/master}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
##############################
# Run setpdy and initialize PDY variables
##############################
#
#export COMIN_master=${COMIN_master:-$COMROOT/$PSLOT/gefs.$PDY/00/mem001/model_data/atmos/master}

export COMIN_master=${COMIN_master:-${COM_ATMOS_MASTER}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we creating so many copies of variables? Just use COMIN_ATMOS_MASTER.

export COMIN_00and03=${HOMEgefs}/anl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMIN_00and03 should not be in HOMEgefs. Since data in this directory is considered fixed (not generated in any global-workflow task), I think it can be somewhere in a common data directory outside of HOMEgefs.


####################################
# Create member list
####################################

#export homesyndir=$HOMEacc

echo
#env | sort
echo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
####################################
# Create member list
####################################
#export homesyndir=$HOMEacc
echo
#env | sort
echo

########################################################
# Execute the acc script.
export cmodel=ens
export ymdh=${date}

#msg="Begin job for $job"
#postmsg "$jlogfile" "$msg"

#echo ${ymdh}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export cmodel=ens
export ymdh=${date}
#msg="Begin job for $job"
#postmsg "$jlogfile" "$msg"
#echo ${ymdh}

"${SCRIPTSens_acc}/gefs_atmos_f0306.sh" "${ymdh}" "${DATA}" > "fv3gefsrf_0306acc.${cmodel}.${ymdh}.out"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let the output through so it appears in the logfile.

Suggested change
"${SCRIPTSens_acc}/gefs_atmos_f0306.sh" "${ymdh}" "${DATA}" > "fv3gefsrf_0306acc.${cmodel}.${ymdh}.out"
"${SCRgfs}/gefs_atmos_f0306.sh"


################################
#msg="JOB COMPLETED NORMALLY"
#postmsg "$jlogfile" "$msg"

exit

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
################################
#msg="JOB COMPLETED NORMALLY"
#postmsg "$jlogfile" "$msg"
exit

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export err=$?; err_chk

##############################
# Remove the Temporary working directory
##############################
#if [[ $KEEPDATA != "YES" ]]; then
# cd $DATAROOT
# rm -rf $DATA
#fi

#echo `date` $job completed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#if [[ $KEEPDATA != "YES" ]]; then
# cd $DATAROOT
# rm -rf $DATA
#fi
#echo `date` $job completed
if [[ "${KEEPDATA}" != "YES" ]] ; then
rm -rf "${DATA}"
fi

24 changes: 24 additions & 0 deletions jobs/rocoto/repairf0306.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#! /usr/bin/env bash

source "${HOMEgfs}/ush/preamble.sh"

###############################################################
echo
echo "=============== START TO SOURCE FV3GFS WORKFLOW MODULES ==============="
. "${HOMEgfs}/ush/load_ufswm_modules.sh"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
. "${HOMEgfs}/ush/load_ufswm_modules.sh"
. "${HOMEgfs}/ush/load_fv3gfs_modules.sh"

status=$?
[[ ${status} -ne 0 ]] && exit "${status}"

export job="repairf0306"
export jobid="${job}.$$"

###############################################################
echo
echo "=============== START TO RUN EXTRACTVARS ==============="
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo
echo "=============== START TO RUN EXTRACTVARS ==============="

# Execute the JJOB
"${HOMEgfs}/jobs/JGEFS_ATMOS_ACC"
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"


exit 0
11 changes: 11 additions & 0 deletions parm/config/gefs/config.repairf0306
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#! /usr/bin/env bash

########## config.repairf0306 #########
# repairf0306 specific

echo "BEGIN: config.repairf0306"


. "${EXPDIR}/config.resources" repairf0306

echo "END: config.repairf0306"
11 changes: 10 additions & 1 deletion parm/config/gefs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,14 @@ case ${step} in
export memory="1GB"
;;

"repairf0306")
export wtime_repairf0306="00:10:00"
export npe_repairf0306=1
export nth_repairf0306=1
export npe_node_repairf0306=$(( npe_node_max / nth_repairf0306 ))
export memory_repairf0306="2GB"
;;

"fcst" | "efcs")
export is_exclusive=True

Expand Down Expand Up @@ -186,14 +194,15 @@ case ${step} in
declare -x "walltime"="06:00:00"
;;
*)
echo "FATAL ERROR: Resources not defined for job ${step} at resolution ${CASE}"
echo "FATAL ERROR: Resources not defined for job ${job} at resolution ${CASE}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this changed?

exit 4
;;
esac

unset NTASKS_TOT
;;


"atmos_products")
export walltime="00:15:00"
export ntasks=24
Expand Down
192 changes: 192 additions & 0 deletions scripts/gefs_atmos_f0306.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the indentation in this script.

Also, there is quite a bit of repetition. Consider consolidating it with loops.

Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
#!/bin/ksh
EricSinsky-NOAA marked this conversation as resolved.
Show resolved Hide resolved

set -xa
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preamble is responsible for handling set.

#cdate=$1
DATA=$2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$DATA, $PDY, and $cyc are already in the environment. We do not need to pass them as arguments.

Suggested change
#cdate=$1
DATA=$2


#foutmax=6 #last lead hour to extract
cd "${DATA}" || exit 1

#ddate="echo $cdate | cut -c1-8"
#YYYY=`echo $cdate | cut -c1-4`
#MONTH=`echo $cdate | cut -c5-6`
#DAY=`echo $cdate | cut -c7-8`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#ddate="echo $cdate | cut -c1-8"
#YYYY=`echo $cdate | cut -c1-4`
#MONTH=`echo $cdate | cut -c5-6`
#DAY=`echo $cdate | cut -c7-8`


# COMIN_00and03 and {COMIN_master} are directory containing the files that we want to extract
#Extract c00 files for f003 from PSL reanalysis data

fnh=03
echo "extracting f${fnh}"
oufile=${DATA}/gefs.t00z.master.grb2f0${fnh}

infile=${COMIN_00and03}/GFSPRS.GrbF03
if [[ -f "${infile}" ]]; then #check if input file exists before extraction

${WGRIB2} "${infile}" | grep "TSNOWP" | ${WGRIB2} -i "${infile}" -grib tmp ||true
${WGRIB2} tmp -for "2:2" -append -grib "${oufile}">/dev/null || true

${WGRIB2} "${infile}" | grep ":APCP:surface" | ${WGRIB2} -i "${infile}" -grib tmp ||true
${WGRIB2} tmp -for "1:1" -append -grib "${oufile}">/dev/null || true

${WGRIB2} "${infile}" | grep ":ACPCP:surface" | ${WGRIB2} -i "${infile}" -grib tmp ||true
${WGRIB2} tmp -for "1:1" -append -grib "${oufile}">/dev/null || true

${WGRIB2} "${infile}" | grep ":NCPCP:surface" | ${WGRIB2} -i "${infile}" -grib tmp ||true
${WGRIB2} tmp -for "1:1" -append -grib "${oufile}">/dev/null || true

${WGRIB2} "${infile}" | grep ":HCDC:high cloud layer:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":MCDC:middle cloud layer:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":LCDC:low cloud layer:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true

rm tmp
else
echo "${infile} does not exist"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this exit with an error?

fi

infile=${COMIN_00and03}/GFSFLX.GrbF03
if [[ -f "${infile}" ]]; then #check if input file exists before extraction
${WGRIB2} "${infile}" | grep "WATR" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "SNOWC" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "SNOHF" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":DLWRF:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ULWRF:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":DSWRF:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":USWRF:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":USWRF:top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ULWRF:top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":UFLX:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":VFLX:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":SHTFL:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":LHTFL:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":PRATE:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":CPRAT:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ALBDO:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":TCDC:entire" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":TCDC:boundary" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":GFLUX:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":U-GWD:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":V-GWD:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMP:middle cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMP:low cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMP:high cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:high cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:middle cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:low cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:high cloud bottom" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:middle cloud bottom" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:low cloud bottom" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "CWORK" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "DUVB" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "CDUVB" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMAX" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMIN" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
else
echo "${infile} does not exist"
fi
# done #fnh

#Extract individual member files for f006 master data
fnh=006

echo "extracting f${fnh}"
infile=${COMIN_master}/gefs.t00z.master.grb2f${fnh}
oufile=${DATA}/gefs.t00z.master.grb2f${fnh}

if [[ -f "${infile}" ]]; then #check if input file exists before extraction
# rm -f ${outfile}/gefs.t00z.master.grb2f${fnh} #remove outfile if it already exists before extraction
${WGRIB2} "${infile}" | grep "TSNOWP" | ${WGRIB2} -i "${infile}" -grib tmp
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
${WGRIB2} tmp -for "2:2" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "WATR" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "SNOWC" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "SNOHF" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":DLWRF:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ULWRF:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":DSWRF:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":USWRF:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":USWRF:top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ULWRF:top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":UFLX:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":VFLX:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":SHTFL:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":LHTFL:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":PRATE:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":CPRAT:surface:0" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ALBDO:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":TCDC:entire" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":TCDC:boundary" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":HCDC:high cloud layer:0-6" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":MCDC:middle cloud layer:0-6" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":LCDC:low cloud layer:0-6" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":GFLUX:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":U-GWD:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":V-GWD:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":APCP:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":ACPCP:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep ":NCPCP:surface" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMP:middle cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMP:low cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMP:high cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:high cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:middle cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:low cloud top" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:high cloud bottom" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:middle cloud bottom" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "PRES:low cloud bottom" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "CWORK" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "DUVB" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "CDUVB" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMAX" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true
${WGRIB2} "${infile}" | grep "TMIN" | ${WGRIB2} -i "${infile}" -append -grib "${oufile}">/dev/null || true

else
echo "${infile} does not exist"
fi

export exec_dir=${EXECacc}
export sorc_dir=${SORCacc}
export sorc_name=gefs_6h_ave_1mem

# cd $DATA

"${exec_dir}/${sorc_name}" >sorc_name.exe.out
cat sorc_name.exe.out

#output f06
infile=${COMIN_master}/gefs.t00z.master.grb2f006

${WGRIB2} "${infile}" | grep "TSNOWP" | ${WGRIB2} -i "${infile}" -grib TSNOWP2.dat
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
${WGRIB2} TSNOWP2.dat -for "1:1" -append -grib TSNOWP1.dat >/dev/null || true

${WGRIB2} "${infile}" -not "(ULWRF|USWRF)" -not "(TSNOWP|DLWRF|DSWRF|UFLX|VFLX|SHTFL|LHTFL|PRATE|CPRAT|ALBDO|GFLUX|U-GWD|V-GWD)" -not "TCDC:(entire|boundary)" -not "HCDC:high cloud layer:0-6" -not "LCDC:low cloud layer:0-6" -not "MCDC:middle cloud layer:0-6" -not "(APCP|ACPCP|NCPCP)" -not "(TMIN|TMAX|DUVB|CDUVB|CWORK|SNOHF|SNOWC|WATR)" -not "(TMP:middle cloud top|TMP:low cloud top|TMP:high cloud top)" -not "(PRES:high cloud top|PRES:middle cloud top|PRES:low cloud top|PRES:high cloud bottom|PRES:middle cloud bottom|PRES:low cloud bottom)" -grib out1.grb2

cat out1.grb2 gefs.t00z.pgrb2af006 TSNOWP1.dat > out2.grb

mv "${COMIN_master}/gefs.t00z.master.grb2f006" "${COMIN_master}/gefs.t00z.master.grb2f006_org"
mv "${COMIN_master}/gefs.t00z.master.grb2if006" "${COMIN_master}/gefs.t00z.master.grb2if006_org"

mv out2.grb "${COMIN_master}/gefs.t00z.master.grb2f006"
${GRB2INDEX} "${COMIN_master}/gefs.t00z.master.grb2f006" "${COMIN_master}/gefs.t00z.master.grb2if006"

rm -fr out1.grb2 out2.grb TSNOWP*.dat

#output f03
infile=${COMIN_master}/gefs.t00z.master.grb2f003
#
${WGRIB2} "${infile}" | grep "TSNOWP" | ${WGRIB2} -i "${infile}" -grib TSNOWP2.dat
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
${WGRIB2} TSNOWP2.dat -for "1:1" -grib out.grb >/dev/null || true
${WGRIB2} out.grb -set_ftime "0-3 hour acc fcst" -grib TSNOWP1.dat

${WGRIB2} "${infile}" -not "(TSNOWP|ULWRF|USWRF)" -not "(DLWRF|DSWRF|UFLX|VFLX|SHTFL|LHTFL|PRATE|CPRAT|ALBDO|GFLUX|U-GWD|V-GWD)" -not "TCDC:(entire|boundary)" -not "HCDC:high cloud layer:182-185 min" -not "LCDC:low cloud layer:182-185 min" -not "MCDC:middle cloud layer:182-185 min" -not "(APCP|ACPCP|NCPCP)" -not "(TMIN|TMAX|DUVB|CDUVB|CWORK|SNOHF|SNOWC|WATR)" -not "(TMP:middle cloud top|TMP:low cloud top|TMP:high cloud top)" -not "(PRES:high cloud top|PRES:middle cloud top|PRES:low cloud top|PRES:high cloud bottom|PRES:middle cloud bottom|PRES:low cloud bottom)" -grib out1.grb

${WGRIB2} out1.grb -set_ftime "3 hour fcst" -grib out2.grb
cat out2.grb gefs.t00z.pgrb2af003 TSNOWP1.dat > out3.grb

mv "${COMIN_master}/gefs.t00z.master.grb2f003" "${COMIN_master}/gefs.t00z.master.grb2f003_org"
mv "${COMIN_master}/gefs.t00z.master.grb2if003" "${COMIN_master}/gefs.t00z.master.grb2if003_org"

mv out3.grb "${COMIN_master}/gefs.t00z.master.grb2f003"
${GRB2INDEX} "${COMIN_master}/gefs.t00z.master.grb2f003" "${COMIN_master}/gefs.t00z.master.grb2if003"

rm -fr out.grb out1.grb out2.grb

exit
4 changes: 3 additions & 1 deletion workflow/applications/gefs.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def _get_app_configs(self):
"""
Returns the config_files that are involved in gefs
"""
configs = ['stage_ic', 'fcst', 'atmos_products', 'arch']
configs = ['stage_ic', 'fcst', 'repairf0306', 'atmos_products', 'arch']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call the new job (and associated scripts) repair_replay_acc or something similar?


if self.nens > 0:
configs += ['efcs', 'atmos_ensstat']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
configs += ['efcs', 'atmos_ensstat']
configs += ['efcs', 'atmos_ensstat']
if self.do_repair_replay:
configs += ['repair_replay']

Expand Down Expand Up @@ -59,6 +59,8 @@ def get_task_names(self):
if self.nens > 0:
tasks += ['efcs']

tasks += ['repairf0306']
EricSinsky-NOAA marked this conversation as resolved.
Show resolved Hide resolved

tasks += ['atmos_prod']

if self.nens > 0:
Expand Down
Loading