Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed absolute paths and tested batch scripts #21

Merged
merged 1 commit into from
Jan 5, 2021
Merged

fixed absolute paths and tested batch scripts #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions scripts/batch_conv_diags.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@
#SBATCH [email protected]

# PyGSIdir=path/to/PyGSI_Directory
PyGSIdir=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI
OUTDIR=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI/
PyGSIdir=../
OUTDIR=../
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only works if the scripts directory is the same directory you submit the job from. I think you want to use something like PyGSIdir=`dirname "$0"```. Note there are two extra because I can't figure out how to make a literal back tick in GitHub.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also outdir should be somewhere else by default, even if it's just $PyGSIdir/out (may need a mkdir -p $OUTDIR)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are several issues with this script:

  1. [email protected] should be eliminated. A non-NOAA user should be able to run this.
  2. This script sources a modulefile meant for hera. We should eliminate that line or make it generic. What if there are no modules?
  3. Establish a Data/ directory and then all paths are relative to that. Relative paths should not be relative to current working directory, and having paths such as ../ are trip hazards.

YAML=$PyGSIdir/test_conv_yaml.yaml

# load environment needed to run python scripts
source $PyGSIdir/modulefiles/modulefile.PyGSI.hera.bash

python $PyGSIdir/scripts/mp_plot_convDiags.py -n 20 -y $YAML -o $OUTDIR
python $PyGSIdir/scripts/mp_plot_conv_diags.py -n 20 -y $YAML -o $OUTDIR
7 changes: 3 additions & 4 deletions scripts/batch_sat_diags.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

#!/bin/bash
#SBATCH -J plot_gsi_diags
#SBATCH -A da-cpu
Expand All @@ -9,11 +8,11 @@
#SBATCH [email protected]

# PyGSIdir=path/to/PyGSI_Directory
PyGSIdir=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI
OUTDIR=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI/
PyGSIdir=../
OUTDIR=../
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comments as above.

YAML=$PyGSIdir/test_sat_yaml.yaml

# load environment needed to run python scripts
source $PyGSIdir/modulefiles/modulefile.PyGSI.hera.bash

python $PyGSIdir/scripts/mp_plot_satDiags.py -n 20 -y $PyGSIdir/sat_test_yaml.yaml -o $OUTDIR
python $PyGSIdir/scripts/mp_plot_sat_diags.py -n 20 -y $YAML -o $OUTDIR
6 changes: 3 additions & 3 deletions scripts/batch_write_ncfile_conv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@
#SBATCH [email protected]

# PyGSIdir=path/to/PyGSI_Directory
PyGSIdir=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI
OUTDIR=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI/ncfiles
PyGSIdir=../
OUTDIR=../
YAML=$PyGSIdir/test_conv_yaml.yaml

# load environment needed to run python scripts
source $PyGSIdir/modulefiles/modulefile.PyGSI.hera.bash

python $PyGSIdir/scripts/create_convNetCDF.py -n 20 -y $YAML -o $OUTDIR
python $PyGSIdir/scripts/create_conv_netcdf.py -n 20 -y $YAML -o $OUTDIR
6 changes: 3 additions & 3 deletions scripts/batch_write_ncfile_sat.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@
#SBATCH [email protected]

# PyGSIdir=path/to/PyGSI_Directory
PyGSIdir=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI
OUTDIR=/scratch1/NCEPDEV/da/$LOGNAME/PyGSI/ncfiles/radiance
PyGSIdir=../
OUTDIR=../
YAML=$PyGSIdir/test_sat_yaml.yaml

# load environment needed to run python scripts
source $PyGSIdir/modulefiles/modulefile.PyGSI.hera.bash

python $PyGSIdir/scripts/create_satNetCDF.py -n 20 -y $YAML -o $OUTDIR
python $PyGSIdir/scripts/create_sat_netcdf.py -n 20 -y $YAML -o $OUTDIR
10 changes: 5 additions & 5 deletions scripts/create_conv_netcdf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def create_netcdf(conv_config):
metadata = diag.get_metadata()

metadata['Data_type'] = data_type
metadata['Obsid'] = obsid
metadata['ObsID'] = obsid
metadata['Subtype'] = subtype
metadata['outDir'] = outdir

Expand All @@ -79,10 +79,10 @@ def create_netcdf(conv_config):

# Get binned data
if diag_components[1] == 'conv' and diag_components[2] == 'uv':
binned_data = spatialBin(
data, lat, lon, binsize='1x1', uvData=True, pressure=pressure)
binned_data = spatial_bin(
data, lat, lon, binsize='1x1', uv_data=True, pressure=pressure)
else:
binned_data = spatialBin(
binned_data = spatial_bin(
data, lat, lon, binsize='1x1', pressure=pressure)

write_netcdf(data, binned_data, metadata)
Expand Down Expand Up @@ -117,7 +117,7 @@ def create_netcdf(conv_config):
print('Binning..')
if diag_components[1] == 'conv' and diag_components[2] == 'uv':
binned_data = spatial_bin(
data, lat, lon, binsize='1x1', uvData=True, pressure=pressure)
data, lat, lon, binsize='1x1', uv_data=True, pressure=pressure)
else:
binned_data = spatial_bin(
data, lat, lon, binsize='1x1', pressure=pressure)
Expand Down
4 changes: 2 additions & 2 deletions scripts/mp_plot_conv_diags.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,12 @@ def plotting(conv_config):
myargs = ap.parse_args()

if myargs.nprocs:
nprocs = int(MyArgs.nprocs)
nprocs = int(myargs.nprocs)
else:
nprocs = 1

input_yaml = myargs.yaml
outdir = MyArgs.outdir
outdir = myargs.outdir

with open(input_yaml, 'r') as file:
parsed_yaml_file = yaml.load(file, Loader=yaml.FullLoader)
Expand Down
4 changes: 2 additions & 2 deletions scripts/mp_plot_sat_diags.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def plotting(sat_config):

metadata = diag.get_metadata()
metadata['Data_type'] = data_type
metadata['channels'] = channel
metadata['Channels'] = channel

if np.isin('histogram', plot_type):
plot_histogram(data, metadata, outdir)
Expand All @@ -55,7 +55,7 @@ def plotting(sat_config):
else:
nprocs = 1

YAML = myargs.yaml
input_yaml = myargs.yaml
outdir = myargs.outdir

with open(input_yaml, 'r') as file:
Expand Down