Add Preporcessing Scripts and a README to the sponges tools #96

theresa-morrison · 2024-09-23T19:34:09Z

This PR includes three pre-processing scripts I wrote and the filling script from Andrew. Together, these scripts manipulate the GLORYS data on uda into a subset region that is consistent with the input expected by write_nudging_data.py.

The README lists the steps for submitting these scripts. It would be gret if they could be merged and generalized to work for multiple domains.

The README also includes the overrides for MOM6 and lines that should be added to an xml to use the sponge data that is produced by these tools.

Partial readme including steps for using the preprocessing scripts and what changes need to be done in the xml to use temperature and salinity sponges in MOM6. Preprocessing scripts subset and average daily data from uda, fill the monthly data, and merge the monthly T and S averages into one file for each year.

Add more details to the readme

Add paths variables for archive and work.

uwagura · 2024-09-23T19:39:21Z

tools/sponge/README.md

+sbatch fill_glorys_nn_monthly.sh <YEAR> <MONTH>
+```
+
+3. Finally, once the filled data for every month in a given yeas have been created, the merge script can be used.


change to "every month in a given year has been created"

@theresa-morrison, I think this PR is almost ready, except for a small comment from @uwagura that hasn’t been addressed yet.

yeah, I was waiting to see if more typos would be found before submitting a commit. I will take care of this!

uwagura · 2024-09-23T19:44:51Z

tools/sponge/preproc_scripts/fill_glorys_nn_monthly.sh

@@ -0,0 +1,38 @@
+#!/bin/tcsh


very minor, but if these are cshell scripts then maybe we should change the file extension to .csh

I can change the extension, would we prefer to have them not be cshell scripts?

@theresa-morrison , since you're using C shell syntax (e.g., set), it might be simpler to rename the script to *.csh to reflect that.

Change file extensions for cshell scripts

andrew-c-ross · 2024-09-23T20:26:30Z

tools/sponge/preproc_scripts/fill_glorys_nn_monthly.csh

+# Regionally-slice and convert daily to monthly GLORYS reanalysis on archive beforehand.
+
+# dmget all of the files for this month from archive.
+dmget /archive/tnm/datasets/glorys/GLOBAL_MULTIYEAR_PHY_001_030/monthly/so/GLORYS_so_arctic_${year}_${month}.nc


These two dmgets could be combined into one (at least, I've always assumed dmget is happier getting everything at once instead of in multiple commands).

andrew-c-ross · 2024-09-23T20:30:35Z

tools/sponge/preproc_scripts/get_so_monthly.csh

+foreach filename (/uda/Global_Ocean_Physics_Reanalysis/global/daily/so/${year}/so_mercatorglorys12v1_gl12_mean_${year}${month}*.nc)
+  echo $filename 
+  set short_name='so_arctic_'$day
+  ncks -d latitude,39.,91. --mk_rec_dmn time $filename ${apath}/so_${year}_${month}/${short_name}'_bd.nc'


Ideally these temporary files would be written to $TMPDIR and only the final result would be copied to /archive

Done - only the final file is written to archive.

andrew-c-ross · 2024-09-23T20:34:31Z

tools/sponge/preproc_scripts/get_so_monthly.csh

+  set day = `expr $day + 1`
+  echo $day
+end
+ncra -O --cnk_plc=r1d --cnk_dmn=time,1  ${apath}/so_${year}_${month}/so_arctic_*.nc ${apath}/GLORYS_so_arctic_${year}_${month}.nc


I know this is doing the time average, but for clarity can you add a comment stating that, and also describing what --cnk_plc=r1d --cnk_dmn=time,1 is doing?

This was meant to help the averaging be faster, but it's not needed any more.

yichengt900

@theresa-morrison, thank you for uploading the example scripts for generating the necessary files for our sponge tools. I have tested them and can confirm that they work as expected. I'll be able to approve this PR once most of @uwagura and @andrew-c-ross's comments have been addressed. Please feel free to reach out if you need any assistance with those comments.

- change get_so and get_thetao to use a TMPDIR as scratch space - remove cnk options since they aren't needed - combine dmget statement

theresa-morrison · 2024-09-24T17:12:07Z

tools/sponge/preproc_scripts/fill_glorys_nn_monthly.csh

+# Regionally-slice and convert daily to monthly GLORYS reanalysis on archive beforehand.
+
+# dmget all of the files for this month from archive.
+dmget ${apath}/so/GLORYS_so_arctic_${year}_${month}.nc ${apath}/thetao/GLORYS_thetao_arctic_${year}_${month}.nc


@andrew-c-ross I think this combines the dmget. Nothing is being dmget in my testing, so I'm not sure if it is working.

update: not working, it is writing over my files as expected.

A typo in get_thetao mean so was being used instead. This has been fixed and not a variable ${var} has been added. This means that lines 18 to 29 should be the same in get_so and get_thetao.

yichengt900 · 2024-09-24T18:37:03Z

tools/sponge/preproc_scripts/merge_so_thetao_year.csh

+cp -f ${wpath}/GLORYS_so_arctic_${year}.nc ${wpath}/GLORYS_arctic_${year}.nc
+
+# Append temperature data to renamed salinity data 
+ncks -A  ${wpath}/GLORYS_thetao_arctic_${year}.nc ${wpath}/GLORYS_arctic_${year}.nc


@theresa-morrison, sorry for being picky, but do you think it’s a good idea to reduce redundant file copies and minimize repetitive processing? We could try something like the following:

!/bin/tcsh #SBATCH --ntasks=1 #SBATCH --job-name=fill_glorys_arctic #SBATCH --time=2880 #SBATCH --partition=batch # Usage: sbatch merge_so_thetao_year.csh <YEAR> module load cdo module load nco/5.0.1 module load gcp set year = $1 set wpath = '/work/Theresa.Morrison/datasets/glorys/GLOBAL_MULTIYEAR_PHY_001_030/monthly/filled' # Define the file variables for salinity and temperature set so_file = "${wpath}/GLORYS_so_arctic_${year}.nc" set thetao_file = "${wpath}/GLORYS_thetao_arctic_${year}.nc" set final_file = "${wpath}/GLORYS_arctic_${year}.nc" # Concatenate monthly averages into single files for salinity and temperature foreach var (so thetao) ncrcat -O ${wpath}/GLORYS_${var}_arctic_${year}_*.nc ${wpath}/GLORYS_${var}_arctic_${year}.nc end # Append temperature data to salinity file directly without copying ncks -A ${thetao_file} ${so_file} # Rename the combined file to final name mv -f ${so_file} ${final_file}

I don't mind, I appreciate the suggestions!

- simplify code based on YCT suggestion - update usage comment and job name

- fix typo -change file names from .sh to .csh

yichengt900 · 2024-09-30T18:29:45Z

tools/sponge/README.md

+
+## Using these files in MOM6
+
+To use the sponges generated by these scripts in MOM6 we reccomend the following settings:


OOPS, It's my bad but I found another one: "recommend"......

theresa-morrison · 2024-09-30T18:31:46Z

There are a few other changes I would like to make before this is merged.
(1) I think that once the merged file is created the individual monthly filled files should be removed
(2) make domain name a variable so that it can be changed in just one place

There is more that could be improved to streamline these scripts, but those are the two that I think make sense before merging.

typos: -fix spelling -add word

yichengt900 · 2024-11-13T18:52:50Z

Hi @theresa-morrison, I understand you have other tasks on your plate, and this one is of secondary importance. Just wanted to check in if you're still planning to make the changes you mentioned earlier, or if you'd prefer to revisit them later. Let me know what works best for you. Thanks!

Theresa Morrison and others added 5 commits September 23, 2024 14:37

Merge branch 'NOAA-GFDL:main' into feature/add_scripts_and_readme

3c23d6a

Update README.md

1ab6d69

Add more details to the readme

Update fill_glorys_nn_monthly.sh

0b95eec

Claen up scripts and add paths

d8812d4

Add paths variables for archive and work.

theresa-morrison requested review from andrew-c-ross, uwagura and yichengt900 September 23, 2024 19:35

uwagura reviewed Sep 23, 2024

View reviewed changes

Theresa Morrison added 2 commits September 23, 2024 16:04

Change file extensions

c008f7f

Change file extensions for cshell scripts

Remove .sh files

69a42c7

andrew-c-ross reviewed Sep 23, 2024

View reviewed changes

yichengt900 reviewed Sep 23, 2024

View reviewed changes

Update paths to use TMPDIR

e5d1d41

- change get_so and get_thetao to use a TMPDIR as scratch space - remove cnk options since they aren't needed - combine dmget statement

theresa-morrison commented Sep 24, 2024

View reviewed changes

Fix Error in get_thetao

3d3808b

A typo in get_thetao mean so was being used instead. This has been fixed and not a variable ${var} has been added. This means that lines 18 to 29 should be the same in get_so and get_thetao.

yichengt900 reviewed Sep 24, 2024

View reviewed changes

Theresa Morrison and others added 3 commits September 24, 2024 15:38

Simplify merge_so_thetao

6b724d0

- simplify code based on YCT suggestion - update usage comment and job name

Merge branch 'main' into feature/add_scripts_and_readme

7a642d3

Update README.md

6a97df2

- fix typo -change file names from .sh to .csh

yichengt900 reviewed Sep 30, 2024

View reviewed changes

theresa-morrison and others added 2 commits September 30, 2024 14:33

Update README.md

858a7da

typos: -fix spelling -add word

Merge branch 'main' into feature/add_scripts_and_readme

04a8e08

yichengt900 added the CEFI_MOM6_RT_gaea_c6 label Oct 6, 2024

yichengt900 added CEFI_MOM6_RT_gaea_c6 and removed CEFI_MOM6_RT_gaea_c6 labels Oct 16, 2024

Merge branch 'main' into feature/add_scripts_and_readme

4869c27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Preporcessing Scripts and a README to the sponges tools #96

Add Preporcessing Scripts and a README to the sponges tools #96

theresa-morrison commented Sep 23, 2024

uwagura Sep 23, 2024

yichengt900 Sep 30, 2024

theresa-morrison Sep 30, 2024

uwagura Sep 23, 2024

theresa-morrison Sep 23, 2024

yichengt900 Sep 23, 2024

theresa-morrison Sep 23, 2024

andrew-c-ross Sep 23, 2024

andrew-c-ross Sep 23, 2024

theresa-morrison Sep 24, 2024

andrew-c-ross Sep 23, 2024

theresa-morrison Sep 24, 2024 •

edited

Loading

yichengt900 left a comment

theresa-morrison Sep 24, 2024

theresa-morrison Sep 24, 2024

yichengt900 Sep 24, 2024

theresa-morrison Sep 24, 2024

yichengt900 Sep 30, 2024

theresa-morrison commented Sep 30, 2024

yichengt900 commented Nov 13, 2024


		## Using these files in MOM6

		To use the sponges generated by these scripts in MOM6 we reccomend the following settings:

Add Preporcessing Scripts and a README to the sponges tools #96

Are you sure you want to change the base?

Add Preporcessing Scripts and a README to the sponges tools #96

Conversation

theresa-morrison commented Sep 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theresa-morrison Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

yichengt900 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theresa-morrison commented Sep 30, 2024

yichengt900 commented Nov 13, 2024

theresa-morrison Sep 24, 2024 •

edited

Loading