Skip to content

Commit

Permalink
Dec 2024 (breichl#5)
Browse files Browse the repository at this point in the history
* Updating build scripts

* Updating .gitmodules

* Adding files to get Baltic example to work

* Adding Baltic grid information for WW3

* Updating the README

* Updating the README--again

* Updating the README--again

* Update README.md

* Updating the wave coupling to run an idealized hurricane (in progress)

* Adding updates for the working idealized hurricane case of wave coupling

* Adding forcing directory and link in INPUT

* Remove mod_def.ww3 from git

* Updating WW3

* Some final updates for a working Idealized_Hurricane example

* Update README.md

* Update README.md

* Updating coupler version

* Update README.md

* Update README.md

* Updating MOM6 version

* Update README.md

* Update README.md

* Updating WW3

* Adding 0p07 and 0p035 test cases

* Updating mkmf

* Updating 1/28th grid files

* Updating MOM6

* Adding a 0p035 deg Atlantic deep water idealized domain

* Adding 2 new Atlantic domains

* Fixing Stokes Band number options in Baltic_OM4_025 test case

* Initial commit of OM4_025.JRA test case

* Update MOM6 version

* Add gitignore

* Update to compile script

* Updates to the README

* Adding datasets link in OM4_025.JRA example

* Update README.md

* Update README.md

* Major update of code and compilers from 2020 -> 2024
- Code source all upgraded to more recent (2024) versions
- Compiler instructions upgraded for Gaea
- New input files included

* Oct2024_Qian (breichl#3)

* Update README.md

* Make all submodule URLs https and remove branches

* Fixing Baltic_OM4_025 test case

* Update MOM6 version

* Update README to reflect change https submodules

---------

Co-authored-by: Zeracesharon <[email protected]>
  • Loading branch information
breichl and Zeracesharon authored Dec 12, 2024
1 parent 696aab9 commit 7563496
Show file tree
Hide file tree
Showing 531 changed files with 7,089 additions and 94 deletions.
33 changes: 33 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Build directory might optionally be placed here
build*
# Ignore .datasets
.datasets
# Ignore model output
*.out
CPU_stats
CPU_stats.*
_read_error.nml

# Ignore most available_diags.* files, but add two as a sample
available_diags.0*
SIS.available_diags
SIS_fast.available_diags
!ocean_only/global_ALE/z/available_diags.000000
!ocean_only/global_ALE/layer/available_diags.000000

exitcode
fort.20
*.nc
*.nc.*
RESTART
ocean.stats
ocean.stats.*
seaice.stats
seaice.stats.*
# Ignore python .pyc files
*.pyc
.ipynb_checkpoints
# Ignore anything labeled ignore
*.ignore
# Ignore lcov files
lcov.*
13 changes: 8 additions & 5 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
url = https://github.com/NOAA-GFDL/atmos_null.git
[submodule "src/FMS"]
path = src/FMS
url = git@github.com:breichl/FMS.git
url = https://github.com/breichl/FMS.git
[submodule "src/icebergs"]
path = src/icebergs
url = https://github.com/NOAA-GFDL/icebergs.git
Expand All @@ -15,16 +15,19 @@
url = https://github.com/NOAA-GFDL/land_null.git
[submodule "src/mkmf"]
path = src/mkmf
url = https://github.com/NOAA-GFDL/mkmf.git
url = https://github.com/breichl/mkmf.git
[submodule "src/MOM6"]
path = src/MOM6
url = git@github.com:breichl/MOM6.git
url = https://github.com/breichl/MOM6.git
[submodule "src/SIS2"]
path = src/SIS2
url = https://github.com/NOAA-GFDL/SIS2.git
[submodule "src/coupler"]
path = src/coupler
url = git@github.com:breichl/FMScoupler.git
url = https://github.com/breichl/FMScoupler.git
[submodule "src/WW3"]
path = src/WW3
url = [email protected]:breichl/WW3.git
url = https://github.com/breichl/WW3.git
[submodule "src/Icepack"]
path = src/Icepack
url = https://github.com/CICE-Consortium/Icepack.git
142 changes: 130 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,143 @@
# FMS Wave Coupling: WAVEWATCH III in FMS

### Clone this repository, update submodules.

To clone this repository from github use the following:

> Using https:
>
> git clone https://github.com/breichl/FMS_Wave_Coupling.git
Download and update the submodules:

> cd FMS_Wave_Coupling
> git submodule update --init --recursive
Now you should be ready to compile.

### Compiling on Gaea

1. First step is to set up the wave model. The wave model source code must be pre-processed using WAVEWATCH provided programs to convert to standard FORTRAN 90 files. A sample script to complete this step is provided in "tools/SetUpWW3_gaea.csh", which works on Gaea to use a particular switch file ("tools/switch") and extract FORTRAN 90 code from the WW3 ftn code. This script sets up code to compile WW3 preprocessor routines for building the grid binary (ww3_grid), the forcing binaries (ww3_prnc, needed for standalone WW3), and the stand alone WW3 driver (ww3_multi). It also sets up code to compile WW3 postprocessor routines for converting the output binary into NetCDF (ww3_ounf).
1. First step is to set up the wave model. The wave model source code must be pre-processed using WAVEWATCH provided programs to convert to standard FORTRAN 90 files. A sample script to complete this step is provided in "tools/SetUpWW3.csh", which works on Gaea and GFDL workstations to use a particular switch file and extract FORTRAN 90 code from the WW3 ftn code. This script sets up code to compile WW3 preprocessor routines for building the grid binary (ww3_grid), the forcing binaries (ww3_prnc, needed for standalone WW3), and the stand alone WW3 driver (ww3_multi). It also sets up code to compile WW3 postprocessor routines for converting the output binary into NetCDF (ww3_ounf). Note that the wave model needs to know a valid compiler to unpack its "ftn" files into "f90" files, but you shouldn't need to use the same fortran compiler that you will use to create executables.

> Gaea Instructions:
> Gaea/GFDL Instructions:
>
> cd FMS_Wave_Coupling
> ln -s tools/SetUpWW3_gaea.csh .
> ./SetUpWW3_gaea.csh
> ./Set_Up_WW3.csh
2. The second step is to compile. Again, a script to do this is provided in "tools/Wave_Compile_gaea.csh". This script will compile 1) FMS library, 2) ww3_grid, 3) ww3_prnc, 4) ww3_multi, 5) WW3 library (for linking within coupled model), 6) the coupled model , and finally 7) ww3_ounf.
2. The second step is to compile. Again, a script to do this is provided for GFDL and Gaea. This script will compile 1) FMS library, 2) ww3_grid, 3) ww3_prnc, 4) ww3_multi, 5) WW3 library (for linking within coupled model), 6) the coupled model , and 7) ww3_ounf. You need to create the soft link to the compile file (this works for both C5 and C6 clusters):
>ln -sf tools/FMScomp.csh FMScomp.csh
> Gaea Instructions:
>ln -sf tools/WW3comp.csh WW3comp.csh
>ln -sf tools/MOM6comp.csh MOM6comp.csh
>ln -sf tools/WW3execcomp.csh WW3execcomp.csh
>
> cd FMS_Wave_Coupling
> ln -s tools/Wave_Compile_gaea.csh .
> ./Wave_Compile_gaea.csh
>ln -sf tools/envs.ncrc6.intel23 envs.ncrc6.intel23

> Gaea Instructions:
>
> cd FMS_Wave_Coupling
>
> ./FMScomp.csh
>
> ./WW3comp.csh
>
> ./MOM6comp.csh
>
> ./WW3execcomp.csh

If working on Gaea or the GFDL workstation, these steps should successfully compile libraries and executables needed to set-up and run the WW3 coupled system with FMS. You can check by run:
>cd FMS_Wave_Coupling
>
>ls build/ncrc6.intel23/wave_ice_ocean/REPRO/MOM6
### Running - OM4_025.JRA example

1. First follow the instructions to download the MOM6-examples input data (https://github.com/NOAA-GFDL/MOM6-examples/wiki/Getting-started#downloading-input-data). Link this directory into the main directory for this repository as ".datasets", exactly as you would in MOM6-examples to use those test cases. On gaea we simply execute "ln -sf <see location below> .datasets". You would replace the source file location with the location you have put the datasets file you download.

On f6:
>cd FMS_Wave_Coupling
>
>ln -sf /gpfs/f6/gfdl/world-shared/gold/datasets .datasets
On f5:
>cd FMS_Wave_Coupling
>
>ln -sf /gpfs/f5/gfdl_o/world-shared/datasets .datasets
2. Change to the OM4_025.JRA test case directory

> cd examples/OM4_025.JRA
3. We next have to set-up the grid.

a. (Optional) If you are on a system that has access to the FMS/FRE tools, you can build your own grids and exchange grids (these tools are available on NOAA/GFDL github: FRE-NCtools). See the GRID directory for some ideas of how these grids can be created to work with the wave model. You would need to run the BuildExchangeGrid.csh first and WaveMosaics.py (cshell and python) scripts both to create all necesssary files (there are some pseudo-hacks needed to set-up the grid_spec.nc and the wave related exchange grids, if you are having trouble getting the wind into the wave model this grid_spec step is critical). The exchange grids can be generated at OM4_025.JRA/INPUT/EXCHANGE_GRIDS. To run BuildExchangeGrid.csh, you have to copy the BUildExchangeGrid.csh at FMS_Wave_Coupling/examples/Baltic_OM4_025/INPUT/ExchangeGrids and make some edits with the correct fre/bronx version. Also, You might not want to run this on a single login node, it is highly recommended to run this on parallel (using r.g. srun --ntasks=30 make_coupler_mosaic_parallel replacing make_coupler_mosaic). After running the BuildExchangeGrid.csh, you will obtain an updated file named with grid_spec_waves.nc. Replace the contents of the original grid_spec.nc file in the OM4_025.JRA/INPUT/ directory with the contents of grid_spec_waves.nc while keeping file name as grid_spec.nc. Alternatively, you can simply create a symbolic link in the OM4_025.JRA/INPUT/ directory using the following command: ln -s EXCHANGE_GRIDS/grid_spec_waves.nc grid_spec.nc. But, this is all optional, you can simply run with the example here to have a 0p25 degree resolution case on the MOM6 native 1/4 degree grid.

b. You will need to update the WW3 grid files in WW3/PreProc, see WW3/PreProc/GenGrid.py for an example (this script should create the basic files for you without modification). This generates the WW3 Depth grid for a curvilinear tripolar grid on the MOM6 native 1/4 degree grid.

> cd WW3/PreProc
> python GenGrid.py
c. Next you need to create the mod_def.ww3 file for WW3's internal grid. Navigate to WW3/PreProc and execute the ww3_grid (note on GFDL/Gaea you need to load the NetCDF libraries used to compile, e.g., on GFDL system: "module load netcdf/4.2" and on Gaea "module load cray-netcdf" before this will work):

> cd WW3/PreProc
> ../../../../build/ncrc6.intel23/ww3_grid/REPRO/ww3_grid
Please replace "ncrc6.intel23" with your compiler environment.

2. Don't forget to create a RESTART directory within the OM4_025.JRA example directory:
>cd examples/OM4_025.JRA
>
> mkdir RESTART
4. Now we should be ready to run. To run on Gaea we can either submit a job through the batch system, or run the job interactively. In this example we are going to run it interactively.

> Gaea Instructions:
>
> cd examples/OM4_025.JRA
> salloc --clusters=c6 --qos=normal --nodes=10 --x11
> srun -n 320 ../../build/ncrc6.intel23/wave_ice_ocean/REPRO/MOM6
5. If this works, then congratulations, you have successful set-up, compiled, and run this example. If you succeded in running the exmaple but run into the errors at the MPI finalize, like below:
>forrtl: severe (174): SIGSEGV, segmentation fault occurred
>
>Image PC Routine Line Source
>
>libpthread-2.31.s 00007F663D2DF910 Unknown Unknown Unknown
>
>libmlx5.so.1.24.4 00007F663AA4DC4B Unknown Unknown Unknown
>
>libmlx5.so.1.24.4 00007F663AA9881A Unknown Unknown Unknown
>
>libfabric.so.1.23 00007F663B0BEA73 Unknown Unknown Unknown
>
>libfabric.so.1.23 00007F663B0C7CF7 Unknown Unknown Unknown
>
>libfabric.so.1.23 00007F663B0D193C Unknown Unknown Unknown
>
>libfabric.so.1.23 00007F663B0D23EB Unknown Unknown Unknown
>
>libfabric.so.1.23 00007F663B0D4DD4 Unknown Unknown Unknown
>
>libmpi_intel.so.1 00007F663F895FFE Unknown Unknown Unknown
>
>libmpi_intel.so.1 00007F663F6E9484 Unknown Unknown Unknown
>
>libmpi_intel.so.1 00007F663E174854 MPI_Finalize Unknown Unknown
>
>libmpifort_intel. 00007F664007D2B9 pmpi_finalize__ Unknown Unknown
You need to add one line before you start running the MOM6 executable:

>export FI_VERBS_PREFER_XRC=0
>
In C6 platform, if you encountered any problems with "fatal OVERFLOW buffer list exhausted", try to use the code in the directory of Examples/OM4_025.JRA before you run the case:

If working on Gaea, these steps should successfully compile libraries and executables needed to set-up and run the WW3 coupled system with FMS.
>export FI_CXI_RX_MATCH_MODE=hybrid
### Running on Gaea
If the error stays, you might need to try to run with a different core number (e.g. try with srun -n 400 or 600). There is much more we can do with this including customizing set-ups, and processing and manipulating output. Adding more instructions to this README.md is always welcome!

1.
6. Basic instructions for creating WW3 output files: MOM6 creates output in an easy to use netCDF format, but WW3 output is stored as binary files which need a further processing step to convert to netCDF. You should have all the tools to do this already in place. All you need to do is navigate to WW3/PostProc, and find the build/intel/wave_ice_ocean/ww3_ounf/ww3_ounf executable. If you run it in this place, the proper files should already be linked to create netCDF from the mod_def and out_grd files. Note that the ww3_ounf.inp can be edited (along with ww3_multi.inp) to specify diagnostics to save and frequency. More information can be found in the WW3 directory.
1 change: 1 addition & 0 deletions Set_Up_WW3.csh
1 change: 1 addition & 0 deletions Wave_Compile.csh
15 changes: 15 additions & 0 deletions examples/Baltic_OM4_025/INPUT/ExchangeGrids/BuildExchangeGrids.csh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/tcsh
#

module load fre/bronx-15

make_solo_mosaic --num_tiles 1 --dir ./ --mosaic_name ocean_mosaic --tile_file ocean_hgrid.nc --periodx 360 --periody 360
make_solo_mosaic --num_tiles 1 --dir ./ --mosaic_name atmos_mosaic --tile_file ocean_hgrid.nc --periodx 360 --periody 360
make_solo_mosaic --num_tiles 1 --dir ./ --mosaic_name land_mosaic --tile_file ocean_hgrid.nc --periodx 360 --periody 360
make_solo_mosaic --num_tiles 1 --dir ./ --mosaic_name wave_mosaic --tile_file ocean_hgrid.nc --periodx 360 --periody 360

make_coupler_mosaic --atmos_mosaic atmos_mosaic.nc --land_mosaic land_mosaic.nc --ocean_mosaic wave_mosaic.nc --ocean_topog wave_topog.nc --mosaic_name grid_spec

cp atmos_mosaic_tile1Xocean_mosaic_tile1.nc atmos_mosaic_tile1Xwave_mosaic_tile1.nc

make_coupler_mosaic --atmos_mosaic atmos_mosaic.nc --land_mosaic land_mosaic.nc --ocean_mosaic ocean_mosaic.nc --wave_mosaic wave_mosaic.nc --ocean_topog ocean_topog.nc --mosaic_name grid_spec
32 changes: 32 additions & 0 deletions examples/Baltic_OM4_025/INPUT/ExchangeGrids/WaveMosaics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import netCDF4 as NC

with NC.Dataset("grid_spec.nc") as src, NC.Dataset("grid_spec_waves.nc","w") as dst:
dst.setncatts(src.__dict__)
for name, dimension in src.dimensions.items():
dst.createDimension(name, (len(dimension) if not dimension.isunlimited() else none))
for name, variable in src.variables.items():
x = dst.createVariable(name, variable.datatype, variable.dimensions)
dst[name][:] = src[name][:]
dst[name].setncatts(src[name].__dict__)

dst.createDimension('nfile_aXw',1)

# a=dst.createVariable('wav_mosaic_dir','c',('string'))
# STRING = './'
# dst.variables['wav_mosaic_dir'][:len(STRING)] = STRING
# a.setncattr('standard_name','directory_storying_wave_mosaic')

# a=dst.createVariable('wav_mosaic_file','c',('string'))
# STRING = 'wave_mosaic.nc'
# dst.variables['wav_mosaic_file'][:len(STRING)] = STRING
# a.setncattr('standard_name','wave_mosaic_file_name')

# a=dst.createVariable('wav_mosaic','c',('string'))
# STRING = 'wave_mosaic'
# dst.variables['wav_mosaic'][:len(STRING)] = STRING
# a.setncattr('standard_name','wave_mosaic_name')

a=dst.createVariable('aXw_file','c',('nfile_aXw','string'))
STRING = 'atmos_mosaic_tile1Xwave_mosaic_tile1.nc'
dst.variables['aXw_file'][0,:len(STRING)] = STRING
a.setncattr('standard_name','atmXwav_exchange_grid_file')
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ExchangeGrids/ocean_hgrid.nc
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ExchangeGrids/ocean_topog.nc
Binary file not shown.
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ExchangeGrids/wave_topog.nc
9 changes: 9 additions & 0 deletions examples/Baltic_OM4_025/INPUT/MOM_channels_global_025
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
! This file specifies restricted channel widths in MOM. The order is:
! [U|V]_width, min_longitude, max_longitude, min_latitude, max_latitude, width

U_width, -5.9, -5.4, 35.8, 36.2, 12000.0 ! Gibraltar
U_width, 26.4, 26.8, 40.4, 40.6, 5000.0 ! Dardanelles

V_width, 28.8, 29.4, 41.1, 41.2, 2500.0 ! Bosporus - should be 1km wide.
V_width, 43.25, 43.5, 12.6, 12.8, 10000.0 ! Red Sea, Bab-el-Mendeb
V_width, 141.5, 141.7, 52.0, 52.15, 2500.0 ! Between Sakhalin & Russia
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/WOA05_pottemp_salt.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/atmos_mosaic.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/geothermal_davies2013_v1.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/grid_spec.nc
Binary file added examples/Baltic_OM4_025/INPUT/hycom1_75_800m.nc
Binary file not shown.
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/land_mask.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/land_mosaic.nc
Binary file added examples/Baltic_OM4_025/INPUT/layer_coord.nc
Binary file not shown.
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ncar_precip.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ncar_rad.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ocean_hgrid.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ocean_mosaic.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/ocean_topog.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/q_10_mod.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/salt_restore.v20140616.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/slp.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/t_10_mod.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/tidal_amplitude.v20140616.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/topog.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/u_10_mod.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/v_10_mod.clim.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/wave_mask.nc
1 change: 1 addition & 0 deletions examples/Baltic_OM4_025/INPUT/wave_mosaic.nc
9 changes: 8 additions & 1 deletion examples/Baltic_OM4_025/MOM_override
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
! Blank file in which we can put "overrides" for parameters !VERBOSITY = 9
USE_WAVES = True
WAVE_METHOD = SURFACE_BANDS
SURFBAND_SOURCE = COUPLER
SURFBAND_SOURCE = COUPLER
STK_BAND_COUPLER = 3

#override USE_LA_LI2016 = False
#override EPBL_LT = True
STOKES_VF = True
STOKES_PGF = True
STOKES_DDT = True
Loading

0 comments on commit 7563496

Please sign in to comment.