Use ACCESS-NRI fork of generic tracers with access couplers

[dougiesquire]

Retry of #26, now from a branch on this repo.

See mom-ocean#388 for context and details.

This PR was originally made to mom-ocean/MOM5 but it was decided that we would make the changes here first while some details about the build system and testing are still being resolved. Comments from @aidanheerdegen's review of the original PR have been incorporated here.

Note, this PR includes the ACCESS-NRI fork of generic_tracers as a git submodule.

Note also, this PR removes the need for the ACCESS-OM-BGC build type (and hence the CSIRO_BGC pp def). The ACCESS-OM build type now always builds with the USE_OCEAN_BGC pp def and generic tracer WOMBAT can be configured at runtime. I’ve verified that always building with USE_OCEAN_BGC doesn’t impact performance when generic tracers are not configured. To do this, I ran two ACCESS-OM2 experiments on the NCI Gadi supercomputer:

• Exp 1 uses this configuration modified to run for 1 year at a time and to use a MOM5 executable built from 59e0aad (the most recent commit on main at the time of opening this PR).
• Exp 2 uses the same configuration, but the MOM5 executable used includes the changes in this PR.

I ran three years of each. The experiments were run at the same time. The chksum outputs in the logs are identical between the experiments and the walltimes reported by PBS for each year are given in the table below.

	Exp 1	Exp 2
year 1	959 s	908 s
year 2	1031 s	1002 s
year 3	1148 s	1028 s
average	1046 s	979 s

Note also also, I've hijacked the ACCESS-ESM build type to build with the ACCESS-NRI fork of generic tracers and with coupling support for the new WOMBATlite. This changes the behaviour of this build type, which used to build with the old MOM5 WOMBAT tracer package. However, we haven't been using this build type to build ACCESS-ESM1.5 anyway - instead ACCESS-ESM1.5 is built with the ACCESS-CM build type using the access-esm1.5 branch. With these changes, we should be able to use the master branch for ACCESS-ESM1.6.

Closes mom-ocean#388

[dougiesquire]

@CodeGat, could you please do the workaround for ACCESS-NRI/build-ci#170?

[dougiesquire]

I'm marking this as a draft because I'm going to see if I can extend this for ACCESS-ESM also.

[CodeGat]

@CodeGat, could you please do the workaround for ACCESS-NRI/build-ci#170?

Done @dougiesquire

[dougiesquire]

Note this PR really requires the changes in mom-ocean#390 for WOMBATlite to run correctly when there is surface salt restoring/correction (as in ACCESS-OM2). Without mom-ocean#390, things will still run, but no virtual flux correction will be applied based on the salt flux added.

Ideally, both mom-ocean#390 and this PR would be in mom-ocean/MOM5, but given how difficult it is to find suitable reviewers we may end up also moving mom-ocean#390 over to this ACCESS-NRI/MOM5.

UPDATE: I've now cherry-picked the changes in mom-ocean#390 into this PR.

[dougiesquire]

Note also, I think the build CI here only builds the ACCESS-OM build type, so the modifications to the ACCESS-ESM build type is this PR are not being tested in the CI.

[dougiesquire]

Okay, this PR now includes the complete set of changes required for ACCESS-OM2 and ACCESS-ESM1.5/1.6 with generic tracer WOMBATlite. In addition to the necessary changes to the accessom/cm_coupler driver code, I've also had to:

• Add the calculation of opacity to the ocean_shortwave_csiro module (58419f4). The old WOMBAT didn't use this, but the generic tracer one does.
• Allow passing salt flux restoring/correction to generic tracers (b1f2a5f). I originally proposed this change to mom-ocean/MOM5 here, but haven't been able to find a reviewer there so I've cherry-picked it into this PR.

@aidanheerdegen, this is ready for review. I've done test runs of both ACCESS-OM2 and ACCESS-ESM1.5 using the changes in this PR and things look sensible afaict.

[dougiesquire]

Note there are prerelease deployments of ACCESS-OM2 and ACCESS-ESM1.6 built from this PR:

• ACCESS-OM2
• ACCESS-ESM1.6

[aidanheerdegen]

I made some comments and pointed out a couple of old historical typos, but I'm really not familiar enough with the coupling stuff to know if you've missed anything. I think we're relying on the fact that it all seems to work to check that.

I assume fields still conserve and fluxes seem reasonable?

[aidanheerdegen]

LGTM.

[dougiesquire]

Thanks @aidanheerdegen, your eyes are very much appreciated. You have lovely eyes.