Performance regression

[simone-silvestri]

A performance regression occurred between some commits of PR#104.

This commit ran in 4hr51
https://buildkite.com/clima/climaocean-examples/builds/103#_

while the latest one takes 5hr46
https://buildkite.com/clima/climaocean-examples/builds/200

Quite some change in performance.
I think this is due to Oceananigans because there has been quite some development going on recently while PR #104 has not really changed so much.

I ll merge #104 but let's keep this in mind. I ll try to backtrack to see what happened.

[glwagner]

What changed in Oceananigans between those commits?

[simone-silvestri]

I am not sure, what comes into mind in terms of big PRs that might have affected the speed are this one
CliMA/Oceananigans.jl#3658
and this one
CliMA/Oceananigans.jl#3482
The former is supposed to have decreased the computational time so I am a bit puzzled unless among the last commits after the benchmarks something changed I have tested it in climaocean.
The latter is only boundary conditions so I am not sure how it could have affected the speed. I ll do some tests

[glwagner]

What version of Oceananigans was performance measured on first, and what version is the second result from? Did you change compat between those?

[simone-silvestri]

Hmm, I tried benchmarks on Oceananigans and it looks like everything is in order. Also I have tried prototype_omip_simulation.jl in the prototype_omip_simulation folder on https://buildkite.com/clima/climaocean-examples/builds/103#_ forcing Oceananigans to main and indeed, the code is about 15% faster so it is a ClimaOcean thing.

The change happened between commit
826208d
and commit
ec6ddd2

but I only see the changes to the logarithmic profile which I don't think can have caused this difference. I ll prepare a MWE between the two commits