Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GSL atmosphere_model will not reproduce NCAR atmosphere_model #66

Open
barlage opened this issue Nov 6, 2024 · 13 comments
Open

GSL atmosphere_model will not reproduce NCAR atmosphere_model #66

barlage opened this issue Nov 6, 2024 · 13 comments
Assignees

Comments

@barlage
Copy link
Collaborator

barlage commented Nov 6, 2024

For out-of-the-box mesoscale_reference and convection_permitting suites, the GSL code does not reproduce NCAR code. For example, CONUS mesoscale_reference testcase differences (using code fixing #65) after 1 hour (5 timesteps):

nccmp -dsSqf bar-455a5a9fa.mesoscale_reference.ncar.ncar.conus.120km.gfs.2023031015/history.2023-03-10_16.00.00.nc ncar-v8.2.2.mesoscale_reference.ncar.ncar.conus.120km.gfs.2023031015/history.2023-03-10_16.00.00.nc

Variable               Group Count          Sum      AbsSum          Min          Max       Range         Mean      StdDev
qv                     /     10347 -5.82311e-06 0.000208967 -1.82134e-05  2.44682e-05 4.26816e-05 -5.62783e-10 4.40722e-07
qc                     /       370  1.03647e-05 1.86199e-05 -1.05407e-06  6.84692e-06 7.90099e-06  2.80127e-08 4.23858e-07
qr                     /       856  6.41402e-06 0.000107672 -1.28703e-05  1.21038e-05 2.49742e-05  7.49302e-09 9.11512e-07
qi                     /      4993 -4.43827e-06 1.42143e-05 -2.46453e-06  6.26966e-07  3.0915e-06 -8.88899e-10 5.44309e-08
qs                     /      3897   5.7315e-06 1.21774e-05 -4.81627e-07  4.72372e-06 5.20535e-06  1.47075e-09 8.52991e-08
qg                     /      3406  2.76559e-08 1.63107e-07 -6.19158e-09  1.77972e-08 2.39888e-08  8.11976e-12 4.73913e-10
u                      /     63447  0.000180574    0.116743 -0.000180483  0.000187874 0.000368357  2.84606e-09 5.37565e-06
w                      /     46769  0.000399901  0.00722084 -7.26432e-06  6.59935e-06 1.38637e-05  8.55055e-09  3.8066e-07
pressure               /      2368      7.90759     196.563     -20.7344      15.1797     35.9141   0.00333936    0.810607
surface_pressure       /        32      5.74219     23.2109     -5.53125      9.98438     15.5156     0.179443     2.16991
rho                    /      2690  -9.0152e-07 0.000141688 -1.07288e-06  1.66893e-06 2.74181e-06 -3.35138e-10 8.91583e-08
theta                  /      2485    0.0183105    0.660706    -0.063324    0.0474854    0.110809  7.36843e-06  0.00236695
relhum                 /     10460    -0.627489     10.8649    -0.677376     0.561337     1.23871 -5.99894e-05   0.0136334
divergence             /     30253 -2.11224e-09 7.54819e-07 -1.15688e-09   1.1014e-09 2.25828e-09 -6.98191e-14  5.1891e-11
vorticity              /     46096 -1.67385e-09 1.90474e-06 -3.60251e-09  3.56295e-09 7.16545e-09 -3.63123e-14 1.15045e-10
ke                     /     20974    0.0046308    0.840786  -0.00610352   0.00390625   0.0100098  2.20788e-07 0.000134643
uReconstructZonal      /     18379  -3.7672e-05   0.0385466 -0.000185966  0.000179291 0.000365257 -2.04973e-09 5.69106e-06
uReconstructMeridional /     22521 -0.000259013   0.0263623 -0.000149727  0.000125885 0.000275612 -1.15009e-08 3.19913e-06
ertel_pv               /     27142   -0.0163986    0.591087   -0.0217799    0.0208461    0.042626 -6.04177e-07 0.000354364
u_pv                   /       368  0.000276625  0.00286525 -7.24792e-05    0.0001297 0.000202179  7.51699e-07 1.45387e-05
v_pv                   /       506  8.92244e-05  0.00188054 -3.24249e-05  6.86646e-05 0.000101089  1.76333e-07 7.37711e-06
theta_pv               /       145  6.10352e-05  0.00549316  -0.00012207  9.15527e-05 0.000213623  4.20932e-07 4.14751e-05
vort_pv                /       609  -9.4181e-10 1.76564e-08 -4.07454e-10  4.65661e-10 8.73115e-10 -1.54649e-12 6.01499e-11
depv_dt_lw             /     24734 -1.07915e-09 4.36438e-08 -1.00408e-09  1.00408e-09 2.00816e-09   -4.363e-14 1.63292e-11
depv_dt_sw             /     25137 -8.64733e-11 2.74772e-08 -4.94765e-10  3.38332e-10 8.33097e-10 -3.44008e-15 6.39317e-12
depv_dt_bl             /     25677 -2.67592e-07 5.87958e-06 -2.67306e-07  1.22556e-07 3.89862e-07 -1.04215e-11 3.49661e-09
depv_dt_cu             /     10439  2.14703e-07 7.74505e-06 -2.43279e-07  2.45578e-07 4.88857e-07  2.05674e-11 7.40202e-09
depv_dt_mix            /     35708 -6.43592e-10 5.01075e-07 -1.38334e-09  1.74805e-09 3.13139e-09 -1.80237e-14 4.88766e-11
dtheta_dt_mp           /      1095  2.22524e-05 0.000805791   -8.795e-05  6.59519e-05 0.000153902  2.03218e-08 4.95232e-06
depv_dt_mp             /     28514 -2.07761e-05 0.000592738 -3.02505e-05  2.89525e-05  5.9203e-05 -7.28629e-10 4.78172e-07
depv_dt_diab           /     33253 -2.08312e-05 0.000599537 -3.02505e-05  2.89525e-05  5.9203e-05 -6.26446e-10 4.42804e-07
depv_dt_fric           /     37552 -2.25347e-06 5.07487e-05   -6.519e-06  3.02762e-06 9.54661e-06 -6.00094e-11 4.35107e-08
depv_dt_diab_pv        /       688  6.55136e-09   3.939e-07 -1.61867e-08    1.321e-08 2.93967e-08  9.52233e-12 1.84542e-09
depv_dt_fric_pv        /       721 -1.27339e-08 5.37006e-08 -9.77889e-09  9.24047e-10 1.07029e-08 -1.76614e-11 3.85797e-10
rainnc                 /       276  -0.00226518  0.00670537  -0.00353809   0.00143323  0.00497132 -8.20717e-06 0.000236026
precipw                /       245  -0.00163984  0.00830603  -0.00219202   0.00242233  0.00461435 -6.69324e-06 0.000244621
cuprec                 /        26   8.3805e-10 1.45554e-08 -2.38833e-09  2.14641e-09 4.53474e-09  3.22327e-11 9.76563e-10
rainc                  /        24  6.09372e-07 1.04721e-05 -1.72015e-06  1.54227e-06 3.26242e-06  2.53905e-08 7.32668e-07
hpbl                   /        21    0.0842514    0.193611   -0.0121536     0.100037     0.11219   0.00401197   0.0228747
hfx                    /        21  -0.00104582  0.00382769  -0.00105286  0.000692368  0.00174522 -4.98011e-05 0.000363773
qfx                    /        29 -1.80762e-11 3.23894e-10 -1.05501e-10  4.72937e-11 1.52795e-10 -6.23317e-13 2.34702e-11
cd                     /         9  7.45058e-09 2.39816e-08 -5.58794e-09  5.12227e-09 1.07102e-08  8.27842e-10  3.3369e-09
cda                    /        11  6.40284e-09 2.66591e-08 -3.72529e-09   6.0536e-09 9.77889e-09  5.82077e-10 2.96025e-09
ck                     /        11  7.21775e-09 1.16415e-08 -9.31323e-10  2.32831e-09 3.25963e-09  6.56159e-10 1.21125e-09
cka                    /        15  7.91624e-09  1.3737e-08 -9.31323e-10   3.0268e-09 3.95812e-09  5.27749e-10 1.31699e-09
lh                     /        30 -9.01222e-05  0.00079298 -0.000267029  0.000118256 0.000385284 -3.00407e-06 5.81692e-05
u10                    /        54  1.04308e-07 2.16812e-05 -2.38419e-06  1.90735e-06 4.29153e-06  1.93163e-09 6.00568e-07
v10                    /        47  2.53553e-06 2.12169e-05 -1.19209e-06  1.43051e-06  2.6226e-06  5.39474e-08 5.85776e-07
q2                     /        19   -0.0110567    0.017515   -0.0142858   0.00118274   0.0154686  -0.00058193  0.00334108
t2m                    /         9     -37.3928     37.3929     -20.4113  6.10352e-05     20.4114     -4.15475     7.51508
th2m                   /         9     -37.4258     37.4258      -20.416  3.05176e-05      20.416     -4.15842     7.52054
acsnow                 /        14 -1.35231e-07 1.53334e-07  -1.2666e-07  3.72529e-09 1.30385e-07 -9.65935e-09 3.38508e-08
skintemp               /         3 -9.15527e-05 9.15527e-05 -3.05176e-05 -3.05176e-05           0 -3.05176e-05           0
snow                   /         2 -4.05312e-06 4.05312e-06 -2.14577e-06 -1.90735e-06 2.38419e-07 -2.02656e-06 1.68587e-07
snowh                  /         2 -2.14204e-08 2.14204e-08 -1.11759e-08 -1.02445e-08 9.31323e-10 -1.07102e-08 6.58545e-10

after one timestep:

Variable Group Count        Sum     AbsSum         Min        Max       Range        Mean      StdDev
q2       /         5 0.00390335 0.00390335 0.000355857 0.00104086 0.000685005 0.000780671 0.000263849
t2m      /         5    11.7124    11.7124    0.702576    3.71896     3.01639     2.34247     1.22392
th2m     /         5    11.7211    11.7211    0.704285    3.72086     3.01657     2.34422     1.22515
@barlage
Copy link
Collaborator Author

barlage commented Nov 7, 2024

There are 5 seaice points in the domain, so exploring the diffs with seaice.

@barlage
Copy link
Collaborator Author

barlage commented Nov 7, 2024

Does anyone know why we are deallocating these four variables?

Compared to the NCAR code?

Seems like that would mean that we aren't carrying the sfclayer calculated values to the downstream calls. Is that intentional? It's causing my current test code to crash (I believe).

@tanyasmirnova
Copy link
Collaborator

Does anyone know why we are deallocating these four variables?
These are the variables, correct:
if(allocated(chs_sea) ) deallocate(chs_sea )
if(allocated(chs2_sea) ) deallocate(chs2_sea )
if(allocated(cqs2_sea) ) deallocate(cqs2_sea )
if(allocated(cqs_sea) ) deallocate(cqs_sea )

Compared to the NCAR code?

I think it is a bug. These variables are not sent to MPAS in sfclayer_to_MPAS, thus they will be not defined when they are needed in mpas_atmphys_driver_seaice.F90

Seems like that would mean that we aren't carrying the sfclayer calculated values to the downstream calls. Is that intentional? It's causing my current test code to crash (I believe).

@barlage
Copy link
Collaborator Author

barlage commented Nov 8, 2024

@tanyasmirnova yes, those are the four that stand out, I haven't looked too closely to see if there are others. I will make some changes to not deallocate these. This may or may not change the hrrrv5 baseline.

@joeolson42
Copy link
Collaborator

joeolson42 commented Nov 8, 2024 via email

@barlage
Copy link
Collaborator Author

barlage commented Nov 8, 2024

Although it has been a bit tedious, it has generally been a positive thing for me to diagnose these things since I think I understand the code a little better each time. I couldn't understand how some of these variables are being used across schemes. Now I see that pretty much all this memory passing is done through mpas_atmphys_vars. I am going to look at the entire sfclayer driver to try and reduce changes with ncar code. I'm encouraged that this is the issue with reproducibility. I'll post the branch here shortly for you all to see.

@tanyasmirnova
Copy link
Collaborator

@joeolson42 Joe, I think that it is confusing that some variables are sent back to MPAS state, and others are not. When Laura moved the 2-m diagnostics to seaice driver, I think it would be cleaner to get all variables used in there from MPAS grid. I guess she had some reasoning behind not doing it for these four *_sea variables, but I have no idea what it was.

@barlage
Copy link
Collaborator Author

barlage commented Nov 8, 2024

I think I understand the logic. Any vars that you want to persist forever are sent to the MPAS state variables. But any vars you only want to persist for the physics timestep are put into these _p and _sea variables. For these phys timestep vars that you only want to persist for your scheme's call, they are allocated before the scheme and deallocated after. For those that are going to be used downstream (but still in the physics timestep), those should not be deallocated after the scheme, but deallocated after the last scheme that uses it. That's why several of these surface layer variables are not deallocated after the sfclayer driver, but deallocated either after the lsm driver or seaice driver. Overall, I think the purpose is to optimize memory management.

@joeolson42
Copy link
Collaborator

joeolson42 commented Nov 8, 2024 via email

@barlage
Copy link
Collaborator Author

barlage commented Nov 8, 2024

It certainly makes the current deallocation process messy since it's not clear what has been deallocated after a scheme driver call and what hasn't. It seems like it would be better to check/deallocate everything in mpas_atmphys_vars at the end of the physics driver calls but I guess that would have a larger memory footprint.

But what you're saying definitely makes sense for the vars that have both state and _p counterparts, otherwise what does the state contain??

@joeolson42
Copy link
Collaborator

joeolson42 commented Nov 8, 2024 via email

@barlage
Copy link
Collaborator Author

barlage commented Nov 8, 2024

It would be nice to not need the vector to 2D translation. That could be taken care of pretty easily at the scheme driver interface level since most of the physics are column. There are a few *_p examples in the noah scheme that don't have state counterparts but I don't see very many elsewhere in a quick check. That's not to say that they shouldn't be there, i.e., maybe those outliers are bugs.

@barlage
Copy link
Collaborator Author

barlage commented Nov 8, 2024

One step closer in the reproducibility. After a little reorg, the model in one test reproduces for four timesteps. In another with double the radiation frequency (test to see if it was something in the radiation), the model goes five timesteps. The results look like this:

nccmp -dsSqf ncar-v8.2.2.mesoscale_reference.ncar.ncar.conus.120km.gfs.2023031015/history.2023-03-10_15.48.00.nc bar-test.mesoscale_reference.ncar.ncar.conus.120km.gfs.2023031015/history.2023-03-10_15.48.00.nc
Variable               Group Count          Sum      AbsSum          Min          Max       Range         Mean      StdDev
qv                     /      1314 -2.71968e-08 6.60205e-07 -6.13509e-08  4.47035e-08 1.06054e-07 -2.06978e-11 3.17115e-09
qc                     /       134 -2.80018e-08  9.2926e-08 -1.56688e-08  1.12632e-08  2.6932e-08 -2.08968e-10  2.1607e-09
qr                     /       267 -2.08312e-09 3.17143e-08 -2.13913e-09  1.06229e-09 3.20142e-09 -7.80195e-12 3.23282e-10
qi                     /      1251  -6.0394e-07 1.24416e-06 -3.78408e-07  1.05858e-07 4.84266e-07 -4.82766e-10 1.41346e-08
qs                     /      1128  6.78719e-07  8.9257e-07 -4.62774e-08  4.03948e-07 4.50225e-07  6.01701e-10 1.39975e-08
qg                     /       929  3.49938e-10 8.86816e-09 -8.47649e-10  7.47605e-10 1.59525e-09  3.76683e-13 6.21748e-11
u                      /     12195  0.000154282  0.00756561 -7.62939e-06  7.62939e-06 1.52588e-05  1.26513e-08 9.92455e-07
w                      /     16136  8.52161e-05  0.00065999 -1.08033e-06  9.12696e-07 1.99303e-06  5.28112e-09 9.02241e-08
pressure               /       233   -0.0178223     2.03052      -0.0625    0.0703125    0.132812 -7.64904e-05   0.0134642
surface_pressure       /         3    0.0078125   0.0390625    -0.015625     0.015625     0.03125   0.00260417    0.016263
rho                    /       295  1.11759e-08 1.62348e-05 -1.19209e-07  1.19209e-07 2.38419e-07  3.78843e-11 6.91684e-08
theta                  /       253  0.000549316   0.0124512 -0.000152588  0.000213623 0.000366211  2.17121e-06 5.72457e-05
relhum                 /      1335   -0.0100263    0.148328  -0.00932312   0.00852203   0.0178452 -7.51036e-06 0.000577774
divergence             /      6661  1.66214e-10 6.74001e-08 -1.38243e-10  1.41881e-10 2.80124e-10  2.49533e-14 1.54325e-11
vorticity              /      9961  9.13281e-10 1.84855e-07 -2.40107e-10  2.00089e-10 4.40195e-10  9.16856e-14 2.82691e-11
ke                     /      3677   0.00320208   0.0509082 -0.000366211  0.000244141 0.000610352  8.70841e-07 3.37185e-05
uReconstructZonal      /      3338  4.40013e-05  0.00265398 -1.14441e-05  7.62939e-06 1.90735e-05  1.31819e-08 1.29065e-06
uReconstructMeridional /      4265  4.54253e-06   0.0020807  -3.8147e-06   3.8147e-06 7.62939e-06  1.06507e-09 6.57777e-07
ertel_pv               /      5118  0.000286166   0.0235342  -0.00114441  0.000808716  0.00195312  5.59136e-08 2.79923e-05
u_pv                   /        79  5.00679e-05 0.000351429 -4.00543e-05  4.19617e-05  8.2016e-05  6.33771e-07 8.34003e-06
v_pv                   /       102 -2.83495e-05 0.000128768 -1.04904e-05   3.8147e-06 1.43051e-05 -2.77936e-07 2.03743e-06
theta_pv               /        11            0 0.000427246 -6.10352e-05  6.10352e-05  0.00012207            0 4.31584e-05
vort_pv                /       142 -3.22871e-11 1.89948e-09 -1.74623e-10  1.18234e-10 2.92857e-10 -2.27374e-13  2.7727e-11
depv_dt_lw             /      4856   2.6657e-10 3.58379e-09 -5.82077e-11  5.82077e-11 1.16415e-10   5.4895e-14 2.85001e-12
depv_dt_sw             /      4973 -1.80546e-11 3.10747e-09 -1.45519e-11  2.91038e-11 4.36557e-11 -3.63052e-15 1.95559e-12
depv_dt_bl             /      5195 -4.11089e-10 5.39206e-09 -1.16415e-10  5.82077e-11 1.74623e-10 -7.91317e-14 4.40537e-12
depv_dt_cu             /      2359  1.11501e-11 3.97929e-09 -1.16415e-10  5.82077e-11 1.74623e-10   4.7266e-15 6.77304e-12
depv_dt_mix            /      8142  5.48619e-11 4.64453e-08 -5.47061e-10  4.87034e-10  1.0341e-09  6.73814e-15 2.33659e-11
dtheta_dt_mp           /       114  4.66276e-07 6.56979e-06 -2.11927e-07  2.54313e-07  4.6624e-07  4.09014e-09 6.88244e-08
depv_dt_mp             /      5980  4.85552e-07 8.43867e-06 -5.30691e-07  5.26375e-07 1.05707e-06   8.1196e-11 1.33158e-08
depv_dt_diab           /      7234  4.85189e-07 8.48497e-06 -5.30679e-07  5.26401e-07 1.05708e-06  6.70706e-11  1.2107e-08
depv_dt_fric           /      8898  4.22833e-09 7.03416e-07 -1.25001e-08  7.45786e-09  1.9958e-08    4.752e-13 3.27245e-10
depv_dt_diab_pv        /       179  2.65976e-08 4.32941e-08 -5.76813e-09  1.42791e-08 2.00472e-08   1.4859e-10 1.32803e-09
depv_dt_fric_pv        /       208  1.31301e-10 3.85081e-09 -1.50067e-10   2.5193e-10 4.01997e-10  6.31256e-13 3.79763e-11
rainnc                 /        46 -9.52277e-08 2.34257e-06 -4.84288e-07  7.15256e-07 1.19954e-06 -2.07017e-09 1.47191e-07
precipw                /        39 -2.38419e-06 9.48906e-05 -1.14441e-05  1.14441e-05 2.28882e-05  -6.1133e-08 3.79296e-06
hfx                    /         1 -0.000114441 0.000114441 -0.000114441 -0.000114441           0 -0.000114441           0
qfx                    /         1  1.27329e-11 1.27329e-11  1.27329e-11  1.27329e-11           0  1.27329e-11           0
lh                     /         1  3.43323e-05 3.43323e-05  3.43323e-05  3.43323e-05           0  3.43323e-05           0
q2                     /         1 -2.32831e-10 2.32831e-10 -2.32831e-10 -2.32831e-10           0 -2.32831e-10           0
t2m                    /         1 -3.05176e-05 3.05176e-05 -3.05176e-05 -3.05176e-05           0 -3.05176e-05           0
th2m                   /         1 -3.05176e-05 3.05176e-05 -3.05176e-05 -3.05176e-05           0 -3.05176e-05           0
skintemp               /         1 -3.05176e-05 3.05176e-05 -3.05176e-05 -3.05176e-05           0 -3.05176e-05           0

The positive thing is that there is one grid in each test that has different surface fields like qfx/lh. They are not the same grid but both grids are vegtyp=15 so permanent snow/ice. That is suspicious so hopefully could be easy to track down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants