Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

global_4denvar ctest seg faults in debug mode #777

Closed
RussTreadon-NOAA opened this issue Jul 28, 2024 · 7 comments · Fixed by #779
Closed

global_4denvar ctest seg faults in debug mode #777

RussTreadon-NOAA opened this issue Jul 28, 2024 · 7 comments · Fixed by #779
Assignees
@azadeh-gh
Milestone
GFS v17 GSI/EnKF ...

Comments

@RussTreadon-NOAA
Copy link
Contributor

As reported in PR #743, the debug mode gsi.x aborts in ctest global_4denvar with the error

GLBSOI: jiter,jiterstart,jiterlast,jiterend=    1    1    2    1
forrtl: error (73): floating divide by zero
Image              PC                Routine            Line        Source
gsi.x              0000000007F2D6AB  Unknown               Unknown  Unknown
libpthread-2.31.s  00001507B280A8C0  Unknown               Unknown  Unknown
libimf.so          00001507B2910AAF  __libm_log_l9         Unknown  Unknown
gsi.x              00000000008853DC  crtm_interface_mp        2773  crtm_interface.f90
gsi.x              000000000078BEBD  crtm_interface_mp        1881  crtm_interface.f90
gsi.x              000000000560E4B1  rad_setup_mp_setu         919  setuprad.f90```

Line 1881 of crtm_interface..f90 is

        call calc_thompson_reff(rho_air,h,cloud(:,ii),cloud_names(iii),cloudefr(:,ii))

Line 2773 is

           lam_i=exp(1.0_r_kind / 3.0_r_kind * log((am_i*ni(k) *gamma(mu_i + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_i+1.0_r_kind))))

NCO builds and runs codes in debug mode as part of their pre-implementation testing. As such the debug mode gsi.x failure must be examined, understood, and resolved. This issue is opened to document this error and work toward resolution of the problem.

Attention: @azadeh-gh , @emilyhcliu , @CatherineThomas-NOAA
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Jul 28, 2024
Merged
6 tasks
@RussTreadon-NOAA
Copy link
Contributor Author

@wx20jjung found the following

There is a write statement in crtm_interface.f90, about line 1224

if (mype==0) write(6,*) myname_, ' imp_physics = ', imp_physics

that is generating > 1000 lines of output.
0: crtm_interface*call_crtm imp_physics = 8

If this is by design, OK. Otherwise, I suggest removing it.

@emilyhcliu
Copy link
Contributor

@wx20jjung found the following

There is a write statement in crtm_interface.f90, about line 1224
if (mype==0) write(6,) myname_, ' imp_physics = ', imp_physics
that is generating > 1000 lines of output.
0: crtm_interfacecall_crtm imp_physics = 8
If this is by design, OK. Otherwise, I suggest removing it.

That's for debugging purposes and should be removed.

@RussTreadon-NOAA
Copy link
Contributor Author

Orion debug test
Build develop at e82365d on Orion in debug mode. Modify regression/global_4denvar.sh so that dump file 1bamua (AMSU-A) is the only observation type assimilated. Run global_4denvar ctest. Test fails with traceback

==== backtrace (tid: 183207) ====
 0 0x000000000005f14c ucs_callbackq_cleanup()  ???:0
 1 0x000000000005f40a ucs_callbackq_cleanup()  ???:0
 2 0x0000000000054d90 __GI___sigaction()  :0
 3 0x0000000007c48809 __libm_log_l9()  ???:0
 4 0x000000000086bcfa crtm_interface_mp_calc_thompson_reff_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/crtm_interface.f90:2773
 5 0x00000000007762ea crtm_interface_mp_call_crtm_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/crtm_interface.f90:1881
 6 0x000000000554212f rad_setup_mp_setuprad_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/setuprad.f90:919
 7 0x0000000003f66ae3 gsi_radoper_mp_setup__()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsi_radOper.F90:100
 8 0x000000000260a571 setuprhsall_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/setuprhsall.f90:492
 9 0x0000000003ec8819 glbsoi_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/glbsoi.f90:323
10 0x000000000107502c gsisub_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsisub.F90:200
11 0x000000000042f053 gsimod_mp_gsimain_run_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsimod.F90:2431
12 0x0000000000416453 MAIN__()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsimain.f90:633
13 0x00000000004162bd main()  ???:0
14 0x000000000003feb0 __libc_start_call_main()  ???:0
15 0x000000000003ff60 __libc_start_main_alias_2()  :0
16 0x00000000004161d5 _start()  ???:0
srun: error: orion-17-49: task 50: Aborted (core dumped)
srun: Terminating StepId=18441484.0
slurmstepd: error: *** STEP 18441484.0 ON orion-17-45 CANCELLED AT 2024-08-07T04:57:17 ***

Line 2788 of src/gsi/crtm_interface.f90 is the lam_r line below

  !Rain
  else if (trim(cloud_name)=='qr') then
     am_r = rho_r*pi/6.0_r_kind
     reff_min = reff_r_min
     reff_max = reff_r_max
     do k = 1, nsig
        qx = qxmr(k) * rho_air(k)  ! convert mixing ratio (kg/kg) to water content (kg/m3)
        if (qx > qmin) then
           lam_r=exp(1.0_r_kind / 3.0_r_kind * log ((am_r*nr(k) *gamma(mu_r + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_r + 1.0_r_kind))))
           reff(k) = 0.5_r_kind *(3.0_r_kind/lam_r)*1.0e6_r_kind
           reff(k) = max(reff_min, min(reff_max, reff(k)))
        else
           reff(k) = zero
        endif
     enddo

Add print statements to src/gsi/crtm_interface.f90. Recompile in debug mode and rerun global_4denvar. The argument inside log( ) on the lam_r line, is zero for k=16 of a specific column. Prints for the variables in log( ) show that the argument is zero because nr(16) is zero.

Below are values for this profile as a function of vertical level`

    50: qr k=  1 am_r=  523.599     qxmr= 0.582161E-15 rho_air=  1.13050     nr= 0.140758E-05 qx= 0.658135E-15 qmin= 0.100000E-11
    50: qr k=  2 am_r=  523.599     qxmr= 0.518425E-15 rho_air=  1.12877     nr= 0.144649E-05 qx= 0.585182E-15 qmin= 0.100000E-11
    50: qr k=  3 am_r=  523.599     qxmr= 0.494004E-15 rho_air=  1.12672     nr= 0.188714E-05 qx= 0.556602E-15 qmin= 0.100000E-11
    50: qr k=  4 am_r=  523.599     qxmr= 0.481493E-15 rho_air=  1.12454     nr= 0.250026E-05 qx= 0.541458E-15 qmin= 0.100000E-11
    50: qr k=  5 am_r=  523.599     qxmr= 0.441871E-15 rho_air=  1.12214     nr= 0.338398E-05 qx= 0.495840E-15 qmin= 0.100000E-11
    50: qr k=  6 am_r=  523.599     qxmr= 0.461052E-28 rho_air=  1.11962     nr= 0.909372E-19 qx= 0.516204E-28 qmin= 0.100000E-11
    50: qr k=  7 am_r=  523.599     qxmr= 0.461030E-28 rho_air=  1.11690     nr= 0.135157E-18 qx= 0.514924E-28 qmin= 0.100000E-11
    50: qr k=  8 am_r=  523.599     qxmr= 0.457989E-28 rho_air=  1.11402     nr= 0.850316E-19 qx= 0.510209E-28 qmin= 0.100000E-11
    50: qr k=  9 am_r=  523.599     qxmr= 0.452205E-28 rho_air=  1.11099     nr= 0.328184E-19 qx= 0.502396E-28 qmin= 0.100000E-11
    50: qr k= 10 am_r=  523.599     qxmr= 0.444311E-28 rho_air=  1.10772     nr= 0.552185E-19 qx= 0.492170E-28 qmin= 0.100000E-11
    50: qr k= 11 am_r=  523.599     qxmr= 0.435068E-28 rho_air=  1.10431     nr= 0.554026E-19 qx= 0.480451E-28 qmin= 0.100000E-11
    50: qr k= 12 am_r=  523.599     qxmr= 0.425320E-28 rho_air=  1.10063     nr= 0.555861E-19 qx= 0.468120E-28 qmin= 0.100000E-11
    50: qr k= 13 am_r=  523.599     qxmr= 0.423111E-28 rho_air=  1.09671     nr= 0.352656E-19 qx= 0.464032E-28 qmin= 0.100000E-11
    50: qr k= 14 am_r=  523.599     qxmr= 0.246025E-12 rho_air=  1.09262     nr= 0.354037E-19 qx= 0.268812E-12 qmin= 0.100000E-11
    50: qr k= 15 am_r=  523.599     qxmr= 0.556802E-12 rho_air=  1.08826     nr=  0.00000     qx= 0.605943E-12 qmin= 0.100000E-11
    50: qr k= 16 am_r=  523.599     qxmr= 0.105476E-11 rho_air=  1.08366     nr=  0.00000     qx= 0.114301E-11 qmin= 0.100000E-11

nr(15) is zero but the qx value on k=15 is less than qmin so exectution does not enter the lam_r block. This changes at k=16. Here the qx value slightly exceeds qmin. nr(16)=0.00000 results in an argument of zero for log( ). This is an invalid floating point operation. Execution terminates.

As a test check the value of nr(k) before entering the lam_r block. Also check ni(k) before entering the lam_i block.

@@ -2769,7 +2769,7 @@ subroutine call_crtm(obstype,obstime,data_s,nchanl,nreal,ich, &
      reff_max = reff_i_max
      do k = 1, nsig
         qx = qxmr(k) * rho_air(k)  ! convert mixing ratio (kg/kg) to water content (kg/m3)
-        if (qx > qmin) then
+        if (qx > qmin .and. ni(k)>zero) then
            lam_i=exp(1.0_r_kind / 3.0_r_kind * log((am_i*ni(k) *gamma(mu_i + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_i+1.0_r_kind))))
         reff(k) = 0.5_r_kind * (3.0_r_kind /lam_i)*1.0e6_r_kind
         reff(k) = max(reff_min, min(reff_max, reff(k)))
@@ -2784,7 +2784,7 @@ subroutine call_crtm(obstype,obstime,data_s,nchanl,nreal,ich, &
      reff_max = reff_r_max
      do k = 1, nsig
         qx = qxmr(k) * rho_air(k)  ! convert mixing ratio (kg/kg) to water content (kg/m3)
-        if (qx > qmin) then
+        if (qx > qmin .and. nr(k)>zero) then
            lam_r=exp(1.0_r_kind / 3.0_r_kind * log ((am_r*nr(k) *gamma(mu_r + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_r + 1.0_r_kind))))
            reff(k) = 0.5_r_kind *(3.0_r_kind/lam_r)*1.0e6_r_kind
            reff(k) = max(reff_min, min(reff_max, reff(k)))

Rebuild in debug mode and rerun global_4denvar. Got past the above segmentation fault but encountered a divide by zero on numerous tasks with the following traceback

[orion-17-47:144027:0:144027] Caught signal 8 (Floating point exception: floating-point divide by zero)
==== backtrace (tid: 183435) ====
 0 0x000000000005f14c ucs_callbackq_cleanup()  ???:0
 1 0x000000000005f40a ucs_callbackq_cleanup()  ???:0
 2 0x0000000000054d90 __GI___sigaction()  :0
 3 0x00000000079c2043 ada_module_mp_crtm_amom_layer_()  /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-crtm-2.4.0.1-jsgqbo2wgepyclizmnmzjfwhchq3qlng/spack-src/libsrc/ADA_Module.f90:295
 4 0x00000000079bd347 ada_module_mp_crtm_ada_()  /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-crtm-2.4.0.1-jsgqbo2wgepyclizmnmzjfwhchq3qlng/spack-src/libsrc/ADA_Module.f90:127
 5 0x0000000007866148 crtm_rtsolution_mp_crtm_compute_rtsolution_()  /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-crtm-2.4.0.1-jsgqbo2wgepyclizmnmzjfwhchq3qlng/spack-src/libsrc/CRTM_RTSolution.f90:253
 6 0x000000000784fcbe crtm_k_matrix_modulecrtm_k_matrix_mp_profile_solution_()  /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-crtm-2.4.0.1-jsgqbo2wgepyclizmnmzjfwhchq3qlng/spack-src/libsrc/CRTM_K_Matrix_Module.f90:1065
 7 0x000000000784b062 crtm_k_matrix_module_mp_crtm_k_matrix_()  /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-crtm-2.4.0.1-jsgqbo2wgepyclizmnmzjfwhchq3qlng/spack-src/libsrc/CRTM_K_Matrix_Module.f90:452
 8 0x000000000015ff13 __kmp_invoke_microtask()  ???:0
 9 0x00000000000cf664 _INTERNALdc23a976::__kmp_serial_fork_call()  /nfs/site/proj/openmp/promo/20230118/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-fxilab153/../../src/kmp_runtime.cpp:1927
10 0x00000000000cf664 __kmp_fork_call()  /nfs/site/proj/openmp/promo/20230118/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-fxilab153/../../src/kmp_runtime.cpp:2188
11 0x0000000000088d23 __kmpc_fork_call()  /nfs/site/proj/openmp/promo/20230118/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-fxilab153/../../src/kmp_csupport.cpp:350
12 0x000000000784a7bd crtm_k_matrix_module_mp_crtm_k_matrix_()  /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-crtm-2.4.0.1-jsgqbo2wgepyclizmnmzjfwhchq3qlng/spack-src/libsrc/CRTM_K_Matrix_Module.f90:442
13 0x00000000007b1e7b crtm_interface_mp_call_crtm_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/crtm_interface.f90:2229
14 0x000000000554263f rad_setup_mp_setuprad_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/setuprad.f90:919
15 0x0000000003f66ff3 gsi_radoper_mp_setup__()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsi_radOper.F90:100
16 0x000000000260aa85 setuprhsall_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/setuprhsall.f90:492
17 0x0000000003ec8d29 glbsoi_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/glbsoi.f90:323
18 0x0000000001075540 gsisub_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsisub.F90:200
19 0x000000000042f053 gsimod_mp_gsimain_run_()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsimod.F90:2431
20 0x0000000000416453 MAIN__()  /work2/noaa/da/rtreadon/git/gsi/debug/src/gsi/gsimain.f90:633
21 0x00000000004162bd main()  ???:0
22 0x000000000003feb0 __libc_start_call_main()  ???:0
23 0x000000000003ff60 __libc_start_main_alias_2()  :0
24 0x00000000004161d5 _start()  ???:0
=================================
forrtl: error (75): floating point exception
Image              PC                Routine            Line        Source
libc.so.6          000014D818F1ED90  Unknown               Unknown  Unknown
gsi.x              00000000079C2043  ada_module_mp_crt         295  ADA_Module.f90
gsi.x              00000000079BD347  ada_module_mp_crt         127  ADA_Module.f90
gsi.x              0000000007866148  crtm_rtsolution_m         253  CRTM_RTSolution.f90
gsi.x              000000000784FCBE  crtm_k_matrix_mod        1065  CRTM_K_Matrix_Module.f90
gsi.x              000000000784B062  crtm_k_matrix_mod         452  CRTM_K_Matrix_Module.f90
libiomp5.so        000014D81BCBCF13  __kmp_invoke_micr     Unknown  Unknown
libiomp5.so        000014D81BC2C664  __kmp_fork_call       Unknown  Unknown
libiomp5.so        000014D81BBE5D23  __kmpc_fork_call      Unknown  Unknown
gsi.x              000000000784A7BD  crtm_k_matrix_mod         442  CRTM_K_Matrix_Module.f90
gsi.x              00000000007B1E7B  crtm_interface_mp        2229  crtm_interface.f90
gsi.x              000000000554263F  rad_setup_mp_setu         919  setuprad.f90
gsi.x              0000000003F66FF3  gsi_radoper_mp_se         100  gsi_radOper.F90
gsi.x              000000000260AA85  setuprhsall_              492  setuprhsall.f90
gsi.x              0000000003EC8D29  glbsoi_                   323  glbsoi.f90
gsi.x              0000000001075540  gsisub_                   200  gsisub.F90
gsi.x              000000000042F053  gsimod_mp_gsimain        2431  gsimod.F90
gsi.x              0000000000416453  MAIN__                    633  gsimain.f90
gsi.x              00000000004162BD  Unknown               Unknown  Unknown
libc.so.6          000014D818F09EB0  Unknown               Unknown  Unknown
libc.so.6          000014D818F09F60  __libc_start_main     Unknown  Unknown
gsi.x              00000000004161D5  Unknown               Unknown  Unknown
forrtl: error (75): floating point exception

The traceback indicates that the divide by zero occurs in the crtm library. I recall mention of a divide by zero issue in previous versions of the crtm. Does this error still remain in crtm/2.4.0.1?

Will pause here and let @azadeh-gh and @emilyhcliu investigate.

@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Aug 7, 2024
Closed
@RussTreadon-NOAA
Copy link
Contributor Author

Hera test
Build debug gsi.x and run 2023060712 case using backgrounds from /scratch2/NCEPDEV/stmp3/Emily.Liu/ROTDIRS/v17allskyens/.

Emily's parallel is a C384L127 / C192L127 using Thompson microphysics. When gmi is present this case seg faults in debug mode as documented in issue #778. Rerun with gmi data removed. The debug gsi.x aborts in this case is it did when running the global_4denvar ctest.

128: ==== backtrace (tid: 172442) ====
128:  0 0x00000000000534e9 ucs_debug_print_backtrace()  ???:0
128:  1 0x0000000000012cf0 __funlockfile()  :0
128:  2 0x0000000007c6d27f __libm_log_l9()  ???:0
128:  3 0x000000000087f10c crtm_interface_mp_calc_thompson_reff_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/crtm_interface.f90:2773
128:  4 0x0000000000788248 crtm_interface_mp_call_crtm_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/crtm_interface.f90:1881
128:  5 0x0000000005598d90 rad_setup_mp_setuprad_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/setuprad.f90:919
128:  6 0x0000000003fb0c58 gsi_radoper_mp_setup__()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/gsi_radOper.F90:100
128:  7 0x000000000263c3c5 setuprhsall_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/setuprhsall.f90:492
128:  8 0x0000000003f117bf glbsoi_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/glbsoi.f90:323
128:  9 0x00000000010989fc gsisub_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/gsisub.F90:200
128: 10 0x000000000042dbd8 gsimod_mp_gsimain_run_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/gsimod.F90:2431
128: 11 0x0000000000414c4b MAIN__()  /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/gsimain.f90:633
128: 12 0x0000000000414aa2 main()  ???:0
128: 13 0x000000000003ad85 __libc_start_main()  ???:0
128: 14 0x00000000004149ae _start()  ???:0

Line 2773 of crtm_interface is the lam_i line mentioned above. The seg fault is likely due to ni(k) being zero resulting in log(0), an invalid floating point operation.

@RussTreadon-NOAA
Copy link
Contributor Author

Hera test (continued)
Add checks to ensure ni and nr are greater than zero in src/gsi/crtm_interface.f90 as shown below

@@ -2769,7 +2769,7 @@ subroutine call_crtm(obstype,obstime,data_s,nchanl,nreal,ich, &
      reff_max = reff_i_max
      do k = 1, nsig
         qx = qxmr(k) * rho_air(k)  ! convert mixing ratio (kg/kg) to water content (kg/m3)
-        if (qx > qmin) then
+        if (qx > qmin .and. ni(k) > zero) then
            lam_i=exp(1.0_r_kind / 3.0_r_kind * log((am_i*ni(k) *gamma(mu_i + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_i+1.0_r_kind))))
         reff(k) = 0.5_r_kind * (3.0_r_kind /lam_i)*1.0e6_r_kind
         reff(k) = max(reff_min, min(reff_max, reff(k)))
@@ -2784,7 +2784,7 @@ subroutine call_crtm(obstype,obstime,data_s,nchanl,nreal,ich, &
      reff_max = reff_r_max
      do k = 1, nsig
         qx = qxmr(k) * rho_air(k)  ! convert mixing ratio (kg/kg) to water content (kg/m3)
-        if (qx > qmin) then
+        if (qx > qmin .and. nr(k) > zero) then
            lam_r=exp(1.0_r_kind / 3.0_r_kind * log ((am_r*nr(k) *gamma(mu_r + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_r + 1.0_r_kind))))
            reff(k) = 0.5_r_kind *(3.0_r_kind/lam_r)*1.0e6_r_kind
            reff(k) = max(reff_min, min(reff_max, reff(k)))

Rebuild debug gsi.x. Both global_4denvar ctest and test using Emily's input successfully run past the point of the previous failure. Neither test finished the analysis due to using the debug queue with a 30 minute wall clock limit. A test on Cactus using a longer wall time limit allowed the debug gsi.x with the above change to successfully complete the global_4denvar ctest.

@azadeh-gh and @emilyhcliu : If the above change is acceptable, we can create a branch, make the change, and open a PR. The PR can also remove line 1209 of crtm_interface.f90. @wx20jjung noted that this line generates a lot of runtime output.

@emilyhcliu
Copy link
Contributor

@RussTreadon-NOAA Thanks so much for fixing this for us.
It is good that you checked both rain and ice (nr and ni) and only calculated their reff when ni and nr are not zero.

@azadeh-gh will run a single cycle test to double-check how much the safeguard added would change the results
The expectation is that the change should be small.

@RussTreadon-NOAA
Copy link
Contributor Author

Work for this issue will be done in RussTreadon-NOAA:feature/thompson_reff

RussTreadon-NOAA added a commit to RussTreadon-NOAA/GSI that referenced this issue Aug 9, 2024
@RussTreadon-NOAA
remove debug print, check ni and nr prior to calculation (NOAA-EMC#777)
408917e
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Aug 9, 2024
Merged
4 tasks
RussTreadon-NOAA added a commit to RussTreadon-NOAA/GSI that referenced this issue Aug 9, 2024
@RussTreadon-NOAA
use 1e-6 as lower threshold for ni and nr (NOAA-EMC#777)
9a3a90d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@azadeh-gh azadeh-gh

Labels
None yet
Projects
None yet
Milestone
GFS v17 GSI/EnKF development
Development

Successfully merging a pull request may close this issue.

add safeguard to thompson_reff RussTreadon-NOAA/GSI
3 participants
@RussTreadon-NOAA @emilyhcliu @azadeh-gh