Model crashes at specific fcst time

[benjamin-cash]

I am running a 6-month, 10-member C192mx025 ensemble, and so far 4 of the 10 members have reached a specfic n_atmsteps value and crashed with the error below (slight variations between runs).

zeroing coupling accumulated fields at kdt=        12527
 zeroing coupling accumulated fields at kdt=        12527
PASS: fcstRUN phase 2, n_atmsteps =            12526 time is         0.326497
  (zap_snow_temperature)zap_snow_temperature: temperature out of bounds!
  (zap_snow_temperature)k:           1
  (zap_snow_temperature)zTsn:  4.636445953738447E-003
  (zap_snow_temperature)Tmin:  -100.000000000000
  (zap_snow_temperature)Tmax:  9.797644457066566E-006
  (zap_snow_temperature)zqsn:  -110216777.762791
 ncells=           5
 nlives=          12
 nthresh=   18.0000000000000

Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2263: comm->shm_numa_layout[my_numa_node].base_addr
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(MPL_backtrace_show+0x24) [0x2b50027c0bf4]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2b5002367ba1]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(+0x368e7e) [0x2b5002168e7e]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(+0x262d66) [0x2b5002062d66]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(+0x2ae65c) [0x2b50020ae65c]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(+0x26a45a) [0x2b500206a45a]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(+0x25f7c9) [0x2b500205f7c9]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(+0x379451) [0x2b5002179451]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/intel-oneapi-mpi-2021.12.1-pvycn2u/mpi/2021.12/lib/libmpi.so.12(MPI_Bcast+0x22c) [0x2b5001f53d8c]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/parallelio-2.6.2-d7wj357/lib/libpioc.so(PIOc_createfile_int+0x336) [0x2b5000bfb186]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/parallelio-2.6.2-d7wj357/lib/libpioc.so(PIOc_createfile+0x41) [0x2b5000bf70c1]
/opt/spack-stack/spack-stack-1.8.0/envs/unified-env/install/intel/2021.10.0/parallelio-2.6.2-d7wj357/lib/libpiof.so(piolib_mod_mp_createfile_+0x1ae) [0x2b5000c5642e]

Looking at the history files, in each case the model was part-way through writing out gefs.ocean.t00z.24hr_avg.f2064.nc when it crashed. All of the crashes were at different times, so it wasn't something simple like a disk issue that caused a temporary error in writing files. Is there some kind of internal limit that I've encountered?

[aerorahul]

@benjamin-cash Thanks for starting this discussion. It seems to me this is likely a model issue. Would you mind transferring this discussion to the appropriate repository?

[benjamin-cash]

That is very good to know, and I did submit this as a single long segment. If this can be done via the segment capability I'd much rather address it that way than get into altering the way CICE is writing out files.

For running in segments does the model run for (say) a month, exit, and then restart and resubmit?

[aerorahul]

Yes. The model restarts from the end of the segment in a new job. The segments run as sequential jobs and are named as gfs_fcst_seg0, gfs_fcst_seg1, etc.

[benjamin-cash]

Denise and Larissa think that I should have still had plenty of files left to go before I hit the limit, so the jury is still out on what caused this. It does look like trying to create gefs.ice.t00z.24hr_avg.f2088.nc was the last thing it attempted before crashing though.

[aerorahul]

Ok. Then I am all out of reasons. If I come across anything, I will post it here.

[benjamin-cash]

If I get a minute I will still set up a run with breakpoints that are shorter than the point at which it failed, just to be sure.