During run = SIGSEGV, segmentation fault occurred

[pgoddard08]

Hello,

I don't believe you support use on non-supported machines. But, I've been trying to run E3SMv2.1 and E3SMv3 on my local machine at Indiana University and cannot get passed the below error. I think it has to do with my mpi library or perhaps some issues with the modules I have loaded - though I'm just guessing. If anyone can help, it would be greatly appreciated.

To describe my problem, I had successful runs of I20TRELM using 1, 4, 8 nodes (128 procs/node) but when I use 16 nodes, I get a similar error message to the one below. I then tried to run Tutorial 2024 Day 1's practicum, but when it came to running the script, run.v3.LR.historical_0101_b2000.pm-cpu.sh, with the custom-2_1x1_ndays, I received errors. First the e3sm error log said "out of memory", so I tried increasing the nodes to custom-4. Then I got the error below which is similar to the error I had previously with I20TRELM.

If anyone wants to be a point of contact, that would be great and I can send you more information about my machine and the config.xml files. Thanks! Paul

271: forrtl: severe (174): SIGSEGV, segmentation fault occurred
271: Image              PC                Routine            Line        Source
271: e3sm.exe           0000000004559D8A  Unknown               Unknown  Unknown
271: libpthread-2.31.s  000014C3238AC8C0  Unknown               Unknown  Unknown
271: libfabric.so.1.14  000014C31C367578  Unknown               Unknown  Unknown
271: libfabric.so.1.14  000014C31C369705  Unknown               Unknown  Unknown
271: libfabric.so.1.14  000014C31C3698D1  Unknown               Unknown  Unknown
271: libfabric.so.1.14  000014C31C31ABCD  Unknown               Unknown  Unknown
271: libfabric.so.1.14  000014C31C319EDE  Unknown               Unknown  Unknown
271: libmpi_intel.so.1  000014C320B3C002  Unknown               Unknown  Unknown
271: libmpi_intel.so.1  000014C3205F4F39  PMPI_Send             Unknown  Unknown
271: e3sm.exe           00000000044C923B  Unknown               Unknown  Unknown
271: e3sm.exe           0000000004470062  Unknown               Unknown  Unknown
271: e3sm.exe           000000000446DBF8  Unknown               Unknown  Unknown
271: e3sm.exe           00000000044685FD  Unknown               Unknown  Unknown
271: e3sm.exe           000000000444EA25  spio_decomp_mp_pi         175  spio_decomp.F90.in
271: e3sm.exe           0000000003EC9A4B  mpas_io_mp_mpas_i        1313  mpas_io.f90
271: e3sm.exe           0000000004005CEB  mpas_io_streams_m        2610  mpas_io_streams.f90
271: e3sm.exe           0000000004006DF2  mpas_io_streams_m         826  mpas_io_streams.f90
271: e3sm.exe           0000000003EF72FA  mpas_stream_manag        4406  mpas_stream_manager.f90
271: e3sm.exe           0000000003EF5B29  mpas_stream_manag        3748  mpas_stream_manager.f90
271: e3sm.exe           0000000003EF41D1  mpas_stream_manag        3546  mpas_stream_manager.f90
271: e3sm.exe           0000000003443680  seaice_core_mp_se         110  mpas_seaice_core.f90
271: e3sm.exe           0000000002DBE662  ice_comp_mct_mp_i         632  ice_comp_mct.f90
271: e3sm.exe           000000000044C5B4  component_mod_mp_         257  component_mod.F90
271: e3sm.exe           00000000004385AA  cime_comp_mod_mp_        1512  cime_comp_mod.F90
271: e3sm.exe           00000000004492F2  MAIN__                    122  cime_driver.F90
271: e3sm.exe           000000000041F512  Unknown               Unknown  Unknown
271: libc-2.31.so       000014C31EC9F2BD  __libc_start_main     Unknown  Unknown
271: e3sm.exe           000000000041F42A  Unknown               Unknown  Unknown

[philipwjones]

From just a quick look at the trace, it's coming from a parallel I/O for the ice component, so I'd start by looking at how you've configured IO tasks. Also, you may be spreading the ice a bit thin for this problem size on that many nodes. Someone else might chime in with other ideas...

[pgoddard08]

Hi Phil,

Thank you very much for your prompt reply. Would you mind sharing how I would go about configuring the IO tasks for the model components?

Also, when I ran run.v3.LR.historical_0101_b2000.pm-cpu.sh, with the custom-2_1x1_ndays (just like the tutorial suggestions), I got this error:
0: PIO: FATAL ERROR: Aborting... An error occured, Opening file (v3.LR.historical_0101.cpl.r.2000-01-01-00000.nc) failed. Invalid iotype (PIO_IOTYPE_PNETCDF:1) specified. Available iotypes are : PIO_IOTYPE_NETCDF (2), PIO_IOTYPE_NETCDF4C (3), PIO_IOTYPE_NETCDF4P (4). Bad IO type (err=-500). Aborting since the error handler was set to PIO_INTERNAL_ERROR... (/N/project/MSB_Project/E3SMv3/externals/scorpio/src/clib/pioc_support.c: 4283)

Seems to also be about parallel I/O -- I'm not sure if it's related to the other error but may be it's a helpful message to you.
Thanks again!

[rljacob]

For IOTYPE_PNETCDF, you need to either install the pnetcdf library or a more recent version of netcdf (NetCDF-C 4.4.0, NetCDF-Fortran 4.5.0 or later).

[mt5555]

if you'd be happy with I/O rates ~100MB/s, you might be able to edit one of the XML files in the case directory and switch to IOTYPE=NETCDF (to avoid having to install pnetcdf)

[pgoddard08]

Thank you all very much for your suggestions. I'll talk with the IT team here and try these options. I'll update afterwards.

[rljacob]

Did this get solved?

[pgoddard08]

Hi Robert,

I did not have an environmental variable "PNETCDF_PATH" set, which I think helped for the PIO error I was getting.
The first issue I brought up, the SIGSEGV error still arises when I use too many nodes or proc/node on our machine at IU. I think this might be an issue I need to resolve with the IT team here. Nonetheless, I was able to run the water cycle compset for 1 year with v2.1 and v3 at a reasonable PES setup, 12 nodes with 32 proc/node which took about 8.5 hours.
Thanks for your help. You may close this. I'll reach out if I have a more specific question regarding the SIGSEGV.