Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ...

[tarunkumar987]

We are hitting following assertion:
Assertion failed in file ...nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ...
I looked at the assert and it looks like this in file socksm.c
MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO ||
hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO);

We have tried multiple cores from 2 to 8 and the behaviour is same. Also process is aborted and a message appear to suggest that. Mostly it's process 0 which is aborted but I have seen other processes as well reporting the crash. We are using mpich-3.2.1. I'm trying to understand possible causes for this error? I have explored the forum and no possible causes, like machine going out of memory etc are applicable here. Please suggest.

[hzhou]

Does it fail for all applications? If not, can you find a small reproducer?

[tarunkumar987]

No it doesn't fail for all the applications. It's the first time we have seen this error in the tool. It currently fails in a customer environment for a big design so there is no easy way to have a small test case. We are currently trying to isolate the issue. Any pointers to what can be tried to isolate?

[hzhou]

Is there other activity going on during the run, for example, a port scan or some other application that is trying to make connection on a seemingly random port? The assertion is part of a initial handshake check and it appears it is receiving an unrecognized packet after the connection.

[tarunkumar987]

Further, does nmap also send bad packets? AFAIK it's a port scanner. We reproduced locally another crash with nmap but that did not appear at customer end. Following is the crash that we saw with nmap locally:
HYD_pmcd_pmiserv_proxy_init_cb (/tmp/mpich//mpich-3.2.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:452): assert (!closed) failed

Customer only observed this crash in netmod as it found illegal header type in the packet.

[tarunkumar987]

We could reproduce this issue locally. Basically we observed two crashes depending on which tcp port is scanned by nmap. In general, we have tcp ports created for mpiexec and our own execs. If nmap is run on tcp port for mpiexec then we see this crash:
HYD_pmcd_pmiserv_proxy_init_cb (/tmp/mpich//mpich-3.2.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:452): assert (!closed) failed

However if nmap scans tcp port for our execuaable then we see the orginal assert as follows:
MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO ||
hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO)

We are looking for both short term and long term solutions. Short term solution since customer is not willing to whitelist our tool from the security app running nmap for long. Long term solution is to move to later(latest) version of mpich as it will go through configuration and testing cycle. For short term, we are exploring certain hotfixes to avoid this assert.

I see following discussion in this thread(#6010):

Is this hotfix safe enough? is this fix present in any later releases? Anything else we can attempt?

[hzhou]

The hotfix effectively ignores unrecognized connection and simply closes the connection. It should work if you assume all such unrecognized packets comes from benign port scan thus safe to ignore.

[tarunkumar987]

For us, as long as packets sent between partitions(or all processes including mpiexec and our executables are recognized) it really won't matter. Can we assume that all other sort of packets are unrecognized/unwanted and are safe to ignore?

[hzhou]

Since you are running on a local node, there isn't any communication using tcp/ip, so it should be safe to ignore all such packets.