bug: message not delivered during interop test

[romanzac]

Problem

At rare occasion, message is not delivered when a node is shut down during the test sequence.

Impact

It is low occurrence, medium impact. The node may shut down before the message in sent to other nodes.

To reproduce

Run interop tests at https://github.com/waku-org/waku-interop-tests/actions/workflows/go_waku_daily.yml. Simillar issue happened also with nwaku during execution of test_publish_after_node1_restarts.

Expected behavior

Node which is shutting down should reject accepting new messages. When the message gets stuck, next time the node starts it retries the delivery. I would assume shutdown was part of a restart. Open question is what to do when node stays shut for long periods of time. Messages lost ?

Related to waku-org/nwaku#2369

Screenshots/logs

data_attachments_8d428324ec7625a3.txt

[chaitanyaprem]

Node which is shutting down should reject accepting new messages, When the message gets stuck, next time the node starts it retries the delivery. I would assume shutdown was part of a restart. Open question is what to do when node stays shut for long periods of time. Messages lost ?

I don't think it is a good idea to implement such synchronization mechanism as it will complicate and create lot more other issues.
It should be the responsibility of caller/user to ensure such scenarios are addressed.

Some of these scenarios will have to be handled at application level, you can refer to discussion here for reference.

Closing this as this is not considered under go-waku scope.

[romanzac]

Another reliability problem happened during different test (not invoking node restart):
test_all_nodes_subscribed_to_the_topic

data_attachments_64e9303acd8f5de5.txt