Unpublished events, sending duplicates on application startup in some cases

[andreasmaubach]

when spring.modulith.events.republish-outstanding-events-on-restart=true

and a bean, using this scheduler (code below) to republish incomplete events exists at the same time. it will produce two duplicate events (duplicate the data atleast) of the unpublished/incomplete events from EVENT_PUBLICATION on application startup.

@Scheduled(fixedDelayString = "${events.resubmitFrequencyInSecs}", timeUnit = TimeUnit.SECONDS) void resubmitUnpublishedEvents() { incompleteEventPublications.resubmitIncompletePublicationsOlderThan( eventsConfigProperties.getResubmitOlderThan()); }

Tested with externalized events to an azure eventhub (using kafka apis).
But also on a local kafka container.

in fact, when calling incompleteEventPublications.resubmitIncompletePublications(eventPublication -> true);
from a test using @ApplicationModuleTest, it will produce 3 similar events pr unpublished event. (1 for resubmit on startup, 1 from the scheduled bean, and 1 for the test)

[davidkarlsen]

I think it should be added here (we work in same company) that this probably happens due to multi-instances - any enterprise system - and especially run in cloud is usually multi-instance - at least for some some sort of time while scaling up / rolling out versions etc.

If the republishing used pessimistic locks (select for update) this can be avoided. At least this fact needs to be documented in the ref-docs.

[odrotbohm]

The property mentioned is documented not to be used in a multi-instance scenario:

Usually not recommended in multi-instance deployments as other instances might still be processing events.

That said, any kind of (At)scheduled operation will need some sort of distributed locking (usually in the form of Spring Integration distributed locks or ShedLock) to make sure the individual application instances do not compete for work or resources. This is not a problem specific to Spring Modulith.

In fact, when calling incompleteEventPublications.resubmitIncompletePublications(eventPublication -> true);
from a test using @ApplicationModuleTest, it will produce 3 similar events per unpublished event. (1 for resubmit on startup, 1 from the scheduled bean, and 1 for the test).

That's “correct” in the sense that that's the result of your code and setup triggering the functionality three times.

If the republishing used pessimistic locks (select for update) this can be avoided.

A simple SELECT FOR UPDATE is unfortunately not going to cut it, as that would mean unconditionally wrapping the event listener into a transaction that'd consume a database connection. As we don't know how long the listener execution is going to take, that would be a recipe for exhausting the connection pool accidentally. There are plans to revamp the current event publication lifecycle to provide more specific means to obtain and manage publications.

[davidkarlsen]

My memory puzzled me - I thought the scheduling was done by modulith, but we actually do this ourselves - so yes, then we can add shedlock to protect it from concurrency.