Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outbox consumers should specify whether they need to process historical events #929

Open
zaychenko-sergei opened this issue Oct 28, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@zaychenko-sergei
Copy link
Contributor

zaychenko-sergei commented Oct 28, 2024

Currently, when a new consumer type registers in the Outbox, it's implicitly assumed consumption boundary is 0.
This effectively means processing all historically accumulated events of the connected producer starting from the very beginning.
This might be a good approach to completely rebuild new consumer's state solely from events.

In some cases, like recent DatasetEntryService,this behavior is undesirable, because there is an explicit initialization job, DatasetEntryIndexer, which is building the initial state of the service from an external source (storage system's state).
Such cases do not require consuming historically accumulated events with overlapping semantics, so consumption should start from the current producer's latest message.

In summary, create a mechanism in Outbox system so that consumer can specify what to do if it's first time registered in the system:

  • start from event 0 (full history reconstruction)
  • start from the latest event of the producer (custom initialization procedure, like indexing).

I.e.:

#[meta(MessageConsumerMeta {
consumer_name: MESSAGE_CONSUMER_KAMU_DATASET_ENTRY_SERVICE,
initial_consumption_boundary: InitialConsumptionBoundary::LatestProduced
...
})]
impl DatasetEntryService {

@zaychenko-sergei zaychenko-sergei added the enhancement New feature or request label Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant