Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eventhistory service should have a storage limit #10173

Open
jvillafanez opened this issue Sep 26, 2024 · 0 comments
Open

Eventhistory service should have a storage limit #10173

jvillafanez opened this issue Sep 26, 2024 · 0 comments

Comments

@jvillafanez
Copy link
Member

Is your feature request related to a problem? Please describe.

The eventhistory service keeps events for 2 weeks (configurable). However, there is no limit on the amount of events that could happen during those 2 weeks. We could consider 1 million or more events that we need to handle during those 2 weeks.

This is also a problem for the activitylog service, which not only would need to store the ids of 1 million events, but also need to transfer the data of those events from the eventhistory service. Include additional processing such as sorting the list of events or ids.

Describe the solution you'd like

The eventhistory service should have a (configurable) limit on the number of events it can store. Once that limit is reached, older events should be removed to leave space to newer events. We can assume that, if the events are ordered by timestamp, it's unlikely that people would want to go through more than 10k events, so a default of 10k events should be good enough.
Note that the current TTL of 2 weeks should be kept. This storage size limit should be applied on top.

Describe alternatives you've considered

A fully persistent eventhistory would require to store data either in files or DB in order to have virtually unlimited storage. Combined with a proper search mechanism, we could check any event that happened 5 years ago because we wouldn't need to delete data (or at least we could storage a much larger event number).

So far, the implementation leans towards showing a partial list of events (the events that happened in the last 2 weeks), so it could make sense to also have a limit on the number of events we can store and show.

Additional context

Taking into account that most of our store implementations don't have an explicit size limit, we could provide a "soft" size limit ourselves. This means that we won't enforce the size limit (the number of events might be higher than the limit at any given time), but we'll try to keep it within the limit (if we detect we're passed the limit, we'll remove events until we're back on track).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant