Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alert on occurrences only when within a time window #27

Open
raags opened this issue Nov 7, 2020 · 2 comments
Open

Alert on occurrences only when within a time window #27

raags opened this issue Nov 7, 2020 · 2 comments

Comments

@raags
Copy link

raags commented Nov 7, 2020

I have a use case of alerting on occurrences, but only if they happen within a time window.

For e.g. alert if there are 3 consecutive events in 1 hour.

This need arises especially when generating alerts via the event API, and not via regular checks. So, let's say you have a cronjob that is known to fail intermittently, and we have set occurrences to 3. Without the time window, an alert will be generated every time occurrences cross 3, even if they happen over a week.

@nixwiz
Copy link
Contributor

nixwiz commented Nov 9, 2020

That is not a use case for this particular filter as its only time based setting is a based on allowing failures to be passed on the handler(s) based on intervals of occurrence. There is no configuration available for it to say x events within y time frame.

If your occurrences for this can reach 3 over the course of a week, but this cron job is seemingly running multiple times per hour (based on your wanting the third occurrence within an hour), it sounds like you are not sending events with status = 0 for successful completions of the cronjob. Are you sending successful status events?

@raags
Copy link
Author

raags commented Nov 10, 2020

Yeah, cron job wasn't a good example. If success can be sent, then I suppose we don't need this. But there are still cases where success cannot be sent. For e.g. application sending alerts due to a failure, where sending success would be too frequent (for e.g a web app).

It's actually doable by using the check history. That has the state for all past occurrences with the time when they occurred. I'll take a stab at it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants