Add fallback_groups feature

[JKHiggins]

No description provided.

[JKHiggins]

@jimmyngo this has been rebased and is passing the build

[jimmyngo]

Do you have an example of when this is useful?

[JKHiggins]

Yes. When a notification group is not supplied in the config, this allows a default fallback group to be configured. Meaning if an alert is setup to monitor services used by all teams, but that team does not configure their notification channel, this fallback group will allow us to send notifications to a common channel.

[mcwqy9]

We have monitors that identify the owning team based on AWS tags on the unhealthy resource, and if there is not a matching group configured in interferon we use this feature to notify a default set of people instead.

@jimmyngo Can this be merged then? :)

[jimmyngo]

I'm not a huge fan, the same thing can be accomplished by doing notify.groups @hostinfo[:groups] || default_group ?

[mcwqy9]

What you suggest would only work if @hostinfo[:groups] was falsey. The problem I ran into with that is if an AWS resource is created with tags that specify a group that doesn't exist in the interferon config, @hostinfo[:groups] is defined and truthy but since no such group exists in interferon that monitor will cause the run of interferon to fail.

At the time the alert files are parsed the groups haven't yet been loaded, so I can't switch on that in the alert file either.