Skip to content

Commit

Permalink
🐛 fixes bug where a muted alert could be included in another active a…
Browse files Browse the repository at this point in the history
…lert
  • Loading branch information
Rohland committed Nov 6, 2023
1 parent afe9097 commit 14803a8
Show file tree
Hide file tree
Showing 5 changed files with 45 additions and 22 deletions.
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ Example configuration:

```yaml
mute-windows: # alerts are silenced if generated in these window periods
- match: mysql:performance # only for monitors matching this regex
- match: mysql.*performance # only for monitors matching this regex
time: 00:00 - 06:00
- date: 2023-08-27 # only matches for this specific date
time: 22:00 - 24:00 # 2PM to 4PM for a specific date
Expand Down Expand Up @@ -421,10 +421,19 @@ Any number of windows can be defined where alerts will be silenced. This is usef

Fields:

- `match`: regex to match monitor names (not required)
- `match`: regex to match monitor identifier(see below for format) (not required)
- `date`: specific date to match (not required)
- `time`: time range to match (required)

The `match` regular expression is used to match against the monitor identifier that is a string value composed of:

- `type` - the type of monitor (example: web, sumo, mysql)
- `label` - the name of the monitor (example: web-performance)
- `identifier` - the name of the failing identifier (example: www.codeo.co.za)

The value is composed as follows: `type|label|identifier`. For example: `web|web-performance|www.codeo.co.za`. The regular expression for match
will thus be compared against this string value (case insensitive).

### SMS

For SMS, any initial change into a failure state for the relevant team, will trigger a single SMS, which will include a summary of what has gone wrong, and will indicate that further updates will be sent via Slack. An update is posted every 15 minutes.
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "barky",
"version": "1.0.48",
"version": "1.0.49",
"description": "A simple cloud services watchdog with digest notification support & no external dependencies",
"homepage": "https://github.com/Rohland/barky#readme",
"main": "dist/cli.js",
Expand Down
42 changes: 27 additions & 15 deletions src/digest/alerter.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -116,21 +116,7 @@ describe("alerter", () => {
]
});
const context = new DigestContext([], []);
const result1 = new Result(
new Date(),
"web",
"health",
"www.codeo.co.za",
false,
"FAIL",
0,
false,
{
alert: {
channels: ["console"]
}
}
);
const result1 = getTestResult();
context.addSnapshotForResult(result1);

// act
Expand All @@ -139,6 +125,32 @@ describe("alerter", () => {
// assert
expect(console.log).not.toHaveBeenCalled();
});
describe("when there is another alert that is not muted", () => {
it("should exclude muted", async () => {
// arrange
const config = new DigestConfiguration({
"mute-windows": [
{
match: "web",
time: "00:00-24:00",
}
]
});
const context = new DigestContext([], []);
const result1 = getTestResult();
const result2 = getTestResult();
result2.type = "mysql";

context.addSnapshotForResult(result1);
context.addSnapshotForResult(result2);

// act
await executeAlerts(config, context);

// assert
expect(console.log).toHaveBeenCalledWith(expect.stringContaining("1 health check affected"));
});
});
});
});
describe("with existing alert", () => {
Expand Down
4 changes: 2 additions & 2 deletions src/digest/alerter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ async function sendNewAlerts(
if (!channel) {
return;
}
const snapshots = context.getSnapshotsForChannel(channel);
const snapshots = context.getAlertableSnapshotsForChannel(config, channel);
alert.start_date = earliestDateFor(snapshots);
await channel.sendNewAlert(
snapshots,
Expand Down Expand Up @@ -62,7 +62,7 @@ async function sendOngoingAlerts(
if (!channel) {
return;
}
const snapshots = context.getSnapshotsForChannel(channel);
const snapshots = context.getAlertableSnapshotsForChannel(config, channel);
if (channel.canSendAlert(alert)) {
await channel.sendOngoingAlert(snapshots, alert);
alert.last_alert_date = new Date();
Expand Down
6 changes: 4 additions & 2 deletions src/digest/digest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,10 @@ export class DigestContext {
return this._snapshots;
}

public getSnapshotsForChannel(channel: ChannelConfig) {
return this.snapshots
public getAlertableSnapshotsForChannel(
config: DigestConfiguration,
channel: ChannelConfig) {
return this.alertableSnapshots(config)
.filter(x => x.isDigestable)
.filter(x => x.alert?.channels?.some(c => channel.isMatchFor(c)));
}
Expand Down

0 comments on commit 14803a8

Please sign in to comment.