Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(alerting): Added Incident.io alerting provider #972

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,7 @@ endpoints:
| `alerting.telegram` | Configuration for alerts of type `telegram`. <br />See [Configuring Telegram alerts](#configuring-telegram-alerts). | `{}` |
| `alerting.twilio` | Settings for alerts of type `twilio`. <br />See [Configuring Twilio alerts](#configuring-twilio-alerts). | `{}` |
| `alerting.zulip` | Configuration for alerts of type `zulip`. <br />See [Configuring Zulip alerts](#configuring-zulip-alerts). | `{}` |
| `alerting.incidentio` | Configuration for alerts of type `incidentio`. <br />See [Configuring Incident.io alerts](#configuring-incidentio-alerts). | `{}` |
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved


#### Configuring AWS SES alerts
Expand Down Expand Up @@ -1256,6 +1257,41 @@ Here's an example of what the notifications look like:

![Slack notifications](.github/assets/slack-alerts.png)

#### Configuring Incident.io alerts
| Parameter | Description | Default |
|:-----------------------------------|:-------------------------------------------------------------------------------------------|:--------------|
| `alerting.incidentio` | Configuration for alerts of type `incidentio` | `{}` |
| `alerting.incidentio.alert-source-config-id` | Which alert source config produced this alert | Required `""` |
| `alerting.incidentio.auth-token` | Token that is used for authentication. | Required `""` |
| `alerting.incidentio.overrides` | List of overrides that may be prioritized over the default configuration | `[]` |
| `alerting.incidentio.default-alert` | Default alert configuration. <br />See [Setting a default alert](#setting-a-default-alert) | N/A |
| `alerting.incidentio.overrides[].group` | Endpoint group for which the configuration will be overridden by this configuration | `""` |
| `alerting.incidentio.overrides[].*` | See `alerting.incidentio.*` parameters | `{}` |

```yaml
alerting:
incidentio:
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
alert-source-config-id: "*****************"
auth-token: "********************************************"

endpoints:
- name: website
url: "https://twin.sh/health"
interval: 30s
conditions:
- "[STATUS] == 200"
- "[BODY].status == UP"
- "[RESPONSE_TIME] < 300"
alerts:
- type: incidentio
description: "healthcheck failed"
send-on-resolved: true
```
in order to get the required alert source config id and authentication token, you must configure an HTTP alert source.

> **_NOTE:_** the source config id is of the form { api.incident.io/v2/alert_events/http/{CONFIG-SOURCE-ID}}

> **_NOTE:_** the auth token is of the form { "Authorization": "Bearer {AUTH-TOKEN}" }

#### Configuring Teams alerts *(Deprecated)*

Expand Down
3 changes: 3 additions & 0 deletions alerting/alert/type.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,4 +73,7 @@ const (

// TypeZulip is the Type for the Zulip alerting provider
TypeZulip Type = "zulip"

// TypeZulip is the Type for the incident.io alerting provider
TypeIncidentio Type = "incidentio"
)
4 changes: 4 additions & 0 deletions alerting/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/TwiN/gatus/v5/alerting/provider/gitlab"
"github.com/TwiN/gatus/v5/alerting/provider/googlechat"
"github.com/TwiN/gatus/v5/alerting/provider/gotify"
"github.com/TwiN/gatus/v5/alerting/provider/incidentio"
"github.com/TwiN/gatus/v5/alerting/provider/jetbrainsspace"
"github.com/TwiN/gatus/v5/alerting/provider/matrix"
"github.com/TwiN/gatus/v5/alerting/provider/mattermost"
Expand Down Expand Up @@ -102,6 +103,9 @@ type Config struct {

// Zulip is the configuration for the zulip alerting provider
Zulip *zulip.AlertProvider `yaml:"zulip,omitempty"`

// IncidentIo is the configuration for the incident.io alerting provider
IncidentIo *incidentio.AlertProvider `yaml:"incidentio,omitempty"`
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
}

// GetAlertingProviderByAlertType returns an provider.AlertProvider by its corresponding alert.Type
Expand Down
205 changes: 205 additions & 0 deletions alerting/provider/incidentio/incident_io.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
package incidentio

import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strconv"

"github.com/TwiN/gatus/v5/alerting/alert"
"github.com/TwiN/gatus/v5/client"
"github.com/TwiN/gatus/v5/config/endpoint"
"gopkg.in/yaml.v3"
)

const (
restAPIUrl = "https://api.incident.io/v2/alert_events/http"
firingStatus = "firing"
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
resolvedStatus = "resolved"
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
)

var (
ErrAlertSourceConfigNotSet = errors.New("alert source config not set.")
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
ErrDuplicateGroupOverride = errors.New("duplicate group override")
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
ErrAuthTokenNotSet = errors.New("authentication token not set.")
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
)

type Config struct {
AlertSourceConfigID string `yaml:"alert-source-config-id,omitempty"`
AuthToken string `yaml:"auth-token,omitempty"`

//Status sent to incident.io, either "firing" or "resolved"
Status string
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved

//key of the alert,initalized on the first event.
DeduplicationKey string
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
}

func (cfg *Config) Validate() error {
if len(cfg.AlertSourceConfigID) == 0 {
return ErrAlertSourceConfigNotSet
}
if len(cfg.AuthToken) == 0 {
return ErrAuthTokenNotSet
}
return nil
}

func (cfg *Config) Merge(override *Config) {
if len(override.AlertSourceConfigID) > 0 {
cfg.AlertSourceConfigID = override.AlertSourceConfigID
}
if len(override.AuthToken) > 0 {
cfg.AuthToken = override.AuthToken
}
}

// AlertProvider is the configuration necessary for sending an alert using incident.io
type AlertProvider struct {
DefaultConfig Config `yaml:",inline"`

// DefaultAlert is the default alert configuration to use for endpoints with an alert of the appropriate type
DefaultAlert *alert.Alert `yaml:"default-alert,omitempty"`

// Overrides is a list of Override that may be prioritized over the default configuration
Overrides []Override `yaml:"overrides,omitempty"`
}

type Override struct {
Group string `yaml:"group"`
Config `yaml:",inline"`
}

func (provider *AlertProvider) Validate() error {
registeredGroups := make(map[string]bool)
if provider.Overrides != nil {
for _, override := range provider.Overrides {
if isAlreadyRegistered := registeredGroups[override.Group]; isAlreadyRegistered || override.Group == "" {
return ErrDuplicateGroupOverride
}
registeredGroups[override.Group] = true
}
}
return provider.DefaultConfig.Validate()
}

func (provider *AlertProvider) Send(ep *endpoint.Endpoint, alert *alert.Alert, result *endpoint.Result, resolved bool) error {
cfg, err := provider.GetConfig(ep.Group, alert)
if err != nil {
return err
}
buffer := bytes.NewBuffer(provider.buildRequestBody(cfg, ep, alert, result, resolved))
url := fmt.Sprintf("%s/%s", restAPIUrl, cfg.AlertSourceConfigID)
req, err := http.NewRequest(http.MethodPost, url, buffer)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+cfg.AuthToken)
response, err := client.GetHTTPClient(nil).Do(req)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode > 399 {
body, _ := io.ReadAll(response.Body)
return fmt.Errorf("call to provider alert returned status code %d: %s", response.StatusCode, string(body))
}
incidentioResponse := Response{}
json.NewDecoder(response.Body).Decode(&incidentioResponse)
cfg.DeduplicationKey = incidentioResponse.DeduplicationKey
return err
}

type Body struct {
AlertSourceConfigID string `json:"alert_source_config_id"`
Status string `json:"status"`
Title string `json:"title"`
DeduplicationKey string `json:"deduplication_key"`
Description string `json:"description"`
}

type Response struct {
DeduplicationKey string `json:"deduplication_key"`
}

func (provider *AlertProvider) buildRequestBody(cfg *Config, ep *endpoint.Endpoint, alert *alert.Alert, result *endpoint.Result, resolved bool) []byte {
var message, formattedConditionResults string
if resolved {
message = "An alert has been resolved after passing successfully " + strconv.Itoa(alert.SuccessThreshold) + " time(s) in a row"
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
cfg.Status = "resolved"
} else {
message = "An alert has been triggered due to having failed " + strconv.Itoa(alert.FailureThreshold) + " time(s) in a row"
cfg.Status = "firing"
}
for _, conditionResult := range result.ConditionResults {
var prefix string
if conditionResult.Success {
prefix = "🟢"
ImTheCurse marked this conversation as resolved.
Show resolved Hide resolved
} else {
prefix = "🔴"
}
// No need for \n since incident.io trims it anyways.
formattedConditionResults += fmt.Sprintf(" %s %s ", prefix, conditionResult.Condition)
}
if len(alert.GetDescription()) > 0 {
message += " with the following description: " + alert.GetDescription()
}

message += fmt.Sprintf(" and the following conditions: %s ", formattedConditionResults)
var body []byte
if len(cfg.DeduplicationKey) > 0 {
body, _ = json.Marshal(Body{
AlertSourceConfigID: cfg.AlertSourceConfigID,
Title: "Gatus: " + ep.DisplayName(),
Status: cfg.Status,
DeduplicationKey: cfg.DeduplicationKey,
Description: message,
})
} else {
body, _ = json.Marshal(Body{
AlertSourceConfigID: cfg.AlertSourceConfigID,
Title: "Gatus: " + ep.DisplayName(),
Status: cfg.Status,
Description: message,
})
}
return body

}
func (provider *AlertProvider) GetConfig(group string, alert *alert.Alert) (*Config, error) {
cfg := provider.DefaultConfig
// Handle group overrides
if provider.Overrides != nil {
for _, override := range provider.Overrides {
if group == override.Group {
cfg.Merge(&override.Config)
break
}
}
}
// Handle alert overrides
if len(alert.ProviderOverride) != 0 {
overrideConfig := Config{}
if err := yaml.Unmarshal(alert.ProviderOverrideAsBytes(), &overrideConfig); err != nil {
return nil, err
}
cfg.Merge(&overrideConfig)
}
// Validate the configuration
err := cfg.Validate()
return &cfg, err
}

// GetDefaultAlert returns the provider's default alert configuration
func (provider *AlertProvider) GetDefaultAlert() *alert.Alert {
return provider.DefaultAlert
}

func (provider *AlertProvider) ValidateOverrides(group string, alert *alert.Alert) error {
_, err := provider.GetConfig(group, alert)
return err
}
Loading