common/prometheus-server: add new alert label incident_group_key

Introducing a new alert label that makes it possible to group alerts in ServiceNow by alertname, region and cluster. This ensures that multiple incidents are no longer generated for one alert, which also led to multiple tickets and notifications etc.
sapcc · May 30, 2024 · 0595d63 · 0595d63
1 parent 3d8ffb8
commit 0595d63
Show file tree

Hide file tree

Showing 3 changed files with 14 additions and 1 deletion.
diff --git a/common/prometheus-server/CHANGELOG.md b/common/prometheus-server/CHANGELOG.md
@@ -1,3 +1,11 @@
+## 7.5.9
+
+* Introducing a new alert label `incident_group_key` that makes it possible to group alerts in ServiceNow by alertname, region and cluster.
+
+## 7.5.8
+
+* Lower default retention time
+
 ## 7.5.7
 
 * softening VPA alerts 

diff --git a/common/prometheus-server/Chart.yaml b/common/prometheus-server/Chart.yaml
@@ -1,7 +1,7 @@
 apiVersion: v2
 description: Prometheus via operator.
 name: prometheus-server
-version: 7.5.8
+version: 7.5.9
 appVersion: v2.47.2
 icon: https://raw.githubusercontent.com/prometheus/prometheus.github.io/master/assets/prometheus_logo-cb55bb5c346.png
 maintainers:

diff --git a/common/prometheus-server/templates/_alertmanager_relabel.yaml.tpl b/common/prometheus-server/templates/_alertmanager_relabel.yaml.tpl
@@ -11,3 +11,8 @@
 - action: replace
   target_label: cluster
   replacement: {{ if $root.Values.global.cluster }}{{ $root.Values.global.cluster }}{{ else }}{{ $root.Values.global.region }}{{ end }}
+
+- action: replace
+  source_labels: [alertname]
+  target_label: incident_group_key
+  replacement: {{ printf "%s_%s_$1" $root.Values.global.region ($root.Values.global.cluster | default $root.Values.global.region) }}