Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

change alertmanager-svc-headless from http to grpc port + expose 9094 TCP and UDP #435

Closed
wants to merge 14 commits into from
Closed

change alertmanager-svc-headless from http to grpc port + expose 9094 TCP and UDP #435

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
dcc7660
change alertmanager-svc-headless from http to grpc port
humblebundledore Feb 16, 2023
d8bdc70
edit CHANGELOG.md
humblebundledore Feb 16, 2023
a001e49
expose 9094 TCP UDP for gossip cluster
humblebundledore Feb 22, 2023
b0df50e
Merge branch 'master' into master
humblebundledore Mar 15, 2023
e483cd5
support configuration of gossip cluster port
humblebundledore Mar 15, 2023
84c1331
Merge branch 'master' into master
humblebundledore Mar 21, 2023
05b3403
WIP: configure alertmanager HA cluster mode for sts
nschad May 12, 2023
d29022e
Merge branch 'cortexproject:master' into master
humblebundledore Jun 28, 2023
8d22e3a
Merge remote-tracking branch 'upstream/feature/alertmanager-cluster'
humblebundledore Jun 28, 2023
1d830da
configure alertmanager cluster peers as comma seperated list
humblebundledore Jul 3, 2023
e909803
clarify values.yaml about cluster enable by default
humblebundledore Jul 4, 2023
b9efaee
replicaset should not set -alertmanager-cluster-peers when cluster is…
humblebundledore Jul 4, 2023
87e6eb4
fix wrong name for grpc targetPort
humblebundledore Jul 26, 2023
62c1222
re-introduce alertmanager-dep.yaml
humblebundledore Jul 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
## 2.1.0 / 2023-03-17

* [FEATURE] add autoscaler for the ruler #430
* [CHANGE] change alertmanager-svc-headless from http to grpc port #420
* [ENHANCEMENT] Add annotations and labels to memberlist service #433
* [ENHANCEMENT] Add podTargetLabels to all Prometheus servicemonitor CRs #439
* [DEPENDENCY] Update Helm release memcached to v6.3.12 #441
Expand Down
2 changes: 1 addition & 1 deletion ci/test-deployment-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ runtimeconfigmap:
annotations:
foo: bar
alertmanager:
replicas: 1
replicas: 3
statefulSet:
enabled: false
extraVolumes:
Expand Down
2 changes: 1 addition & 1 deletion ci/test-sts-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ runtimeconfigmap:
annotations:
foo: bar
alertmanager:
replicas: 1
replicas: 3
statefulSet:
enabled: true
extraVolumes:
Expand Down
169 changes: 0 additions & 169 deletions templates/alertmanager/alertmanager-dep.yaml
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a breaking change. I would rather not remove it. Also the configuration below then does make little sense.

The {{- if .Values.alertmanager.statefulSet.enabled -}} would become obsolete.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also currently if you set statefulSet to false (the default) and deploy cortex then you only get the non-headless service of alertmanager. Makes no sense.

Suggestion:

Keep the alertmanager deployment (maybe clean that up too if necessary) but don't enable support for clustering out-of-the-box. Let's keep it as it is.

This file was deleted.

20 changes: 20 additions & 0 deletions templates/alertmanager/alertmanager-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{{- $svcClusterAddress := ((.Values.config.alertmanager.cluster).listen_address) | default "0.0.0.0:9094" }}
{{- $svcClusterPort := (split ":" $svcClusterAddress)._1 }}
{{- if .Values.alertmanager.enabled -}}
{{- if .Values.alertmanager.statefulSet.enabled -}}
apiVersion: apps/v1
Expand Down Expand Up @@ -152,6 +154,15 @@ spec:
args:
- "-target=alertmanager"
- "-config.file=/etc/cortex/cortex.yaml"
{{- if and (gt (int .Values.alertmanager.replicas) 1) (ne .Values.config.alertmanager.cluster.listen_address "") }}
{{- $fullName := include "cortex.alertmanagerFullname" . }}
{{- $peers := list }}
{{- range $i := until (int .Values.alertmanager.replicas) }}
{{- $peer := printf "%s-%d.%s-headless.%s.svc.cluster.local:%s" $fullName $i $fullName $.Release.Namespace $svcClusterPort }}
{{- $peers = append $peers $peer }}
{{- end }}
- "-alertmanager.cluster.peers={{ join "," $peers }}"
{{- end }}
{{- range $key, $value := .Values.alertmanager.extraArgs }}
- "-{{ $key }}={{ $value }}"
{{- end }}
Expand All @@ -175,6 +186,15 @@ spec:
- name: gossip
containerPort: {{ .Values.config.memberlist.bind_port }}
protocol: TCP
- name: grpc
containerPort: {{ .Values.config.server.grpc_listen_port }}
protocol: TCP
- containerPort: {{ $svcClusterPort }}
name: alert-clu-tcp
protocol: TCP
- containerPort: {{ $svcClusterPort }}
name: alert-clu-udp
protocol: UDP
startupProbe:
{{- toYaml .Values.alertmanager.startupProbe | nindent 12 }}
livenessProbe:
Expand Down
10 changes: 9 additions & 1 deletion templates/alertmanager/alertmanager-svc-headless.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,18 @@ spec:
clusterIP: None
publishNotReadyAddresses: true
ports:
- port: {{ .Values.config.server.http_listen_port }}
- port: {{ .Values.config.server.grpc_listen_port }}
protocol: TCP
name: http-metrics
targetPort: http-metrics
humblebundledore marked this conversation as resolved.
Show resolved Hide resolved
- port: 9094
protocol: UDP
name: alert-clu-udp
targetPort: alert-clu-udp
- port: 9094
protocol: TCP
name: alert-clu-tcp
targetPort: alert-clu-tcp
selector:
{{- include "cortex.alertmanagerSelectorLabels" . | nindent 4 }}
{{- end -}}
Expand Down
4 changes: 4 additions & 0 deletions values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,10 @@ config:
runtime_config:
file: /etc/cortex-runtime-config/runtime_config.yaml
alertmanager:
# -- Enable alertmanager gossip cluster
# -- Disable alertmanager gossip cluster by setting empty listen_address to empty string
cluster:
listen_address: '0.0.0.0:9094'
# -- Enable the experimental alertmanager config api.
enable_api: false
external_url: '/api/prom/alertmanager'
Expand Down