Skip to content

Commit

Permalink
ci: disable readinessProbes in maintenanceMode
Browse files Browse the repository at this point in the history
While testing upgrades as part of [0], updating the docs in [1],
it became clear that during manual maintenance for munging state during
chain upgrades procedures, we want all the nodes to be reset together.
Therefore if maintenanceMode=true, we'll also disable the
readinessProbes, otherwise the rollout will be staged, forcing us to
upgrade each node in term, rather than being able to parallelize them.
Now, we can `pd export` and backup state, then update the statefulset to bump
the container version (to the post-upgrade version), and restart it in
maintenance mode to continue with the `pd migrate` step and resulting
copying around of emitted files.

[0] #4055
[1] https://github.com/penumbra-zone/penumbra/wiki/Performing-upgrades
  • Loading branch information
conorsch committed Mar 22, 2024
1 parent 45f2b21 commit efb364e
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,14 @@ spec:
- name: pd-abci
containerPort: 26658
protocol: TCP
{{- if .Values.maintenanceMode }}
{{- else }}
readinessProbe:
tcpSocket:
port: pd-grpc
timeoutSeconds: 5
initialDelaySeconds: 20
{{- end }}
resources:
{{- toYaml $.Values.resources | nindent 12 }}
securityContext:
Expand Down Expand Up @@ -132,11 +135,14 @@ spec:
- name: tm-metrics
containerPort: 26660
protocol: TCP
{{- if .Values.maintenanceMode }}
{{- else }}
readinessProbe:
tcpSocket:
port: tm-rpc
timeoutSeconds: 10
initialDelaySeconds: 10
{{- end }}
resources:
{{- toYaml $.Values.resources | nindent 12 }}
volumeMounts:
Expand Down
6 changes: 6 additions & 0 deletions deployments/charts/penumbra-node/templates/statefulset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -208,11 +208,14 @@ spec:
mountPath: /penumbra-config
securityContext:
runAsUser: {{ .Values.maintenanceMode | ternary 0 .Values.securityContext.runAsUser }}
{{- if .Values.maintenanceMode }}
{{- else }}
readinessProbe:
tcpSocket:
port: pd-grpc
timeoutSeconds: 10
initialDelaySeconds: 10
{{- end }}

- name: cometbft
securityContext:
Expand All @@ -235,11 +238,14 @@ spec:
- name: tm-rpc
containerPort: 26657
protocol: TCP
{{- if .Values.maintenanceMode }}
{{- else }}
readinessProbe:
tcpSocket:
port: tm-rpc
timeoutSeconds: 10
initialDelaySeconds: 10
{{- end }}
resources:
{{- toYaml .Values.cometbft.resources | nindent 12 }}
volumeMounts:
Expand Down

0 comments on commit efb364e

Please sign in to comment.