fix(volume): Reclaim persistent volumes when the StatefulSet is deleted

[jsirianni]

Description of Changes

When upgrading a StatefulSet to use a Prometheus sidecar, the StatefulSet needs to be deleted first. We need the volume reclaim settings to be set to "retain" to avoid dataloss. When the new StatefulSet is created, it will attach to the same volume and avoid dataloss.

You can learn more here: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention

Testing

Start minikube: minikube start

Create the following values.yaml file.

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true

Deploy

helm template --values ./values.yaml charts/bindplane | kubectl apply -f -

Port forward and connect to the UI:

kubectl port-forward pod/release-name-bindplane-0 3011:3001 --address 0.0.0.0

http://localhost:3011

Create a config. You do not need to roll it out or anything, we just need a way to prove that we have not lost data.

With a config created, update the values file to include the Prometheus sidecar.

prometheus:
  enable: true
  enableSideCar: true
config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true

If you re-deploy with helm, you will see this error:

The StatefulSet "release-name-bindplane" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Delete the StatefulSet in order to re-create it.

kubectl delete sts release-name-bindplane
helm template --values ./values.yaml charts/bindplane | kubectl apply -f -

The bindplane pod should be re-created and contain two containers. Re-run the port forward command and check the UI for your configuration. If the configuration exists, we have not lost data despite deleting the StatefulSet.

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes

[Mrod1598]

LGTM testing got expected outcome.