Skip to content

How to expand PVC created by statefulsets?

Jump to bottom
Masih H. Derkani edited this page Sep 19, 2022 · 8 revisions

Statefulset can be configured to create persistent volume claims (PVC) for the pods it manages. Though convenient, such volumes created by statefulset are not modifiable. This is because the volumeClaims section in the statefulset is immutable even if the storage class used by the PVC permits expansion of volumes.

Storetheindex prod and dev instances that are created based on storetheindex kustomization create PVCs in this way. Both use expandable volumes of type io2 but their disk cannot be expanded by modifying the relevant statefulset patches for each of the environments. Note that we do not need to do any of this for instances that are created based on storetheindex-single kustomization, and we can simply edit the PVC size in pvc.yaml and get that merged. So before beginning, check that the PVCs are indeed created by storetheindex kustomization.

To work around the immutability issue, the statefulset can be manually re-created with the expanded volumes without affecting the running pods. Before explaining the steps on how to do this, make sure:

To expand the volumes manually and update the statefulset:

  1. Make sure kubectl is authenticated to the right cluster:

    kubectl config current-context

  2. Suspend the CD pipeline in storetheindex namespace:

    flux suspend -n storetheindex kustomization storetheindex

  3. Create a PR to change the PVC size in the relevant deployment. See this PR as an example.

  4. Get the PR reviewed and merged. Note that since the CD pipeline is suspended in earlier steps the changes in the PR will not be applied to the cluster.

  5. Delete the existing statefulset without deleting the pods it manages; this is done by passing --cascade=orphan flag when deleting the statefulset:

    kubectl delete sts -n storetheindex --cascade=orphan indexer

    You should see that the statefulset is no longer present after the command above is executed by running:

    kubectl get sts -n storetheindex

  6. Manually edit the PVCs that are created and managed by the statefulset, and change the volume size to match the value in the PR. Note that the number of PVCs matches the replicas count on the statefulset and the PVCs need to be edited one by one. The name of the PVC depends on the statefulset configuration. Make sure to only edit the PVCs that correspond to the statefulset. In the case of storetheindex, the PVC names will be data-indexer-0, data-indexer-1 and so on where the total number of PVCs matches the configured replica count of indexer statefulset. Start editing the first PVC by running:

    kubectl edit pvc -n storetheindex data-indexer-0 This will open up the PVC in your configured EDITOR. Change the storage value under spec key to the new size. Example:

    ...
spec:
 resources:
   requests:
     storage: 20Ti
...

    Change the line, save your changes and close down your editor. Upon closing your editor the changes are applied to the remote K8S cluster. Carry on editing the rest of the PVCs one by one.

  7. Confirm that the PVCs are resized. This might take a bit of time as the controller applies the changes. Check that all is complete by running:

    $ kubectl get pvc -n storetheindex
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-indexer-0   Bound    pvc-000000000000000000000000000000000000   20Ti       RWO            io2            125d
data-indexer-1   Bound    pvc-000000000000000000000000000000000000   20Ti       RWO            io2            125d

    The CAPACITY column should show the new size.

  8. Resume the CD pipeline, which will recreate the statefulset with the new size and will retake charge of existing pods:

    flux resume -n storetheindex kustomization storetheindex

  9. Confirm that statefulset is re-created and ready:

    kubectl get sts -n storetheindex
NAME      READY   AGE
indexer   2/2     41m

    Note the READY column showing 2/2 i.e. 2 out of 2 pods are ready.

Troubleshooting

PVC volume size change is not applied

This can happen when the controller responsible for applying the changes cannot perform them, say because hitting disk quota on the region. To investigate further probe the state of the PVC in question (e.g. data-indexer-0) by running:

kubectl describe pvc -n storetheindex data-indexer-0

Under Events section you should be able to see the log of events that occurred while the controller attempted to apply the changes. That should serve as clues on what might be wrong. If no such events are listed, either the PVC was not edited correctly or the controller is not running.

Exceeded maximum storage limit

Each AWS region has a maximum storage limit which may be hit during volume expansion. Here is an example of the event message signalling this:

You have exceeded your maximum io2 storage limit of 200 TiB in this region. Please contact AWS Support to request an Elastic Block Store service limit increase

  • Open the AWS Console
  • Go to Service Quotas
  • Extract the volume type from the event message and find the corresponding quota section under Amazon EBS. The example message above pointes at io2 volumes, the corresponding quota section for which is Storage for Provisioned IOPS SSD (io2) volumes, in TiB.
  • Click on "Request quota increase" and create a new case. There is a judgment call on the amount to request. Usually twice the current amount is a reasonable number. See previous case as an example.
  • Submit the case and wait for it to be actioned.
  • Once actioned, go back to checking the affected PVCs. They should automatically expand.
  • Follow on with any remaining steps.
Clone this wiki locally