Skip to content

Commit

Permalink
Fix regression when scaling up
Browse files Browse the repository at this point in the history
The default PodManagementPolicy for StatefulSet is set to
OrderedReadyPodManagement, which means it only creates new pods when the
previous one is ready.

However, to start the pods the Swift ring file ConfigMap is needed, and
this in turn is only created once all requested pods are started and
their PVCs are bound (because it needs information on the PVCs to be
used). If the instance is scaled by more than 1 (which is the case also
for starting with 3 replicas), it failed.

This is a regression to previous behaviour, introduced by changes to the
ring setup. Previously the ring rebalance did not wait until all PVCs
where bound, but estimated their size and updated the rings afterwards
with the correct size. This required ring rebalances and data movements
even for new deployments.

Therefore changing the policy to ParallelPodManagement, wich starts the
number of requested pods immediately. This does only affect scaling, not
updates.

The kuttl tests has been updated to avoid regressions in the future.
  • Loading branch information
cschwede committed Mar 14, 2024
1 parent 88c4c30 commit 741a368
Show file tree
Hide file tree
Showing 5 changed files with 30 additions and 1 deletion.
3 changes: 2 additions & 1 deletion pkg/swiftstorage/statefulset.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,8 @@ func StatefulSet(
Selector: &metav1.LabelSelector{
MatchLabels: labels,
},
Replicas: swiftstorage.Spec.Replicas,
Replicas: swiftstorage.Spec.Replicas,
PodManagementPolicy: appsv1.ParallelPodManagement,
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: labels,
Expand Down
21 changes: 21 additions & 0 deletions tests/kuttl/tests/basic-deploy/02-assert-scaleup-swift.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
apiVersion: v1
kind: Pod
metadata:
name: swift-storage-0
status:
phase: Running
---
apiVersion: v1
kind: Pod
metadata:
name: swift-storage-1
status:
phase: Running
---
apiVersion: v1
kind: Pod
metadata:
name: swift-storage-2
status:
phase: Running
5 changes: 5 additions & 0 deletions tests/kuttl/tests/basic-deploy/02-scaleup-swift.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
oc patch -n swift-kuttl-tests swift/swift --type='json' -p='[{"op": "replace", "path": "/spec/swiftStorage/replicas", "value":3}]'
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@ delete:
commands:
- script: |
oc delete --ignore-not-found=true -n $NAMESPACE pvc srv-swift-storage-0
oc delete --ignore-not-found=true -n $NAMESPACE pvc srv-swift-storage-1
oc delete --ignore-not-found=true -n $NAMESPACE pvc srv-swift-storage-2

0 comments on commit 741a368

Please sign in to comment.