K8SPSMDB-1237: add incremental backups #1836

pooknull · 2025-02-18T20:44:06Z

https://perconadev.atlassian.net/browse/K8SPSMDB-1237

DESCRIPTION

This PR introduces support for incremental physical backups.

A new backup types for the field .spec.type have been added to the PerconaServerMongoDBBackup resource:

incremental: makes an incremental backup
incremental-base: makes a full incremental backup

CHECKLIST

Jira

Is the Jira ticket created and referenced properly?
Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

Is an E2E test/test case added for the new feature/change?
Are unit tests added where appropriate?
Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

Are all needed new/changed options added to default YAML files?
Are all needed new/changed options added to the Helm Chart?
Did we add proper logging messages for operator actions?
Did we ensure compatibility with the previous version or cluster upgrade process?
Does the change support oldest and newest supported MongoDB version?
Does the change support oldest and newest supported Kubernetes version?

https://perconadev.atlassian.net/browse/K8SPSMDB-1237

hors

@pooknull please add example into https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml

gkech · 2025-02-26T08:40:05Z

pkg/controller/perconaservermongodbbackup/backup.go

+			Type:             t,
+			IncrBase:         cr.Spec.Type == backup.BackupTypeIncrementalBase,


Maybe it is worth having here

cmd := ctrl.Cmd{ Cmd: ctrl.CmdBackup, Backup: &ctrl.BackupCmd{ Name: name, Type: cr.Spec.BackupType(), IncrBase: cr.Spec.IsBackupTypeIncrementalBase(), Compression: cr.Spec.Compression, CompressionLevel: compLevel, }, }

And having inside the spec some functions like the following:

func (p PerconaServerMongoDBBackupSpec) IsBackupTypeIncrementalBase() bool func (p PerconaServerMongoDBBackupSpec) BackupType() defs.BackupType

Mainly because this logic belongs to the backup spec type and less to the backup and cmd creation itself.

gkech · 2025-02-26T08:43:31Z

pkg/psmdb/backup/pbm.go

+const (
+	BackupTypeIncrementalBase defs.BackupType = defs.IncrementalBackup + "-base"
+)
+


We can also move this type to pkg/apis/psmdb/v1/perconaservermongodbbackup_types.go since it is more related to

type PerconaServerMongoDBBackupSpec struct { ... ... .... // +kubebuilder:validation:Enum={logical,physical,incremental} Type defs.BackupType `json:"type,omitempty"` }

egegunes · 2025-02-27T10:57:54Z

pkg/apis/psmdb/v1/perconaservermongodbbackup_types.go

@@ -17,7 +17,7 @@ type PerconaServerMongoDBBackupSpec struct {
 	Compression      compress.CompressionType `json:"compressionType,omitempty"`
 	CompressionLevel *int                     `json:"compressionLevel,omitempty"`

-	// +kubebuilder:validation:Enum={logical,physical}
+	// +kubebuilder:validation:Enum={logical,physical,incremental}


don't we need to add incremental-base here?

egegunes · 2025-02-27T10:59:26Z

e2e-tests/demand-backup-incremental/run

+
+desc 'Testing on not sharded cluster'
+
+echo "Creating PSMDB cluster"


please use log function instead of echo

egegunes · 2025-02-28T12:28:06Z

pkg/apis/psmdb/v1/perconaservermongodbbackup_types.go

@@ -17,7 +17,7 @@ type PerconaServerMongoDBBackupSpec struct {
 	Compression      compress.CompressionType `json:"compressionType,omitempty"`
 	CompressionLevel *int                     `json:"compressionLevel,omitempty"`

-	// +kubebuilder:validation:Enum={logical,physical,incremental}
+	// +kubebuilder:validation:Enum={logical,physical,incremental,incremental-base}


please regenerate CRDs

egegunes · 2025-02-28T12:28:18Z

e2e-tests/demand-backup-incremental/run

@@ -43,18 +43,18 @@ run_recovery_check() {
 	local base=${3:-true}

 	wait_restore "${backup_name}" "${cluster}" "requested" "0" "3000"
-	echo
+	log ""


we can continue using echo for empty lines

Operator always supported multiple storages but didn't have native support for having multiple backup storages until v2.6.0. In operator we were reconfiguring PBM every time user selects a storage for their backups/restores different than the previous storage. This was causing long wait periods esp. in storages with lots of backups due to resync operation. Another limitation was forcing users to have only 1 backup storage if they want to enable point-in-time-recovery. In case of multiple storages, PBM would upload oplog chunks to whatever storage is last used by a backup/restore and this would make consistent recovery impossible. PBM v2.6.0 added native support for multiple storages and these changes introduce it to our operator: * User can have one main storage in PBM configuration. Any other storages can be added as profiles. Main storage can be found in: ``` kubectl exec cluster1-rs0-0 -c backup-agent -- pbm config ``` This commit introduces a new field `main` in storage spec: ``` storages: s3-us-west: main: true type: s3 s3: bucket: operator-testing credentialsSecret: cluster1-s3-secrets region: us-west-2 ```` If user only has 1 storage configured in `cr.yaml`, operator will automatically use it as main storage. If more than 1 storage is configured, one of them must have `main: true`. User can't have more than 1 storage with `main: true`. If user changes main storage in cr.yaml, operator will configure PBM with the new storage and start resync. Any other storage in `cr.yaml` will be added to PBM as a profile. User can see profiles using cli: ``` kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile list ``` When user adds a new profile to `cr.yaml`, operator will add it to PBM but won't start resync. **`pbm config --force-resync` only start resync for the main storage.** To manually resync a profile: ``` kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile sync <storage-name> ``` If user starts a restore using a backup in a storage configured as profile, operator will start resync operation for profile and block restore until resync finishes. Note: Profiles are also called external storages in PBM documentation. If user has multiple storages in `cr.yaml` and changes main storage between them, operator: 1. configures PBM with the new main storage. 2. adds the old main as a profile. 2. deletes profile for the new main storage. If user configures `backupSource` in backups/restores: * if `cr.yaml` has no storages configured, operator configures PBM with storage data in `backupSource` field. This storage will effectively be the main storage until user adds a storage to `cr.yaml`. After a storage is configured PBM configuration will be overwritten and `backupSource` storage will be gone. * if `cr.yaml` has a storage configured, operator adds `backupSource` storage as a profile. * Oplog chunks will be only be uploaded to main storage. User can use any backup as base backup for point-in-time-recovery. * Incremental backup chains all need to be stored in the same storage. TBD after #1836 merged. --- Other significant changes in operator behavior: * Operator now configures automatically configures PBM on a fresh cluster. Before this changes, PBM was not configured until user starts a backup/restore after deploying a fresh cluster. Now, PBM will directly configured with main storage in `cr.yaml` and resync will be started in the background. There's a new field in `PerconaServerMongoDB` status: `backupConfigHash`. Operator will maintain hash of the current PBM configuration and reconfigures PBM if hash is changed. Fields in `spec.backup.pitr` are excluded from hash calculation, they're handled separately. * If `PerconaServerMongoDB` is annotated with `percona.com/resync-pbm=true`, operator will start resync operation both for main storage and profiles. Resync for profiles are started with the equivalent of `pbm profile sync --all`. These resync operations will be run in the background and will not block the reconciliation. * If a backup that has `percona.com/delete-backup` finalizer is deleted, operator will only delete oplogs chunks if it's in main storage.

JNKPercona · 2025-03-02T19:40:00Z

Test name	Status
arbiter	passed
balancer	passed
cross-site-sharded	passed
custom-replset-name	passed
custom-tls	passed
custom-users-roles	passed
custom-users-roles-sharded	passed
data-at-rest-encryption	passed
data-sharded	passed
demand-backup	passed
demand-backup-eks-credentials-irsa	passed
demand-backup-fs	passed
demand-backup-incremental	failure
demand-backup-incremental-sharded	failure
demand-backup-physical	passed
demand-backup-physical-sharded	passed
demand-backup-sharded	passed
expose-sharded	passed
finalizer	passed
ignore-labels-annotations	passed
init-deploy	passed
ldap	passed
ldap-tls	passed
limits	passed
liveness	passed
mongod-major-upgrade	passed
mongod-major-upgrade-sharded	passed
monitoring-2-0	passed
multi-cluster-service	passed
non-voting	passed
one-pod	passed
operator-self-healing-chaos	passed
pitr	passed
pitr-physical	passed
pitr-sharded	passed
preinit-updates	passed
pvc-resize	passed
recover-no-primary	passed
replset-overrides	passed
rs-shard-migration	passed
scaling	passed
scheduled-backup	passed
security-context	passed
self-healing-chaos	passed
service-per-pod	passed
serviceless-external-nodes	passed
smart-update	passed
split-horizon	passed
stable-resource-version	passed
storage	passed
tls-issue-cert-manager	passed
upgrade	passed
upgrade-consistency	passed
upgrade-consistency-sharded-tls	passed
upgrade-sharded	passed
users	passed
version-service	passed
We run 57 out of 57

commit: 9b5d933
image: perconalab/percona-server-mongodb-operator:PR-1836-9b5d933d

K8SPSMDB-1237: add incremental backups

5a9fd78

https://perconadev.atlassian.net/browse/K8SPSMDB-1237

pull-request-size bot added the size/XXL 1000+ lines label Feb 18, 2025

pooknull and others added 6 commits February 19, 2025 00:08

Merge branch 'main' into K8SPSMDB-1237

3642e59

fix tests

3089c9f

Merge branch 'main' into K8SPSMDB-1237

d027dc4

Merge branch 'main' into K8SPSMDB-1237

cb39268

Merge branch 'main' into K8SPSMDB-1237

fdf064a

Merge branch 'main' into K8SPSMDB-1237

2e048f7

hors added this to the v1.20.0 milestone Feb 25, 2025

use incremental-base backup type

12cb4da

github-actions bot added the tests label Feb 25, 2025

pooknull marked this pull request as ready for review February 25, 2025 13:45

Merge branch 'main' into K8SPSMDB-1237

20a094a

pooknull requested review from jvpasinatto, eleo007, hors, egegunes, nmarukovich and gkech as code owners February 25, 2025 13:45

fix

fd681cf

hors requested changes Feb 25, 2025

View reviewed changes

pooknull added 2 commits February 25, 2025 15:53

add examples to deploy/backup.yaml

be557c2

add proper example to deploy/cr.yaml

9cf30c3

hors self-requested a review February 25, 2025 14:42

gkech reviewed Feb 26, 2025

View reviewed changes

egegunes requested changes Feb 27, 2025

View reviewed changes

pooknull added 2 commits February 28, 2025 14:15

small improvements

f533760

Merge branch 'main' into K8SPSMDB-1237

1386647

pooknull requested a review from egegunes February 28, 2025 12:16

pooknull requested a review from gkech February 28, 2025 12:16

egegunes requested changes Feb 28, 2025

View reviewed changes

pooknull added 2 commits February 28, 2025 14:47

make manifests

60235d1

update tests

2280fca

egegunes previously approved these changes Feb 28, 2025

View reviewed changes

fix tests

9b5d933

pooknull dismissed egegunes’s stale review via 9b5d933 March 1, 2025 00:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

K8SPSMDB-1237: add incremental backups #1836

K8SPSMDB-1237: add incremental backups #1836

pooknull commented Feb 18, 2025 •

edited

Loading

hors left a comment

gkech Feb 26, 2025

pooknull Feb 28, 2025

gkech Feb 26, 2025 •

edited

Loading

pooknull Feb 28, 2025

egegunes Feb 27, 2025

pooknull Feb 28, 2025

egegunes Feb 27, 2025

pooknull Feb 28, 2025

egegunes Feb 28, 2025

pooknull Feb 28, 2025

egegunes Feb 28, 2025

pooknull Feb 28, 2025

JNKPercona commented Mar 2, 2025

		Type: t,
		IncrBase: cr.Spec.Type == backup.BackupTypeIncrementalBase,


		desc 'Testing on not sharded cluster'

		echo "Creating PSMDB cluster"

K8SPSMDB-1237: add incremental backups #1836

Are you sure you want to change the base?

K8SPSMDB-1237: add incremental backups #1836

Conversation

pooknull commented Feb 18, 2025 • edited Loading

DESCRIPTION

CHECKLIST

hors left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gkech Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JNKPercona commented Mar 2, 2025

pooknull commented Feb 18, 2025 •

edited

Loading

gkech Feb 26, 2025 •

edited

Loading