Backup/Restore database

[leborchuk]

Description

Lets discuss, how are you going to backup/restore CBDB?

gpbackup/gprestore https://github.com/cloudberrydb/gpbackup by their realisation are logical backup/restore utility.

Logical backup is quite useful for data migration purposes but also lack of important properties:

• No point in time recovery
• Long database restore

Use case/motivation

In Greenplum 6.x we use physical database backups (use https://github.com/wal-g/wal-g for that)

Maybe it would be great to add support CBDB in wal-g too. And so CBDB could also use physical backups.

Or maybe I'm wrong and you'll talked me out of this: physical backup not needed anymore for cloud database installations )

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[gspiegelberg]

Greenplum 6 & 7 GPDR component for physical backups and PITR leverages pgbackrest not WAL-g. I have been using extensively on a contract. Works very well and recommend pgbackrest over WAL-g.

Should this be picked up, I as a user would prefer the ability to place cluster in backup mode (pg_start_backup()) with the ability to either stop there or make an external call for physical backup. Use case I have in mind is as follows:

1. pg_start_backup()
2. WAL arching continues to repository
3. Snapshot all data volumes
4. pg_stop_backup()
   Let underlying storage do heavy lifting. Will support cloud, VM and even perhaps k8s clusters better/more efficiently.

Application of this case is:

1. Start backup & snapshot volumes
2. Spin up another cluster quickly using snapshots
3. Replay WAL's from repo to consistent state
4. Make cluster a hot_standby (RO) while continuously restoring ideally streaming but WAL shipping would suffice

[x4m]

@gspiegelberg pgbackrest is focused on Postgres. They will never support GP in the upstream, let alone CBDB. WAL-G is on the other side - we were first PITR system to support Greenplum officially.

speaking about snapshot backups - yeah, this feature is desired and was requested many times. PRs are welcome.

[gspiegelberg]

@x4m Whether pgbackrest or WAL-g, support and coordination must come from within the database to say at xid segments are at WAL 1..N and LSN 1..N and recorded such that recovery segments 1..N understands where it must start and end in a restore. Coordinated recording is the Greenplum concept of a restore point. The rest is file shuffling whether archiving (archive_command), restoring (restore_command), or backing up be it tar, one of the many 3rd party tools, or snapshotting data volumes.

Perhaps a more agnostic approach should be preferred. Why lock in a consumer of CBDB?

[reshke]

@x4m Whether pgbackrest or WAL-g, support and coordination must come from within the database to say at xid segments are at WAL 1..N and LSN 1..N and recorded such that recovery segments 1..N understands where it must start and end in a restore. Coordinated recording is the Greenplum concept of a restore point. The rest is file shuffling whether archiving (archive_command), restoring (restore_command), or backing up be it tar, one of the many 3rd party tools, or snapshotting data volumes.

Perhaps a more agnostic approach should be preferred. Why lock in a consumer of CBDB?

Wal-G already leverages restore points for PITR feature
https://github.com/wal-g/wal-g?tab=readme-ov-file#greenplum-work-in-progress

[my-ship-it]

wal-g already supports GP. Maybe we could leverage wal-g to support Clouberry as well.

[x4m]

There’s no need “to lock”. Both tools (WAL-G and some fork of backrest) already work with GP. But it make sense to focus community on a specific tool, because community resources are limited.

[x4m]

FWIW the work on adapting WAL-G to CBDB is going on here's wal-g/wal-g#1796
And it seems to be in a pretty good shape, I think we can have a WAL-G release soon.

[jianlirong]

PRs are welcome!

[jianlirong]

@x4m Any update on this?

[x4m]

@ostinru did you merge CBDB support to WAL-G? If so let’s have an official release.

[ostinru]

Yes. We have added Cloudberry support to wal-g. All tests are green.

Note 1: documentation is not good enough to build whole backup/restore pipeline, so check tests.

Note 2: There is known issue - race in backup unpacking when WALG_DOWNLOAD_CONCURRENCY != 1 (and by default it set to 10). We have PR to fix it, however I'd like to get all approvals from PostgreSQL/OrioleDB developers. Fix here: wal-g/wal-g#1815

[ostinru]

Good news:

• in order to reproduce an issue - you should run exclusive backup concurrently with wal-g and have restore point close to backup.
• it is restore phase issue. If it is broke - restart restore with WALG_DOWNLOAD_CONCURRENCY=1.

So, you shouldn't observe this issue in most cases... but I prefer to fix it before release... I know, that a lot of people uses official wal-g build rather then use night builds.