Restoring Greenplum backup data to a new Cloudberry 1.5.3 cluster

[xanadu-dev]

I am investigating whether CloudberryDB would be good option to replace our current Greenplum 6.x cluster
I have a prototype system up and running with Cloudberry gpbackup version 1.0.3 and Greenplum s3_plugin version 1.10.2

I am trying to restore my latest production Greenplum backup and get an error
[CRITICAL]:-gprestore 1.2.7-beta1+dev.7 cannot restore a backup taken with gpbackup 1.28.1; please use gprestore 1.28.1 or later"

So its seems that the current version of Cloudberry GPBACKUP was forked some time ago and is now incomptible with Greenplum GPBACKUP

Any plans to address this ?

[lmugnano4537]

In the CloudberryDB repository there is a gpbackup repo that looks like it removed this check. The error comes from utils/util.go from:

func ValidateGPDBVersionCompatibility(connectionPool *dbconn.DBConn) {
if connectionPool.Version.Before(MINIMUM_GPDB5_VERSION) {
gplog.Fatal(errors.Errorf(GPDB version %s is not supported. Please upgrade to GPDB %s or later., connectionPool.Version.VersionString, MINIMUM_GPDB5_VERSION), "")
}
}

In the cloudberrydb version of the same code I can see that this func is a no-opt so you shouldn't get this error if you build the gpbackup from the cloudberrydb repo:

https://github.com/cloudberrydb/gpbackup

I am also evaluating cloudberrydb as a replacement for GPDB and plan to compile gpbackup next and test it but, as best I can tell, I don't think you'll get this error if you build gpbackup from the above repo.

Hope this helps

[xanadu-dev]

Thanks for your reply - I actually posted the wrong error message initially and corrected by post

The repo I used is the one you suggested but appears to have been forked in late 2023 and gpbackup has had several further releases since then which has made it incompatible

The error I am getting now is due to version compatibility and is thrown by report.go

[CRITICAL]:-gprestore 1.2.7-beta1+dev.7 cannot restore a backup taken with gpbackup 1.28.1; please use gprestore 1.28.1 or later"

func EnsureBackupVersionCompatibility(backupVersion string, restoreVersion string) {
	backupSemVer, err := semver.Make(backupVersion)
	gplog.FatalOnError(err)
	restoreSemVer, err := semver.Make(restoreVersion)
	gplog.FatalOnError(err)
	if backupSemVer.GT(restoreSemVer) {
		gplog.Fatal(errors.Errorf("gprestore %s cannot restore a backup taken with gpbackup %s; please use gprestore %s or later.",
			restoreVersion, backupVersion, backupVersion), "")
	}
}

[xanadu-dev]

Modifiying the 2 version checks in report.go allows the restore process to start
I'll run a full production restore from our GP6 production database and see if its succesfull

func EnsureBackupVersionCompatibility(backupVersion string, restoreVersion string) {

}

func EnsureDatabaseVersionCompatibility(backupGPDBVersion string, restoreGPDBVersion dbconn.GPDBVersion) {
	pattern := regexp.MustCompile(`\d+\.\d+\.\d+`)
	threeDigitVersion := pattern.FindStringSubmatch(backupGPDBVersion)[0]
	backupGPDBSemVer, err := semver.Make(threeDigitVersion)
	gplog.FatalOnError(err)
	if backupGPDBSemVer.Major < restoreGPDBVersion.SemVer.Major {
		gplog.Fatal(errors.Errorf("Cannot restore from GPDB version %s to %s due to catalog incompatibilities.", backupGPDBVersion, restoreGPDBVersion.VersionString), "")
	}
}

[xanadu-dev]

Restores from Greenplum 6.24.5 seem to be working fine using this slightly modified fork
https://github.com/xanadu-dev/cloudberry-gpbackup/tree/xanadu-temp

[tuhaihe]

Great work! Thanks @xanadu-dev

[Terry1504]

Greenplum 6.x is based on PostgreSQL 9.x, and CloudberryDB 1.5.3 is based on PostgreSQL 14.x. There might have non-trivial difference. Maybe use some other way e.g. data migration from GP6 to CloudberryDB?

[tuhaihe]

@xanadu-dev would you like to submit your changes in one pull request for gpbackup? I think it would be helpful to other community members.

[xanadu-dev]

@xanadu-dev would you like to submit your changes in one pull request for gpbackup? I think it would be helpful to other community members.

PR created now - apache/cloudberry-gpbackup#8

[xanadu-dev]

Warnings can now be seen in the restore logs for the version incompatibility but the restore process succeeds regardless

20240627:09:29:43 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-Restore Key = 20240625200000
20240627:09:29:43 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-Reading Plugin Config /tmp/gpbackup_s3_config_mbr.yml
20240627:09:29:43 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-plugin config path: /tmp/20240627092943_gpbackup_s3_config_mbr.yml
20240627:09:29:43 gprestore:gpadmin:gpmaster01dev:010617-[WARNING]:-cannot recover plugin version from history using timestamp 20240625200000, so using current plugin version. This is fine unless there is a backwards compatibility consideration within the plugin
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[WARNING]:-gprestore 1.2.7-beta1+dev.7 cannot restore a backup taken with gpbackup 1.28.1; please use gprestore 1.28.1 or later.
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[WARNING]:-Cannot restore from GPDB version 6.24.3 build commit:25d3498a400ca5230e81abb94861f23389315213 Open Source to oudberry Database 1.5.3+dev.53.gccfdc18cb8 build dev due to catalog incompatibilities.
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[WARNING]:-Backup segment count matches restore segment count; the --resize-cluster flag is not needed. Proceeding with a normal restore.
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-gpbackup version = 1.28.1
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-gprestore version = 1.2.7-beta1+dev.7
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-Database Version = oudberry Database 1.5.3+dev.53.gccfdc18cb8 build dev
20240627:09:29:44 gprestore:gpadmin:gpmaster01dev:010617-[INFO]:-Restoring pre-data metadata

[tuhaihe]

@Terry1504 could your team help cherry-pick the commit xanadu-dev/cloudberry-gpbackup@f290d91 to cloudberry/gpbackup?

[xanadu-dev]

@Terry1504 could your team help cherry-pick the commit xanadu-dev/cloudberry-gpbackup@f290d91 to cloudberry/gpbackup?

There is a newer branch now which changes the error level rather than disabling checks completely - apache/cloudberry-gpbackup#8

[tuhaihe]

Thanks @xanadu-dev !

[xanadu-dev]

The changes I made need to be refined before I can submit them as a pull request and I am on holiday at the moment. I’ll take a look when I am back in office.

…

On Fri 21 Jun 2024 at 12:23, 'Dianjin Wang' via xanadu.dev < ***@***.***> wrote: @xanadu-dev <https://github.com/xanadu-dev> would you like to submit your changes in one pull request for gpbackup? I think it would be helpful to other community members.

[tuhaihe]

Cool. Thanks for your reply. Enjoy your holidays!

[reshke]

gpbackup/gprestore is essentially logical backup, which is cost linear to total data size, isn't it? Can we instead use pg_upgrade, which is linear of catalog size?
pg_upgrade works fine for upgrading from 9.4 to 14.x for PostgreSQL, however, there are surely some caveats for greenplum->cloudberry case.