Skip to content

Commit

Permalink
Merge pull request #18 from woblerr/fix_delete_in_progress_hang_backup
Browse files Browse the repository at this point in the history 
Refactoring the backup deletion process to cleanup backup info on master host.
  • Loading branch information
@woblerr
woblerr authored Oct 13, 2024
2 parents d1feaed + 4f5998d commit adcbaea
Show file tree
Hide file tree
Showing 18 changed files with 213 additions and 309 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/build.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ jobs:
- name: Available platforms
run: echo ${{ steps.buildx.outputs.platforms }}

- name: Run end-to-end tests
run: |
make test-e2e
# - name: Run end-to-end tests
# run: |
# make test-e2e

- name: Build image and push master tag to ghcr.io and Docker Hub
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
Expand Down
26 changes: 10 additions & 16 deletions COMMANDS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@
- [Display information about backups (`backup-info`)](#display-information-about-backups-backup-info)
- [Examples](#examples-2)
- [Using container](#using-container-2)
- [Clean failed and deleted backups from the history database (`history-clean`)](#clean-failed-and-deleted-backups-from-the-history-database-history-clean)
- [Clean deleted backups from the history database (`history-clean`)](#clean-deleted-backups-from-the-history-database-history-clean)
- [Examples](#examples-3)
- [Delete information about failed and deleted backups from history database older than n days](#delete-information-about-failed-and-deleted-backups-from-history-database-older-than-n-days)
- [Delete all backups using storage plugin older than timestamp](#delete-all-backups-using-storage-plugin-older-than-timestamp-1)
- [Delete information about deleted backups from history database older than n days](#delete-information-about-deleted-backups-from-history-database-older-than-n-days)
- [Delete information about deleted backups from history database older than timestamp](#delete-information-about-deleted-backups-from-history-database-older-than-timestamp)
- [Using container](#using-container-3)
- [Migrate history database (`history-migrate`)](#migrate-history-database-history-migrate)
- [Examples](#examples-4)
Expand Down Expand Up @@ -442,19 +442,17 @@ docker run \
--history-db /data/master/gpseg-1/gpbackup_history.db
```
# Clean failed and deleted backups from the history database (`history-clean`)
# Clean deleted backups from the history database (`history-clean`)
Available options for `history-clean` command and their description:
```bash
./gpbackman history-clean -h
Clean failed and deleted backups from the history database.
Clean deleted backups from the history database.
Only the database is being cleaned up.
By default, information is deleted only about failed backups from gpbackup_history.db.
To delete information about deleted backups, use the --deleted option.
Information is deleted only about deleted backups from gpbackup_history.db. Each backup must be deleted first.
To delete information about backups older than the given timestamp, use the --before-timestamp option.
To delete information about backups older than the given number of days, use the --older-than-day option.
Expand All @@ -475,7 +473,6 @@ Usage:
Flags:
--before-timestamp string delete information about backups older than the given timestamp
--deleted delete information about deleted backups
-h, --help help for history-clean
--older-than-days uint delete information about backups older than the given number of days
Expand All @@ -488,20 +485,18 @@ Global Flags:
```
## Examples
### Delete information about failed and deleted backups from history database older than n days
Delete information about failed and deleted backups from history database older than 7 days:
### Delete information about deleted backups from history database older than n days
Delete information about deleted backups from history database older than 7 days:
```bash
./gpbackman history-clean \
--older-than-days 7 \
--deleted
```
### Delete all backups using storage plugin older than timestamp
Delete information about failed and deleted backups from history database older than timestamp `20240101100000`:
### Delete information about deleted backups from history database older than timestamp
Delete information about deleted backups from history database older than timestamp `20240101100000`:
```bash
./gpbackman history-clean \
--before-timestamp 20240101100000 \
--deleted
```
## Using container
Expand All @@ -517,7 +512,6 @@ docker run \
gpbackman history-clean \
--older-than-days 7 \
--history-db /data/master/gpseg-1/gpbackup_history.db \
--deleted
```
# Migrate history database (`history-migrate`)
Expand Down
6 changes: 3 additions & 3 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,10 @@ define e2e_command
endef

define run_docker_compose
GPBACKMAN_UID=$(UID) GPBACKMAN_GID=$(GID) docker-compose -f e2e_tests/docker-compose.yml build --force-rm --parallel ${1}
GPBACKMAN_UID=$(UID) GPBACKMAN_GID=$(GID) docker-compose -f e2e_tests/docker-compose.yml run --rm --name ${1} ${1}
GPBACKMAN_UID=$(UID) GPBACKMAN_GID=$(GID) docker compose -f e2e_tests/docker-compose.yml build --force-rm --parallel ${1}
GPBACKMAN_UID=$(UID) GPBACKMAN_GID=$(GID) docker compose -f e2e_tests/docker-compose.yml run --rm --name ${1} ${1}
endef

define down_docker_compose
GPBACKMAN_UID=$(UID) GPBACKMAN_GID=$(GID) docker-compose -f e2e_tests/docker-compose.yml down -v
GPBACKMAN_UID=$(UID) GPBACKMAN_GID=$(GID) docker compose -f e2e_tests/docker-compose.yml down -v
endef
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The utility works with both history database formats: `gpbackup_history.yaml` fi
* display the backup report for existing backups;
* delete existing backups from local storage or using storage plugins (for example, [S3 Storage Plugin](https://github.com/greenplum-db/gpbackup-s3-plugin));
* delete all existing backups from local storage or using storage plugins older than the specified time condition;
* clean failed and deleted backups from the history database;
* clean deleted backups from the history database;
* migrate history database from `gpbackup_history.yaml` format to `gpbackup_history.db` SQLite format.

## Commands
Expand All @@ -34,7 +34,7 @@ Available Commands:
backup-info Display information about backups
completion Generate the autocompletion script for the specified shell
help Help about any command
history-clean Clean failed and deleted backups from the history database
history-clean Clean deleted backups from the history database
history-migrate Migrate history database
report-info Display the report for a specific backup

Expand All @@ -56,7 +56,7 @@ Description of each command:
* [Delete all existing backups older than the specified time condition (`backup-clean`)](./COMMANDS.md#delete-all-existing-backups-older-than-the-specified-time-condition-backup-clean)
* [Delete a specific existing backup (`backup-delete`)](./COMMANDS.md#delete-a-specific-existing-backup-backup-delete)
* [Display information about backups (`backup-info`)](./COMMANDS.md#display-information-about-backups-backup-info)
* [Clean failed and deleted backups from the history database (`history-clean`)](./COMMANDS.md#clean-failed-and-deleted-backups-from-the-history-database-history-clean)
* [Clean deleted backups from the history database (`history-clean`)](./COMMANDS.md#clean-deleted-backups-from-the-history-database-history-clean)
* [Migrate history database (`history-migrate`)](./COMMANDS.md#migrate-history-database-history-migrate)
* [Display the report for a specific backup (`report-info`)](./COMMANDS.md#display-the-report-for-a-specific-backup-report-info)
Expand Down
125 changes: 60 additions & 65 deletions cmd/backup_delete.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -360,16 +360,35 @@ func backupDeleteDBPluginFunc(backupName, pluginConfigPath string, pluginConfig
gplog.Error(stderr)
}
if errdel != nil {
gplog.Error(textmsg.ErrorTextUnableDeleteBackup(backupName, errdel))
err = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedPluginFailed, hDB)
if err != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedPluginFailed, backupName, err))
}
handleErrorDB(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, errdel), gpbckpconfig.DateDeletedPluginFailed, hDB)
if !ignoreErrors {
return errdel
}
}
gplog.Info(stdout)
backupData, err := gpbckpconfig.GetBackupDataDB(backupName, hDB)
if err != nil {
handleErrorDB(backupName, textmsg.ErrorTextUnableGetBackupInfo(backupName, err), gpbckpconfig.DateDeletedPluginFailed, hDB)
if !ignoreErrors {
return err
}
}
bckpDir, _, _, err := getBackupMasterDir("", backupData.BackupDir, backupData.DatabaseName)
if err != nil {
handleErrorDB(backupName, textmsg.ErrorTextUnableGetBackupPath("backup directory", backupName, err), gpbckpconfig.DateDeletedPluginFailed, hDB)
if !ignoreErrors {
return err
}
}
gplog.Debug(textmsg.InfoTextCommandExecution("delete directory", gpbckpconfig.BackupDirPath(bckpDir, backupName)))
// Delete local files on master.
err = os.RemoveAll(gpbckpconfig.BackupDirPath(bckpDir, backupName))
if err != nil {
handleErrorDB(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, err), gpbckpconfig.DateDeletedPluginFailed, hDB)
if !ignoreErrors {
return err
}
}
err = gpbckpconfig.UpdateDeleteStatus(backupName, dateDeleted, hDB)
if err != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(dateDeleted, backupName, err))
Expand All @@ -390,31 +409,19 @@ func backupDeleteDBLocalFunc(backupName, backupDir string, maxParallelProcesses
}
backupData, err := gpbckpconfig.GetBackupDataDB(backupName, hDB)
if err != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupInfo(backupName, err))
errUpdate = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedLocalFailed, hDB)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, err))
}
handleErrorDB(backupName, textmsg.ErrorTextUnableGetBackupInfo(backupName, err), gpbckpconfig.DateDeletedLocalFailed, hDB)
return err
}
bckpDir, segPrefix, isSingleBackupDir, err := getBackupMasterDir(backupDir, backupData.BackupDir, backupData.DatabaseName)
if err != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupPath("backup directory", backupName, err))
errUpdate = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedLocalFailed, hDB)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
}
handleErrorDB(backupName, textmsg.ErrorTextUnableGetBackupPath("backup directory", backupName, err), gpbckpconfig.DateDeletedLocalFailed, hDB)
return err
}
gplog.Debug(textmsg.InfoTextBackupDirPath(bckpDir))
gplog.Debug(textmsg.InfoTextSegmentPrefix(segPrefix))
backupType, err := backupData.GetBackupType()
if err != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupValue("type", backupData.Timestamp, err))
errUpdate = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedLocalFailed, hDB)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
}
handleErrorDB(backupName, textmsg.ErrorTextUnableGetBackupValue("type", backupName, err), gpbckpconfig.DateDeletedLocalFailed, hDB)
return err
}
// If backup type is not "metadata-only", we should delete files on segments and master.
Expand All @@ -423,33 +430,25 @@ func backupDeleteDBLocalFunc(backupName, backupDir string, maxParallelProcesses
var errSeg error
segConfig, errSeg := getSegmentConfigurationClusterInfo(backupData.DatabaseName)
if errSeg != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupPath("segment configuration", backupName, errSeg))
errUpdate = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedLocalFailed, hDB)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
handleErrorDB(backupName, textmsg.ErrorTextUnableGetBackupPath("segment configuration", backupName, errSeg), gpbckpconfig.DateDeletedLocalFailed, hDB)
if !ignoreErrors {
return errSeg
}
return errSeg
}
// Execute on segments.
errSeg = executeDeleteBackupOnSegments(backupDir, backupData.BackupDir, backupName, segPrefix, isSingleBackupDir, ignoreErrors, segConfig, maxParallelProcesses)
if errSeg != nil {
gplog.Error(textmsg.ErrorTextUnableDeleteBackup(backupName, errSeg))
errUpdate = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedLocalFailed, hDB)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
handleErrorDB(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, errSeg), gpbckpconfig.DateDeletedLocalFailed, hDB)
if !ignoreErrors {
return errSeg
}
return errSeg
}
}
// Delete files on master.
gplog.Debug(textmsg.InfoTextCommandExecution("delete directory", gpbckpconfig.BackupDirPath(bckpDir, backupName)))
err = os.RemoveAll(gpbckpconfig.BackupDirPath(bckpDir, backupName))
if err != nil {
gplog.Error(textmsg.ErrorTextUnableDeleteBackup(backupName, err))
errUpdate = gpbckpconfig.UpdateDeleteStatus(backupName, gpbckpconfig.DateDeletedLocalFailed, hDB)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
}
handleErrorDB(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, err), gpbckpconfig.DateDeletedLocalFailed, hDB)
if !ignoreErrors {
return err
}
Expand Down Expand Up @@ -556,16 +555,28 @@ func backupDeleteFilePluginFunc(backupData gpbckpconfig.BackupConfig, parseHData
gplog.Error(stderr)
}
if errdel != nil {
gplog.Error(textmsg.ErrorTextUnableDeleteBackup(backupName, errdel))
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, gpbckpconfig.DateDeletedPluginFailed)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedPluginFailed, backupName, errUpdate))
}
handleErrorFile(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, errdel), gpbckpconfig.DateDeletedPluginFailed, parseHData)
if !ignoreErrors {
return errdel
}
}
gplog.Info(stdout)
// Delete local files on master.
bckpDir, _, _, err := getBackupMasterDir("", backupData.BackupDir, backupData.DatabaseName)
if err != nil {
handleErrorFile(backupName, textmsg.ErrorTextUnableGetBackupPath("backup directory", backupName, err), gpbckpconfig.DateDeletedPluginFailed, parseHData)
if !ignoreErrors {
return err
}
}
gplog.Debug(textmsg.InfoTextCommandExecution("delete directory", gpbckpconfig.BackupDirPath(bckpDir, backupName)))
err = os.RemoveAll(gpbckpconfig.BackupDirPath(bckpDir, backupName))
if err != nil {
handleErrorFile(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, err), gpbckpconfig.DateDeletedPluginFailed, parseHData)
if !ignoreErrors {
return err
}
}
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, dateDeleted)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(dateDeleted, backupName, errUpdate))
Expand All @@ -587,22 +598,14 @@ func backupDeleteFileLocalFunc(backupData gpbckpconfig.BackupConfig, parseHData
}
bckpDir, segPrefix, isSingleBackupDir, err := getBackupMasterDir(backupDir, backupData.BackupDir, backupData.DatabaseName)
if err != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupPath("backup directory", backupName, err))
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, gpbckpconfig.DateDeletedLocalFailed)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
}
handleErrorFile(backupName, textmsg.ErrorTextUnableGetBackupPath("backup directory", backupName, err), gpbckpconfig.DateDeletedLocalFailed, parseHData)
return err
}
gplog.Debug(textmsg.InfoTextBackupDirPath(bckpDir))
gplog.Debug(textmsg.InfoTextSegmentPrefix(segPrefix))
backupType, err := backupData.GetBackupType()
if err != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupValue("type", backupData.Timestamp, err))
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, gpbckpconfig.DateDeletedLocalFailed)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
}
handleErrorFile(backupName, textmsg.ErrorTextUnableGetBackupValue("type", backupName, err), gpbckpconfig.DateDeletedLocalFailed, parseHData)
return err
}
// If backup type is not "metadata-only", we should delete files on segments and master.
Expand All @@ -611,33 +614,25 @@ func backupDeleteFileLocalFunc(backupData gpbckpconfig.BackupConfig, parseHData
var errSeg error
segConfig, errSeg := getSegmentConfigurationClusterInfo(backupData.DatabaseName)
if errSeg != nil {
gplog.Error(textmsg.ErrorTextUnableGetBackupPath("segment configuration", backupName, errSeg))
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, gpbckpconfig.DateDeletedLocalFailed)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
handleErrorFile(backupName, textmsg.ErrorTextUnableGetBackupPath("segment configuration", backupName, errSeg), gpbckpconfig.DateDeletedLocalFailed, parseHData)
if !ignoreErrors {
return errSeg
}
return errSeg
}
// Execute on segments.
errSeg = executeDeleteBackupOnSegments(backupDir, backupData.BackupDir, backupName, segPrefix, isSingleBackupDir, ignoreErrors, segConfig, maxParallelProcesses)
if errSeg != nil {
gplog.Error(textmsg.ErrorTextUnableDeleteBackup(backupName, errSeg))
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, gpbckpconfig.DateDeletedLocalFailed)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
handleErrorFile(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, errSeg), gpbckpconfig.DateDeletedLocalFailed, parseHData)
if !ignoreErrors {
return errSeg
}
return errSeg
}
}
// Delete files on master.
gplog.Debug(textmsg.InfoTextCommandExecution("delete directory", gpbckpconfig.BackupDirPath(bckpDir, backupName)))
err = os.RemoveAll(gpbckpconfig.BackupDirPath(bckpDir, backupName))
if err != nil {
gplog.Error(textmsg.ErrorTextUnableDeleteBackup(backupName, err))
errUpdate = parseHData.UpdateBackupConfigDateDeleted(backupName, gpbckpconfig.DateDeletedLocalFailed)
if errUpdate != nil {
gplog.Error(textmsg.ErrorTextUnableSetBackupStatus(gpbckpconfig.DateDeletedLocalFailed, backupName, errUpdate))
}
handleErrorFile(backupName, textmsg.ErrorTextUnableDeleteBackup(backupName, err), gpbckpconfig.DateDeletedLocalFailed, parseHData)
if !ignoreErrors {
return err
}
Expand Down
Loading

0 comments on commit adcbaea

Please sign in to comment.