-
Notifications
You must be signed in to change notification settings - Fork 3.4k
HBASE-29687: Extend IntegrationTestBackupRestore to handle continuous backups #7417
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-29687: Extend IntegrationTestBackupRestore to handle continuous backups #7417
Conversation
…s and non-continuous subclasses Change-Id: I0c70c417b86c7732b58642a51c75897c35b16cb6
Change-Id: Id043400bf85c7b696bb94bef7cb17ed9dad13334
…a for loop Change-Id: I5ba3276919e6bbdf343c134fa287c69f3854a8a2
Change-Id: I25fe484e9c227b7a31cb3768def3c12f66d617ac
Change-Id: Ie9aece8a3ec55739d618ebf2d2f173a41a116eb6
Change-Id: Ie345e623089979f028b13aed13e5ec93e025eff8
Change-Id: I98eb019dd93dfc8e21b6c730e0e2e60314102724
Change-Id: I4de6fc485aa1ff6e0d8d837e081f8dde20bb3f67
Change-Id: I911180a8f263f801a5c299d43d0215fe444f22d3
Change-Id: I78fe59f800cde7c89b11760a49d774c5173a862c
Change-Id: Ia150d21f48bb160d9e8bcf922799dc18c0b7c77c
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR extends the backup integration tests to support continuous backup functionality by refactoring IntegrationTestBackupRestore into an abstract base class and creating separate test implementations for continuous and non-continuous backup scenarios. The primary addition is a new continuous backup test that validates backup WAL directory structure, replication peer subscription, and snapshot creation.
Key changes:
- Abstract base class
IntegrationTestBackupRestoreBasecontaining shared backup/restore test logic - New
IntegrationTestContinuousBackupRestoreclass for continuous backup scenarios - Refactored
IntegrationTestBackupRestoreclass extending the base class for non-continuous scenarios
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| IntegrationTestContinuousBackupRestore.java | New test class implementing continuous backup integration tests with single region server constraint |
| IntegrationTestBackupRestoreBase.java | Abstract base class containing shared backup/restore test logic including snapshot verification, WAL verification, and backup deletion |
| IntegrationTestBackupRestore.java | Refactored to extend base class for non-continuous backup testing |
| IntegrationTestBackupRestore.java (deleted) | Original file relocated to backup package |
| hbase-it/pom.xml | Added hbase-backup test-jar dependency |
| TestContinuousBackup.java | Refactored to use shared utility method for replication peer verification |
| BackupTestUtil.java | Added utility methods for enabling backup and verifying replication peer subscription |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Outdated
Show resolved
Hide resolved
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Outdated
Show resolved
Hide resolved
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Show resolved
Hide resolved
|
Great stuff @kgeisz ! I think with the new base class, the code has become pretty clean and easy to read. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
left few comments
...-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestContinuousBackupRestore.java
Outdated
Show resolved
Hide resolved
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Outdated
Show resolved
Hide resolved
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Outdated
Show resolved
Hide resolved
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Show resolved
Hide resolved
...-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestContinuousBackupRestore.java
Outdated
Show resolved
Hide resolved
|
@kgeisz We're planning to ship integration tests separately from the feature to reduce the overhead on the review process. So, we probably don't want to merge this into the feature branch. |
@anmolnar Thanks! Also, I have not seen any issues even without the synchronization.
Sounds good. Should I just have this merge into master instead and we just won't merge this until the HBASE-28957 feature branch has been merged into master? |
Change-Id: I9d5b55e36b44367ac8ace08a5859c42b796fefd4
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…h latest Put timestamp Change-Id: Ic438ca292bc01827d46725e006bfa0c21bc95f01
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Outdated
Show resolved
Hide resolved
Change-Id: I9d52e774e84dc389d42aa63315529a2590c40cb8
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
The patch compile tests resulted in a -1 due to Maven spotless, but I have already run I also found a |
Change-Id: I27eec25091842376ee7a059a9688c6f5ab385ac7
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Change-Id: I18ab629df4af4e93b42ec1b0d576fd411279c775
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
hbase-it/src/test/java/org/apache/hadoop/hbase/backup/IntegrationTestBackupRestoreBase.java
Outdated
Show resolved
Hide resolved
Change-Id: Ibc96fd712e384cc3ca5a2c4575e47e65e62c60fa
|
I made another small change to this PR. The original IntegrationTestBackupRestore file had a bug in how it configures command line parameters. In |
Change-Id: Ie8e94ce978836b1314525138726a13641360aae6
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Change-Id: Ibeea379a65e801b60ec5124938b7aa17087025f0
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
https://issues.apache.org/jira/browse/HBASE-29687
The original
IntegrationTestBackupRestore.javaclass tests the functionality of full and incremental backups. It creates a full backup, and then iteratively performs an incremental backup plus multiple restores. Later, it merges the incremental backups and performs a final restore.With the introduction of the continuous backup feature, this pull request extends
IntegrationTestBackupRestoreto also handle testing full and incremental backups with continuous backup enabled. This change turnsIntegrationTestBackupRestoreinto an abstract base class. Much of the functionality within this newIntegrationTestBackupRestoreBaseclass is the same. Several new test cases have been added that are only performed for continuous backups (backup WAL directory verification, backup WAL file verification, etc.). In addition, snapshot verification and backup deletion has been added for both the continuous and the non-continuous test case. The existingtestBackupRestore()method has been moved to its ownIntegrationTestBackupRestore.javaclass that extends the new base class. Similarly, a newIntegrationTestContinuousBackupRestore.javatest class has been added for the continuous backup case.Note: The continuous backup test case only allows one region server per minicluster. When multiple region servers are used, there are issues with waiting for all of the region servers to catch up with replication. This results in only most of the rows being restored when a backup restore is performed, regardless of how long the test waits for.
There are some additional changes made to
BackupTestUtil.javaandTestContinuousBackup.javathat allowIntegrationTestBackupRestoreBaseto reuse some code.hbase-backuphad to be added as a test artifact tohbase-it/pom.xmlas a result.