Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-29003 Proper bulk load tracking #6506

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

HBASE-29003 Proper bulk load tracking #6506

wants to merge 3 commits into from
+420 −340

Conversation

DieterDP-ng
Copy link
Contributor

Note that this build on top of my older PR: #6134

@DieterDP-ng
HBASE-28767 Simplify backup bulk-loading code
343a11d 
No functional changes.
@DieterDP-ng
HBASE-28767 Simplify backup bulk-loading code
5932ae3 
No functional changes.

Introduces the BulkLoad class as data container for the tracking
of which files were bulk loaded (for backup purposes).
This class could be a record class, but that would make backporting
less straightforward.
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@DieterDP-ng
HBASE-29003 Proper bulk load tracking
fbb7080 
The HBase backup mechanism keeps track of which HFiles
were bulk loaded, so they can be included in incremental
backups.

Before this ticket, these bulk load records were only
deleted when an incremental backup is created. This
commit adds 2 more locations:

1) after a full backup. Since a full backup already
captures all data, this meant that unnecessary HFiles
were included in the next incremental backup.

2) after a table delete/truncate. Previously, if an
HFile was loaded before a table was cleared, the next
incremental backup would effectively still include the
HFile. This lead to incorrect data being restored.

This commit also completely refactors & simplifies the
test for this functionality.
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 27s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for branch
+1 💚 mvninstall 3m 5s master passed
+1 💚 compile 7m 57s master passed
+1 💚 checkstyle 1m 10s master passed
+1 💚 spotbugs 9m 16s master passed
+0 🆗 refguide 2m 14s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 spotless 0m 45s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for patch
+1 💚 mvninstall 2m 51s the patch passed
+1 💚 compile 9m 31s the patch passed
+1 💚 javac 9m 31s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 12s /buildtool-patch-checkstyle-root.txt The patch fails to run checkstyle in root
+1 💚 spotbugs 10m 54s the patch passed
+0 🆗 refguide 2m 20s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 hadoopcheck 13m 13s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚 spotless 0m 47s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 29s The patch does not generate ASF License warnings.
75m 52s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6506
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless refguide
uname Linux 3f250e00f0d4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / fbb7080
Default Java Eclipse Adoptium-17.0.11+9
refguide https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-6506/2/yetus-general-check/output/branch-site/book.html
refguide https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-6506/2/yetus-general-check/output/patch-site/book.html
Max. process+thread count 192 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/2/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for branch
+1 💚 mvninstall 2m 45s master passed
+1 💚 compile 1m 52s master passed
+1 💚 javadoc 2m 37s master passed
+1 💚 shadedjars 4m 48s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for patch
+1 💚 mvninstall 2m 41s the patch passed
+1 💚 compile 1m 55s the patch passed
+1 💚 javac 1m 55s the patch passed
+1 💚 javadoc 2m 36s the patch passed
+1 💚 shadedjars 4m 44s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 287m 15s root in the patch passed.
319m 8s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6506
Optional Tests javac javadoc unit compile shadedjars
uname Linux 036ae9fc1048 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / fbb7080
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/2/testReport/
Max. process+thread count 8381 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DieterDP-ng @Apache-HBase