HBASE-29662 - Avoid regionDir/tableDir creation as part of .regioninfo file creation #7406

gvprathyusha6 · 2025-10-22T14:18:33Z

The test added currently fails on master branch and empty regionDir/regioninfo file are still created, with the fix it is throwing an excepted exception .

apurtell · 2025-10-27T17:21:40Z

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java

    FsPermission perms = CommonFSUtils.getFilePermissions(fs, conf, HConstants.DATA_FILE_UMASK_KEY);
    // Write the RegionInfo file content
-    try (FSDataOutputStream out = FSUtils.create(conf, fs, regionInfoFile, perms, null)) {
+    try (FSDataOutputStream out = FSUtils.create(conf, fs, regionInfoFile, perms, null, false)) {


Add a comment here why this change, providing the JIRA identifier. The reason not to recursively create directories is important, we don't want someone refactoring this code later to miss that.

virajjasani · 2025-10-27T18:19:40Z

I know it's difficult but any possibility of creating some tests around this?

ujjawal4046 · 2025-10-29T15:12:14Z

I know it's difficult but any possibility of creating some tests around this?

+1, as discussed offline this can actually be tested with same scenario @gvprathyusha6

Restore the snapshot into a temp directory
Delete the snapshot from the temp directory (other option would be to call the MR job with non-existing restore directory)
Create MR job for snapshot (ref IntegrationTestTableSnapshotInputFormat class on this), it should fail

gvprathyusha6 · 2025-10-29T21:27:14Z

I know it's difficult but any possibility of creating some tests around this?

+1, as discussed offline this can actually be tested with same scenario @gvprathyusha6

Restore the snapshot into a temp directory

Delete the snapshot from the temp directory (other option would be to call the MR job with non-existing restore directory)

Create MR job for snapshot (ref IntegrationTestTableSnapshotInputFormat class on this), it should fail

@ujjawal4046 why not just add the test for HRegion initialise to fail if there is no regionDir? that way all downstreams are covered?

apurtell · 2025-11-02T19:46:19Z

lgtm, I plan to apply this back to 2.5 tomorrow, unless objection.
Thanks @gvprathyusha6 for digging in to this issue.

virajjasani

Left few minor comments, looks good overall

virajjasani · 2025-11-03T02:11:30Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/janitor/MetaFixer.java

+            mfs.getFileSystem(), tableDir, regionInfo);
+        } catch (IOException e) {
+          LOG.warn("Failed to create region directory for {}: {}",
+            regionInfo.getRegionNameAsString(), e.getMessage());


Let's print whole stacktrace here instead of error message?

Also, if we suppress the error here, how can fixHoles progress? I hope that's the only place using this utility?

how can fixHoles progress

It shall continue to fix hole for other regions, right now also that part is taken care of by the procedure logic, as part of Assign it is creating the region directory, so if region directory creation fails, it would only effect that region and not rest of them, so I tried to keep it similar

I hope that's the only place using this utility?

yes and its private

virajjasani · 2025-11-03T02:13:01Z

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java

+        throw new IOException("Region directory does not exist: " + getRegionDir());
+      }
+      if (!fs.exists(getTempDir())) {
+        fs.mkdirs(getTempDir());


Curious, this was not covered so far?

this was created previously as part of recursion in FS while creating file, now that I removed it and now we dont just blindly create all the directories that are present in path, we would need to create tmp directory explicitly

…o file creation in HRegion initialize

Apache-HBase · 2025-11-03T17:32:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 34s		master passed
+1 💚	compile	6m 1s		master passed
+1 💚	checkstyle	2m 8s		master passed
+1 💚	spotbugs	3m 44s		master passed
+1 💚	spotless	1m 5s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 56s		the patch passed
+1 💚	compile	6m 11s		the patch passed
+1 💚	javac	6m 11s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 59s		the patch passed
+1 💚	spotbugs	4m 0s		the patch passed
+1 💚	hadoopcheck	14m 22s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	1m 0s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		60m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7406/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7406
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux d6ded2a3d85b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5bce7ca`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server hbase-mapreduce U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7406/8/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-03T21:38:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 19s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 45s		master passed
+1 💚	compile	1m 57s		master passed
+1 💚	javadoc	1m 12s		master passed
+1 💚	shadedjars	7m 15s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 4s		the patch passed
+1 💚	compile	2m 2s		the patch passed
+1 💚	javac	2m 2s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed
+1 💚	shadedjars	6m 55s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	3m 0s		hbase-common in the patch passed.
+1 💚	unit	242m 59s		hbase-server in the patch passed.
+1 💚	unit	25m 31s		hbase-mapreduce in the patch passed.
		306m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7406/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7406
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux ad21094c44a3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5bce7ca`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7406/8/testReport/
Max. process+thread count	3746 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server hbase-mapreduce U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7406/8/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

…o file creation in HRegion initialize (#7406) Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>

…o file creation in HRegion initialize (#7406) Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/master/janitor/MetaFixer.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java

…o file creation in HRegion initialize (#7406) Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/master/janitor/MetaFixer.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateRegionProcedure.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java

gvprathyusha6 marked this pull request as draft October 22, 2025 14:56

gvprathyusha6 force-pushed the HBASE-29662 branch from 8cf73b6 to 33362aa Compare October 25, 2025 03:48

gvprathyusha6 marked this pull request as ready for review October 25, 2025 03:48

apurtell requested changes Oct 27, 2025

View reviewed changes

gvprathyusha6 force-pushed the HBASE-29662 branch from 33362aa to d482b64 Compare October 29, 2025 21:24

gvprathyusha6 force-pushed the HBASE-29662 branch from d482b64 to cdf26b3 Compare October 29, 2025 21:30

gvprathyusha6 requested a review from apurtell October 29, 2025 21:34

gvprathyusha6 force-pushed the HBASE-29662 branch from cdf26b3 to 8bd7056 Compare October 29, 2025 21:47

This comment has been minimized.

Sign in to view

gvprathyusha6 force-pushed the HBASE-29662 branch from 8bd7056 to bc2f551 Compare October 30, 2025 19:50

This comment has been minimized.

Sign in to view

gvprathyusha6 force-pushed the HBASE-29662 branch 5 times, most recently from 0579809 to c2cb21c Compare October 30, 2025 23:32

gvprathyusha6 marked this pull request as draft October 31, 2025 00:10

This comment has been minimized.

Sign in to view

gvprathyusha6 force-pushed the HBASE-29662 branch from c2cb21c to d8657bc Compare November 1, 2025 06:59

gvprathyusha6 marked this pull request as ready for review November 1, 2025 07:02

This comment has been minimized.

Sign in to view

apurtell approved these changes Nov 2, 2025

View reviewed changes

virajjasani reviewed Nov 3, 2025

View reviewed changes

HBASE-29662 - Avoid regionDir/tableDir creation as part of .regioninf…

5bce7ca

…o file creation in HRegion initialize

gvprathyusha6 force-pushed the HBASE-29662 branch from d8657bc to 5bce7ca Compare November 3, 2025 16:28

apurtell merged commit bc54a7e into apache:master Nov 3, 2025
1 check passed

HBASE-29662 - Avoid regionDir/tableDir creation as part of .regioninfo file creation #7406

HBASE-29662 - Avoid regionDir/tableDir creation as part of .regioninfo file creation #7406

Uh oh!

Conversation

gvprathyusha6 commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apurtell Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Oct 27, 2025

Uh oh!

ujjawal4046 commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvprathyusha6 commented Oct 29, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

apurtell commented Nov 2, 2025

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

virajjasani Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

gvprathyusha6 Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

virajjasani Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

gvprathyusha6 Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Nov 3, 2025

Uh oh!

Apache-HBase commented Nov 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gvprathyusha6 commented Oct 22, 2025 •

edited

Loading

apurtell Oct 27, 2025 •

edited

Loading

ujjawal4046 commented Oct 29, 2025 •

edited

Loading

gvprathyusha6 Nov 3, 2025 •

edited

Loading

gvprathyusha6 Nov 3, 2025 •

edited

Loading