HBASE-28988 Enhance WALPlayer for restore of BulkLoad #6523
+174
−13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Need to update for newly suggested backup directory structure |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
@ankitsol You need to run |
There was a problem hiding this comment.
In general, I found the following issues:
- All the new log lines are at the INFO level, which may not be necessary. Consider reducing them to DEBUG or TRACE in some cases.
- Javadoc/comments have not been updated to reflect the latest changes.
- There are code style issues that need to be fixed so we can run the unit tests and update them if necessary.
- New unit tests need to be added.
| @@ -156,7 +164,7 @@ protected static enum Counter { | |||
| * A mapper that writes out {@link Mutation} to be directly applied to a running HBase instance. | |||
| */ | |||
| protected static class WALMapper | |||
| extends Mapper<WALKey, WALEdit, ImmutableBytesWritable, Mutation> { | |||
| extends Mapper<WALKey, WALEdit, ImmutableBytesWritable, Pair<Mutation, List<String>>> { | |||
There was a problem hiding this comment.
Instead of Pair here, can we use a Custom Class? So that The exclusivity between Mutation and BulkLoadFiles is enforced programmatically.
|
|
||
| // Retrieve configuration and set up file systems for backup and staging locations | ||
| Configuration conf = context.getConfiguration(); | ||
| Path backupLocation = new Path(conf.get(BULKLOAD_BACKUP_LOCATION)); |
There was a problem hiding this comment.
check for if backupLocation is not specified.
|
|
||
| try { | ||
| for (String file : bulkloadFilesWithFullPath) { | ||
| // Full file path from S3 |
There was a problem hiding this comment.
remove the hardcoded S3 here
| List<String> stagingPaths = new ArrayList<>(); | ||
|
|
||
| try { | ||
| for (String file : bulkloadFilesWithFullPath) { |
There was a problem hiding this comment.
these are not full paths, but the relative paths from namespace
| @@ -172,6 +180,52 @@ public void map(WALKey key, WALEdit value, Context context) throws IOException { | |||
| ExtendedCell lastCell = null; | |||
| for (ExtendedCell cell : WALEditInternalHelper.getExtendedCells(value)) { | |||
| context.getCounter(Counter.CELLS_READ).increment(1); | |||
|
|
|||
| if (CellUtil.matchingQualifier(cell, WALEdit.BULK_LOAD)) { | |||
There was a problem hiding this comment.
I believe the processing of bulkloaded files can be simplified, and we could reduce the log level from INFO to DEBUG or TRACE in some cases.
| @@ -293,6 +398,8 @@ public Job createSubmittableJob(String[] args) throws IOException { | |||
| setupTime(conf, WALInputFormat.START_TIME_KEY); | |||
| setupTime(conf, WALInputFormat.END_TIME_KEY); | |||
| String inputDirs = args[0]; | |||
| String walDir = new Path(inputDirs, "WALs").toString(); | |||
There was a problem hiding this comment.
I don't think this is correct. We are hard-coding the directories here.
We could introduce a new optional parameter that the user can specify if they have bulkloaded files for us to process.
For example:
hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2 -Dwal.bulk.backup.location=/bulkload-files-dir
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhance WALPlayer for restore of BulkLoad WAL entries
https://issues.apache.org/jira/browse/HBASE-28988