HIVE-29028: Iceberg: Implement auto compaction #5886
+858
−377
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
difin
force-pushed
the
iceberg_auto_compaction
branch
from
7c50524 to
b75da71
Compare
difin
force-pushed
the
iceberg_auto_compaction
branch
from
b75da71 to
05f666a
Compare
asf-ci-hive
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
difin
force-pushed
the
iceberg_auto_compaction
branch
from
05f666a to
c6cbb86
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
difin
force-pushed
the
iceberg_auto_compaction
branch
from
c6cbb86 to
dc06a7c
Compare
|
|
plenty of reactions, but no reviews :) |
deniskuzZ
reviewed
Jun 26, 2025
| @@ -2381,4 +2384,15 @@ private static List<FieldSchema> schema(List<VirtualColumn> exprs) { | |||
| private static List<FieldSchema> orderBy(VirtualColumn... exprs) { | |||
| return schema(Arrays.asList(exprs)); | |||
| } | |||
|
|
|||
| @Override | |||
| public CompactionType determineCompactionType(org.apache.hadoop.hive.metastore.api.Table table, CompactionInfo ci) | |||
There was a problem hiding this comment.
it doesn't belong here. maybe CompactionEvaluator?
deniskuzZ
reviewed
Jun 26, 2025
| * @throws IOException If an I/O error occurs during the compaction evaluation process. | ||
| * @throws UnsupportedOperationException if the implementing storage handler does not support this operation. | ||
| */ | ||
| default CompactionType determineCompactionType(Table table, CompactionInfo ci) throws IOException { |
There was a problem hiding this comment.
that is not responsibility of StorageHandler
deniskuzZ
reviewed
Jun 26, 2025
| * A class to initiate Iceberg compactions. This will run in a separate thread. | ||
| * It's critical that there exactly 1 of these in a given warehouse. | ||
| */ | ||
| public class IcebergInitiator extends InitiatorBase { |
There was a problem hiding this comment.
please move it under iceberg-handler, org.apache.iceberg.mr.hive.compaction.
deniskuzZ
reviewed
Jun 26, 2025
| */ | ||
| public abstract class InitiatorBase extends MetaStoreCompactorThread { | ||
|
|
||
| protected ExecutorService compactionExecutor; |
There was a problem hiding this comment.
Is there a way to disable certain Initiators individually? I don't think we need 2 processes, but 1 running multiple tasks:
- AcidTableOptimizer
- IcebergTableOptimizer
There was a problem hiding this comment.
both should return a list of tasks the Initiator needs to schedule
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Iceberg auto compaction. In this initial version,
IcebergInitiatorremembers the last checked snapshot id of all Iceberg tables and it compares the current snapshot id to the cached snapshot id to determine if further evaluation is needed.Why are the changes needed?
Currently, Iceberg compaction can only be triggered manually.
This PR introduces automatic Iceberg compaction which periodically checks all Iceberg tables and schedules compactions when necessary.
Does this PR introduce any user-facing change?
No
How was this patch tested?
New unit tests, precommit tests.