Core: Merge conflicting deletion vectors #11693
Conversation
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
Outdated
Show resolved
Hide resolved
amogh-jahagirdar
force-pushed
the
merge-conflicting-dvs
branch
from
558bee2 to
fb07a75
Compare
amogh-jahagirdar
force-pushed
the
merge-conflicting-dvs
branch
from
fb07a75 to
a8c5a59
Compare
| * @param spec the data file partition spec | ||
| * @param partition the data file partition | ||
| */ | ||
| default void delete( |
There was a problem hiding this comment.
We could also make this interface take in a List positions (kind of like a batch delete API on the writer) but it felt more natural to just pass in the index imo, cc @aokolnychyi @nastra thoughts?
| public static Builder builderFor( | ||
| TableOperations ops, PartitionSpec spec, FileFormat format, int partitionId, long taskId) { | ||
| return new Builder(ops, spec, format, partitionId, taskId); | ||
| } | ||
|
|
There was a problem hiding this comment.
Needed a way to create an output file factory from table operations since that's all that exists in MergingSnapshotProducer, this is public so we need to be careful.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
This PR adds the ability to merge conflicting deletion vectors for a given data file instead of failing the commit. In parallel, every conflicting DV will be merged with the committed DV, and a new Puffin with the merged DV will be output per datafile which had conflicting DVs.
This PR also takes care of any tables that wrote multiple DVs for a given data file (which is not spec compliant) by opportunistically merging those as well