Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: attempt to workaround problems reading actions as empty rather than null #3045

Merged
merged 4 commits into from
Dec 4, 2024

Conversation

rtyler
Copy link
Member

@rtyler rtyler commented Dec 3, 2024

See #3030

@github-actions github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Dec 3, 2024
@rtyler rtyler changed the title chore: add more robust regression tests for the checkpoint related failure(s) fix: attempt to workaround problems reading actions as empty rather than null Dec 4, 2024
Copy link

codecov bot commented Dec 4, 2024

Codecov Report

Attention: Patch coverage is 48.45361% with 50 lines in your changes missing coverage. Please review.

Project coverage is 72.62%. Comparing base (ba671c9) to head (48ba19f).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/kernel/snapshot/parse.rs 43.58% 17 Missing and 5 partials ⚠️
crates/core/src/kernel/snapshot/log_data.rs 9.52% 17 Missing and 2 partials ⚠️
crates/core/src/kernel/snapshot/replay.rs 60.00% 4 Missing and 2 partials ⚠️
crates/core/src/protocol/checkpoints.rs 86.36% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3045      +/-   ##
==========================================
- Coverage   72.68%   72.62%   -0.06%     
==========================================
  Files         129      129              
  Lines       41462    41509      +47     
  Branches    41462    41509      +47     
==========================================
+ Hits        30137    30147      +10     
- Misses       9376     9415      +39     
+ Partials     1949     1947       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rtyler rtyler marked this pull request as ready for review December 4, 2024 16:47
@rtyler rtyler enabled auto-merge December 4, 2024 16:49
@rtyler rtyler force-pushed the fix/checkpoint-3030 branch 2 times, most recently from 54487ea to 09f1376 Compare December 4, 2024 20:13
rtyler and others added 4 commits December 4, 2024 20:15
The root cause remains elusive but with the recent arrow upgrade the
nullable deletionVector struct, which contains non-nullable fields, is
being read out as "empty" in the default case rather than as null. This
causes the "seen" code in log replay to fail to correctly identify files
which should be marked as removed when computing the state of the
snapshot.

This change introduces another workaround that I am still not thrilled
about 😒

Fixes delta-io#3030

Signed-off-by: R. Tyler Croy <[email protected]>
@rtyler rtyler force-pushed the fix/checkpoint-3030 branch from 09f1376 to 48ba19f Compare December 4, 2024 20:15
@rtyler rtyler added this pull request to the merge queue Dec 4, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 4, 2024
@ion-elgreco ion-elgreco added this pull request to the merge queue Dec 4, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 4, 2024
@rtyler rtyler added this pull request to the merge queue Dec 4, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 4, 2024
@rtyler rtyler merged commit 98f8b0b into delta-io:main Dec 4, 2024
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants