Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tracking issue] Change Data Feed read path #440

Open
12 of 21 tasks
OussamaSaoudi-db opened this issue Oct 28, 2024 · 1 comment
Open
12 of 21 tasks

[tracking issue] Change Data Feed read path #440

OussamaSaoudi-db opened this issue Oct 28, 2024 · 1 comment
Assignees
@OussamaSaoudi-db

Comments

@OussamaSaoudi-db
Copy link
Collaborator

OussamaSaoudi-db commented Oct 28, 2024 •
edited
Loading

Change Data Feed is a feature of the Delta protocol that allows users to scan all row-level changes that happened for each commit.

Completed

Remaining

Future work (out of scope)

  • (later) Support schema evolution for tables with column mapping enabled
  • (later) improve error message when performing CDF using a predicate that references a CDF generated column. Currently this fails, but the error message is not very clear.
@OussamaSaoudi-db OussamaSaoudi-db mentioned this issue Oct 28, 2024
Merged
OussamaSaoudi-db added a commit that referenced this issue Nov 8, 2024
@OussamaSaoudi-db @scovich
Move log segment into separate module (#438)
534a4e4 
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md
2. Run `cargo t --all-features --all-targets` to get started testing,
and run `cargo fmt`.
  3. Ensure you have added or run the appropriate tests for your PR.
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  5. Be sure to keep the PR description updated to reflect all changes.
-->

## What changes are proposed in this pull request?
<!--
Please clarify what changes you are proposing and why the changes are
needed.
The purpose of this section is to outline the changes, why they are
needed, and how this PR fixes the issue.
If the reason for the change is already explained clearly in an issue,
then it does not need to be restated here.
1. If you propose a new API or feature, clarify the use case for a new
API or feature.
  2. If you fix a bug, you can clarify why it is a bug.
-->
This PR is a prefactor that moves `LogSegment` into its own module. This
will be useful to implement `TableChanges` for [change data
feed](#440).
`TableChanges` will represent CDF scans and will depend on `LogSegment`.
Since `TableChanges` and `Snapshot` both depend on `LogSegment`, it
makes sense to keep it separated.

I also use this opportunity to address some nits in the codebase.
<!--
Uncomment this section if there are any changes affecting public APIs:
### This PR affects the following public APIs

If there are breaking changes, please ensure the `breaking-changes`
label gets added by CI, and describe why the changes are needed.

Note that _new_ public APIs are not considered breaking.
-->


## How was this change tested?
<!--
Please make sure to add test cases that check the changes thoroughly
including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please
clarify how you tested, ideally via a reproducible test documented in
the PR description.
-->
The changes compile.

---------

Co-authored-by: Ryan Johnson <[email protected]>
@OussamaSaoudi-db OussamaSaoudi-db mentioned this issue Nov 12, 2024
Open
@OussamaSaoudi-db
Copy link
Collaborator Author

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@OussamaSaoudi-db OussamaSaoudi-db

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@OussamaSaoudi-db