feat: extend write_deltalake to accept Deltalake schema #1879
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
r3stl355
requested review from
wjones127,
fvaleye,
roeap and
ion-elgreco
as code owners
|
interesting, caught by type check in Python :) |
ion-elgreco
reviewed
Nov 18, 2023
python/deltalake/writer.py
Outdated
| @@ -179,6 +180,8 @@ def write_deltalake( | |||
| raise ValueError("You must provide schema if data is Iterable") | |||
| else: | |||
| schema = data.schema | |||
| elif isinstance(schema, Schema): | |||
| schema = schema.to_pyarrow() | |||
There was a problem hiding this comment.
This DeltaLake.schema to pyarrow schema needs happen also when we do pa.Table.from_pandas().
Also, probably best to merge it after this PR since I've been moving some things around in the writer https://github.com/delta-io/delta-rs/pull/1820/files#diff-557b42d8017266a17216274b0b47022a587045905f0990e2723c4a2369a037e8
There was a problem hiding this comment.
good point, missed that, will fix
There was a problem hiding this comment.
Agree, makes sense to wait until that PR is merged as it changes the same file. I'll refresh the PR once the other one is merged
wjones127
previously approved these changes
Nov 18, 2023
Signed-off-by: Nikolay Ulmasov <[email protected]>
Signed-off-by: Nikolay Ulmasov <[email protected]>
Signed-off-by: Nikolay Ulmasov <[email protected]>
|
closing this as something got messed up with rebase, will create a different PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Extended write_deltalake to accept either PyArrow or Deltalake schema.
Added a test
Related Issue(s)
closes #1862