fix(python): add support for pyarrow 13+ #1804
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ion-elgreco
requested review from
wjones127,
fvaleye,
roeap and
rtyler
as code owners
wjones127
reviewed
Nov 4, 2023
python/deltalake/table.py
Outdated
Comment on lines
1440
to
1452
| def _cast_to_equal_batch( | ||
| batch: pyarrow.RecordBatch, schema: pyarrow.Schema | ||
| ) -> pyarrow.RecordBatch: | ||
| """ | ||
| Cast a batch to a schema, if it is already considered equal. | ||
|
|
||
| This is mostly for mapping things like list field names, which arrow-rs | ||
| checks when looking at schema equality, but pyarrow does not. | ||
| """ | ||
| if batch.schema == schema: | ||
| return pyarrow.Table.from_batches([batch]).cast(schema).to_batches()[0] | ||
| else: | ||
| return batch |
There was a problem hiding this comment.
@ion-elgreco If we take this out, do the tests still pass?
There was a problem hiding this comment.
Yes, everything passes still. Shall I take it out?
Or would we require it later once we enable the compliant types again.
edit: Taking it out for now.
There was a problem hiding this comment.
Ideally I think we shouldn’t need it. This was an earlier attempt at fixing the compliant types thing
There was a problem hiding this comment.
We should have backwards compatibility during reads and writes when we use the compliant types, I think spark-delta also doesn't write these compliant types but I may be wrong here
wjones127
approved these changes
Nov 5, 2023
|
Thanks for finishing this up! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
I build on top of the branch of @wjones127 #1602. In pyarrow v13+ the ParquetWriter by default uses the
compliant_nested_types = True(see related PR: https://github.com/apache/arrow/pull/35146/files)and the docs: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html).In arrow/parquet-rs it fails when it compares schemas because it expected the old non-compliant ones. For now we can have pyarrow 13+ supported by disabling it or updating the file options provided by a user.
Related Issue(s)
pyarrow>=13#1744Documentation