Spark 3.5: Support default values in vectorized reads #11815
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdblue
commented
Dec 18, 2024
| @@ -49,7 +49,6 @@ | |||
| import org.apache.parquet.schema.MessageType; | |||
| import org.apache.parquet.schema.Type; | |||
| import org.apache.spark.sql.vectorized.ColumnarBatch; | |||
| import org.junit.jupiter.api.Disabled; | |||
There was a problem hiding this comment.
I switched this to use assumptions like the other tests that are based on AvroDataTest. I just wanted to be consistent.
Fokko
approved these changes
Dec 19, 2024
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java
Outdated
Show resolved
Hide resolved
nastra
approved these changes
Dec 19, 2024
rdblue
commented
Dec 19, 2024
| @@ -248,17 +269,6 @@ public void testMixedTypes() throws IOException { | |||
| writeAndValidate(schema); | |||
| } | |||
|
|
|||
| @Test | |||
| public void testTimestampWithoutZone() throws IOException { | |||
There was a problem hiding this comment.
Removing this test for TimestampNTZ by adding the type to SUPPORTED_PRIMITIVES (so that it is handled like any other primitive) is what broke the ORC tests. It looks like the problem is that Spark 3.5's ColumnarRow doesn't support TimestampNTZType. As a temporary work-around, I've added validation code that checks the value by accessing it as a TimestampType instead.
This isn't a change to read behavior, just how we access the data to validate it. I expect to be able to remove this workaround in the next Spark version.
There was a problem hiding this comment.
yeah I noticed that too and was planning on fixing that in Spark. I've opened https://issues.apache.org/jira/browse/SPARK-50624
This was referenced Jan 10, 2025
cloud-fan
pushed a commit
to apache/spark
that referenced
this pull request
Jan 13, 2025
…rRow ### What changes were proposed in this pull request? Noticed that this was missing when using this in Iceberg. See additional details in apache/iceberg#11815 (comment) ### Why are the changes needed? To be able to read `TimestampNTZType` when using `ColumnarRow` ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Added some unit tests that failed without the fix ### Was this patch authored or co-authored using generative AI tooling? No Closes #49437 from nastra/SPARK-50624. Authored-by: Eduard Tudenhoefner <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
cloud-fan
pushed a commit
to apache/spark
that referenced
this pull request
Jan 13, 2025
…rRow ### What changes were proposed in this pull request? Noticed that this was missing when using this in Iceberg. See additional details in apache/iceberg#11815 (comment) ### Why are the changes needed? To be able to read `TimestampNTZType` when using `ColumnarRow` ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Added some unit tests that failed without the fix ### Was this patch authored or co-authored using generative AI tooling? No Closes #49437 from nastra/SPARK-50624. Authored-by: Eduard Tudenhoefner <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit d7545d0) Signed-off-by: Wenchen Fan <[email protected]>
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Jan 17, 2025
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Jan 17, 2025
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Jan 17, 2025
This was referenced Jan 17, 2025
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Jan 17, 2025
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Jan 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This follows on #11803 and adds default value support to vectorized reads.