[Kernel] Remove the ColumnVector::getStruct API #2131
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scottsand-db
reviewed
Oct 3, 2023
allisonport-db
force-pushed
the
struct-type-changes
branch
from
ec680ff to
9e25ae4
Compare
5 tasks
allisonport-db
added a commit
that referenced
this pull request
Oct 10, 2023
## Description Provides implementations for `getChild` for column vectors that are missing them. ## How was this patch tested? Adds simple tests for `DefaultViewVector` and `DefaultGenericVector` (used by complex types in the JSON handler). #2131 also is based off these changes and uses `getChild` instead of `getStruct` everywhere in the code.
allisonport-db
added a commit
that referenced
this pull request
Oct 10, 2023
## Description Provides implementations for `getChild` for column vectors that are missing them. ## How was this patch tested? Adds simple tests for `DefaultViewVector` and `DefaultGenericVector` (used by complex types in the JSON handler). #2131 also is based off these changes and uses `getChild` instead of `getStruct` everywhere in the code.
allisonport-db
force-pushed
the
struct-type-changes
branch
from
9e25ae4 to
330e9f3
Compare
allisonport-db
commented
Oct 10, 2023
| /** | ||
| * Wrapper around list of {@link Row}s to expose the rows as a column vector | ||
| */ | ||
| private static class RowBasedVector implements ColumnVector { |
There was a problem hiding this comment.
had to add this until/if we decide to remove JsonHandlerTestImpl
allisonport-db
changed the title
vkorukanti
reviewed
Oct 12, 2023
kernel/kernel-api/src/main/java/io/delta/kernel/internal/actions/Format.java
Show resolved
Hide resolved
| Format.fromRow(requireNonNull(row, 0, "id").getStruct(3)), | ||
| requireNonNull(vector.getChild(0), rowId, "id").getString(rowId), | ||
| Optional.ofNullable(vector.getChild(1).isNullAt(rowId) ? null : | ||
| vector.getChild(1).getString(rowId)), |
There was a problem hiding this comment.
what is the cost of getChild? Wondering with this approach are we creating too many objects?
There was a problem hiding this comment.
I think depends on the implementation; some create wrapper objects, we can update those implementations to only create the child vectors once.
It's no more object creation than the row API (and often less) since we'd create a wrapper row each getStruct call
kernel/kernel-api/src/main/java/io/delta/kernel/utils/Utils.java
Outdated
Show resolved
Hide resolved
allisonport-db
added a commit
to allisonport-db/delta
that referenced
this pull request
Oct 13, 2023
## Description Removes the `getStruct` API from `ColumnVector`. We will use a wrapper to convert to rows only for the ColumnarBatch/FilteredColumnarBatch row-based processing APIs. ## How was this patch tested? Existing tests should suffice.
xupefei
pushed a commit
to xupefei/delta
that referenced
this pull request
Oct 31, 2023
…-io#2133) ## Description Provides implementations for `getChild` for column vectors that are missing them. ## How was this patch tested? Adds simple tests for `DefaultViewVector` and `DefaultGenericVector` (used by complex types in the JSON handler). delta-io#2131 also is based off these changes and uses `getChild` instead of `getStruct` everywhere in the code.
xupefei
pushed a commit
to xupefei/delta
that referenced
this pull request
Oct 31, 2023
## Description Removes the `getStruct` API from `ColumnVector`. We will use a wrapper to convert to rows only for the ColumnarBatch/FilteredColumnarBatch row-based processing APIs. ## How was this patch tested? Existing tests should suffice.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Removes the
getStructAPI fromColumnVector. We will use a wrapper to convert to rows only for the ColumnarBatch/FilteredColumnarBatch row-based processing APIs.How was this patch tested?
Existing tests should suffice.