-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: Ignore nullability of list elements when consuming Substrait (#1…
…0874) * Ignore nullability of list elements when consuming Substrait DataFusion (= Arrow) is quite strict about nullability, specifically, when using e.g. LogicalPlan::Values, the given schema must match the given literals exactly - including nullability. This is non-trivial to do when converting schema and literals separately. The existing implementation for from_substrait_literal already creates lists that are always nullable (see ScalarValue::new_list => array_into_list_array). This reverts part of #10640 to align from_substrait_type with that behavior. This is the error I was hitting: ``` ArrowError(InvalidArgumentError("column types must match schema types, expected List(Field { name: \"item\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: \"item\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0"), None) ``` * use `Field::new_list_field` in `array_into_(large_)list_array` just for consistency, to reduce the places where "item" is written out * add a test for non-nullable lists
- Loading branch information
Showing
5 changed files
with
114 additions
and
21 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
71 changes: 71 additions & 0 deletions
71
datafusion/substrait/tests/testdata/non_nullable_lists.substrait.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
{ | ||
"extensionUris": [], | ||
"extensions": [], | ||
"relations": [ | ||
{ | ||
"root": { | ||
"input": { | ||
"read": { | ||
"common": { | ||
"direct": { | ||
} | ||
}, | ||
"baseSchema": { | ||
"names": [ | ||
"col" | ||
], | ||
"struct": { | ||
"types": [ | ||
{ | ||
"list": { | ||
"type": { | ||
"i32": { | ||
"typeVariationReference": 0, | ||
"nullability": "NULLABILITY_REQUIRED" | ||
} | ||
}, | ||
"typeVariationReference": 0, | ||
"nullability": "NULLABILITY_REQUIRED" | ||
} | ||
} | ||
], | ||
"typeVariationReference": 0, | ||
"nullability": "NULLABILITY_REQUIRED" | ||
} | ||
}, | ||
"virtualTable": { | ||
"values": [ | ||
{ | ||
"fields": [ | ||
{ | ||
"list": { | ||
"values": [ | ||
{ | ||
"i32": 1, | ||
"nullable": false, | ||
"typeVariationReference": 0 | ||
}, | ||
{ | ||
"i32": 2, | ||
"nullable": false, | ||
"typeVariationReference": 0 | ||
} | ||
] | ||
}, | ||
"nullable": false, | ||
"typeVariationReference": 0 | ||
} | ||
] | ||
} | ||
] | ||
} | ||
} | ||
}, | ||
"names": [ | ||
"col" | ||
] | ||
} | ||
} | ||
], | ||
"expectedTypeUrls": [] | ||
} |