Allow logical expressions to express a cast to an extension type #18136
Conversation
github-actions
bot
added
sql
datafusion/expr/src/expr_schema.rs
Outdated
| f.as_ref() | ||
| .clone() | ||
| .with_data_type(data_type.data_type().clone()) | ||
| .with_metadata(f.metadata().clone()) | ||
| // TODO: should nullability be overridden here or derived from the | ||
| // input expression? |
There was a problem hiding this comment.
I am not sure if this type of cast should be able to express nullability or not.
There was a problem hiding this comment.
Currently this PR does not consider the nullability of the cast field, because the destination physical expression won't consider it either (and thus the return fields would be out of sync).
| data_type.clone(), | ||
| // TODO: this drops extension metadata associated with the cast | ||
| data_type.data_type().clone(), |
There was a problem hiding this comment.
I don't actually need physical expressions to be able to cast things...my vauge plan is to use a logical plan transformation or perhaps optimizer rule to replace casts to extension types with a ScalarUDF call. This should possibly error if there is mismatched metadata between the input and destination (i.e., a physical cast would only ever represent a storage cast, which is usually OK).
There was a problem hiding this comment.
I updated this to error at the logical expr -> physical expr stage if there is metadata on the cast field. I think this is better than dropping it (and won't break any existing code because before this PR such a cast could not exist).
1 participant
Which issue does this PR close?
I am sorry that I missed the previous PR implementing this ( #18120 ) and I'm also happy to review that one instead of updating this!
Rationale for this change
Other systems that interact with the logical plan (e.g., SQL, Substrait) can express types that are not strictly within the arrow DataType enum.
What changes are included in this PR?
For the Cast and TryCast structs, the destination data type was changed from a DataType to a FieldRef.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes, any code using
Cast { .. }to create an expression would need to useCast::new()instead (or pass on field metadata if it has it). Existing matches will need to be upated for thedata_type->fieldmember rename.