-
Notifications
You must be signed in to change notification settings - Fork 447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[VL] Support map_concat spark function #5093
Conversation
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format?
See also: |
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Could be merged after CI passes.
4653a1e
to
6dd97b8
Compare
Run Gluten Clickhouse CI |
The CAST function is then transformed into Gluten to CASTTransformer but fails in validation since the cast is not supported for map type in Gluten yet.
|
@rui-mo can you please check this patch again. I have added few comments and rewritten the spark test case since there is no exception thrown for duplicate values. Velox already handles it by keeping last value. |
Hi @Surbhi-Vijay, I notice |
Yes, Maybe we can introduce change in Velox to throw exception based on a config which will only be used in Gluten. Please let me know your thoughts. |
@rui-mo @PHILO-HE can you please check this comment? and we can finalize the next set of action for this patch. |
@Surbhi-Vijay, yes, better to introduce a config in Velox to support those two behaviors. I guess Spark's default EXCEPTION behavior is commonly used, so please firstly fix it in Velox. cc @rui-mo |
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
I will be updating these PRs in few days. |
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks. |
Adds support for
map_concat
spark function as part of #4039Added supporting test case and updated documentation