feat: Add Spark concat_ws function #8854

PHILO-HE · 2024-02-26T09:01:59Z

Add concat_ws Spark function which returns the concatenation for the
input, separated by a separator (the first argument). It allows variable
number of VARCHAR or ARRAY<VARCHAR> arguments. And these two
types can be used in combination.

This function is a bit similar to ConcatFunction, except that concat_ws
requires separator and supports using ARRAY type and mixed types.

This PR is based on #6292 (author: @unigof). There are a few bug fixes
and improvements. Also made some changes to align with Spark.

Doc link.

netlify · 2024-02-26T09:02:18Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`3bcce79`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/6752a36cb48c4c0008b6a9a8

PHILO-HE · 2024-02-28T08:03:56Z

Found Spark supports arguments with String/Array type mixed used. Needs to enhance this patch.

SELECT concat_ws(,, a, b, array(c, d), e);
"a,b,c,d,e"

PHILO-HE · 2024-04-10T03:08:04Z

Found Spark supports arguments with String/Array type mixed used. Needs to enhance this patch.
SELECT concat_ws(,, a, b, array(c, d), e);
"a,b,c,d,e"

Supported such case in the latest code.

PHILO-HE · 2024-04-10T03:08:27Z

@rui-mo, could you take a review?

velox/docs/functions/spark/string.rst

PHILO-HE · 2024-04-12T03:37:28Z

@rui-mo, could you review again? Thanks!

rui-mo

Thanks.

velox/functions/sparksql/String.cpp

rui-mo

Thanks.

velox/functions/sparksql/String.cpp

PHILO-HE · 2024-11-29T02:23:55Z

@jinchengchenghh, @rui-mo, thank you so much for your time on this PR. I have fixed your comments. Could you review again?

rui-mo

Thanks. Looks good to me overall % some minors.

rui-mo · 2024-11-29T08:16:19Z

velox/expression/fuzzer/ExpressionFuzzer.cpp

+            "concat_ws",
+            std::vector<facebook::velox::exec::FunctionSignaturePtr>{
+                // Signature: concat_ws (separator, input,...) -> output:
+                // varchar, varchar, varchar,.. -> varchar


Do we need to cover the case of mixed string and array?

When array is enabled, the fuzzer test framework reports one issue when max_level_of_nesting=10. It seems not related to our implementation logic.

Can you show the exception? Maybe we should fix it first.

See below error message. I run the test with --velox_fuzzer_enable_complex_types. Seems this option has not been enabled for spark fuzzer test? cc @rui-mo

@ 0000000003cbcd71 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*) /root/PHILO/workspace/velox/deps-download/folly/folly/debugging/symbolizer/SignalHandler.cpp:453 @ 000000000001441f (unknown) @ 0000000001f13a7a facebook::velox::exec::LocalSelectivityVector::~LocalSelectivityVector() @ 000000000129f0f6 facebook::velox::exec::Expr::evalSimplified(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) [clone .cold] @ 00000000038ca794 facebook::velox::exec::Expr::evalSimplifiedImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) @ 00000000038cb077 facebook::velox::exec::Expr::evalSimplified(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) @ 00000000038ca794 facebook::velox::exec::Expr::evalSimplifiedImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) @ 00000000038cb077 facebook::velox::exec::Expr::evalSimplified(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) @ 00000000038cb1bd facebook::velox::exec::ExprSetSimplified::eval(int, int, bool, facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector> > >&) @ 0000000001fd4743 facebook::velox::test::ExpressionVerifier::verify(std::vector<std::shared_ptr<facebook::velox::core::ITypedExpr const>, std::allocator<std::shared_ptr<facebook::velox::core::ITypedExpr const> > > const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&, std::shared_ptr<facebook::velox::BaseVector>&&, bool, facebook::velox::fuzzer::InputRowMetadata const&) @ 00000000013dd3a5 facebook::velox::fuzzer::ExpressionFuzzerVerifier::go()

@PHILO-HE Would you open an issue so we can take a further look when enabling the velox_fuzzer_enable_complex_types option?

@rui-mo, just filed one issue: #11727.

velox/functions/sparksql/tests/ConcatWsTest.cpp

velox/functions/sparksql/ConcatWs.cpp

velox/functions/sparksql/tests/ConcatWsTest.cpp

jinchengchenghh · 2024-12-02T03:31:42Z

velox/functions/sparksql/ConcatWs.cpp

+    return totalResultBytes;
+  }
+
+  /// Initialize some vectors for inputs. And concatenate consecutive


// for private function, not revel to users

rui-mo · 2024-12-03T01:41:02Z

velox/expression/fuzzer/ExpressionFuzzer.cpp

@@ -253,6 +253,17 @@ static const std::unordered_map<
            /// them to fuzzer instead of hard-coding signatures here.
            getSignaturesForCast(),
        },
+        {
+            "concat_ws",


nit: maybe include a note indicating that this signature is only for Spark SQL. We will separate the Presto and Spark signatures in #9949. Thanks.

@rui-mo, just updated. Thanks!

Yohahaha · 2024-12-03T15:34:01Z

velox/functions/sparksql/registration/RegisterSpecialForm.cpp

@@ -44,6 +44,9 @@ void registerSpecialFormGeneralFunctions(const std::string& prefix) {
      "cast", std::make_unique<SparkCastCallToSpecialForm>());
  registerFunctionCallToSpecialForm(
      "try_cast", std::make_unique<SparkTryCastCallToSpecialForm>());
+  registerFunctionCallToSpecialForm(
+      ConcatWsCallToSpecialForm::kConcatWs,


I suppose concat_ws should be in RegisterString.cpp.

@Yohahaha, seems all special form functions should be registered in RegisterSpecialForm.cpp?

concat_ws is a string function in spark, https://github.com/apache/spark/blob/13315eeec07e2aebcc05dfee762bbd060ae192ec/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L77, and you documented it in string.srt too.

@Yohahaha, just updated this pr according to your suggestion. Thanks!

PHILO-HE · 2024-12-04T12:22:46Z

@xiaoxmeng, could you merge this pr if you have no comment?

PHILO-HE · 2024-12-06T13:27:18Z

Friendly ping @xiaoxmeng. Could you merge this pr? The CI failure is not related.

facebook-github-bot · 2024-12-06T21:26:30Z

@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

PHILO-HE · 2024-12-09T15:55:33Z

@bikramSingh91, thanks for importing the pr! I note internal test is red. If it's not related to this pr, could you merge it?

PHILO-HE · 2024-12-11T01:59:20Z

Friendly ping @bikramSingh91, could you merge this pr if no other change is required?

PHILO-HE · 2024-12-13T11:42:29Z

Hi @kgpai, can you merge this pr if you have no comment?

kgpai · 2024-12-13T17:42:56Z

@PHILO-HE Following up with @bikramSingh91

facebook-github-bot · 2024-12-13T21:28:08Z

@bikramSingh91 merged this pull request in bddddf8.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 26, 2024

PHILO-HE mentioned this pull request Feb 26, 2024

Add concat_ws Spark Function #6292

Closed

PHILO-HE force-pushed the concat_ws branch 3 times, most recently from b2bca03 to 7efd217 Compare March 26, 2024 14:49

PHILO-HE force-pushed the concat_ws branch 9 times, most recently from 3033eb8 to a1f105b Compare April 10, 2024 02:58

PHILO-HE force-pushed the concat_ws branch 2 times, most recently from 379108a to 8203b08 Compare April 10, 2024 03:11

rui-mo reviewed Apr 10, 2024

View reviewed changes

velox/docs/functions/spark/string.rst Outdated Show resolved Hide resolved

velox/docs/functions/spark/string.rst Outdated Show resolved Hide resolved

velox/docs/functions/spark/string.rst Outdated Show resolved Hide resolved

PHILO-HE force-pushed the concat_ws branch 2 times, most recently from c2638ed to bbfa05e Compare April 12, 2024 03:35

PHILO-HE mentioned this pull request Apr 12, 2024

[VL] Unsupported spark function list [please leave a comment if you plan to pick some] apache/incubator-gluten#4039

Open

99 tasks

PHILO-HE force-pushed the concat_ws branch 2 times, most recently from 6651c0d to 20df5f9 Compare April 12, 2024 09:39

rui-mo reviewed Apr 15, 2024

View reviewed changes

velox/functions/sparksql/String.cpp Outdated Show resolved Hide resolved

velox/functions/sparksql/String.cpp Outdated Show resolved Hide resolved

velox/functions/sparksql/String.cpp Outdated Show resolved Hide resolved

velox/functions/sparksql/String.cpp Outdated Show resolved Hide resolved

rui-mo reviewed Apr 15, 2024

View reviewed changes

velox/functions/sparksql/String.cpp Outdated Show resolved Hide resolved

PHILO-HE force-pushed the concat_ws branch from b3ee1cc to cabeb65 Compare April 16, 2024 02:22

PHILO-HE force-pushed the concat_ws branch from 814eaa3 to d5d2655 Compare November 29, 2024 02:27

PHILO-HE mentioned this pull request Nov 29, 2024

[VL] Support concat_ws function apache/incubator-gluten#8101

Merged

rui-mo reviewed Nov 29, 2024

View reviewed changes

PHILO-HE force-pushed the concat_ws branch 2 times, most recently from 9687831 to 3d3e184 Compare December 2, 2024 03:23

jinchengchenghh reviewed Dec 2, 2024

View reviewed changes

PHILO-HE force-pushed the concat_ws branch 2 times, most recently from 16adc62 to c305598 Compare December 2, 2024 03:44

rui-mo changed the title ~~feat: Add concat_ws Spark function~~ feat: Add Spark concat_ws function Dec 2, 2024

rui-mo approved these changes Dec 2, 2024

View reviewed changes

jinchengchenghh approved these changes Dec 2, 2024

View reviewed changes

rui-mo reviewed Dec 3, 2024

View reviewed changes

rui-mo added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Dec 3, 2024

PHILO-HE force-pushed the concat_ws branch from 8b4d031 to 7c39c8e Compare December 3, 2024 15:11

Yohahaha reviewed Dec 3, 2024

View reviewed changes

Initial commit

3bcce79

PHILO-HE force-pushed the concat_ws branch from 530becf to 3bcce79 Compare December 6, 2024 07:10

kgpai approved these changes Dec 13, 2024

View reviewed changes

facebook-github-bot closed this in bddddf8 Dec 13, 2024

facebook-github-bot added the Merged label Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Spark concat_ws function #8854

feat: Add Spark concat_ws function #8854

PHILO-HE commented Feb 26, 2024 •

edited

Loading

netlify bot commented Feb 26, 2024 •

edited

Loading

PHILO-HE commented Feb 28, 2024

PHILO-HE commented Apr 10, 2024

PHILO-HE commented Apr 10, 2024

PHILO-HE commented Apr 12, 2024

rui-mo left a comment

rui-mo left a comment

PHILO-HE commented Nov 29, 2024

rui-mo left a comment

rui-mo Nov 29, 2024

PHILO-HE Dec 2, 2024 •

edited

Loading

jinchengchenghh Dec 2, 2024

PHILO-HE Dec 2, 2024 •

edited

Loading

rui-mo Dec 3, 2024

PHILO-HE Dec 3, 2024

jinchengchenghh Dec 2, 2024

rui-mo Dec 3, 2024

PHILO-HE Dec 3, 2024

Yohahaha Dec 3, 2024

PHILO-HE Dec 4, 2024 •

edited

Loading

Yohahaha Dec 4, 2024

PHILO-HE Dec 4, 2024

PHILO-HE commented Dec 4, 2024

PHILO-HE commented Dec 6, 2024

facebook-github-bot commented Dec 6, 2024

PHILO-HE commented Dec 9, 2024

PHILO-HE commented Dec 11, 2024

PHILO-HE commented Dec 13, 2024

kgpai commented Dec 13, 2024

facebook-github-bot commented Dec 13, 2024

feat: Add Spark concat_ws function #8854

feat: Add Spark concat_ws function #8854

Conversation

PHILO-HE commented Feb 26, 2024 • edited Loading

netlify bot commented Feb 26, 2024 • edited Loading

✅ Deploy Preview for meta-velox canceled.

PHILO-HE commented Feb 28, 2024

PHILO-HE commented Apr 10, 2024

PHILO-HE commented Apr 10, 2024

PHILO-HE commented Apr 12, 2024

rui-mo left a comment

Choose a reason for hiding this comment

rui-mo left a comment

Choose a reason for hiding this comment

PHILO-HE commented Nov 29, 2024

rui-mo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE commented Dec 4, 2024

PHILO-HE commented Dec 6, 2024

facebook-github-bot commented Dec 6, 2024

PHILO-HE commented Dec 9, 2024

PHILO-HE commented Dec 11, 2024

PHILO-HE commented Dec 13, 2024

kgpai commented Dec 13, 2024

facebook-github-bot commented Dec 13, 2024

PHILO-HE commented Feb 26, 2024 •

edited

Loading

netlify bot commented Feb 26, 2024 •

edited

Loading

PHILO-HE Dec 2, 2024 •

edited

Loading

PHILO-HE Dec 2, 2024 •

edited

Loading

PHILO-HE Dec 4, 2024 •

edited

Loading