[GLUTEN-2163][CH] support aggregate function approx_percentile #4829

taiyang-li · 2024-03-01T10:23:28Z

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #2163)

vanilla takes 2.665 seconds
gluten takes 1.502 seconds

github-actions · 2024-03-01T10:23:45Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2024-03-01T10:24:05Z

Run Gluten Clickhouse CI

github-actions · 2024-03-01T10:46:07Z

Run Gluten Clickhouse CI

github-actions · 2024-03-02T12:23:51Z

Run Gluten Clickhouse CI

github-actions · 2024-03-04T04:29:35Z

Run Gluten Clickhouse CI

taiyang-li · 2024-03-04T06:47:53Z

Notice: This bugfix(ClickHouse/ClickHouse#60740) must be merged first otherwise we have the following issue:

0: jdbc:hive2://localhost:10000/> 
0: jdbc:hive2://localhost:10000/> 
0: jdbc:hive2://localhost:10000/> CREATE TEMPORARY VIEW lineitem
. . . . . . . . . . . . . . . . > USING org.apache.spark.sql.parquet
. . . . . . . . . . . . . . . . > OPTIONS (
. . . . . . . . . . . . . . . . >   path "/data1/liyang/cppproject/gluten/gluten-core/src/test/resources/tpch-data/lineitem"
. . . . . . . . . . . . . . . . > ) ; 
+---------+
| Result  |
+---------+
+---------+
No rows selected (3.193 seconds)
0: jdbc:hive2://localhost:10000/> select l_linenumber % 10, approx_percentile(l_extendedprice, 0.5) from lineitem group by l_linenumber % 10;  
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 2.0 failed 1 times, most recent failure: Lost task 11.0 in stage 2.0 (TID 6) (bigo executor driver): io.glutenproject.exception.GlutenException: io.glutenproject.exception.GlutenException: The number of elements 11869 for quantileGK exceeds 10000
0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000012067b9d in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
1. ./build_gcc/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000b03b0bf in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long&, unsigned long&) @ 0x0000000006ccf93e in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
3. ./build_gcc/./src/AggregateFunctions/AggregateFunctionQuantileGK.cpp:0: DB::(anonymous namespace)::QuantileGK<double>::deserialize(DB::ReadBuffer&) @ 0x000000000bfb0fac in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
4. ./src/AggregateFunctions/AggregateFunctionQuantile.h:233: DB::AggregateFunctionQuantile<double, DB::(anonymous namespace)::QuantileGK<double>, DB::NameQuantileGK, false, void, false, true>::deserialize(char*, DB::ReadBuffer&, std::optional<unsigned long>, DB::Arena*) const @ 0x000000000bfad96f in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
5. ./src/AggregateFunctions/Combinators/AggregateFunctionNull.h:177: DB::AggregateFunctionNullBase<true, true, DB::AggregateFunctionNullUnary<true, true>>::deserialize(char*, DB::ReadBuffer&, std::optional<unsigned long>, DB::Arena*) const @ 0x000000000d29dde6 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
6. ./build_gcc/./src/DataTypes/Serializations/SerializationAggregateFunction.cpp:0: DB::SerializationAggregateFunction::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x000000000e6ce44e in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
7. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:211: DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000000e6c9656 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
8. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:202: local_engine::readNormalComplexData(DB::ReadBuffer&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, local_engine::NativeReader::ColumnParseUtil&) @ 0x000000000b37fb57 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
9. ./contrib/llvm-project/libcxx/include/__utility/swap.h:37: local_engine::NativeReader::prepareByFirstBlock() @ 0x000000000b37e6ca in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
10. ./build_gcc/./utils/extern-local-engine/Storages/IO/NativeReader.cpp:0: local_engine::NativeReader::read() @ 0x000000000b37d115 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
11. ./build_gcc/./utils/extern-local-engine/Shuffle/ShuffleReader.cpp:52: local_engine::ShuffleReader::read() @ 0x000000000b4369a7 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
12. ./build_gcc/./utils/extern-local-engine/local_engine_jni.cpp:587: Java_io_glutenproject_vectorized_CHStreamReader_nativeNext @ 0x0000000005eed2bb in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000012067b9d in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
1. ./build_gcc/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000b03b0bf in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::createRuntime(int, String&) @ 0x0000000005efccd2 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
3. ./utils/extern-local-engine/jni/jni_common.h:79: unsigned char local_engine::safeCallBooleanMethod<>(JNIEnv_*, _jobject*, _jmethodID*) @ 0x0000000005efdd70 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
4. ./build_gcc/./utils/extern-local-engine/Storages/SourceFromJavaIter.cpp:55: local_engine::SourceFromJavaIter::peekBlock(JNIEnv_*, _jobject*) @ 0x000000000b36a3dd in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
5. ./build_gcc/./utils/extern-local-engine/Parser/SerializedPlanParser.cpp:318: local_engine::SerializedPlanParser::parseReadRealWithJavaIter(substrait::ReadRel const&) @ 0x000000000b31578a in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
6. ./build_gcc/./utils/extern-local-engine/Parser/SerializedPlanParser.cpp:0: local_engine::SerializedPlanParser::parseOp(substrait::Rel const&, std::list<substrait::Rel const*, std::allocator<substrait::Rel const*>>&) @ 0x000000000b31b3f9 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
7. ./build_gcc/./utils/extern-local-engine/Parser/RelParser.cpp:70: local_engine::RelParser::parseOp(substrait::Rel const&, std::list<substrait::Rel const*, std::allocator<substrait::Rel const*>>&) @ 0x000000000b2d3a39 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: local_engine::SerializedPlanParser::parseOp(substrait::Rel const&, std::list<substrait::Rel const*, std::allocator<substrait::Rel const*>>&) @ 0x000000000b31a71d in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
9. ./build_gcc/./utils/extern-local-engine/Parser/SerializedPlanParser.cpp:398: local_engine::SerializedPlanParser::parse(std::unique_ptr<substrait::Plan, std::default_delete<substrait::Plan>>) @ 0x000000000b3193e7 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
10. ./build_gcc/./utils/extern-local-engine/Parser/SerializedPlanParser.cpp:1790: local_engine::SerializedPlanParser::parse(String const&) @ 0x000000000b327dc5 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so
11. ./build_gcc/./utils/extern-local-engine/local_engine_jni.cpp:277: Java_io_glutenproject_vectorized_ExpressionEvaluatorJniWrapper_nativeCreateKernelWithIterator @ 0x0000000005ee75a0 in /data1/liyang/cppproject/kyli/ClickHouse/build_gcc/utils/extern-local-engine/libch.so

        at io.glutenproject.vectorized.ExpressionEvaluatorJniWrapper.nativeCreateKernelWithIterator(Native Method)
        at io.glutenproject.vectorized.CHNativeExpressionEvaluator.createKernelWithBatchIterator(CHNativeExpressionEvaluator.java:93)
        at io.glutenproject.backendsapi.clickhouse.CHIteratorApi.genFinalStageIterator(CHIteratorApi.scala:265)
        at io.glutenproject.execution.WholeStageZippedPartitionsRDD.$anonfun$compute$1(WholeStageZippedPartitionsRDD.scala:58)
        at io.glutenproject.utils.Arm$.withResource(Arm.scala:25)
        at io.glutenproject.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
        at io.glutenproject.execution.WholeStageZippedPartitionsRDD.compute(WholeStageZippedPartitionsRDD.scala:46)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
        at org.apache.spark.sql.execution.CHColumnarToRowRDD.compute(CHColumnarToRowExec.scala:92)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

github-actions · 2024-03-04T06:52:19Z

#2163

github-actions · 2024-03-04T06:56:39Z

Run Gluten Clickhouse CI

github-actions · 2024-03-14T06:24:33Z

Run Gluten Clickhouse CI

github-actions · 2024-03-14T10:03:08Z

Run Gluten Clickhouse CI

github-actions · 2024-03-18T08:45:43Z

Run Gluten Clickhouse CI

github-actions · 2024-03-18T15:50:40Z

Run Gluten Clickhouse CI

taiyang-li · 2024-03-19T02:35:23Z

DataFrame.summary()("summary" in DataFrameSuite) is not related to spark-sql, ignore it temporarily.

github-actions · 2024-03-19T02:37:43Z

Run Gluten Clickhouse CI

taiyang-li · 2024-03-19T06:48:15Z

A velox ut failed, do you know why? @rui-mo it is strange because approx_percentile is already in kBlackList. Thanks

cpp-ch/local-engine/Parser/SerializedPlanParser.cpp

taiyang-li · 2024-03-19T08:56:16Z

@zzcclp @liuneng1994 could you help review this pr, thanks very much !

rui-mo · 2024-03-19T12:17:51Z

@taiyang-li I assume it passes the validation here https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/io/glutenproject/backendsapi/velox/VeloxBackend.scala#L326-L327 because approx_percentile is an aggregate expression. Could you try to modify the logic here to let it fallback on Velox backend?

github-actions · 2024-03-20T03:27:19Z

Run Gluten Clickhouse CI

backends-velox/src/main/scala/io/glutenproject/backendsapi/velox/VeloxBackend.scala

github-actions · 2024-03-20T03:33:28Z

Run Gluten Clickhouse CI

github-actions · 2024-03-20T12:03:49Z

Run Gluten Clickhouse CI

taiyang-li · 2024-03-20T13:35:32Z

velox build failed cc @rui-mo

github-actions · 2024-03-20T13:37:07Z

Run Gluten Clickhouse CI

rui-mo

Could you rebase this PR? Thanks.

github-actions · 2024-03-25T07:33:58Z

Run Gluten Clickhouse CI

zzcclp · 2024-03-26T00:31:34Z

LGTM, @rui-mo could you help to review this pr, thanks.

backends-velox/src/main/scala/io/glutenproject/backendsapi/velox/VeloxBackend.scala

github-actions · 2024-03-29T03:19:40Z

Run Gluten Clickhouse CI

taiyang-li changed the title ~~[CH-2163] Resupport aggregate function approx_percentile~~ [CH-2163] support aggregate function approx_percentile Mar 4, 2024

taiyang-li changed the title ~~[CH-2163] support aggregate function approx_percentile~~ [GLUTEN-2163][CH] support aggregate function approx_percentile Mar 4, 2024

taiyang-li mentioned this pull request Mar 4, 2024

Remove wrong sanitize checking in aggregate function quantileGK ClickHouse/ClickHouse#60740

Merged

taiyang-li force-pushed the gluten_2163_new branch from 7773ecf to 0e9e9ca Compare March 4, 2024 06:56

taiyang-li marked this pull request as draft March 5, 2024 08:50

taiyang-li force-pushed the gluten_2163_new branch from 0e9e9ca to 2a92e73 Compare March 14, 2024 06:24

taiyang-li marked this pull request as ready for review March 14, 2024 06:24

taiyang-li force-pushed the gluten_2163_new branch from bf046d9 to 3fbd99a Compare March 18, 2024 08:45

taiyang-li commented Mar 19, 2024

View reviewed changes

cpp-ch/local-engine/Parser/SerializedPlanParser.cpp Show resolved Hide resolved

taiyang-li commented Mar 20, 2024

View reviewed changes

backends-velox/src/main/scala/io/glutenproject/backendsapi/velox/VeloxBackend.scala Outdated Show resolved Hide resolved

taiyang-li force-pushed the gluten_2163_new branch from 30c5cc1 to 58e8a02 Compare March 20, 2024 13:36

rui-mo reviewed Mar 21, 2024

View reviewed changes

taiyang-li force-pushed the gluten_2163_new branch from 58e8a02 to 2923096 Compare March 25, 2024 07:33

rui-mo reviewed Mar 26, 2024

View reviewed changes

backends-velox/src/main/scala/io/glutenproject/backendsapi/velox/VeloxBackend.scala Show resolved Hide resolved

rui-mo approved these changes Mar 26, 2024

View reviewed changes

taiyang-li added 11 commits March 29, 2024 11:04

support aggregate function approx_percentile

cb0a3a0

fix bugs

1ccd52b

fix bugs

5b5b51d

fix bugs

c4cd782

apply s3 options

856ab35

fix failed uts

eb686e4

fix failed uts

3d405c6

ignore summary ut

47bbc79

fix failed velox uts

b108c10

use ref instead of value copy

4154926

fix building velox

3deab1f

taiyang-li force-pushed the gluten_2163_new branch from 2923096 to 3deab1f Compare March 29, 2024 03:19

zzcclp merged commit c0bad12 into apache:main Mar 29, 2024
32 checks passed

taiyang-li mentioned this pull request Aug 14, 2024

[GLUTEN-6819][CH] Refactor source from jave iter && make casting happens before materializing #6830

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-2163][CH] support aggregate function approx_percentile #4829

[GLUTEN-2163][CH] support aggregate function approx_percentile #4829

taiyang-li commented Mar 1, 2024 •

edited

Loading

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 2, 2024

github-actions bot commented Mar 4, 2024

taiyang-li commented Mar 4, 2024 •

edited

Loading

github-actions bot commented Mar 4, 2024

github-actions bot commented Mar 4, 2024

github-actions bot commented Mar 14, 2024

github-actions bot commented Mar 14, 2024

github-actions bot commented Mar 18, 2024

github-actions bot commented Mar 18, 2024

taiyang-li commented Mar 19, 2024 •

edited

Loading

github-actions bot commented Mar 19, 2024

taiyang-li commented Mar 19, 2024

taiyang-li commented Mar 19, 2024

rui-mo commented Mar 19, 2024

github-actions bot commented Mar 20, 2024

github-actions bot commented Mar 20, 2024

github-actions bot commented Mar 20, 2024

taiyang-li commented Mar 20, 2024

github-actions bot commented Mar 20, 2024

rui-mo left a comment

github-actions bot commented Mar 25, 2024

zzcclp commented Mar 26, 2024

github-actions bot commented Mar 29, 2024

[GLUTEN-2163][CH] support aggregate function approx_percentile #4829

[GLUTEN-2163][CH] support aggregate function approx_percentile #4829

Conversation

taiyang-li commented Mar 1, 2024 • edited Loading

What changes were proposed in this pull request?

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 2, 2024

github-actions bot commented Mar 4, 2024

taiyang-li commented Mar 4, 2024 • edited Loading

github-actions bot commented Mar 4, 2024

github-actions bot commented Mar 4, 2024

github-actions bot commented Mar 14, 2024

github-actions bot commented Mar 14, 2024

github-actions bot commented Mar 18, 2024

github-actions bot commented Mar 18, 2024

taiyang-li commented Mar 19, 2024 • edited Loading

github-actions bot commented Mar 19, 2024

taiyang-li commented Mar 19, 2024

taiyang-li commented Mar 19, 2024

rui-mo commented Mar 19, 2024

github-actions bot commented Mar 20, 2024

github-actions bot commented Mar 20, 2024

github-actions bot commented Mar 20, 2024

taiyang-li commented Mar 20, 2024

github-actions bot commented Mar 20, 2024

rui-mo left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 25, 2024

zzcclp commented Mar 26, 2024

github-actions bot commented Mar 29, 2024

taiyang-li commented Mar 1, 2024 •

edited

Loading

taiyang-li commented Mar 4, 2024 •

edited

Loading

taiyang-li commented Mar 19, 2024 •

edited

Loading