Gluten 2025 Roadmap

[FelixYBW]

Description

2024 is almost gone. Time to define 2025 features. Here is some from my mind. Feel free to add more or comment
For more information about the achievements in 2024 Roadmap, please check #4709

Velox backend:

• Spark 4.0 support with minimal JDK 11 support, Deprecate Spark 3.2 and JDK 8
• Spark 3.3.x/3.4.x/3.5.x upgrade
• Switch to Upstream Velox's official release
• Stage level resource management(Stage level resource management to handle offheap/onheap memory conflict #4392 )
• Spill enhancement and performance ([VL] Spill related issues tracker #3030 )
• Hash table broadcast support in BHJ (https://docs.google.com/document/d/1upEby9aJnBcMKul7HS5agSr1zwFR63Iq4QL1nYxuFvU/edit?usp=sharing)
• Dictionary support in shuffle
• Multile core per task ([VL] Use multiple threads in the same executor #7810 )
• Pyspark Support including Python UDF/Arrow UDF
• Other Accelerator such as GPU, FPGA Support
• ARM architecture support
• Stability improvement, full fuzzer support, Result Mismatch issues ([VL] Result mismatch issues tracker #4652 )
• Spark Coverage including Operators and Functions
• Full Datalake support
• File Format support (Json)
• Performance improvement: Spill, Sort, HashAgg, HashJoin, ColumnarShuffle, and others
• Complex type in R2C ([VL] row2VeloxColumnar bad performance #7223)
• Gluten Indicator
• Gluten + Flink + Velox POC (experimental)

Community:

• Major Release: 1.4/1.5/1.6/1.7 in Q1/Q2/Q3/Q4 2025
• Minor Release: On Demand
• Apache TLP Submission in Q4 2025

[zhanglistar]

What about supporting Flink framework?

[Myracle]

Any plan or discussion about flink?

[zhztheplayer]

@zhanglistar @Myracle

It waits for discussion.

Does someone want to open / host a discussion for the topic? cc @apache/gluten-committers

[jinchengchenghh]

Do we need to do complex data type support and optimization such as row to column optimization for complex data type?

[FelixYBW]

Do we need to do complex data type support and optimization such as row to column optimization for complex data type?

Yes, currently R2C with complex type is too expensive

[MacChen01]

What about support openEuler OS ?

[FelixYBW]

What about support openEuler OS ?

Can vcpkg=on work on openEuler?