Gluten Roadmap in 2024

[weiting-chen]

Description

Here listed Gluten roadmap in 2024. We will use this list to have an overview for Gluten, an easy way to track feature support plan, and to avoid duplicate work before starting the implementation.

• Migrate to Apache Gluten: https://issues.apache.org/jira/browse/INFRA-25506
• Partially Done: Installation Method: pip(Pending)/conda/docker
• Spark version support: Spark3.4 UT([VL] Track all the failed unit test in Spark 3.4. #3559 )(done), Spark 3.5 API/UT([GLUTEN-4424] Explore adding Spark 35 w/ Scala 2.12 only (WIP) #4425 )
• Partially Done: Data Type Support: complex, decimal32, timestamp, timestampNTZ, Calendar, Interval, UDT
• Partially Done, fuzz tests support decimal/agg, Spark query runner in Velox(Enable decimal fuzzer test in experimental workflow facebookincubator/velox#10948 build: Enable Spark query runner as reference in aggregation fuzzer test facebookincubator/velox#9559)
• OS support: move to use static link via vcpkg to avoid any OS support issue
• Partially Done: File Format Support: Parquet(gap on some datatypes), ORC, Text, CSV, Json(Pending)
• Data Source Support: ABFS, S3, GCS, Alluxio, HDFS viewfs
• Spark Functions Support, please check ([VL] Unsupported spark function list [please leave a comment if you plan to pick some] #4039 )
• Partially Done: Spark Operator Support: SMJ, PyArrow
• Spill Support including Partial Agg with single agg, Join Probe side Spill, Window + Spill, Sort + Spill, Spill performance improvement, Issue fixing([VL] Spill related issues tracker #3030 )
• Shuffle Support including Sort based Shuffle, Merge before Compress, Reducer Optimization
• Data Lake Support([VL] Unified design for data lake read support in Gluten + Velox #3378 ): DeltaLake, Iceberg, Hudi, Paimon
• RSS Support: Apache Uniffle([VL] Add uniffle integration #3767 )
• Pyspark Support including Python UDF/Arrow UDF (Pending)
• Stage level resource management(Stage level resource management to handle offheap/onheap memory conflict #4392 ), (Pending)
• Other Big Data Framework Support (Pending)
• Partially Done: Performance Optimizations: Sort, Join, Agg., Parquet Write, Scan
• Other Accelerator such as GPU, FPGA Support (Pending)
• Document and Website Update
• Planner: optimization for multiple fallback cases (RAS framework)

[felipepessoto]

@weiting-chen, for item

• Data Lake Support([VL] Unified design for data lake read support in Gluten + Velox #3378 ): DeltaLake, Iceberg, Hudi, Paimon

Do you know what changes to expect for Delta?

thanks

[weiting-chen]

@weiting-chen, for item

• Data Lake Support([VL] Unified design for data lake read support in Gluten + Velox #3378 ): DeltaLake, Iceberg, Hudi, Paimon

Do you know what changes to expect for Delta?

thanks

The plan is to support both COW table and MOR table this year, which means DeltaLake with and without Deletion Vector support.

[felipepessoto]

Thanks @weiting-chen

By Delta Deletion Vector support you mean for reads and writes?

[zemin-piao]

Hello @weiting-chen,

Thanks a lot for concluding the roadmap.

Regarding supporting Spark 3.5.1 in general, I saw there are two PRs merged.

#4822
#4845

May I ask there are any other works to be done in order to support Spark 3.5.1 on the plan, or these MRs concludes the work to support Spark 3.5.1?

[yjshen]

Thanks @weiting-chen for sharing the roadmap.

Regarding the item Other Big Data Framework Support, are there any designs or action items that could be shared here?
Additionally, I saw an existing but archived effort at https://github.com/oap-project/Gluten-Trino, could you please provide more details or share your thoughts about it?

[FelixYBW]

I just reviewed the list. It's a fruitful year, we did enhanced lots features but left some. Time to move to 2025.

Feel free to modify or comment