Review Backlog and Plan - Andrew Lamb - Nov 2024

[alamb]

I have been off for a few days and I am (again) feeling overwhelmed by the number of PRs waiting on review. This ticket is my attempt to organize myself and provide visibility into my own plans

Anyone should feel free to help work down the review backlog or post here if they would like to draw additional attention to their changes.

At a high level I prioritize reviews like this:

1. Bug fixes
2. Things that make DataFusion easier to use (documentation, code cleanup, etc)
3. Finishing up outstanding projects (notably [EPIC] Automatically generate all function documentation from code #12740 and [Epic] Unify WindowFunction Interface (remove built in list of BuiltInWindowFunction s) #8709)
4. Performance improvements
5. New features

I am hoping to work on myself:

• [DISCUSS] Document criteria for adding new features / what belongs in core DataFusion (e.g. sql syntax, functions, etc) #12357
• Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Blog post with DataFusion July - Sep 2024 #11631
• Release arrow-rs / parquet minor version 53.3.0 (November 2024) arrow-rs#6597
• Document Schema metadata expectations #12736

[alamb]

Review Queue

DataFusion: Bug Fixes

• Update sqlparser requirement from 0.51.0 to 0.52.0 #13373

DataFusion: Ease of Use

• Convert nth_value builtIn function to User Defined Window Function #13201
• Split the GroupColumn Implementations into smaller modules #13352

DataFusion: Features

• feat: Optimize SortPreservingMergeExec to avoid merging non-overlapping partitions #13296
• feat: scalar regex match physical expr #12270

arrow-rs bugs:

• Prevent FlightData overflowing max size limit whenever possible. arrow-rs#6690

arrow-rs features:

• feat: add GenericListViewBuilder arrow-rs#6552