DataFusion weekly project plan (Andrew Lamb) - Dec 18, 2023

[alamb]

Follow on to Dec 11 2023:

Boilerplate Overview

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Try some different ways to communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well

My (personal) plans for this week

• Shepherd next PRs through for Support general pruning based on <col> = 'const' in PruningPredicate #8376
• Draft proposal to move DataFusion to a new top level apache project #8491
• Reproduce other bugs / limitations we have hit while upgrading DataFusion in InfluxDB IOx

Project Queue (list of future projects)

• Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)

Projects I hope to help review and coordinate

• FIFO JoinHashMap for HashJoin #8130, in support of memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848
• Next steps towards function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 with @2010YOUY01 and @edmondop . This week I hope to help get Support Expr creation for ScalarUDF: Resolve function calls by name during planning #8157 complete, and move on to RFC: Demonstrate what a function package might look like -- encoding expressions #8046
• Help improve planning speed (e.g with benchmarks Benchmarks for planning queries #8638 and overall epic [Epic] A collection of issues to improve planning performance / speed / efficiency #5637)

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I am using

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality

The current strategically important projects in my head are:

• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Improving performance of high cardinality aggregation (I have dreams of working on this over Christmas break)

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation / etc.

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

[alamb]

Review queue

Datafusion

• Add support for conflicting order sensitive aggregates in ARRAY_AGG aggregate function #8558
• Port tests in parquet.rs to sqllogictest #8560
• Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend #8317
• Port tests in parquet.rs to sqllogictest #8560
• Substrait insubquery #8363
• [MINOR]: Parametrize sort-preservation tests to exercise all situations (unbounded/bounded sources and flag behavior) #8575
• feat: support LargeList in array_element #8570

Arrow

• feat(ord): Support equality of StructArray arrow-rs#5217

[alamb]

Review queue

Datafusion

• Minor: support complex expr as the arg in the ApproxPercentileCont function #8580
• Port tests in parquet.rs to sqllogictest #8560
• Refactor TreeNode recursions #7942
• Change ScalarValue::{List, LargeList, FixedSizedList} to take specific types rather than ArrayRef #8562
• Add new configuration item listing_table_ignore_subdirectory #8565

Arrow

• Introduce Schema::field_names method arrow-rs#5186

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062
• feat(mysql): Increased compatibility for MySQL datafusion-sqlparser-rs#1059
• Add support for generated columns skipping 'GENERATED ALWAYS' keywords datafusion-sqlparser-rs#1058
• Document that comments aren't preserved for round trip datafusion-sqlparser-rs#1060
• Replace type_id() by trait method to allow wrapping dialects datafusion-sqlparser-rs#1065
• Add support for DEFERRED, IMMEDIATE, and EXCLUSIVE in SQLite's BEGIN TRANSACTION command datafusion-sqlparser-rs#1067

[alamb]

(moved down)

[matthewgapp]

@alamb any chance that you'll get around to reviewing this soon-ish? #7581. Really appreciate it!

[alamb]

DataFusion:

• Fix regression in regenerating protobuf source #8603 (review)
• Add examples of DataFrame::write* methods without S3 dependency #8606 (review)

[alamb]

DataFusion

• feat: support LargeList in array_remove #8595 (review)
• Optimize the parameter types of ParamValues's methods #8613
• Do not panic on zero placeholders in ParamValues::get_placeholders_with_values #8615
• Add schema_err! error macros with optional backtrace #8620
• Fix #8507: Non-null sub-field on nullable struct-field has wrong nullity #8623
• Support Writing Arrow files #8608
• Rename expr::window_function::WindowFunction to WindowFunctionDefinition, make structure consistent with ScalarFunction #8382
• Improve regexp_match performance by avoiding cloning Regex #8631
• Change ScalarValue::{List, LargeList, FixedSizedList} to take specific types rather than ArrayRef #8562
• Filter pushdown into cross join #8626
• Add support for conflicting order sensitive aggregates in ARRAY_AGG aggregate function #8558
• Add support for recursive CTEs #7581

sqlparse-rs:

• snowflake: fix rendering of SELECT TOP datafusion-sqlparser-rs#1070
• Support Mysql REPLACE statement and PRIORITY clause of INSERT datafusion-sqlparser-rs#1072