DataFusion weekly project plan (Andrew Lamb) - Jan 22, 2024

[alamb]

Follow on to Jan 15, 2024

Boilerplate Introduction

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)

My (personal) plans for this week

• Help test / optimize HashJoin outputs generation: feat: emitting partial join results in HashJoinStream #8020 from @korowa
• Complete Optimize COUNT( DISTINCT ...) for strings (up to 9x faster) #8849 with @jayzhan211
• Begin work on Improve aggregate performance with specialized groups accumulator for single string group by #7064
• Get first steps towards breaking up packages Create datafusion-functions crate, extract encode and decode to  #8705 reviewed, address comments, file follow on tickets

Project Queue (list of future projects)

• Improved Statistics: Epic: Statistics improvements #8227 (specifically, Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)

Projects I am prioritizing reviewing / helping

• More advanced pruning predicates Don't error on unknown column when pruning if predicate can still be proven false #7869 with @appletreeisyellow
• Improve planning speed [Epic] A collection of issues to improve planning performance / speed / efficiency #5637 with @matthewmturner
• Supporting various work to improve user defined functions and make functions packages more modular: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 (thank you @guojidan)
• Supporting recursive CTEs Add support for recursive CTEs #462 with @matthewgapp and @jonahgao

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I use to prioritize my own time

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality (note this is the lowest)

The current strategically important projects in my head are:

• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Getting started guide for new users (who want to use DataFusion in their project)
• Improving performance of aggregation - especially high cardinality groupings, where we still lag state of the art by 2x

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation or the projects named above

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

[alamb]

Review Queue:
DataFusion

• Simplify windows builtin functions return type #8920
• test: Port tests in partitioned_csv.rs to sqllogictest #8919 (review)
• Consolidate TreeNode transform and rewrite APIs #8891 (review)
• Fix handling of nested leaf columns in parallel parquet writer #8923
• fix: common_subexpr_eliminate rule should not apply to short-circuit expression #8928
• Add syntax highlight to datafusion-cli #8918
• Fix expr partial ord test #8908

[alamb]

DataFusion

• fix: issue #8922 make row group test more readable #8941
• Find the correct fields when using page filter on struct fields in parquet #8848
• feat: Support parquet bloom filter pruning for decimal128 #8930
• feat:implement sql style 'ends_with' and 'instr' string function #8862
• [fix #8904] Improved output of datafuson-cli with empty results #8947
• Combine multiple IN lists in ExprSimplifier #8949
• Support GroupsAccumulator accumulator for udaf #8892 (review)
• [MINOR]: Update create_window_expr to refer only input schema #8945
• fix: common_subexpr_eliminate rule should not apply to short-circuit expression #8928

Sqlparser

• Add Postgres operators for the LIKE expression variants datafusion-sqlparser-rs#1096

[alamb]

DataFusion

• feat: Support parquet bloom filter pruning for decimal128 #8930 (review)
• fix: issue #8922 make row group test more readable #8941
• Don't error in simplify_expressions rule #8957
• Add helper function for processing scalar function input #8962
• Combine multiple IN lists in ExprSimplifier #8949 (review)

Arrow

• Add JSON writer benchmarks (#5314) arrow-rs#5317 (review)

Sqlparser

• ISSUE-1088: Fix array_agg wildcard behavior datafusion-sqlparser-rs#1093
• Add support for constraint characteristics clause datafusion-sqlparser-rs#1099
• Add support for BigQuery table and view options datafusion-sqlparser-rs#1061
• Support for unquoted hyphenated identifiers on bigquery datafusion-sqlparser-rs#1109
• Add Qrlew as a user in README.md datafusion-sqlparser-rs#1107
• Error on dangling NO in CREATE SEQUENCE options datafusion-sqlparser-rs#1104
• Use Option<Expr> for Min and Max vals in Seq Opts, fix alter col seq display datafusion-sqlparser-rs#1106

[alamb]

DataFusion

• Enhance simplifier by adding Canonicalize #8780
• fix: allow placeholders to be substituted when coercible #8977
• NOT operator not return internal error when args are not boolean value #8982
• Improve to_timestamp docs #8981
• Fix optimize projections bug #8960
• NOT operator not return internal error when args are not boolean value #8982

[alamb]

DataFusion:

• Support compute return types from argument values (not just their DataTypes) #8985

[gruuya]

Can we also add #8894 to the queue please?

[alamb]

This is my review queue -- it would be great if other contributors can help review PRs. I am unlikely to be able to review all of these

DataFusion

• [MINOR]: Fix Optimize Projections Bug #8992
• [task #8917] Implement information_schema.schemata #8993 (review)
• TreeNode refactor code deduplication: Part 3 #8817
• Optimize MIN/MAX when relation is empty #8940
• [task #8213]Port tests in select.rs to sqllogictest #8967
• Add support for Postgres LIKE operators #8894
• feat: issue #8969 adding position function #8988
• [task #8203] Port tests in joins.rs to sqllogictest #8996
• test: Port (last) repartition.rs query to sqllogictest #8936
• Lead and Lag window functions should support default value with datatype other than Int64 #9001
• feat: Parallel Arrow file format reading #8897

Comet:

• Initial PR datafusion-comet#1

Arrow

• Add ListBuilder::with_field to support non nullable list fields (#5330) arrow-rs#5331 (review)

Sqlparser

• Fix :start and :end json accesses on snowflake datafusion-sqlparser-rs#1110

Watching

• Feat/make dfschema wrap schemaref #8905

[alamb]

Can we also add #8894 to the queue please?

FYI @gruuya -- I plan to do a sqlparser release today or tomorrow, so we should be able to get that integrated into DataFusion over the next few days and get #8894 mergeable

[gruuya]

I plan to do a sqlparser release today or tomorrow, so we should be able to get that integrated into DataFusion over the next few days and get #8894 mergeable

Sounds good, thanks!

[alamb]

DataFusion

• Recursive CTEs: Stage 3 - add execution support #8840
• Minor: fix: #9010 - Optimizer schema change assert error is incorrect #9012
• Safeguard against potential inexact row count being smaller than exact null count #9007
• Support compute return types from argument values (not just their DataTypes) #8985
• Relax join keys constraint from Column to any physical expression for physical join operators #8991
• Cache common referred expression at the window input #9009
• feat: support the ergonomics of getting list slice with stride  #8946
• Support FixedSizeList type coercion #8902
• Add ForeignKey constraint type #8566
• Change ScalarValue::Struct to ArrayRef #7893

Watching

• Feat/make dfschema wrap schemaref #8905

[alamb]

• [task #8201] Port tests in expr.rs to sqllogictest, finish the left c… #9014
• feat: Disable client console highlight by default #9013

[alamb]

Next week: #9030