Weekly Plan (Andrew Lamb) March 3, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Help @xudong963 to get Release DataFusion 46.0.0 #14123 ready (mostly testing / herding bugs)
• Complete a sql-parser-rs release Release sqlparser-rs version 0.55.0 datafusion-sqlparser-rs#1671
• object_store releaase: Release object_store 0.12.0 (API breaking) Around Feb 30 2025 arrow-rs#6903
• write a new "this week in datafusion": March 4, 2025: This week(s) in DataFusion #15005

(BTW if anyone wants to help with writing or releases that would be 🤗 )

Other projects I think are strategically important

These are the ones I plan to look at / review PRs for first

• Anything performance related (though really I am interested in things that help many/all DataFusion users)
• Productionizing / hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01
• Pretty tree explain plans [EPIC] Complete SQL EXPLAIN Tree Rendering #14914 (thanks to @irenjj for starting us off!)
• Splitting out datasources with @AdamGS and @logan-keede [Epic] Split datasources out from datafusion crate (datafusion/core) #14444
• pushing expressions down: Support Push down expression evaluation in TableProviders #14993

Weekly Stretch / Nice to have goals

• Figure out next steps for Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Figure out where next to improve our performance
• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom

[alamb]

DataFusion: Bugs/UX/Performance

• feat: Add tree / pretty explain mode #14677
• feat: instrument spawned tasks with current tracing span when tracing feature is enabled #14547

arrow-rs

• ObjectStore WASM32 Support arrow-rs#7226

[alamb]

DataFusion: Bugs/UX/Performance

• chore: Update SessionStateBuilder::with_default_features does not replace existing features #14935
• Fix documentation warnings and error if anymore occur #14952
• doc: update RecordBatchReceiverStreamBuilder::spawn_blocking task behaviour #14995
• Simplify Between expression to Eq #14994

DataFusion: New Features/ Functions

• Split out avro, parquet, json and csv into individual crates #14951

[alamb]

DataFusion: Bugs/UX/Performance

• Move UnwrapCastInComparison into Simplifier #15012
• Refactor test suite in EnforceDistribution, to use standard test config. #15010
• fix: External sort failing on an edge case #15017
• BUG: schema_force_view_type configuration not working for CREATE EXTERNAL TABLE #14922
• Reuse last projection layer when renaming columns #14684

[alamb]

DataFusion: Bugs/UX/Performance

• fix: nested window function #15033
• add manual trigger for extended tests in pull requests #14331

DataFusion: New Features/ Functions

• feat: Add config max_temp_directory_size to limit max disk usage for spilling queries #14975
• Bug: Fix multi-lines printing issue for datafusion-cli #14954
• [POC] feat: Add datafusion-storage #15018
• feat: Add Aggregate UDF to FFI crate #14775
• Bug: Fix multi-lines printing issue for datafusion-cli #14954
• Add support for DISTINCT + ORDER BY in ARRAY_AGG #14413
• support simple/cross lateral joins #14595
• Support multiple results comparison in benchmark scripts #14196
• Add union_tag scalar function #14687
• STRING_AGG missing functionality #14412
• Introduce Async User Defined Functions  #14837

arrow-rs

• arrow-cast: Support cast to Dictionary<_, FixedSizeBinary> and add FixedSizeBinaryDictionaryBuilder arrow-rs#6666
• Add Parquet Modular decryption (read) support + encryption flag arrow-rs#6637
• feat: impl display for DataType::List arrow-rs#7051
• Avro codec enhancements arrow-rs#6965
• Test : move tests for parse_string_decimal_native to parse_decimal  arrow-rs#7177
• feat: add rounding logic and scale zero fix fro parse_decimal to match parse_string_to_decimal_native behavior arrow-rs#7179
• feat:use parse_decimal for generic_string_to_decimal conversion. arrow-rs#7191

Stuck PRs that need help pushing to completion

• Substrait support for propagating TableScan.filters to Substrait ReadRel.filter #14194
• hash join: add build-side join keys to memory accounting #14222
• Support marking columns as system columns via Field's metadata #14362 / feat: metadata columns #14057

[alamb]

• Next week Weekly Plan (Andrew Lamb) March 10, 2025 #15121