Release DataFusion `49.0.0` (July 2025)

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be https://crates.io/crates/datafusion/47.0.0 (June, 2025) so next major release would be late June or Early July 2025

Steps:

• Update version and changelog: [branch-49] Update version to 49.0.0, add changelog #16822
• Create a release branch: https://github.com/apache/datafusion/tree/branch-49
• Test with DataFusion Python:
• Test with DataFusion Comet: chore: migrate to DF 49.0.0 datafusion-comet#2040
• Test with delta.rs: chore: update to DataFusion 49.0.0 delta-io/delta-rs#3603
• Test with iceberg-rust:
• Test with LakeSail: chore: Upgrade to DataFusion 49.0.0 lakehq/sail#653
• Write upgrade guide:
• Test with parquet viewer
• Test with datafusion-materialized-views: Upgrade to DF49 datafusion-contrib/datafusion-materialized-views#75
• Voting Thread:
• Create ticket for next release: Release DataFusion 50.0.0 (Aug/Sep 2025) #16799
• Forward port version and changelog to main: [main] Update version to 49.0.0, add 49.0.0 changelog #16855

TODOs

• Update upgrade md for new unified config for sql string mapping to utf8view when we release datafusion 49.0.0 #16428
• Blog post for the DataFusion 49 release #16758

Prior release tickets:

• 48.0.0: Release DataFusion 48.0.0 (June 2025) #15771

Changes to add to upgrade guide

These PRs made changes that deserve a mention in the upgrade guide

• [MAJOR] Equivalence System Overhaul #16217
• Add compression option to SpillManager #16268

Features to mention in the blog (if they make it)

• Introduce Async User Defined Functions  #14837
• [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
• TopK dynamic filter pushdown attempt 2 #15770
• Support integration with Parquet modular encryption #15216
• [MAJOR] Equivalence System Overhaul #16217
• Use Tokio's task budget consistently, better APIs to support task cancellation #16398
• feat: Parquet modular encryption #16351

Bugs that need to be fixed

• Panic in datafusion_expr::window_state::WindowAggState::update #16308
• SortQueryFuzzer found a failing case on main #16452
• Wrong results when pushdown_filters is enabled (starting in 48.0.0) #16588
• restore topk pre-filtering of batches and make sort query fuzzer less sensitive to expected non determinism  #16501
• Extended tests are failing #16595

Bugs that would be good to fix / investigate

• 1000x slowdown opening parquet file due to partitions #16676
• How to write csv file to disk from a empty dataframe? #16240
• Release DataFusion 49.0.0 (July 2025) #16235 (comment)

Community Wishlist

• Format Date32 to string given timestamp specifiers #15361
• [Epic] Pipeline breaking cancellation support and improvement #16353

[alamb]

I am thinking we should begin the 49.0.0 release next week. I'll publicize to the lists / etc

[xudong963]

@alamb Yeah, let's do it (time flies

[adriangb]

We introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.

[xudong963]

We introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.

I'm okay to wait for these changes. It's always good to not break users.