chore: upgrade to DataFusion 46.0.0 #3261

alamb · 2025-02-24T19:29:55Z

Description

chore: upgrade to DataFusion 46.0.0

Related Issue(s)

related to Release DataFusion 46.0.0 apache/datafusion#14123

github-actions · 2025-02-24T19:30:16Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

ion-elgreco · 2025-02-24T19:36:39Z

crates/sql/src/planner.rs

@@ -44,6 +44,7 @@ impl<'a, S: ContextProvider> DeltaSqlToRel<'a, S> {
                        enable_ident_normalization: self.options.enable_ident_normalization,
                        support_varchar_with_length: false,
                        enable_options_value_normalization: false,
+                        collect_spans: false,


What is spans?

The source code locations for expressions:

https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html

We are starting to gather / plumb this into DataFusion. This particular setting means the sql planner won't try and pass the span information along. The only thing the spans are used for now is debug / help messages

Ah ok thanks for explaining :) I learned something new about datafusion!

FWIW this is a new feature that was starting to be added in DataFusion 46

alamb · 2025-02-24T19:44:50Z

Looks like from CI we also need to update rust to be 1.82

https://github.com/delta-io/delta-rs/actions/runs/13506223447/job/37736308124?pr=3261

error: rustc 1.81.0 is not supported by the following packages:
[email protected] requires rustc 1.82.0
[email protected] requires rustc 1.82.0
...

alamb · 2025-02-24T21:57:26Z

I have a few cleanup PRs I would plan to make as I work on this. Here is the first one:

chore: fix some compiler warnings #3262

alamb · 2025-02-25T17:58:43Z

I also started breaking this PR up into some smaller ones:

alamb · 2025-02-26T12:29:03Z

crates/core/src/delta_datafusion/mod.rs

    #[derive(Default)]
-    struct ParquetPredicateVisitor {
+    struct ParquetVisitor {


I had to change the way the parquet scan information was found in the two visitors, and I combined them together at the same time as they were mostly boiler plate copy/paste

alamb · 2025-02-26T20:40:41Z

I filed a ticket in datafusion explaining the current test failures

Error projecting statistics in DataSourceExec apache/datafusion#14905

alamb · 2025-02-28T15:58:56Z

crates/sql/src/planner.rs

-                        enable_options_value_normalization: false,
-                    },
-                );
+                let parser_options = ParserOptions::new()


This uses the nice new API from @kosiew in

Implement builder style API for ParserOptions apache/datafusion#14887

alamb · 2025-02-28T16:16:57Z

.github/workflows/build.yml

@@ -20,7 +20,7 @@ jobs:
        uses: actions-rs/toolchain@v1
        with:
          profile: default
-          toolchain: '1.81'
+          toolchain: '1.82'


DataFusion 46 requires Rust 1.82

You can see CI fail without these changes
https://github.com/delta-io/delta-rs/actions/runs/13591787541/job/38000195037?pr=3261

I don't know what the MSRV policy in delta is so we probably can't merge this PR until it is ok to increase MSRV in delta

alamb · 2025-02-28T18:42:07Z

I think this PR will pass -- maybe someone could trigger the CI so I can show a clean run?

codecov · 2025-02-28T18:53:49Z

Codecov Report

Attention: Patch coverage is 89.77273% with 9 lines in your changes missing coverage. Please review.

Project coverage is 72.16%. Comparing base (9d98ac2) to head (e9a56aa).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/operations/load_cdf.rs	87.50%	2 Missing and 2 partials ⚠️
crates/sql/src/parser.rs	0.00%	1 Missing and 1 partial ⚠️
crates/core/src/delta_datafusion/expr.rs	66.66%	0 Missing and 1 partial ⚠️
crates/core/src/delta_datafusion/mod.rs	97.50%	1 Missing ⚠️
python/src/lib.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3261   +/-   ##
=======================================
  Coverage   72.16%   72.16%           
=======================================
  Files         144      144           
  Lines       45771    45784   +13     
  Branches    45771    45784   +13     
=======================================
+ Hits        33030    33042   +12     
+ Misses      10651    10650    -1     
- Partials     2090     2092    +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb · 2025-02-28T19:13:50Z

🤔 there are some python failures here. Seeing if I can figure out what is going on

https://github.com/delta-io/delta-rs/actions/runs/13592071811/job/38008047004?pr=3261

FAILED tests/test_cdf.py::test_read_cdf_partitioned_projection - pyo3_runtime.PanicException: index out of bounds: the len is 3 but the index is 3
FAILED tests/test_cdf.py::test_read_cdf_last_version - pyo3_runtime.PanicException: index out of bounds: the len is 2 but the index is 2
==== 2 failed, 495 passed, 7 skipped, 35 deselected, 95 warnings in 40.15s =====

alamb · 2025-02-28T22:00:16Z

🤔 there are some python failures here. Seeing if I can figure out what is going on

https://github.com/delta-io/delta-rs/actions/runs/13592071811/job/38008047004?pr=3261
FAILED tests/test_cdf.py::test_read_cdf_partitioned_projection - pyo3_runtime.PanicException: index out of bounds: the len is 3 but the index is 3
FAILED tests/test_cdf.py::test_read_cdf_last_version - pyo3_runtime.PanicException: index out of bounds: the len is 2 but the index is 2
==== 2 failed, 495 passed, 7 skipped, 35 deselected, 95 warnings in 40.15s =====

Could someone tell me how to run these tests locally or give me a rust stack trace? I don't really understand what is failing here

blaginin · 2025-02-28T22:34:41Z

cd python
make develop
RUST_BACKTRACE=1 uv run pytest tests/test_cdf.py -s -k "test_read_cdf_partitioned_projection"

trace.txt

alamb · 2025-03-02T12:57:54Z

Update here is @blaginin has a fix:

Do not swap with projection when file is partitioned apache/datafusion#14956

alamb · 2025-03-02T13:37:07Z

Ok, updated with latest release candidate. Here is hoping for a clean CI run 🙏

alamb · 2025-03-02T20:11:38Z

SUccess! Thank you so much @blaginin for making this happen

alamb

Thanks @ion-elgreco -- I'll update the to use the released verson of datafusion now

Signed-off-by: Andrew Lamb <[email protected]>

ion-elgreco

Thanks @alamb @blaginin !

alamb · 2025-03-09T19:02:04Z

🎉

alamb requested review from wjones127, roeap, rtyler, hntd187 and ion-elgreco as code owners February 24, 2025 19:29

alamb marked this pull request as draft February 24, 2025 19:29

github-actions bot added the binding/rust Issues for the Rust crate label Feb 24, 2025

alamb mentioned this pull request Feb 24, 2025

Release DataFusion 46.0.0 apache/datafusion#14123

Closed

26 tasks

alamb changed the title ~~core: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS)~~ chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) Feb 24, 2025

ion-elgreco reviewed Feb 24, 2025

View reviewed changes

alamb mentioned this pull request Feb 24, 2025

chore: fix some compiler warnings #3262

Merged

This was referenced Feb 25, 2025

chore: fix clippy warnings on main #3264

Merged

chore: update tests to use Column::new and other expr_fn functions #3265

Merged

chore: use builder API to create FileScanConfig #3266

Merged

alamb mentioned this pull request Feb 25, 2025

Implement builder style API for ParserOptions apache/datafusion#14879

Closed

alamb force-pushed the alamb/df_46 branch from ce87ed1 to 2036ef1 Compare February 25, 2025 21:36

alamb force-pushed the alamb/df_46 branch from 2036ef1 to 47ea65c Compare February 26, 2025 12:26

github-actions bot added the binding/python Issues for the Python package label Feb 26, 2025

alamb commented Feb 26, 2025

View reviewed changes

This was referenced Feb 26, 2025

Rename DataSource and FileSource fields for consistency apache/datafusion#14898

Merged

Error projecting statistics in DataSourceExec apache/datafusion#14905

Closed

alamb commented Feb 28, 2025

View reviewed changes

alamb mentioned this pull request Feb 28, 2025

chore: remove unused dev dependency on home #3277

Merged

blaginin mentioned this pull request Mar 1, 2025

Do not swap with projection when file is partitioned apache/datafusion#14956

Merged

alamb mentioned this pull request Mar 1, 2025

index out of bounds: the len is 2 but the index is 2 in some data sources apache/datafusion#14957

Closed

alamb mentioned this pull request Mar 2, 2025

Minor: Add more projection pushdown tests, clarify comments apache/datafusion#14963

Merged

alamb force-pushed the alamb/df_46 branch from bd76bbe to 082f9a8 Compare March 2, 2025 13:36

alamb marked this pull request as ready for review March 3, 2025 15:54

alamb requested a review from fvaleye as a code owner March 3, 2025 15:54

ion-elgreco force-pushed the alamb/df_46 branch from 1ec6ed6 to e0301e3 Compare March 9, 2025 13:14

alamb changed the title ~~chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS)~~ chore: upgrade to DataFusion 46.0.0 Mar 9, 2025

alamb commented Mar 9, 2025

View reviewed changes

ion-elgreco changed the title ~~chore: upgrade to DataFusion 46.0.0~~ chore: upgrade to DataFusion 46.0.0 Mar 9, 2025

chore: Update to DataFusion 46.0.0, update for API chanages

e9a56aa

Signed-off-by: Andrew Lamb <[email protected]>

alamb force-pushed the alamb/df_46 branch from e0301e3 to e9a56aa Compare March 9, 2025 13:22

ion-elgreco approved these changes Mar 9, 2025

View reviewed changes

ion-elgreco enabled auto-merge March 9, 2025 13:24

ion-elgreco added this pull request to the merge queue Mar 9, 2025

Merged via the queue into delta-io:main with commit c149502 Mar 9, 2025
28 checks passed

alamb deleted the alamb/df_46 branch March 9, 2025 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: upgrade to DataFusion 46.0.0 #3261

chore: upgrade to DataFusion 46.0.0 #3261

alamb commented Feb 24, 2025 •

edited

Loading

github-actions bot commented Feb 24, 2025

ion-elgreco Feb 24, 2025

alamb Feb 24, 2025

ion-elgreco Feb 24, 2025

alamb Feb 25, 2025

alamb commented Feb 24, 2025

alamb commented Feb 24, 2025

alamb commented Feb 25, 2025

alamb Feb 26, 2025

alamb commented Feb 26, 2025 •

edited

Loading

alamb Feb 28, 2025

alamb Feb 28, 2025

alamb commented Feb 28, 2025

codecov bot commented Feb 28, 2025 •

edited

Loading

alamb commented Feb 28, 2025

alamb commented Feb 28, 2025

blaginin commented Feb 28, 2025

alamb commented Mar 2, 2025

alamb commented Mar 2, 2025

alamb commented Mar 2, 2025

alamb left a comment

ion-elgreco left a comment

alamb commented Mar 9, 2025

chore: upgrade to DataFusion 46.0.0 #3261

chore: upgrade to DataFusion 46.0.0 #3261

Conversation

alamb commented Feb 24, 2025 • edited Loading

Description

Related Issue(s)

github-actions bot commented Feb 24, 2025

ion-elgreco Feb 24, 2025

Choose a reason for hiding this comment

alamb Feb 24, 2025

Choose a reason for hiding this comment

ion-elgreco Feb 24, 2025

Choose a reason for hiding this comment

alamb Feb 25, 2025

Choose a reason for hiding this comment

alamb commented Feb 24, 2025

alamb commented Feb 24, 2025

alamb commented Feb 25, 2025

alamb Feb 26, 2025

Choose a reason for hiding this comment

alamb commented Feb 26, 2025 • edited Loading

alamb Feb 28, 2025

Choose a reason for hiding this comment

alamb Feb 28, 2025

Choose a reason for hiding this comment

alamb commented Feb 28, 2025

codecov bot commented Feb 28, 2025 • edited Loading

Codecov Report

alamb commented Feb 28, 2025

alamb commented Feb 28, 2025

blaginin commented Feb 28, 2025

alamb commented Mar 2, 2025

alamb commented Mar 2, 2025

alamb commented Mar 2, 2025

alamb left a comment

Choose a reason for hiding this comment

ion-elgreco left a comment

Choose a reason for hiding this comment

alamb commented Mar 9, 2025

alamb commented Feb 24, 2025 •

edited

Loading

alamb commented Feb 26, 2025 •

edited

Loading

codecov bot commented Feb 28, 2025 •

edited

Loading