Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Use projected_table_schema for projection in DeltaSchemaAdapter
After upgrading from deltalake 0.20.1 to 0.22.3 it looks like Parquet column projection is broken when using DeltaTable::scan. Instead of scanning only the a single column, it looks like all columns are fetched from storage. Inspection with a debugger revelas that the adapted_projections are wrong here: https://github.com/apache/datafusion/blob/88f58bf929167c5c5e2250ad87caa88d4dff11e5/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L153-L159 The adapted_projections are obtained in https://github.com/delta-io/delta-rs/blob/5b2f46b06e0eb508f932a8b39feb11b568a78a32/crates/core/src/delta_datafusion/schema_adapter.rs#L46-L60 Changing line 49 to use the projected_table_schema seems to solve the problem.
- Loading branch information