20.0.0.md

20.0.0 (2023-03-10)

[Full Changelog](https://github.com/apache/arrow-datafusion/compare/19.0.0...20.0.0

Breaking changes:

• Minor: Move TableProviderFactories up out of RuntimeEnv and into SessionState #5477 (alamb)
• chore: Remove references from SessionState from physical_plan #5455 (alamb)
• Implement Debug for ExecutionProps and VarProvider #5489 (alamb)

Implemented enhancements:

• Add UserDefinedLogicalNodeCore #5521 (mslapek)
• feat: add arrow_cast function to support supports arbitrary arrow types #5166 (alamb)
• feat: interval add timestamp #5491 (Weijun-H)
• feat: ParquetExec predicate preservation #5495 (crepererum)
• feat: add optimization rules for bitwise operations #5423 (izveigor)
• feat: Support bitwise operations for unsigned integer types #5476 (izveigor)
• feat: eliminate the duplicated sort keys in Order By clause #5462 (jackwener)
• feat: add name() method to UserDefinedLogicalNode #5450 (waynexia)
• feat: express unsigned literal in substrait #5448 (waynexia)
• feat: extensions_options macro #5442 (crepererum)
• [feat]:fast check has column #5328 (suxiaogang223)
• feat: eliminate unnecessary projection. #5366 (jackwener)

Fixed bugs:

• revert accidently deleted size code in count_distinct #5533 (comphead)
• fix: return schema of ExtensionPlan instead of its children's #5514 (waynexia)
• fix: logical merge conflict -- hash_join tests with passing boolean by value #5531 (korowa)
• fix: build union schema with child has same column name but qualifier… #5452 (yukkit)
• Fix is_distinct from for float NaN values #5446 (comphead)
• Bug/union wrong casting #5342 (berkaysynnada)
• fix nested loop join with literal join filter #5431 (ygf11)
• Fix filter pushdown for extension plans #5425 (thinkharderdev)
• Bug fix: Window frame range value outside the type range #5384 (mustafasrepo)
• fix: misc phys. expression display bugs #5387 (crepererum)

Documentation updates:

• Minor: Improve docs for UserDefinedLogicalNode dyn_eq and dyn_hash #5515 (alamb)
• chore: add known project ZincObserve #5376 (hengfeiyang)
• docs: clarify spark #5391 (hyoklee)

Merged pull requests:

• Manual changelog for 20.0.0 #5551 (andygrove)
• Prepare for 20.0.0 release [Part 1] #5539 (andygrove)
• chore: deduplicate workspace fields in Cargo.toml #5519 (waynexia)
• Add necessary features to optimizer #5540 (viirya)
• Minor: add the concise way for matching numerics #5537 (izveigor)
• Add UserDefinedLogicalNodeCore #5521 (mslapek)
• revert accidently deleted size code in count_distinct #5533 (comphead)
• fix: return schema of ExtensionPlan instead of its children's #5514 (waynexia)
• Minor: Move ObjectStoreRegistry to datafusion_execution crate #5478 (alamb)
• Minor: Add db-benchmark URL to db-benchmark readme #5503 (alamb)
• minor: fix clippy problem in new version. #5532 (jackwener)
• fix: logical merge conflict -- hash_join tests with passing boolean by value #5531 (korowa)
• Memory limited hash join #5490 (korowa)
• minor: improve error style #5510 (alamb)
• feat: add arrow_cast function to support supports arbitrary arrow types #5166 (alamb)
• build(deps): update sqlparser requirement from 0.30 to 0.32 w/ API update #5457 (alamb)
• Allow setting config extensions for TaskContext #5497 (mpurins-coralogix)
• Minor: Improve docs for UserDefinedLogicalNode dyn_eq and dyn_hash #5515 (alamb)
• feat: interval add timestamp #5491 (Weijun-H)
• Pass booleans by value instead of by reference #5487 (maxburke)
• Minor: Move TableProviderFactories up out of RuntimeEnv and into SessionState #5477 (alamb)
• feat: ParquetExec predicate preservation #5495 (crepererum)
• feat: add optimization rules for bitwise operations #5423 (izveigor)
• chore: Remove references from SessionState from physical_plan #5455 (alamb)
• Implement Debug for ExecutionProps and VarProvider #5489 (alamb)
• feat: Support bitwise operations for unsigned integer types #5476 (izveigor)
• Apply workaround for #5444 to DataFrame::describe #5468 (jiangzhx)
• feat: eliminate the duplicated sort keys in Order By clause #5462 (jackwener)
• Propagate timezone to created arrays #5481 (maxburke)
• refactor: make GeometricMean not to have update and merge #5469 (Weijun-H)
• feat: add name() method to UserDefinedLogicalNode #5450 (waynexia)
• Comment out description text in issue templates #5482 (Jefffrey)
• feat: express unsigned literal in substrait #5448 (waynexia)
• fix: build union schema with child has same column name but qualifier… #5452 (yukkit)
• refactor: make sum_distinct not to have update and merge #5474 (Weijun-H)
• compute_decimal_op_dyn_scalar should not cast lhs array to decimal array #5465 (viirya)
• feat: extensions_options macro #5442 (crepererum)
• Enable hash joins on FixedSizeBinary columns #5461 (maxburke)
• Fix is_distinct from for float NaN values #5446 (comphead)
• Implement/fix Eq and Hash for Expr and LogicalPlan #5421 (mslapek)
• [feat]:fast check has column #5328 (suxiaogang223)
• Parquet sorting benchmark #5433 (jaylmiller)
• refactor count_distinct to not to have update and merge #5408 (Weijun-H)
• build(deps): update zstd requirement from 0.11 to 0.12 #5458 (alamb)
• Upgrade bytes to 1.4 #5460 (viirya)
• add std,median result to describe method #5445 (jiangzhx)
• minor: Port more window tests to sqlogictests #5434 (alamb)
• Use compute_op_dyn_scalar for datatime #5315 (viirya)
• add a unit test that cover cast bug. #5443 (jackwener)
• create new datafusion-execution crate, start splitting code out #5432 (alamb)
• minor: fix clippy in nightly. #5440 (jackwener)
• Support for Sliding Windows Joins with Symmetric Hash Join (SHJ) #5322 (metesynnada)
• refactor: ParquetExec logical expr. => phys. expr. #5419 (crepererum)
• Update README.md fix [DataFusion] links #5438 (jiangzhx)
• add mean result for describe method #5435 (jiangzhx)
• add expr_fn::median #5437 (jiangzhx)
• Bug/union wrong casting #5342 (berkaysynnada)
• reimplement push_down_projection and prune_column. #4465 (jackwener)
• Add expr_fn::stddev #5409 (jiangzhx)
• fix nested loop join with literal join filter #5431 (ygf11)
• add a describe method on DataFrame like Polars #5226 (jiangzhx)
• Memory reservation & metrics for cross join #5339 (korowa)
• Optimize count_distinct.size #5377 (comphead)
• Fix filter pushdown for extension plans #5425 (thinkharderdev)
• Also push down all filters in TableProvider #5420 (avantgardnerio)
• Update arrow 34 #5375 (tustvold)
• Parquet limit pushdown (#5404) #5416 (tustvold)
• Move file format config.rs to live with the rest of the datasource code #5406 (alamb)
• Support Zstd compressed files #5397 (dennybritz)
• Add example of catalog API usage (#5291) #5326 (jaylmiller)
• Add support for protobuf serialisation of Arrow Map type #5359 (ahmedriza)
• minor: port window tests to slt (part 2) #5399 (alamb)
• fix(docs): fix typos #5403 (WenyXu)
• Try to push down full filter before break-up #5367 (avantgardnerio)
• enhance: remove more projection. #5402 (jackwener)
• refactor push_down_filter to fix dead-loop and use optimizer_recurse. #5337 (jackwener)
• feat: eliminate unnecessary projection. #5366 (jackwener)
• minor: add forgotten large_utf8 #5393 (jackwener)
• Minor: add tests for subquery to join #5363 (ygf11)
• bugfix: fix master bors problem. #5395 (jackwener)
• Rule ReplaceDistinctWithAggregate #5354 (mingmwang)
• chore: add known project ZincObserve #5376 (hengfeiyang)
• refactor: parquet pruning simplifications #5386 (crepererum)
• Minor: intersect expressions optimization #5388 (izveigor)
• docs: clarify spark #5391 (hyoklee)
• UDF zero params #5378 #5380 (jaylmiller)
• Minor: added some tests for coercion type #5389 (izveigor)
• minor: make table resolution an independent function ... #5373 (MichaelScofield)
• minor: port predicates tests to sqllogictests #5374 (jackwener)
• Bug fix: Window frame range value outside the type range #5384 (mustafasrepo)
• Fixed small typos in files of the optimizer #5356 (izveigor)
• fix: misc phys. expression display bugs #5387 (crepererum)
• Prepare for 19.0.0 release #5381 (andygrove)
• minor: disable tpcds-q41 due to not support decorrelate disjunction subquery #5369 (jackwener)