-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Simplify is ins to an OR chain of eqs #3800
Conversation
CodSpeed Performance ReportMerging #3800 will degrade performances by 12.76%Comparing Summary
Benchmarks breakdown
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3800 +/- ##
==========================================
+ Coverage 77.82% 77.87% +0.04%
==========================================
Files 745 750 +5
Lines 94416 94796 +380
==========================================
+ Hits 73480 73819 +339
- Misses 20936 20977 +41
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
src/daft-algebra/src/simplify/mod.rs
Outdated
let chain_of_eqs = list | ||
.iter() | ||
.map(|item| e.clone().eq(item.clone())); | ||
Transformed::yes(chain_of_eqs.reduce(|a, b| a.or(b)).unwrap()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tiny nit: you can use combine_disjunction
here instead of the reduce
Closes #3662
Implement simplify expr rule to transform
is_in
into an OR chain ofeq
, e.g.e IN (1, 2, 3) -> e = 1 OR e = 2 OR e = 3
.This can be faster than using the
is_in
kernel, which has to hash the items and does a lookup per row in the base column.Results on Q12 and Q19 of TPCH SF 10:
Q12: 410.57s -> 313.44s
Q19: 489.13s -> 401.24s
I configured the threshold for max number of items to transform into the OR chain is 5. I chose the number 5 because of the significant symbolism it has in nature and culture. Humans have 5 fingers on each hand and 5 toes on each foot. Most flowering plants have 5 petals. It's the number of Platonic solids (regular polyhedra). Many animals have pentameral (5-fold) symmetry, like starfish. In Chinese culture, there are five blessings (longevity, happiness, wealth, luck and prosperity), as well as five elements (wood, water, fire, Earth and metal).