Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api): add .delta method for computing difference in units between two temporal values #7273

Merged
merged 3 commits into from
Oct 2, 2023

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Oct 2, 2023

This PR adds a new API to temporal types, delta, that allows users to compute
the difference between two temporal types (times, dates, and timestamps).

This implemented for all the backends for which it was relatively
straightforward to get matching behavior out of.

Backends differ in many annoying and meaningful ways in how they implement this
operation:

  1. Naming: datediff (MS SQL), date_diff (DuckDB)
  2. Whether the inputs work on all temporal types or there's a separate function
    for each type: date_diff (DuckDB, MS SQL, etc) vs time_diff,
    date_diff, timestamp_diff (BigQuery)
  3. The meaning of the order of operands. (ClickHouse follows subtraction
    order, DuckDB subtracts the first argument from the second argument)
  4. Whether the unit is a string or magic identifier 'day' (ClickHouse) vs
    DAY (MySQL)
  5. Whether the computation truncates the inputs and then computes the delta,
    or truncates afterward
    . (Trino and MySQL truncate after, others before)

Tests hit each of these cases.

Backends that support one or more of these operations:

  1. BigQuery
  2. ClickHouse
  3. DuckDB
  4. MS SQL
  5. MySQL
  6. Snowflake
  7. Trino

Notably, postgres and oracle do not have functions for computing the delta between two temporal types in integer units.

  • Postgres only has functions for computing the delta that return intervals
  • Oracle has this blog post showing how to use advanced database features to implement support for this operation.

Closes #7195

@cpcloud cpcloud added feature Features or general enhancements timestamps Issues related to the timestamp API labels Oct 2, 2023
@cpcloud cpcloud force-pushed the delta-api branch 5 times, most recently from 173bcae to ab64b4c Compare October 2, 2023 12:23
@cpcloud
Copy link
Member Author

cpcloud commented Oct 2, 2023

Clouds are passing

…/ibis on  delta-api is 📦 v6.1.0 via 🐍 v3.10.12 via ❄️  impure (ibis-3.10.12-env)
❯ pytest -m 'bigquery or snowflake' --snapshot-update --allow-snapshot-deletion -n auto -q
bringing up nodes...
....x.....xx.x......x.x......xx..x...............x.......x.....x.x..x.x.x...............x.x...xx....x..x.....x....x.......x..x......x...x.......x...x.x..xx..x..x.....x.....x..x....x......... [  6%]
.xxx..x.x.x..x...x..x..x.....xx...............x.......x......x.............xx......x..x.......x.......xx..xx.........x................x....................x....x...............xx..........xx [ 13%]
.x.x....x.................x..........x...x.....................x.xx.x.x.............x.....x...x..xx....xx........x...x.......x.x...x.........x.s.............x..............................x. [ 20%]
s.......x..........x........................x...............x...x....x......xxxx......x........x......x....x.......xx.....x.x.x....x..x............x...x......x......x.................ssss... [ 26%]
x......x...xxx.....x.......x.x.xx.x....x.x.x........x.x....x..........x....................x.....xx.....x..x...............x.......xx..x.........xx.....xx..xx.x.x.x....x.x...x.xxx.x.x....... [ 33%]
.......x..x....x.xxxxx....x..xxx.x...x..xxx...sssss...xx.x.xx.x.xxxx.xxxx.x.xxx..........x..x.x.......xxsxx...xxxs......x...x...x..x...x...............x.....x.x.................xxx....x..... [ 40%]
.......x..............x...x...x....................x.x..x.x.................x..............x.....x.....................ss..........s....x..x..x....x..x.xx....................x......xx....... [ 47%]
....x...x......x.....x...x..x........ss.x.......x.....sx.........xx..x..x...x.x....x.................x.x.........x.......ss..x.......x..x..........x.x.x.....x......x...xx..x.xx.......xxx..x. [ 53%]
.........xxxxxx.....x.xx.xxxxs.xxxxxxxx...x.xx.xx.x.xxx.......x.x......x.x..........s.xx.xxxx....xx.......xx...........x...x.....x..x.............x.x..x....x.......s......................... [ 60%]
.....x..x.x............x.......x............x.x.......................................x........x.x................x.........ssss.............xx..x...........xxxxxxxx..x...................... [ 67%]
..............xx.....x.....xx....xx.x.x..x.x.x.....x...x...ss..x..x....x...x.....x....x........x.....xx.......x...x...............x......x.x.xx............s.x....x................x......x..x [ 73%]
.......x...x...x......x.........xs............................x.............................x.....................x...........................................................x...s..x........ [ 80%]
...x...xx..xss....x......x...........................x......xxx.xx....x.............x...................x.......x.............x................x.xx..x........................................ [ 87%]
.................................................................................................................................................x.............x.............................. [ 94%]
...................................x...................x......................x...............x...x..s.xxsx..........xx.................x........xx...............s...x.x                      [100%]
2353 passed, 38 skipped, 438 xfailed in 445.06s (0:07:25)

@cpcloud
Copy link
Member Author

cpcloud commented Oct 2, 2023

Just to show that the clouds are running the relevant tests successfully:

❯ pytest -m 'bigquery or snowflake' ibis/backends/tests/test_temporal.py::test_delta -vv
...
ibis/backends/tests/test_temporal.py::test_delta[snowflake-timestamp] PASSED [ 16%]
ibis/backends/tests/test_temporal.py::test_delta[snowflake-date] PASSED  [ 33%]
ibis/backends/tests/test_temporal.py::test_delta[snowflake-time] PASSED  [ 50%]
ibis/backends/tests/test_temporal.py::test_delta[bigquery-timestamp] PASSED [ 66%]
ibis/backends/tests/test_temporal.py::test_delta[bigquery-date] PASSED   [ 83%]
ibis/backends/tests/test_temporal.py::test_delta[bigquery-time] PASSED   [100%]

@gforsyth
Copy link
Member

gforsyth commented Oct 2, 2023

3. DuckDB subtracts the first argument from the second argument

WUT

Adding this one to the "SQL is everywhere" talk

@cpcloud
Copy link
Member Author

cpcloud commented Oct 2, 2023

This whole operation replaces TRIM as my go-to for things that look similar but are wildly different.

The temporal delta is a juicier example because there are multiple correct ways to implement it :)

Copy link
Member

@gforsyth gforsyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me -- one (non-blocking) question about the usage of _ in docstrings

@cpcloud
Copy link
Member Author

cpcloud commented Oct 2, 2023

@gforsyth I don't see your comment!

@cpcloud cpcloud added this to the 7.0 milestone Oct 2, 2023
@cpcloud cpcloud merged commit 79788c7 into ibis-project:master Oct 2, 2023
94 checks passed
@cpcloud cpcloud deleted the delta-api branch October 2, 2023 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Features or general enhancements timestamps Issues related to the timestamp API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat(api): add API for computing the difference between two temporal objects
2 participants