You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ibis has a common test suite found in ibis/backends/tests/ for ensuring methods have identical behavior across backends. Most of these tests make use of one (or more) tables preloaded into the backends (and accessed via fixtures). A common one is functional_alltypes, but there are others.
The functional_alltypes dataset is very old and isn’t sufficiently interesting to catch all bugs. For example, the string_col column is currently just the numbers 0-9 as strings (so all ascii, 1-char length strings). This has resulted in some bugs in functionality making it through to releases, only to be discovered later by users (or us when trying to make changes later). As a recent example, the NULL handling behavior of collect/first/last wasn’t consistent across backends, but wasn’t caught since our test data didn’t contain a rich enough mix of null/non-null data.
We've been responding to bugs where not all backends have matching behavior for some functions. I propose we spend some time improving our test suite and doing some proactive bug hunting.
Improvements might include:
Adding one (or several) more interesting datasets that would better cover features tested
Updating the backend tests to use these datasets (and fixing/filing issues for any bugs discovered)
Evaluating the existing test functions for each method to ensure edge cases have coverage
Making more use of memtable inputs for some functions to give more targeted inputs for testing edge cases
Fuzzing using hypothesis
Intentionally not being prescriptive here, open to any methods that might incrementally improve our testing process.
The text was updated successfully, but these errors were encountered:
Ibis has a common test suite found in
ibis/backends/tests/
for ensuring methods have identical behavior across backends. Most of these tests make use of one (or more) tables preloaded into the backends (and accessed via fixtures). A common one isfunctional_alltypes
, but there are others.The
functional_alltypes
dataset is very old and isn’t sufficiently interesting to catch all bugs. For example, thestring_col
column is currently just the numbers 0-9 as strings (so all ascii, 1-char length strings). This has resulted in some bugs in functionality making it through to releases, only to be discovered later by users (or us when trying to make changes later). As a recent example, theNULL
handling behavior ofcollect/first/last
wasn’t consistent across backends, but wasn’t caught since our test data didn’t contain a rich enough mix of null/non-null data.We've been responding to bugs where not all backends have matching behavior for some functions. I propose we spend some time improving our test suite and doing some proactive bug hunting.
Improvements might include:
memtable
inputs for some functions to give more targeted inputs for testing edge caseshypothesis
Intentionally not being prescriptive here, open to any methods that might incrementally improve our testing process.
The text was updated successfully, but these errors were encountered: