Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: improve backends test suite coverage #9446

Open
jcrist opened this issue Jun 26, 2024 · 0 comments
Open

test: improve backends test suite coverage #9446

jcrist opened this issue Jun 26, 2024 · 0 comments
Labels
tests Issues or PRs related to tests
Milestone

Comments

@jcrist
Copy link
Member

jcrist commented Jun 26, 2024

Ibis has a common test suite found in ibis/backends/tests/ for ensuring methods have identical behavior across backends. Most of these tests make use of one (or more) tables preloaded into the backends (and accessed via fixtures). A common one is functional_alltypes, but there are others.

The functional_alltypes dataset is very old and isn’t sufficiently interesting to catch all bugs. For example, the string_col column is currently just the numbers 0-9 as strings (so all ascii, 1-char length strings). This has resulted in some bugs in functionality making it through to releases, only to be discovered later by users (or us when trying to make changes later). As a recent example, the NULL handling behavior of collect/first/last wasn’t consistent across backends, but wasn’t caught since our test data didn’t contain a rich enough mix of null/non-null data.

We've been responding to bugs where not all backends have matching behavior for some functions. I propose we spend some time improving our test suite and doing some proactive bug hunting.

Improvements might include:

  • Adding one (or several) more interesting datasets that would better cover features tested
  • Updating the backend tests to use these datasets (and fixing/filing issues for any bugs discovered)
  • Evaluating the existing test functions for each method to ensure edge cases have coverage
  • Making more use of memtable inputs for some functions to give more targeted inputs for testing edge cases
  • Fuzzing using hypothesis

Intentionally not being prescriptive here, open to any methods that might incrementally improve our testing process.

@jcrist jcrist added the tests Issues or PRs related to tests label Jun 26, 2024
@jcrist jcrist added this to the Q3 2024 milestone Jun 26, 2024
@jcrist jcrist changed the title Improve backends test suite coverage test: improve backends test suite coverage Jun 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Issues or PRs related to tests
Projects
Status: backlog
Development

No branches or pull requests

1 participant