Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up testing by automatically passing if certain files don't change #1221

Closed
f0uriest opened this issue Aug 23, 2024 · 4 comments · Fixed by #1239
Closed

Speed up testing by automatically passing if certain files don't change #1221

f0uriest opened this issue Aug 23, 2024 · 4 comments · Fixed by #1239
Assignees
Labels
testing Adding a new test or fixing an existing one

Comments

@f0uriest
Copy link
Member

We talked about this before and in general it's tricky, but I think we can add a few simple cases:

  • if nothing in desc or tests changes then unit and regression tests aren't needed
  • if nothing in docs/notebooks or desc or tests changes then notebook tests aren't needed
  • if nothing in desc or tests/benchmarks changes then benchmarks aren't needed
@unalmis
Copy link
Collaborator

unalmis commented Aug 24, 2024

Might be nice to also have a run_all_tests label that we can use to override the heuristics given above

@dpanici dpanici added the testing Adding a new test or fixing an existing one label Aug 27, 2024
@YigitElma YigitElma self-assigned this Aug 27, 2024
@YigitElma
Copy link
Collaborator

What if we change dependencies? That requires the whole test to run. I thought about this kind of check before but I think the best we can do is skipping notebook tests if nothing in docs/notebooks, desc or requirements changed. And also we can skip all tests except notebook ones, if the only change is in docs/notebooks.

@YigitElma
Copy link
Collaborator

So, I think the number of PR's that would benefit this kind of change is really low. But I can look into it

@YigitElma
Copy link
Collaborator

#1235 (comment)

YigitElma added a commit that referenced this issue Sep 10, 2024
Resolves #1221 

I assume it will not affect a lot of PR's but some small ones (like this
one or just docs update) can be faster. The conditions are,

- if no change in `desc` or `requirements` don't run `notebook` tests
- if no change in `desc`, `tests (without benchmarks)` or `requirements`
don't run `unit` and `regression` tests
- if no change in `desc`, `tests/benchmarks` or `requirements` don't run
`benchmarks`

If there is a change in the workflow file itself, it will run. If unit
and regression tests are skipped no coverage report will be uploaded and
someone has to override these PRs to merge to master. I have tried
uploading 100% fake coverage report but it could have unintentional
effects as Rory mentioned.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Adding a new test or fixing an existing one
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants