-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixed errors when running tests in random order #5533
Conversation
Running I've added code to reset |
In normal order, With pytest-reverse, it crashes at $ pytest --reverse
======================================================= test session starts ========================================================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.9.0, pluggy-0.13.1
rootdir: /Users/hugo/github/Pillow, configfile: setup.cfg, testpaths: Tests
plugins: cov-2.12.1, hypothesis-6.9.2, requests-mock-1.8.0, rerunfailures-9.1.1, flaky-3.7.0, mockito-0.0.4, xdist-2.2.1, reverse-1.2.0, testmon-1.0.3, timeout-1.4.2, forked-1.3.0
collected 2812 items
Tests/oss-fuzz/test_fuzzers.py ............................................................................................. [ 3%]
............................................................................................................................ [ 7%]
............................................................................................................................ [ 12%]
............................................................................................................................ [ 16%]
............................................................................................................................ [ 20%]
............................................................................................................................ [ 25%]
............................................................................................................................ [ 29%]
.................................................................................................................... [ 33%]
Tests/test_webp_leaks.py . [ 33%]
Tests/test_util.py ...... [ 34%]
Tests/test_tiff_ifdrational.py .... [ 34%]
Tests/test_tiff_crashes.py .............ss [ 34%]
Tests/test_shell_injection.py .... [ 34%]
Tests/test_sgi_crash.py ........... [ 35%]
Tests/test_qt_image_toqimage.py . [ 35%]
Tests/test_qt_image_qapplication.py % With pytest-randomly, $ pytest
======================================================= test session starts ========================================================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.9.0, pluggy-0.13.1
Using --randomly-seed=824403459
rootdir: /Users/hugo/github/Pillow, configfile: setup.cfg, testpaths: Tests
plugins: cov-2.12.1, hypothesis-6.9.2, requests-mock-1.8.0, rerunfailures-9.1.1, flaky-3.7.0, randomly-3.8.0, mockito-0.0.4, xdist-2.2.1, reverse-1.2.0, testmon-1.0.3, timeout-1.4.2, forked-1.3.0
collected 2812 items
Tests/test_file_gimppalette.py .. [ 0%]
Tests/test_pickle.py ............... [ 0%]
Tests/test_file_jpeg2k.py .....s....................sss.... [ 1%]
Tests/test_box_blur.py .............. [ 2%]
Tests/test_image_rotate.py .............. [ 2%]
Tests/test_file_apng.py .................... [ 3%]
Tests/test_psdraw.py ... [ 3%]
Tests/test_image_point.py .... [ 3%]
Tests/test_image_array.py .. [ 3%]
Tests/test_lib_pack.py .................................... [ 5%]
Tests/test_file_eps.py ................. [ 5%]
Tests/test_image_getbands.py . [ 5%]
Tests/test_image_filter.py ......... [ 6%]
Tests/test_qt_image_qapplication.py % As that file isn't touched here, skipping it with Would it be worth adding random or reverse (reverse being more "stable") to one of the CI jobs? |
Ok, I've created PR #5542. |
Helps #5526
With pytest-randomly installed, Tests/test_decompression_bomb.py can fail, because we are currently setting
Image.MAX_IMAGE_PIXELS
in three methods, and not resetting it until the tests in TestDecompressionBomb are done inteardown_class
.This PR instead resets it after each method.