Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed errors when running tests in random order #5533

Merged
merged 2 commits into from
Jun 14, 2021

Conversation

radarhere
Copy link
Member

Helps #5526

With pytest-randomly installed, Tests/test_decompression_bomb.py can fail, because we are currently setting Image.MAX_IMAGE_PIXELS in three methods, and not resetting it until the tests in TestDecompressionBomb are done in teardown_class.

This PR instead resets it after each method.

@radarhere
Copy link
Member Author

radarhere commented Jun 10, 2021

Running py.test-3.8 Tests/test_imagefile.py Tests/oss-fuzz/test_fuzzers.py, test_truncated_with_errors and test_broken_datastream_with_errors can fail with "DID NOT RAISE <class 'OSError'>".

I've added code to reset enable_decompression_error.

@radarhere radarhere changed the title Reset MAX_IMAGE_PIXELS after each TestDecompressionBomb test Fixed errors when running tests in random order Jun 10, 2021
@hugovk
Copy link
Member

hugovk commented Jun 14, 2021

In normal order, pytest passes.

With pytest-reverse, it crashes at Tests/test_qt_image_qapplication.py:

$ pytest --reverse
======================================================= test session starts ========================================================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.9.0, pluggy-0.13.1
rootdir: /Users/hugo/github/Pillow, configfile: setup.cfg, testpaths: Tests
plugins: cov-2.12.1, hypothesis-6.9.2, requests-mock-1.8.0, rerunfailures-9.1.1, flaky-3.7.0, mockito-0.0.4, xdist-2.2.1, reverse-1.2.0, testmon-1.0.3, timeout-1.4.2, forked-1.3.0
collected 2812 items

Tests/oss-fuzz/test_fuzzers.py ............................................................................................. [  3%]
............................................................................................................................ [  7%]
............................................................................................................................ [ 12%]
............................................................................................................................ [ 16%]
............................................................................................................................ [ 20%]
............................................................................................................................ [ 25%]
............................................................................................................................ [ 29%]
....................................................................................................................         [ 33%]
Tests/test_webp_leaks.py .                                                                                                   [ 33%]
Tests/test_util.py ......                                                                                                    [ 34%]
Tests/test_tiff_ifdrational.py ....                                                                                          [ 34%]
Tests/test_tiff_crashes.py .............ss                                                                                   [ 34%]
Tests/test_shell_injection.py ....                                                                                           [ 34%]
Tests/test_sgi_crash.py ...........                                                                                          [ 35%]
Tests/test_qt_image_toqimage.py .                                                                                            [ 35%]
Tests/test_qt_image_qapplication.py %

With pytest-randomly, pytest also crashes at Tests/test_qt_image_qapplication.py

$ pytest
======================================================= test session starts ========================================================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.9.0, pluggy-0.13.1
Using --randomly-seed=824403459
rootdir: /Users/hugo/github/Pillow, configfile: setup.cfg, testpaths: Tests
plugins: cov-2.12.1, hypothesis-6.9.2, requests-mock-1.8.0, rerunfailures-9.1.1, flaky-3.7.0, randomly-3.8.0, mockito-0.0.4, xdist-2.2.1, reverse-1.2.0, testmon-1.0.3, timeout-1.4.2, forked-1.3.0
collected 2812 items

Tests/test_file_gimppalette.py ..                                                                                            [  0%]
Tests/test_pickle.py ...............                                                                                         [  0%]
Tests/test_file_jpeg2k.py .....s....................sss....                                                                  [  1%]
Tests/test_box_blur.py ..............                                                                                        [  2%]
Tests/test_image_rotate.py ..............                                                                                    [  2%]
Tests/test_file_apng.py ....................                                                                                 [  3%]
Tests/test_psdraw.py ...                                                                                                     [  3%]
Tests/test_image_point.py ....                                                                                               [  3%]
Tests/test_image_array.py ..                                                                                                 [  3%]
Tests/test_lib_pack.py ....................................                                                                  [  5%]
Tests/test_file_eps.py .................                                                                                     [  5%]
Tests/test_image_getbands.py .                                                                                               [  5%]
Tests/test_image_filter.py .........                                                                                         [  6%]
Tests/test_qt_image_qapplication.py %

As that file isn't touched here, skipping it with --ignore=Tests/test_qt_image_qapplication.py means both reverse and random pass. 👍

Would it be worth adding random or reverse (reverse being more "stable") to one of the CI jobs?

@radarhere radarhere merged commit 3066c48 into python-pillow:master Jun 14, 2021
@radarhere radarhere deleted the random branch June 14, 2021 12:43
@radarhere
Copy link
Member Author

Ok, I've created PR #5542.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants