Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make more tests robust & independent #797

Open
1 task
glatterf42 opened this issue Mar 11, 2024 · 8 comments
Open
1 task

Make more tests robust & independent #797

glatterf42 opened this issue Mar 11, 2024 · 8 comments
Labels
bug Doesn't work as advertised/unintended effects ci Continuous integration
Milestone

Comments

@glatterf42
Copy link
Member

glatterf42 commented Mar 11, 2024

Unfortunately, even though #784 improved the tests a lot, we still seem to see flaky behaviour. So let's continue the same kind of work from #784 on these tests:

  • FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_add_share_mode_lo - assert False
     +  where False = <function isclose at 0x111d84700>(0.5416666666666666, 0.525)
     +    where <function isclose at 0x111d84700> = np.isclose
@glatterf42 glatterf42 added bug Doesn't work as advertised/unintended effects ci Continuous integration labels Mar 11, 2024
@glatterf42
Copy link
Member Author

It seems like this flaky behaviour wasn't completely resolved by the first PR:

  • ERROR message_ix/tests/tools/test_add_year.py::test_add_year - RuntimeError: unhandled Java exception: Error opening GDX file: Expected data marker (BOI) not found in GDX file
    On: windows-latest-py3.8

@glatterf42
Copy link
Member Author

glatterf42 commented Mar 13, 2024

In addition, I'm seeing another error for a test that we thought we made non-flaky:

  • ERROR message_ix/tests/test_feature_bound_activity_shares.py::test_b_a_u_all_modes[299-5] - FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/work/message_ix/message_ix/message_ix/model/225a'
    On: macos-latest-py3.8, macos-latest-py3.9, macos-latest-py3.10, ubuntu-latest-py3.9, ubuntu-latest-py3.11

This is happening for macos-latest-py3.8, macos-latest-py3.9, and macos-latest-py3.10, but for 3.9 and 3.10, the test case in question is rather [299-2] (and everything else is exactly the same).
Edit: now this is also happening with [299-3] on macos-latest-py3.9 and [299-1] on macos-latest-py3.10.

@glatterf42
Copy link
Member Author

There's a new flaky error for test_add_year:

  • ERROR message_ix/tests/tools/test_add_year.py::test_add_year_cli - RuntimeError: unhandled Java exception: Error opening GDX file: Expected data marker (SYMBOL_2) not found in GDX file
    On: windows-latest-py3.10

@glatterf42 glatterf42 added this to the 3.9 milestone Mar 22, 2024
@glatterf42
Copy link
Member Author

glatterf42 commented Mar 29, 2024

Another one bites the dust:

  • FAILED message_ix/tests/test_cli.py::test_copy_model - AssertionError: assert 1 == 0
     +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
    On: macos-latest-py3.9, macos-latest-py3.11

@glatterf42
Copy link
Member Author

Another flaky test appeared today:

  •  FAILED message_ix/tests/test_util.py::test_testing_make_scenario - RuntimeError: unhandled Java exception: The index set 'emission' does not have an element 'CO2'!
    On: macos-13-py3.10

@glatterf42
Copy link
Member Author

There goes the next one:

  • FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_add_share_mode_lo - assert False
     +  where False = <function isclose at 0x000001BF5AF800F0>(0.5416666666666666, 0.525)
     +    where <function isclose at 0x000001BF5AF800F0> = np.isclose
    On: windows-latest-py3.11

@glatterf42 glatterf42 modified the milestones: 3.9, 3.10 May 23, 2024
@glatterf42
Copy link
Member Author

Postponing as internal PRs are not that time-sensitive anyway.

@glatterf42
Copy link
Member Author

  •  FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_commodity_share_lo - assert False
     +  where False = <function isclose at 0x10fe1d8b0>(0.0, 0.15384615384615385)
     +    where <function isclose at 0x10fe1d8b0> = np.isclose
     +    and   0.15384615384615385 = <function test_commodity_share_lo.<locals>.calc_share at 0x161a71ca0>(<message_ix.core.Scenario object at 0x158b22370>)
    On: macos-latest-py3.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Doesn't work as advertised/unintended effects ci Continuous integration
Projects
None yet
Development

No branches or pull requests

1 participant