Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make tests deterministic #329

Open
hugobuddel opened this issue Jan 8, 2024 · 1 comment
Open

Make tests deterministic #329

hugobuddel opened this issue Jan 8, 2024 · 1 comment
Labels
tests Related to unit or integration tests

Comments

@hugobuddel
Copy link
Collaborator

Some tests can randomly fail, e.g.

https://github.com/AstarVienna/ScopeSim/actions/runs/7443459990/job/20248391071

================================== FAILURES ===================================
________ TestApplyTo.test_noise_reduces_with_square_root_of_ndit[9-1] _________

self = <scopesim.tests.tests_effects.test_BasicReadoutNoise.TestApplyTo object at 0x0000027593CEF8B0>
ndit = 9, noise_std = 1

    @pytest.mark.parametrize("noise_std", [1, 9, 100])
    @pytest.mark.parametrize("ndit", [1, 9, 100])
    def test_noise_reduces_with_square_root_of_ndit(self, ndit, noise_std):
        dtcr = _basic_detector(width=256)
        ron = PoorMansHxRGReadoutNoise(noise_std=noise_std, n_channels=64, ndit=ndit)
        dtcr = ron.apply_to(dtcr)
        noise_real = np.std(dtcr._hdu.data)
    
>       assert noise_real == approx(noise_std * ndit**0.5, rel=0.1)
E       assert 3.3188548593454184 == 3.0 � 3.0e-01
E         comparison failed
E         Obtained: 3.3188548593454184
E         Expected: 3.0 � 3.0e-01

scopesim\tests\tests_effects\test_BasicReadoutNoise.py:36: AssertionError

It would be better to make these deterministic.

PoorMansHxRGReadoutNoise can be made deterministic by passing the random_seed parameter, and np.std probably has something similar.

@teutoburg
Copy link
Contributor

Yes, I've encountered cases like this before. Usually pass if run again. But indeed, deterministic should be the goal!
Perhaps we can fix this in the course of #298

@teutoburg teutoburg added the tests Related to unit or integration tests label Jan 8, 2024
@teutoburg teutoburg moved this from 🆕 New to 📋 Backlog in ScopeSim-development Jan 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Related to unit or integration tests
Projects
Status: 📋 Backlog
Development

No branches or pull requests

2 participants