Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lin Bytes negative test fails to trigger #518

Open
jmid opened this issue Jan 15, 2025 · 1 comment · Fixed by #519
Open

Lin Bytes negative test fails to trigger #518

jmid opened this issue Jan 15, 2025 · 1 comment · Fixed by #519
Labels
test suite reliability Issue concerns tests that should behave more predictably

Comments

@jmid
Copy link
Collaborator

jmid commented Jan 15, 2025

#516 changed the Lin Bytes test, turning it into a negative test.

The merge to main of the PR then failed to find a counterexample within 1000 attempts on MSVC bytecode 5.3:
https://github.com/ocaml-multicore/multicoretests/actions/runs/12791598373/job/35660087124

random seed: 278988859
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin Bytes test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin Bytes test with Domain (generating)
[ ]   75    0    0   75 / 1000    60.1s Lin Bytes test with Domain
[ ]  343    0    0  343 / 1000   121.0s Lin Bytes test with Domain
[ ]  926    0    0  926 / 1000   181.0s Lin Bytes test with Domain
[✗] 1000    0    0 1000 / 1000   188.4s Lin Bytes test with Domain

[ ]    0    0    0    0 /  250     0.0s Lin Bytes test with Thread
[ ]   69    0    0   69 /  250   243.1s Lin Bytes test with Thread
[✓]  250    0    0  250 /  250   300.4s Lin Bytes test with Thread

--- Failure --------------------------------------------------------------------

Test Lin Bytes test with Domain failed:

Negative test Lin Bytes test with Domain succeeded but was expected to fail
================================================================================
failure (1 tests failed, 0 tests errored, ran 2 tests)
File "src/bytes/dune", line 12, characters 7-16:
12 |  (name lin_tests)
            ^^^^^^^^^
(cd _build/default/src/bytes && ./lin_tests.exe --verbose)
Command exited with code 1.

We should just increase the count for a start, as it is not taking particularly long to execute - and doesn't cost anything for platforms finding a counterexample earlier 🤷

@jmid jmid added the test suite reliability Issue concerns tests that should behave more predictably label Jan 15, 2025
@jmid jmid closed this as completed in #519 Jan 16, 2025
@jmid jmid reopened this Jan 22, 2025
@jmid
Copy link
Collaborator Author

jmid commented Jan 22, 2025

Reopening this as this happened on the push to main in connection with the 0.7 release for MSVC bytecode trunk:
https://github.com/ocaml-multicore/multicoretests/actions/runs/12891458000/job/35943440732

random seed: 318271471
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 5000     0.0s Lin Bytes test with Domain
[ ]    0    0    0    0 / 5000     0.0s Lin Bytes test with Domain (generating)
[ ]  609    0    0  609 / 5000    88.8s Lin Bytes test with Domain
[ ]  902    0    0  902 / 5000   157.6s Lin Bytes test with Domain
[ ] 1208    0    0 1208 / 5000   258.0s Lin Bytes test with Domain
[ ] 1735    0    0 1735 / 5000   903.8s Lin Bytes test with Domain
[ ] 2176    0    0 2176 / 5000  1013.6s Lin Bytes test with Domain
[ ] 3118    0    0 3118 / 5000  1077.0s Lin Bytes test with Domain
[ ] 3493    0    0 3493 / 5000  1138.1s Lin Bytes test with Domain
[ ] 4085    0    0 4085 / 5000  1199.2s Lin Bytes test with Domain
[ ] 4247    0    0 4247 / 5000  1373.2s Lin Bytes test with Domain
[✗] 5000    0    0 5000 / 5000  1422.3s Lin Bytes test with Domain

I wonder if there's something fundamental with (MSVC) bytecode or whether it can be solved, e.g., by adjusting weights a bit like src/dynarray/lin_tests.ml does... 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test suite reliability Issue concerns tests that should behave more predictably
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant