You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Normal fuzz tests check that a fuzzer passes the test every time (∀x: p(x)).
Sometimes it's helpful to test whether a fuzzer passes the test at least once (∃x: p(x)).
API sketch (with bad naming):
Test.fuzz :Fuzzer a ->String->(a ->Expectation)->TestTest.fuzzCanPass :Fuzzer a ->String->(a ->Expectation)->Test
This can be inefficiently done in userspace like this:
{-| Will run the fuzzer up to 100 times and check if it passes the test at least once.-}canPass:Fuzzera-> (a->Bool) ->ExpectationcanPass fuzzer pred =let
tries =100
go n nLeft =if nLeft <=0thenExpect.fail <|"Expected the fuzzer to pass the test at least once out of "++String.fromInt tries ++" tries."elselet
seenItPass =Fuzz.examples n fuzzer
|>List.any pred
inif seenItPass thenExpect.pass
else
go (n +1)(nLeft - n)in
go 1 tries
since the exposed API lacks the ability to run a single fuzzer with a varying seed. And it's kinda abusing Fuzz.examples.
The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.
The text was updated successfully, but these errors were encountered:
Alternatively we could expose a function that negates an Expectation, makes failures into passes and vice versa. Then the test "fuzzer X can pass predicate Y at least once" would become "test !Y with fuzzer X doesn't pass"
Conceptually Test.fuzz checks ∀x: p(x)
And we want ∃x: p(x)
The original proposal here would give us that directly, while the negation would give us that via the identity ∃x: p(x) === not ∀x: not p(x)
Janiczek
changed the title
Allow testing that a test _can_ pass at least once out of N tries
Allow testing ∃x: p(x): a test passes at least once out of N tries
Oct 18, 2024
Normal fuzz tests check that a fuzzer passes the test every time (
∀x: p(x)
).Sometimes it's helpful to test whether a fuzzer passes the test at least once (
∃x: p(x)
).API sketch (with bad naming):
This can be inefficiently done in userspace like this:
since the exposed API lacks the ability to run a single fuzzer with a varying seed. And it's kinda abusing
Fuzz.examples
.The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.
The text was updated successfully, but these errors were encountered: