Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow testing ∃x: p(x): a test passes at least once out of N tries #235

Open
Janiczek opened this issue Oct 18, 2024 · 3 comments
Open

Comments

@Janiczek
Copy link
Collaborator

Janiczek commented Oct 18, 2024

Normal fuzz tests check that a fuzzer passes the test every time (∀x: p(x)).
Sometimes it's helpful to test whether a fuzzer passes the test at least once (∃x: p(x)).

API sketch (with bad naming):

Test.fuzz        : Fuzzer a -> String -> (a -> Expectation) -> Test
Test.fuzzCanPass : Fuzzer a -> String -> (a -> Expectation) -> Test

This can be inefficiently done in userspace like this:

{-| Will run the fuzzer up to 100 times and check if it passes the test at least once.
-}
canPass : Fuzzer a -> (a -> Bool) -> Expectation
canPass fuzzer pred =
    let
        tries =
            100

        go n nLeft =
            if nLeft <= 0 then
                Expect.fail <| "Expected the fuzzer to pass the test at least once out of " ++ String.fromInt tries ++ " tries."

            else
                let
                    seenItPass =
                        Fuzz.examples n fuzzer
                            |> List.any pred
                in
                if seenItPass then
                    Expect.pass

                else
                    go (n + 1) (nLeft - n)
    in
    go 1 tries

since the exposed API lacks the ability to run a single fuzzer with a varying seed. And it's kinda abusing Fuzz.examples.

The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.

@Janiczek
Copy link
Collaborator Author

Alternatively we could expose a function that negates an Expectation, makes failures into passes and vice versa. Then the test "fuzzer X can pass predicate Y at least once" would become "test !Y with fuzzer X doesn't pass"

@Janiczek
Copy link
Collaborator Author

Janiczek commented Oct 18, 2024

Conceptually Test.fuzz checks ∀x: p(x)
And we want ∃x: p(x)

The original proposal here would give us that directly, while the negation would give us that via the identity ∃x: p(x) === not ∀x: not p(x)

@Janiczek Janiczek changed the title Allow testing that a test _can_ pass at least once out of N tries Allow testing ∃x: p(x): a test passes at least once out of N tries Oct 18, 2024
@Janiczek
Copy link
Collaborator Author

Example of where this would be useful: testing Random.Generators: https://github.com/Janiczek/nu-ashworld/blob/2b31f64fb303de4d29d172d704e6d456885954f9/tests/Data/Fight/GeneratorTest.elm#L471-L526

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant