Skip to content

Latest commit

 

History

History
274 lines (219 loc) · 14.1 KB

CHANGELOG.md

File metadata and controls

274 lines (219 loc) · 14.1 KB
Raw

Releases

Version Notes
2.2.0 Fuzz: Add Fuzz.filterMap
2.1.2 Fuzz: Remove arbitrary limit on amount of randomness drawn
2.1.1 Test.Html.Query: Change how boolean attributes are rendered
2.1.0 Add Test.Html.Selector.exactText
2.0.1 Documentation fixes
2.0.0 Reimplements fuzzing+shrinking, adds fuzzer distribution reporting. Most notably readds Fuzz.andThen. See "Changes in 2.0.0"
1.2.2 Fixes a crash in Test.Html when the HTML contains nested Html.Lazy nodes. #78
1.2.1 Many small documentation fixes. Improve error messages when failing to simulate an event.
1.2.0 Add HTML tests. #41
1.0.0 Update for Elm 0.19. Remove Fuzz.andThen, Fuzz.conditional, and Test.Runner.getFailure. Fail on equating floats to encourage checks with tolerance. Test.Runner.fuzz now returns a Result.
renamed from elm-community/elm-test (below) to elm-explorations/test (above)
4.0.0 Add only, skip, todo; change Fuzz.frequency to fail rather than crash on bad input, disallow tests with blank or duplicate descriptions.
3.1.0 Add Expect.all
3.0.0 Update for Elm 0.18; switch the argument order of Fuzz.andMap.
2.1.0 Switch to rose trees for Fuzz.andThen, other API additions.
2.0.0 Scratch-rewrite to project-fuzzball
1.0.0 ElmTest initial release

Changes in 2.0.0

The changes can be grouped into these categories:

  1. Fuzzing and shrinking reimplementation (re-adding Fuzz.andThen etc.)
  2. Test.Distribution: fuzzer distribution reporting
  3. Test.Html.Event additions
  4. Expect.true and Expect.false removal
  5. Test.Runner.Failure.format removal
  6. Fuzzer behaviour changes

1. Fuzzing and shrinking reimplementation

Fuzzing and shrinking has been reimplemented: the rose tree approach has been replaced with the "internal shrinking" approach found in the Python test library Hypothesis.

In short, shrinking is now done on the PRNG history instead of on the generated values themselves. This is hidden from the user: the Shrink module has now been removed.

This new approach allows us to reintroduce Fuzz.andThen and remove Fuzz.custom: in case you were forced to use Fuzz.custom and a Random generator, you'll now be able to express this logic with Fuzz alone.

We've also taken the opportunity to expand the Fuzz module API with functions that you'd normally reach into *-extra packages for: sequence, traverse, listOfLength, intAtLeast etc., and some quality of life helpers like examples and labelExamples:

> import Fuzz
> Fuzz.examples 5 (Fuzz.intRange 1 10)
[6,7,7,5,4] : List Int

Float fuzzers will now generate NaN and infinities by default (use Fuzz.niceFloat instead if you don't want these values) and shrink towards simple human-readable fractions like 0.5 instead of small floats like 1.1102230246251567e-15.

Full list of changes:

  • Generic helpers

    • ➕ andThen : (a -> Fuzzer b) -> Fuzzer a -> Fuzzer b
    • ➕ filter : (a -> Bool) -> Fuzzer a -> Fuzzer a
    • ➕ lazy : (() -> Fuzzer a) -> Fuzzer a
    • ➕ map6
    • ➕ map7
    • ➕ map8
    • ➕ sequence : List (Fuzzer a) -> Fuzzer (List a)
    • ➕ traverse : (a -> Fuzzer b) -> List a -> Fuzzer (List b)
    • ➕ shuffledList : List a -> Fuzzer (List a)

  • oneOf helpers

    • ➕ oneOfValues : List a -> Fuzzer a
    • ➕ frequencyValues : List ( Float, a ) -> Fuzzer a

  • REPL helpers

    • ➕ examples : Int -> Fuzzer a -> List a
    • ➕ labelExamples : Int -> List ( String, a -> Bool ) -> Fuzzer a -> List ( List String, Maybe a )

  • Tuples

    • 📝 tuple changed into pair : Fuzzer a -> Fuzzer b -> Fuzzer ( a, b )
    • 📝 tuple3 changed into triple : Fuzzer a -> Fuzzer b -> Fuzzer c -> Fuzzer ( a, b, c )

  • Bools

    • ➕ weightedBool : Float -> Fuzzer Bool

  • Ints

    • ➕ intAtLeast : Int -> Fuzzer Int
    • ➕ intAtMost : Int -> Fuzzer Int
    • ➕ uniformInt : Int -> Fuzzer Int

  • Floats

    • ➕ floatAtLeast : Float -> Fuzzer Float
    • ➕ floatAtMost : Float -> Fuzzer Float
    • ➕ niceFloat : Fuzzer Float

  • ASCII

    • ➕ asciiChar : Fuzzer Char
    • ➕ asciiString : Fuzzer String

  • Collections of a given length

    • ➕ listOfLength : Int -> Fuzzer a -> Fuzzer (List a)
    • ➕ listOfLengthBetween : Int -> Int -> Fuzzer a -> Fuzzer (List a)
    • ➕ stringOfLength : Int -> Fuzzer String
    • ➕ stringOfLengthBetween : Int -> Int -> Fuzzer String
    • ➕ asciiStringOfLength : Int -> Fuzzer String
    • ➕ asciiStringOfLengthBetween : Int -> Int -> Fuzzer String

  • Escape hatches

    • ➖ custom : Generator a -> Shrinker a -> Fuzzer a
    • ➕ (discouraged escape hatch) fromGenerator : Generator a -> Fuzzer a

2. Test.Distribution

You can now report or enforce the value distribution of your fuzzers with Test.reportDistribution and Test.expectDistribution.

For more information on this technique, we recommend watching the talk "Building on developers' intuitions to create effective property-based tests" by John Hughes.

Plug these functions into Test.fuzzWith:

  Test.fuzzWith
      { runs = 10000
      , distribution =
          Test.reportDistribution
              [ ( "low", \n -> n == 1 )
              , ( "high", \n -> n == 20 )
              , ( "in between", \n -> n > 1 && n < 20 )
              , ( "outside", \n -> n < 1 || n > 20 )
              ]
      }
      (Fuzz.intRange 1 20)
      "Example for Test.reportDistribution"
      (\n -> Expect.pass)

Reporting will never change the outcome of a test, but will always output a distribution report table next to the other test results:

↓ Distribution.ReportDistributionPassing
✓ Example for Test.reportDistribution

    Distribution report:
    ====================
      in between:  90.5%  (9046x)  ███████████████████████████░░░
      low:          4.9%   (485x)  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
      high:         4.7%   (469x)  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
      outside:        0%     (0x)  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

On the other hand, the function Test.expectDistribution will make sure the fuzzers have the distribution you expect:

    Test.fuzzWith
        { runs = 10000
        , distribution =
            Test.expectDistribution
                [ -- Expecting the number 1 to be generated at least 10% of the time, which is too much!
                  ( Test.Distribution.atLeast 10, "low", \n -> n == 1 )
                , ( Test.Distribution.atLeast 4, "high", \n -> n == 20 )
                , ( Test.Distribution.atLeast 80, "in between", \n -> n > 1 && n < 20 )
                , ( Test.Distribution.zero, "outside", \n -> n < 1 || n > 20 )
                , ( Test.Distribution.moreThanZero, "one", \n -> n == 1 )
                ]
        }
        (Fuzz.intRange 1 20)
        "Example for Test.expectDistribution (with insufficient distribution)"
        (\n -> Expect.pass)

Resulting in a failure (although the fuzz test itself does pass):

↓ Distribution.ExpectDistributionFailingDistribution
✗ Example for Test.expectDistribution (with insufficient distribution)

    Distribution report:
    ====================
      in between:  90%  (9004x)  ███████████████████████████░░░
      high:         5%   (498x)  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
      low:          5%   (498x)  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
      one:          5%   (498x)  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
      outside:      0%     (0x)  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

    Combinations (included in the above base counts):
      low, one:     5%   (498x)  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

    Distribution of label "low" was insufficient:
      expected:  10.000%
      got:       4.980%.

    (Generated 10000 values.)

Note that to reduce the flakiness of such assertion (particularly when your expected percentage is very close to the actual percentage, eg. 4.9% expected and 5% actual), the test runner can occasionally run more tests than specified in FuzzOptions.runs.

📖 This check is implemented using the Wilson score interval and is tuned to accept one flaky result roughly every 10^9 runs.

3. Test.Html.Event additions

These new expectations have been added:

expectNotPreventDefault  : Test.Html.Event.Event msg -> Expectation
expectNotStopPropagation : Test.Html.Event.Event msg -> Expectation
expectPreventDefault     : Test.Html.Event.Event msg -> Expectation
expectStopPropagation    : Test.Html.Event.Event msg -> Expectation

4. Expect.true and Expect.false removal

These functions were encouraging an easy but not idiomatic way to create an Expectation in situations where other Expect.* functions would have been a better choice.

You can still achieve the same result with:

-- BEFORE
foo
  |> Expect.true "string saying what went wrong"

-- AFTER
foo
  |> Expect.equal True
  |> Expect.onFail "string saying what went wrong"

5. Test.Runner.Failure.format removal

The function Test.Runner.Failure.format was deprecated since 1.2.0; with 2.0.0 we're now fully removing it.

6. Fuzzer behaviour changes

String fuzzers now generate Unicode text (including emojis and combining marks). Use asciiString or asciiChar if you only want the ASCII subset.

> Fuzz.examples 3 Fuzz.string
["y\n\t@]̂z❤B","O🔥","̈4🌈&"] : List String

Reported equality failures now show strings both verbatim and with Unicode characters escaped if needed, to better show things like non-breaking spaces.

↓ Expectations
↓ Expect.equal on unicode strings should show pretty output
✗ ascii
           ▼        ▼▼       ▼▼
    "\u{1f63b}\u{1f640}\u{1f47b}" (same string but with unicode characters escaped)
     ▼
    "😻🙀👻"
    ╵
    │ |> Expect.equal
    ╷
    "🙀👻😻🙈"
       ▲▲
    "\u{1f640}\u{1f47b}\u{1f63b}\u{1f648}" (same string but with unicode characters escaped)
           ▲▲▲▲▲▲▲▲▲▲        ▲▲      ▲ ▲