Skip to content

Commit

Permalink
Documentation: Rework the documentation and the examples
Browse files Browse the repository at this point in the history
  • Loading branch information
saroupille committed May 12, 2024
1 parent 0a126b9 commit a7ea8f5
Show file tree
Hide file tree
Showing 22 changed files with 1,022 additions and 630 deletions.
132 changes: 110 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,51 @@

## Overview

Bam is an OCaml library designed for writing property-based tests. It
simplifies the process of defining and using generators for tests,
offering a monad-like structure that integrates seamlessly with
shrinking strategies. This design aims to make shrinking both
predictable and effective, thereby enhancing the debugging experience.

Bam is an OCaml library designed for property-based testing (PBT).
Property-based testing is a methodology where properties are verified
across numerous randomly generated values. When a counterexample is
found, a shrinker can be employed to produce smaller, more
comprehensible counterexamples, thus facilitating easier debugging.

Bam is an OCaml library designed for property-based testing (PBT).
Property-based testing is a methodology where properties are verified
across numerous randomly generated values. When a counterexample is
found, a shrinker can be employed to produce smaller, more
comprehensible counterexamples, thus facilitating easier debugging.

## Key Features

- Standard Module: The {!module:Std} module provides some basic
generators with predefined shrinking strategies
- **Monad-like Generators**: The library facilitates the easy creation of
new generators, adhering to a monad-like pattern that works
seamlessly with shrinking mechanisms.

- **Standard Library**: The Std module includes basic generators
equipped with multiple shrinking strategies. The modularity of the
monad-like generators makes it straightforward to develop custom
generators for your types.

- **PPX support**: Automatically derives generators based on type
description. The deriver is customizable to ensure smooth
integration into your codebase.

- Monad-like Generators: The library enables easy creation of new
generators, following a monad-like pattern that works harmoniously
with shrinking mechanisms.
- **Tezt Integration**: Features integration with the
[Tezt](https://gitlab.com/nomadic-labs/tezt) test framework,
aiming to provide a user-friendly experience, especially notable
in debugging scenarios

- Shrinking Strategies: Various default shrinking strategies are
available, aiding in the efficient identification of minimal
counter-examples.
- **Internal shrinking**: Offers various default shrinking strategies
that help efficiently pinpoint minimal counterexamples. Each
strategy is "internal," ensuring that shrinking processes do not
generate new values, only smaller iterations of the original
values.

- Custom Shrinkers: The {!Gen} module allows for the definition of
ad-hoc shrinkers.
- **Custom Shrinking**: The library supports the definition of custom
shrinkers that combine well with the already shrinking strategies

- Documentation on Shrinking: For those interested in understanding
the intricacies of shrinking within this library, a detailed primer
is available [here](https://francoisthire.github.io/bam/bam/shrinking.html).
- **Documentation on Shrinking**: A detailed primer on the intricacies
of shrinking within this library is available for those interested
in deepening their understanding. Access it
[here](https://francoisthire.github.io/bam/bam/shrinking.html).

## Installation

Expand All @@ -37,6 +56,8 @@ predictable and effective, thereby enhancing the debugging experience.
opam install bam tezt-bam
```

Compatibility is ensured for OCaml versions `4.14.2`, `5.0.0` and
`5.1.0`.

## Usage

Expand All @@ -45,16 +66,83 @@ A simple test can be run as follows:
```ocaml
open Tezt_bam
type t = Foo of {a: int; b : string} | Bar of int list[@@deriving gen]
(** The deriver creates a value [val gen : t Bam.Std.t]. *)
let register () =
let gen = Std.int () in
let property _x = Ok () in
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~__FILE__ ~title:"Simple example of bam" ~tags:["bam"; "simple"]
~gen ~property ()
```

![executation of bam example](media/bam.png)


or without using the PPX deriver, the same example could be written as follows:

```ocaml
open Tezt_bam
type t = Foo of {a: int; b : string} | Bar of int list
let gen =
let open Bam.Std.Syntax in
let gen_Foo =
let* a = Bam.Std.int () in
let* b = Bam.Std.string ~size:(Bam.Std.int ~max:10 ()) () in
return (Foo {a; b})
in
let gen_Bar =
let* arg_0 =
Bam.Std.list ~size:(Bam.Std.int ~max:10 ()) (Bam.Std.int ())
in
return (Bar arg_0)
in
Bam.Std.oneof [(1, gen_Foo); (1, gen_Bar)]
let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~__FILE__ ~title:"Simple example of bam" ~tags:["bam"; "simple"]
~gen ~property ()
```

This example and how it can be run are explained through the various
[examples](https://github.com/francoisthire/bam/tree/master/example). We invite
you to read them if you are interested in starting with the library!

## PPX

At the moment, the PPX support is partial but should cover a vast majority of
use cases. Moreover, the deriver supports many attributes to tune its behavior.
In particular, one can specify a generator when it is not supported by the
deriver.

More examples can be found [here](https://github.com/francoisthire/bam/tree/master/example).
There is no proper documentation for the PPX yet. Instead, we invite you to look
at the many examples
[here](https://github.com/francoisthire/bam/blob/master/test/ppx.ml) to see what
types are supported and how the deriver can be tuned.

Contributions are welcome!

## License

Expand Down
18 changes: 8 additions & 10 deletions example/2-simple-failure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,11 @@ dune exec example/main.exe -- simple_failure
[15:30:55.229] Try again with: _build/default/example/main.exe --verbose --file example/2-simple-failure/simple_failure.ml --title 'Simple failure example with bam' --seed 555586205
```

This time `Tezt` is more verbose. On a failure, the counter-example
found is printed. By default, shrinking is not enabled, hence the
counter-example found can be rather big.
This time `Tezt` is more verbose. On a failure, the counter-example found is
printed. By default, shrinking is not enabled, hence the counter-example found
can be rather big and quite complex to use for debugging.

Moreover, the error raised associated to the counter-example is
printed.

If you want to run this example again using the shrinking of bam, you can use the option `--shrink`. To be sue the very same initial counter-example will be printed, you can use the seed given by Tezt:
If you want to run this example again using the shrinking of *Bam*, you can use the option `--shrink`. To be sure the very same initial counter-example will be printed, you can use the seed given by Tezt:

```ocaml
$ dune exec example/main.exe -- simple_failure --shrink --seed 555586205
Expand All @@ -57,9 +54,10 @@ $ dune exec example/main.exe -- simple_failure --shrink --seed 555586205

At this stage, we see the shrinking heuristic allowed us to find a
smaller-counter example which is `100`. Such a counter-example can be
a good candidate for starting debugging the test.
a good candidate when debugging the test.


**Next steps:**
- The next example
[3-debugging](https://github.com/francoisthire/bam/tree/master/example/3-debugging)
will investigate how bam can help for debugging
[3-writing-generators](https://github.com/francoisthire/bam/tree/master/example/3-writing-generators)
will explain how the monadic interface of *bam* is useful to write generators easily.
76 changes: 76 additions & 0 deletions example/3-writing-generators/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# 4-writing-generators

One can take advantage of the monadic interface of *bam* to define
generators.

```ocaml
open Tezt_bam
(** The deriver creates a value [val gen : t Bam.Std.t]. *)
type t = Foo of {a: int; b: string} | Bar of int list [@@deriving gen]
let gen =
let open Bam.Std.Syntax in
let gen_Foo =
let* a = Bam.Std.int () in
let* b = Bam.Std.string ~size:(Bam.Std.int ~max:10 ()) () in
return (Foo {a; b})
in
let gen_Bar =
let* arg_0 = Bam.Std.list ~size:(Bam.Std.int ~max:10 ()) (Bam.Std.int ()) in
return (Bar arg_0)
in
Bam.Std.oneof [(1, gen_Foo); (1, gen_Bar)]
let pp fmt = function
| Foo {a; b} ->
Format.fprintf fmt "Foo {x=%d;%s}" a b
| Bar list ->
Format.fprintf fmt "Bar %a"
(Format.pp_print_list ~pp_sep:Format.pp_print_space Format.pp_print_int)
list
let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~pp ~__FILE__ ~title:"Writing generators"
~tags:["bam"; "writing_generators"]
~gen ~property ()
```

And now run it:

```bash
$ dune exec example/main.exe -- writing_generators --shrink
[11:31:02.335] [pbt] \ | /
[11:31:02.335] [pbt] - BAM -
[11:31:02.335] [pbt] / | \
[11:31:02.335] [pbt] Counter example found:
[11:31:02.335] [pbt] A {x=1001;aaaaz}
[11:31:02.335] [error] Test failed with error:
[11:31:02.335] [error] A counter-example was found
[11:31:02.335] [FAILURE] (1/1, 1 failed) Writing generators
[11:31:02.335] Try again with: _build/default/example/main.exe --verbose --file example/4-writing-generators/writing_generators.ml --title 'Writing generators' --seed 493027010
```

With *bam* it is encouraged to extensively use and abuse of the
monadic interface. The library ensures it will behave well with
respect to shrinking.

However, one can notice that *bam* did not find the smallest counter-example on this example. The reason is due to the default shrinking strategy which is efficient. Fortunately, *bam* provides different shrinking strategies to help you.

**Next steps:**
- The next example
[4-deriving-generators](https://github.com/francoisthire/bam/tree/master/example/4-deriving-generators)
will investigate how `bam-ppx` can be used to generate derivers automatically.
- The next example
[5-shrinking-strategy](https://github.com/francoisthire/bam/tree/master/example/5-shrinking-strategy)
will investigate how we can modify the shrinking strategies to find better counter-examples.
File renamed without changes.
40 changes: 40 additions & 0 deletions example/3-writing-generators/writing_generators.ml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
open Tezt_bam

(** The deriver creates a value [val gen : t Bam.Std.t]. *)
type t = Foo of {a: int; b: string} | Bar of int list [@@deriving gen]

let gen =
let open Bam.Std.Syntax in
let gen_Foo =
let* a = Bam.Std.int () in
let* b = Bam.Std.string ~size:(Bam.Std.int ~max:10 ()) () in
return (Foo {a; b})
in
let gen_Bar =
let* arg_0 = Bam.Std.list ~size:(Bam.Std.int ~max:10 ()) (Bam.Std.int ()) in
return (Bar arg_0)
in
Bam.Std.oneof [(1, gen_Foo); (1, gen_Bar)]

let pp fmt = function
| Foo {a; b} ->
Format.fprintf fmt "Foo {x=%d;%s}" a b
| Bar list ->
Format.fprintf fmt "Bar %a"
(Format.pp_print_list ~pp_sep:Format.pp_print_space Format.pp_print_int)
list

let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~pp ~__FILE__ ~title:"Writing generators"
~tags:["bam"; "writing_generators"]
~gen ~property ()
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ One can take advantage of the `bam-ppx` library to derive automatically generato
```ocaml
open Tezt_bam
type t = A of {x: int; b: bool} | B of string
type t = A of {x: int; b: bool} | B of string [@weight 4]
[@@deriving gen] [@@size.min 1] [@@size.max 10]
let pp fmt = function
Expand All @@ -26,7 +26,16 @@ let register () =
~tags:["bam"; "deriving_generators"]
~gen ~property ()
```
The attribute `[@@deriving gen]` derives automatically a generator for the type attached. The generator derived with `gen` can be tuned via other `attributes` as shown in the example above. In this example, we have specified that the minimum size and the maximum size that should be used are `1` and `10`.
The attribute `[@@deriving gen]` derives automatically a generator for the type
attached. The generator derived with `gen` can be tuned via other `attributes`
as shown in the example above. In this example, we have specified that the
minimum size and the maximum size that should be used are `1` and `10`. And that
the weight of the constructor `B` if `4` (by default it is `1`). Meaning that
the variant `B` will be generated `4` times more often than variant `A`.

More examples can be found
[here](https://github.com/francoisthire/bam/blob/master/test/ppx.ml) to
understand how the deriver can be tuned.

And now if your un it:

Expand All @@ -43,5 +52,11 @@ $ dune exec example/main.exe -- deriving_generators --shrink
[23:37:37.020] Try again with: _build/default/example/main.exe --verbose --file example/5-deriving-generators/deriving_generators.ml --title 'Deriving generators' --seed 268040292
```


**Next steps:**
- The next example
[5-shrinking-strategy](https://github.com/francoisthire/bam/tree/master/example/5-shrinking-strategy)
will investigate how we can modify the shrinking strategies to find better counter-examples.
- The next example
[6-debugging](https://github.com/francoisthire/bam/tree/master/example/6-debugging)
will investigate how *tezt-bam* can help for debugging

27 changes: 27 additions & 0 deletions example/4-deriving-generators/deriving_generators.ml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
open Tezt_bam

type t = Foo of {a: int; b: string} | Bar of int list
[@@deriving gen] [@@size.min 1] [@@size.max 10]

let pp fmt = function
| Foo {a; b} ->
Format.fprintf fmt "Foo {x=%d;%s}" a b
| Bar list ->
Format.fprintf fmt "Bar %a"
(Format.pp_print_list ~pp_sep:Format.pp_print_space Format.pp_print_int)
list

let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~pp ~__FILE__ ~title:"Deriving generators"
~tags:["bam"; "deriving_generators"]
~gen ~property ()
File renamed without changes.
Loading

0 comments on commit a7ea8f5

Please sign in to comment.