Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further QCheck2 list shrinker improvements #321

Merged
merged 4 commits into from
Feb 14, 2025

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented Feb 7, 2025

This PR is a small adjustment to the QCheck2 list shrinker, so that lists counterexamples of size 1-3 are treated uniformly,
removing one element in turn. For example [1;2;3] will trigger [2;3], [1;3], [1;2] (in that order).

This makes a difference for the test suite where, e.g., Test bytes have unique chars failed will return minimal 2-char counterexamples as expected (rather than the 3-char ones that really bugged me).

With this, the QCheck2 shrinkers are now in good shape, witnessed by the expect test outputs on all OCaml{4,5} {64,32}-bit combinations.

The exception is the shrinker for QCheck2.Gen.list_size (and derivatives) which I've not attempted to improve.
For reference below I include a shrink_bench.log output for OCaml 4.14.2 on my laptop dominated by

  • long_shrink - 2.070s
  • lists shorter than 432 - 1.034s
  • lists shorter than 4332 - 7.541s
  • fold_left test, fun first - 1.908s

Of these the 3 first use QCheck2.Gen.list_size - and the last uses a function first (a known pain point also for QCheck(1)),

                                                         iteration seed 1234                   iteration seed 8743                   iteration seed 6789               total
Shrink test name                                  Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att    Q1/s   Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
big bound issue59                                - skipped as generator is stateful, making it non-repeatable
long_shrink                                       0.027  149/352     0.782 3086/3148    0.006  147/345     0.563 3090/3150    0.008  145/339     0.725 3051/3111    0.040  2.070
ints arent 0 mod 3                                0.000   84/216     0.000   85/317     0.000   71/204     0.000   82/286     0.000   88/215     0.000    1/1       0.000  0.000
ints are 0                                        0.000   62/63      0.000   60/121     0.000   61/62      0.000   61/122     0.000   62/63      0.000   59/119     0.000  0.000
int32s arent 0l rem 3l                            0.000   41/101     0.000    1/1       0.000   38/109     0.000    1/1       0.000   35/116     0.000    1/1       0.000  0.000
int32s are 0l                                     0.000   31/32      0.000   30/61      0.000   31/32      0.000   30/61      0.000   28/29      0.000   27/55      0.000  0.000
int64s arent 0L rem 3L                            0.000   90/282     0.000    1/1       0.000   77/203     0.000    1/1       0.000   95/330     0.000    1/1       0.000  0.000
int64s are 0L                                     0.000   63/64      0.000   62/125     0.000   63/64      0.000   62/125     0.000   60/61      0.000   59/119     0.000  0.000
ints < 209609                                    - skipped as generator is stateful, making it non-repeatable
nat < 5001                                        0.000    6/56      0.000    7/77      0.000    6/47      0.000    7/69      0.000    3/24      0.000    8/85      0.000  0.000
char never produces 'abcdef'                      0.000    3/3       0.000    1/1       0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000  0.000
printable never produces '!"#$%&'                 0.000    2/15      0.000    1/10      0.000    1/10      0.000    2/15      0.000    2/14      0.000    1/10      0.000  0.000
printable never produces less than '5             0.000    3/3       0.000    1/1       0.000    2/2       0.000    1/1       0.000    3/3       0.000    1/1       0.000  0.000
bytes are empty                                   0.000   15/24      0.000    7/8       0.000   20/29      0.000   10/11      0.000    7/15      0.000    4/5       0.000  0.000
bytes never has a \000 char                       0.000    8/22      0.002   13/27      0.001   17/34      0.003   14/26      0.000    6/18      0.000    8/21      0.001  0.005
bytes never has a \255 char                       0.000   14/34      0.000    6/18      0.000   15/30      0.000    9/22      0.001   21/44      0.000    6/18      0.001  0.000
bytes have unique chars                           0.000   13/38      0.000   13/50      0.001   15/39      0.000    9/34      0.000    5/21      0.000   16/31      0.001  0.000
strings are empty                                 0.000   15/24      0.000    7/8       0.000   20/29      0.000   10/11      0.000    7/15      0.000    4/5       0.000  0.000
string never has a \000 char                      0.000    8/22      0.002   13/27      0.000   17/34      0.002   14/26      0.000    6/18      0.000    8/21      0.000  0.004
string never has a \255 char                      0.000   14/34      0.000    6/18      0.001   15/30      0.000    9/22      0.001   21/44      0.000    6/18      0.002  0.000
strings have unique chars                         0.000   13/38      0.000   13/50      0.002   15/39      0.000    9/34      0.000    5/21      0.000   16/31      0.002  0.000
pairs have different components                   0.000    0/4       0.000    0/6       0.000    0/4       0.000    0/6       0.000    0/6       0.000    0/6       0.000  0.000
pairs have same components                        0.000  125/126     0.000   63/124     0.000  124/125     0.000   64/126     0.000  119/120     0.000   63/125     0.000  0.000
pairs have a zero component                       0.000  124/188     0.000  123/307     0.000  123/186     0.000  124/310     0.000  118/182     0.000  123/308     0.000  0.000
pairs are (0,0)                                   0.000  125/126     0.000   63/124     0.000  124/125     0.000   64/126     0.000  119/120     0.000   63/125     0.000  0.000
pairs are ordered                                 0.000  827/17624   0.000   94/1192    0.000  690/12888   0.000   91/1059    0.000  687/13960   0.000   88/1033    0.001  0.000
pairs are ordered reversely                       0.000  124/125     0.000   62/124     0.000  123/124     0.000   61/122     0.000  122/123     0.000   62/124     0.000  0.000
pairs sum to less than 128                        0.000  116/129     0.000   58/130     0.000  120/146     0.000   58/137     0.000  119/141     0.000   55/126     0.000  0.000
pairs lists rev concat                            0.020  147/350     0.000   66/129     0.008  135/334     0.000   79/143     0.000  131/320     0.000   70/134     0.028  0.000
pairs lists no overlap                            0.001   26/48      0.000   13/15      0.000   18/39      0.000   17/20      0.001    8/18      0.000    9/28      0.001  0.000
triples have pair-wise different components       0.000    7/23      0.000    2/2       0.000    6/6       0.000    1/1       0.000    2/4       0.000    3/3       0.000  0.000
triples have same components                      0.000  188/252     0.000   65/129     0.000  177/240     0.000   66/130     0.000  182/246     0.000   62/123     0.000  0.000
triples are ordered                               0.000  188/252     0.000    4/5       0.000  177/178     0.000    5/6       0.000  187/250     0.000    3/4       0.000  0.000
triples are ordered reversely                     0.000  188/189     0.000  125/248     0.000  177/240     0.000   64/126     0.000  182/183     0.000   64/126     0.000  0.000
quadruples have pair-wise different components    0.000   23/35      0.000    3/3       0.000   11/11      0.000    4/19      0.000   14/38      0.000    2/2       0.000  0.000
quadruples have same components                   0.000  250/377     0.000  125/311     0.000  237/424     0.000  127/315     0.000  242/425     0.000  126/313     0.000  0.000
quadruples are ordered                            0.000  251/315     0.000    5/6       0.000  239/240     0.000    6/7       0.000  244/308     0.000    5/6       0.000  0.000
quadruples are ordered reversely                  0.000  251/252     0.000  126/249     0.000  239/302     0.000   63/123     0.000  244/245     0.000   66/128     0.000  0.000
forall (a, b) in nat: a < b                       0.000   13/22      0.000    1/1       0.000   10/15      0.000    5/14      0.000    5/6       0.000    2/5       0.000  0.000
forall (a, b, c) in nat: a < b < c                0.000   15/21      0.000    2/2       0.000   26/52      0.000    3/3       0.000    9/9       0.000    5/10      0.000  0.000
forall (a, b, c, d) in nat: a < b < c < d         0.000   23/28      0.000    3/3       0.000   30/55      0.000    4/4       0.000   13/13      0.000    4/4       0.000  0.000
forall (a, b, c, d, e) in nat: a < b < c < d < e  0.000   28/28      0.000    4/4       0.000   33/33      0.000    4/4       0.000   14/14      0.000    5/5       0.000  0.000
forall (a, b, c, d, e, f) in nat: a < b < c < d   0.000   30/30      0.000    5/5       0.000   38/38      0.000    6/6       0.000   16/16      0.000    6/6       0.000  0.000
forall (a, b, c, d, e, f, g) in nat: a < b < c <  0.000   31/31      0.000    6/6       0.000   41/41      0.000    7/7       0.000   22/22      0.000    7/7       0.000  0.000
forall (a, b, c, d, e, f, g, h) in nat: a < b <   0.000   35/35      0.000    7/7       0.000   48/48      0.000    8/8       0.000   22/22      0.000    7/7       0.000  0.000
forall (a, b, c, d, e, f, g, h, i) in nat: a < b  0.000   42/42      0.000    8/8       0.000   55/55      0.000    9/9       0.000   26/26      0.000    9/9       0.000  0.000
bind ordered pairs                                0.000  125/125     0.000    1/1       0.000  124/124     0.000    1/1       0.000  120/120     0.000    1/1       0.000  0.000
bind list_size constant                           0.000   49/200     0.000   12/25      0.000   50/197     0.000   12/25      0.000   51/205     0.000   13/28      0.000  0.000
lists are empty                                   0.000   12/18      0.000    7/8       0.003   17/23      0.000   10/11      0.000    4/9       0.000    3/4       0.003  0.000
lists shorter than 10                             0.000   39/213     0.000   18/34      0.000   53/277     0.000   16/22      0.000   47/283     0.000   14/28      0.001  0.000
lists shorter than 432                            0.145 1632/17856   0.509  403/429     0.162 1677/18346   0.205  406/441     0.053 1735/19013   0.321  412/455     0.360  1.034
lists shorter than 4332                           0.091   13/67      2.724 4022/4068    0.082   10/35      2.480 3987/4046    0.009    8/55      2.337 4001/4061    0.181  7.541
lists have unique elems                           0.000    8/17      0.000    8/23      0.002   12/20      0.000    8/19      0.000    8/17      0.000    4/17      0.003  0.000
int option are none                               0.000   11/23      0.000    1/3       0.000    9/19      0.000    1/3       0.000   11/23      0.000    1/3       0.000  0.000
int option are some 100 or more                   0.000    1/1       0.000    1/1       0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000  0.000
(int,string) result are Ok                        0.000    0/0       0.000    4/4       0.000    2/2       0.000    4/4       0.000    6/6       0.000    4/4       0.000  0.000
(int,string) result are Error                     0.000   11/11      0.000    1/1       0.000    9/9       0.000    1/1       0.000   11/11      0.000    1/1       0.000  0.000
tree contains only 42                             0.000   10/10      0.000    1/1       0.000   10/10      0.000    2/2       0.000   12/13      0.000    2/2       0.000  0.000
sum list = 0                                      0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000  0.000
fail_pred_map_commute_int                         0.000   77/238     0.000  107/364     0.000   74/228     0.000   19/69      0.000   87/269     0.000   28/71      0.000  0.001
fail_pred_map_commute_int32                       0.001  149/553     0.000   57/193     0.001  113/446     0.001  113/360     0.000  106/380     0.001   71/212     0.002  0.002
fail_pred_map_commute_int64                       0.001  181/600     0.001  107/383     0.001  184/575     0.001  136/404     0.001  181/536     0.002  133/408     0.002  0.004
fail_pred_strings                                 0.000    2/6       0.000    1/3       0.000    1/5       0.000    2/4       0.000    1/5       0.000    2/4       0.001  0.000
fold_left fold_right                              0.000   34/104     0.000   20/43      0.000   39/120     0.000   16/37      0.000   18/54      0.000   15/43      0.001  0.001
fold_left fold_right uncurried                    0.003   44/198     0.000   17/142     0.037   55/170     0.003   26/56      0.000    5/14      0.000   17/84      0.040  0.003
fold_left fold_right uncurried fun last           0.000   34/101     0.001   48/157     0.000   42/129     0.000   16/38      0.000   19/57      0.000   19/74      0.001  0.001
fold_left test, fun first                         0.002   36/65      0.322   87/25914   1.052   87/1062    1.482  395/20346   0.028   76/673     0.103   40/5860    1.081  1.908
                                                                                                                                                                    1.753 12.578

@jmid jmid merged commit 1ca1186 into c-cube:main Feb 14, 2025
13 checks passed
@jmid jmid deleted the further-qcheck2-list-improvements branch February 14, 2025 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant