Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

String generator breaks if test not always true #289

Closed
starbase527 opened this issue Jun 17, 2019 · 2 comments
Closed

String generator breaks if test not always true #289

starbase527 opened this issue Jun 17, 2019 · 2 comments

Comments

@starbase527
Copy link

Version

077c096

Environment

macOS 10.14.5, Xcode 10.2.1, Swift 5.0.1, SwiftCheck 0.12.0

Description

When forAll sometimes evaluates false using the generator in the code below, things get weird.

Steps To Reproduce

import SwiftCheck

let metreGen = Gen<String>.fromElements(of: ["m", "metre", "metres"])

property("works fine") <- forAll(metreGen) { (unit: String) in
    print(unit)
    return true
}

property("doesn't work") <- forAll(metreGen) { (unit: String) in
    print(unit)
    return false
}

The first property produces expected output of the form

metre
metres
metres
m
metre
m
m
metre
metre
metre
metres
m
m
metres
metres
metre
m
metres
metre
metre
m
metre
metre
metres
metre
metre
metres
metre

etc.

The second seems to produce a single result as expected, then things get weird. It also seems to run indefinitely, sometimes terminating with a error: Execution was interrupted, reason: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0). Output:

metres
*** Failed! es
tres
mees
metr
etres
mtres
meres
metes
metrs
metre
1etres
aetres
2etres
 etres
Cetres
cetres
betres
Aetres
Betres

etres
3etres
mctres
m
tres
m2tres
matres
m tres
m1tres
mCtres
m3tres

etc.

I'm only just learning Swift, so I'm struggling to find any leads in the source code at the moment.

@sebastiangrail
Copy link
Member

sebastiangrail commented Jun 17, 2019

This is expected behaviour: The shrinker is not aware of the generator.
You are using a custom generator, but once your forAll fails, SwiftCheck tries to find a minimal failing example by shrinking the failing test case. SwiftCheck is using the default shrinker for the type (see https://github.com/typelift/SwiftCheck/blob/master/Sources/SwiftCheck/Arbitrary.swift#L316) unless you specify a custom shrinker with forAllShrink. If you don't want SwiftCheck to shrink your failing test cases, you can also use forAllNoShrink, but you'll lose a lot of the power of property based testing.

See also this question on shrinking behaviour

I couldn't reproduce the crash with the code you posted above.

@CodaFi
Copy link
Member

CodaFi commented Aug 4, 2019

I'm going to close this because there hasn't been an update. Thank you @sebastiangrail for such a quick and thorough response to this issue. Please feel free to reopen it if I've made a mistake.

@CodaFi CodaFi closed this as completed Aug 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants