Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: explain-mode for parts of arguments #3624

Closed
wants to merge 2 commits into from

Conversation

Zac-HD
Copy link
Member

@Zac-HD Zac-HD commented Apr 26, 2023

This currently-broken spike aims to extend #3555 (original issue: #3411) to report on sub-parts of arguments. Because this only makes sense for fixed-size collections - otherwise the minimal failing example would omit that part, so there'd be nothing to report on - we'll implement it for arguments to st.builds(), elements of st.tuples(), and for the non-optional keys inst.fixed_dictionaries().

Not a top priority right now, so I'm opening this just to mark where it's up to and ensure that we don't lose any progress.


Because this means we might have many more reportable spans to report on, it might also require some performance optimizations. The two obvious big ones are:

  1. refactor ConjectureData to natively support vary-this-span, which would save up to 50% of runtime when we retry on changed span size
    • easier: for changed-length spans, we can skip the retry if it's shorter on the assumption that valid shrinks have already been tried. Maybe we can also use generate_novel_prefix() here, and exploit lexical ordering the same way?
  2. reduce the number of failing examples we require before concluding that a part can vary.
    • this is currently 100, but we could accept a higher false-report rate to go faster
    • or incorporate a time-based heuristic so that we only make that trade when the runtime would otherwise be too long. For example, order the parts to check by nesting depth, abort at $timeout, and discard reports for the incomplete depth (or abort after finishing the current depth or a further timeout).

@Zac-HD Zac-HD added enhancement it's not broken, but we want it to be better performance go faster! use less memory! legibility make errors helpful and Hypothesis grokable labels Apr 26, 2023
@Zac-HD Zac-HD force-pushed the inquisitor-collections branch 2 times, most recently from ca2cd2f to fd3ee41 Compare September 17, 2023 09:51
@Zac-HD
Copy link
Member Author

Zac-HD commented Mar 31, 2024

Closing with intent to revive on top of #3864.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement it's not broken, but we want it to be better legibility make errors helpful and Hypothesis grokable performance go faster! use less memory!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant