Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Counting witnesses #1562

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Counting witnesses #1562

wants to merge 10 commits into from

Conversation

bugarela
Copy link
Collaborator

@bugarela bugarela commented Dec 13, 2024

Hello :octocat:

On recent projects, we found ourselves wanting to estimate how many of our simulations were "interesting", as in hitting interesting scenarios with specific interactions. To that purpose, I implemented a simple counter for witnesses, which I'm including here.

I see this as a start of a feature. We should have ways to dump witnesses to JSON as well, but this requires a bit more design work as I'm unsure how the interface should look like for that, and how it should interact with traces for violations.

Also, at first, I was also printing the state count (not only the trace count), but that information is harder to interpret and wasn't useful in the instances I looked at it.

This is how the coloring looks like (adding it here as it doesn't show up in the integration test):

image

  • Tests added for any new code
  • Documentation added for any new functionality
  • Entries added to the respective CHANGELOG.md for any new functionality
  • Feature table on README.md updated for any listed functionality

@bugarela bugarela marked this pull request as ready for review December 16, 2024 19:59
@bugarela bugarela requested a review from ivan-gavran December 16, 2024 19:59
@ivan-gavran
Copy link
Contributor

I would like to see one more thing reported with witnesses, and that is how many traces were inspected in total.
If the invariant of a run is not violated, then we control it with max-samples. But if it is violated, we don't know how many traces were run, and this is a useful info.

(An alternative solution to the same problem may be to display the info on traces run before the violation, but I still prefer to have explicit info on witnessed in x cases out of total y cases.)

Copy link
Contributor

@ivan-gavran ivan-gavran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me.
I'd suggest adding the feature that shows total number of explored traces.

@bugarela
Copy link
Collaborator Author

Good point, I haven't thought of that. I should also add a test case for printing witnesses in violations.

Not sure if I'll have time to pick this up before the holidays and I'll be on vacation for almost a month, so if anyone wants to work on this in the meantime, feel free.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants