Skip to content

Proposal: Switch to update_bench.py #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

arkocal
Copy link

@arkocal arkocal commented May 23, 2025

This PR includes a rewritten update bench script in python, that is almost identical to the existing one, with some new functionality. I have created this for my own convenience, but if there is interest I can polish it up a little more to include here.

Rationale:

  • Many more people are fluent in python than ruby, for better future maintenance
  • Tidied up the code a bit, more can be done, especially regarding command line parameters and error messages in parsing the bench.txt
  • Currently, running a larger set of different benchmarks and keeping track of all the results is tricky. This creates one json file (very large, contains everything) in addition to the html and the other outputs.
  • Also, I end up analyzing results in python anyway, and this exports a pickle for extra convenience.

@sim642
Copy link
Member

sim642 commented May 23, 2025

We started https://github.com/goblint/GobExec though, which should be more-or-less feature complete with the Ruby scripts.
It has remained dormant for a year, but it might make more sense to add new features there instead.

One feature of GobExec was running benchmarks in parallel, but that doesn't make sense for your parallel solver benchmarking, so that might need to be made disable-able.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants