Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow factorial specification of tests based on sets of options #10

Open
whart222 opened this issue Nov 15, 2015 · 1 comment
Open

Allow factorial specification of tests based on sets of options #10

whart222 opened this issue Nov 15, 2015 · 1 comment

Comments

@whart222
Copy link
Contributor

The pyutilib.testdriver should allow specifying multiple sets of options for a solver. It would then run a "full-factorial" suite of tests. For example:

  solvers:
    solver_a:
      np:
        - 1
        - 2
        - 4
      strategy:
        - dive
        - pseudocost
    solver_b:
      np: 1
  problems:
    problem_1:
    problem_2:

would specify 14 tests.

This is on the wishlist because this capability has lots of difficult implications, including how it gets implemented, how fractional specifications are managed (in the 'tests' block), and how tests get named. If this was spelled out in more detail, I'd be willing to look into this. However, this might be a nice extension for John or Dave to dig into to help familiarize themselves with this code.

It is an open question how to consistently name the individual tests.

@whart222 whart222 added this to the Wish List milestone Nov 15, 2015
@whart222
Copy link
Contributor Author

The pyutilib.testdriver YAML input specification could support a section that allows us to specify a subset of the full factorial set of tests. For example:

testsuites:
   suite_a:
      solvers:
         solver_a:
         solver_b:
      problems:
         problem_1:
         problem_2:
      tests:
         - solver_a_problem_1
         - solver_b_problem_2

would only run solver_a on problem_1 and solver_b on problem_2. Additionally,

testsuites:
   suite_a:
      solvers:
         solver_a:
         solver_b:
         solver_c:
      problems:
         problem_1:
         problem_2:
      tests: */4

would run 1/4 of the full-factorial design (distributed in some statistically meaningful manner).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant