Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement regression testing and performance testing in CI #247

Open
4 tasks
gskorokhod opened this issue Feb 23, 2024 · 0 comments
Open
4 tasks

Implement regression testing and performance testing in CI #247

gskorokhod opened this issue Feb 23, 2024 · 0 comments
Assignees
Labels
area::ci Related to CI, coverage, Github, etc. area::conjure-oxide Related to conjure_oxide. kind::testing Testing and Correctness priority::soon We'll need this soon, or other features may be blocked.

Comments

@gskorokhod
Copy link
Contributor

Currently, our integration tests verify that a model has been parsed and rewritten correctly, get Minion solutions, and validate them against a hard coded correct set of solutions.

Overview

Instead, we need to run the same model through conjure-oxide and conjure proper, and compare their solutions, as well as various performance metrics (for now, probably just the CPU time it takes to solve).
This will achieve regression testing ("our solution is not different from old conjure, which is assumed to be correct") and performance testing (measuring how much worse (or better!) our performance is compared to conjure)

Tasks

  • Expand out script that installs conjure to also install solvers
  • Read conjure docs to find a way to get performance information and metadata out of it
  • Write code to parse the solution that conjure gives us and the metadata mentioned above
  • Expand our CI to compare our solution and our performance to conjure

As a separate task, we would probably need a nice web page to visualise the results of performance testing.
This will be a separate issue

@gskorokhod gskorokhod self-assigned this Feb 23, 2024
@gskorokhod gskorokhod added area::conjure-oxide Related to conjure_oxide. area::ci Related to CI, coverage, Github, etc. kind::testing Testing and Correctness priority::soon We'll need this soon, or other features may be blocked. and removed area::conjure-oxide Related to conjure_oxide. labels Feb 23, 2024
@gskorokhod gskorokhod added this to the Better integration testing milestone Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area::ci Related to CI, coverage, Github, etc. area::conjure-oxide Related to conjure_oxide. kind::testing Testing and Correctness priority::soon We'll need this soon, or other features may be blocked.
Projects
None yet
Development

No branches or pull requests

1 participant