Implement regression testing and performance testing in CI #247
Labels
area::ci
Related to CI, coverage, Github, etc.
area::conjure-oxide
Related to conjure_oxide.
kind::testing
Testing and Correctness
priority::soon
We'll need this soon, or other features may be blocked.
Milestone
Currently, our integration tests verify that a model has been parsed and rewritten correctly, get Minion solutions, and validate them against a hard coded correct set of solutions.
Overview
Instead, we need to run the same model through
conjure-oxide
andconjure
proper, and compare their solutions, as well as various performance metrics (for now, probably just the CPU time it takes to solve).This will achieve regression testing ("our solution is not different from old conjure, which is assumed to be correct") and performance testing (measuring how much worse (or better!) our performance is compared to conjure)
Tasks
As a separate task, we would probably need a nice web page to visualise the results of performance testing.
This will be a separate issue
The text was updated successfully, but these errors were encountered: