-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request: Persist comparison tolerances with test metadata #15
Comments
Hi Michael, Just to confirm, you are asking for comparison tolerance fields in the test configuration files? This would be nice because tests can then have personalized tolerances. If we do this, I'd probably prefer to remove the ability to set the comparison tolerance from the command line. This would need to override the tolerance in all tests, which is just asking for trouble. |
@davidchall Yes like you say, comparison tolerance fields in the test configuration file. I also agree that setting the comparison tolerance from the command line could become problematic in that scenario. How were you thinking of implementing this? I was thinking maybe add another optional field to the test configuration file. Perhaps, a dictionary were the key was the file type and the value was a list of tolerance values. It would be nice to enclose tolerance values in a data structure. That way test functions with different input parameters besides |
I've been thinking about this some more. The only truly comprehensive way to do this would be to change the Test config files in a backwards-incompatible way, so we can add metadata to each result file. Then ( Something like:
|
Hey @davidchall
nrtest
is wonderful! We are putting it to great use for our projects. We are thinking about using it in our CI tool chain and had a thought on how to better facilitate that.It would be great if default comparison tolerances could be persisted with the test metadata. The defaults could be overridden with values passed from the command line. This would preserve the current functionality and make it more straight forward to automate the regression test. Make sense?
Would you be open to including this feature? Thanks.
The text was updated successfully, but these errors were encountered: