Ensure all Haskell test cases still run within same file when there are failed test cases #543
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
When running Haskell tests within a file, and there is at least 1 failed tests, all tests fail in the file even though some of them or passing. The issue is If any test case fails, the stack runghc ... command will have an exit status of 1, not 0. This will cause the subprocess.run call to raise an error, because of the check=True argument.
Solution:
A try-except block was used around the subprocess.run call. If the return code is 1 we won't raise an error but will still raise it for other errors.
Testing
Currently markus-autotesting does not have a framework to do unit-testing for Haskell, and due to time, I did not write unit tests for this.
However, to do a manual test, I used a Haskell test file that has 1 passing and 1 failing test case where I named the file Test.hs, with the following contents (note: ".hs" extension file is not supported on GitHub, so I am displaying the contents here.):
It should show 1 passing and 1 failing test in the results after running a test.