Improve error message when forced fail test fails #354
Conversation
| @@ -885,14 +885,14 @@ def print_stats(source_file_mutation_data_by_path, force_output=False): | |||
| print_status(f'{(s.total - s.not_checked)}/{s.total} 🎉 {s.killed} 🫥 {s.no_tests} ⏰ {s.timeout} 🤔 {s.suspicious} 🙁 {s.survived} 🔇 {s.skipped}', force_output=force_output) | |||
|
|
|||
|
|
|||
| def run_forced_fail(runner): | |||
| def run_forced_fail_test(runner): | |||
There was a problem hiding this comment.
Renamed to better differentiate it from the runner's run_force_fail method.
mutmut/__main__.py
Outdated
| print("FAILED") | ||
| os._exit(1) | ||
| print("Error: Unable to force a test failure during the forced fail test") | ||
| raise SystemExit(1) |
There was a problem hiding this comment.
I would suggest "FAILED: Unable to force test failures". To be consistent, and for brevity.
Why did you change the exit? I use os._exit() for fast exit, so we don't spend time deallocating at the end of the process.
There was a problem hiding this comment.
I have update the error message.
I switched to SystemExit because it is much easier to write a test for. Re the performance thing, I think this is something that will only happen once, early on in a run, if something has gone wrong, so some milliseconds taken occasionally doesn't seem like a problem.
There was a problem hiding this comment.
Not sure why test_run_forced_fail_test_with_all_tests_passing is passing locally, but not in GitHub Actions. I'll come back to that when I next have some time.
There was a problem hiding this comment.
All the unit tests I added are failing in GitHub Actions because the output emitted by run_forced_fail_test is not making it to stdout.
I have tried adding some sleeps to rule out a race condition and changing the runner to MacOS in case that was a factor. No banana.
There was a problem hiding this comment.
@boxed I have poked around and set things up so that there is a test that does not capture the output and running single tests with verbose etc.
If you look at this GitHub Action run, it seems that CatchOutput is only capturing the first line of the output in GitHub Actions for some reason. You can see "done" (and "FAILED" for come reason without the new text) in the test output.
My time for playing this morning has run out. If you have any thoughts on why this might be I would be interested in hearing them.
There was a problem hiding this comment.
Temporarily running the tests on MacOS as well as Ubuntu in GitHub Actions indicates it's not an OS difference.
I'm now wondering whether there is some difference in the way https://github.com/actions/setup-python installs Python compared to how I installed it (via asdf) on my laptop.
@boxed (or anyone else) If you checkout this branch and run the following, does it pass or fail, and how did you install Python?
python -m pytest -vvv --capture=no tests/test_mutation.py::test_run_forced_fail_test_with_failing_test
There was a problem hiding this comment.
It passes. I use basic venv and pip.
There was a problem hiding this comment.
Thanks. But... damn. I wonder what the difference is in GitHub Actions.
There was a problem hiding this comment.
If I comment out the with CatchOutput part in run_forced_fail_test, capfd works as expected.
I suspect CatchOutput and capfd are doing much the same thing behind the scenes and CatchOutput is usurping capfd. I'll need to read the source code for capfd, but boarding a plane imminently.
@boxed What is the intent/purpose of CatchOutput?
There was a problem hiding this comment.
Yea that makes sense.
The purpose is to not show a lot of output when running mutmut, but being able to dump it on errors. Showing all output to the user would make the tool basically unusable.
WillGibson
force-pushed
the
improve-failed-forced-fail-error
branch
from
2edd3ea to
70cbf80
Compare
WillGibson
force-pushed
the
improve-failed-forced-fail-error
branch
from
b2a014d to
6fd2e3b
Compare
I started with something small relating to #349 in order to start familiarising myself with the Mutmut code/tests.