Dealing with mutants

If a mutant is missed and you feel it should have been caught, you should probably add a test!

But first: think about why no existing test caught it. Is this code supposed to be covered at all?

Your first impulse might be to add a test for the specific function, checking the specific return value isn't the mutated value. Maybe that's right.

But, I'd suggest to think a bit more about whether this behavior could reasonably be caught in a larger-scope test. Isn't there an integration test that ought to depend on this function working properly?

For example in cargo-mutants, the Mutant::unapply function was reported as missed. This function restores the original source code in a tree after testing a mutant. If it did nothing, the build directory would gradually accumulate more and more mutants in different files, and the results would be incorrect. The most-direct way to squash the mutant would be to just add a test checking that unapply does what it's meant to do, and restores the original content.

But wait.. there are already several tests that run cargo-mutants and that look at the results. Why wouldn't any of them notice that files weren't being restored after they were tested?

The problem is, I think: if the mutant isn't removed from the tree, it's likely to cause some tests to fail: in other words, it will cause other mutants tested later to be falsely reported as caught, and probably many tests are expecting the mutants to be caught. Also, many of the test cases have relatively few files: even if the mutant isn't unapplied from one file, that file will anyhow be rewritten to try the next mutant.

So my theory is: a tree with several files, where some mutants are expected to be caught and others are to be missed, might find a bug where unapply doesn't work properly.