Running sample-test does not give the same results as described in the readme.

[PiotrJustyna]

Rahul,

First of all, thanks for the great work, you must have invested a lot of time into putting this library together. Much appreciated.

Just started using your library, but not sure I'm getting the results I would expect from the readme file. Running sample-test (according to the readme) gives me the following output:

$ cabal run mucheck -- -tix sample-test.tix Examples/AssertCheckTest.hsPreprocessing executable 'mucheck' for MuCheck-0.4.0.0...
Running mucheck...
Total mutants: 19 (basis for %)
    Covered: 3
    Sampled: 3
    Errors: 3  (15%)
    Alive: 0/16
    Killed: 0/16 (0%)

While the readme states I should expect:

Total mutants: 19 (basis for %)
        Covered: 13
        Sampled: 13
        Errors: 0  (0%)
        Alive: 1/19
        Killed: 12/19 (63%)

Am I doing something wrong?

[vrthra]

Hi @PiotrJustyna, thank you for the bug report. I am working on a deadline, and may take a few days to respond. I will reply once I have had a chance to look at the difference.

[PiotrJustyna]

Thanks, @vrthra. Looking forward to your update! 😃

[vrthra]

@PiotrJustyna could you please share your sample-test.tix ? It contains the line coverage information. Also could you please paste output of cksum .mutants/*.hs and cat .mutants/*.log?

[vrthra]

Could you please apply this and run again, and paste any console output?

diff --git a/src/Test/MuCheck/Interpreter.hs b/src/Test/MuCheck/Interpreter.hs
index 3536ff8..c84be7d 100644
--- a/src/Test/MuCheck/Interpreter.hs
+++ b/src/Test/MuCheck/Interpreter.hs
@@ -16,6 +16,7 @@ import Test.MuCheck.Utils.Common
 import Test.MuCheck.Utils.Print
 import Test.MuCheck.AnalysisSummary

+import Debug.Trace

 -- | Data type to hold results of a single test execution
 data MutantSummary = MSumError Mutant String [Summary]         -- ^ Capture the error if one occured
@@ -137,7 +138,7 @@ fullSummary m _tests results = MAnalysisSummary {
   _maNumMutants = length results,
   _maAlive = length alive,
   _maKilled = length fails,
-  _maErrors= length errors}
+  _maErrors= length (trace (show errors) errors)}
   where res = map (map _io) results
         lasts = map last res -- get the last test runs
         (errors, completed) = partitionEithers lasts

[vrthra]

@PiotrJustyna Could you please also try installing mucheck with cabal install, then cabal clean, and then cabal install --only-dependencies --enable-tests and the rest?

[PiotrJustyna]

@vrthra Thanks for the update. So, I tried clearing the whole repo and sandbox folder, got the code and installed the packages again in a fresh environment as you suggested, but still getting the same problem.

*output:

Total mutants: 19 (basis for %)
    Covered: 13
    Sampled: 13
    Errors: 13  (68%)
    Alive: 0/6
    Killed: 0/6 (0%)

• sample-test.tix:

Tix [ TixModule "Examples.AssertCheckTest" 6348160 93 [0,0,0,0,1,10,10,15,15,15,10,10,15,15,15,15,15,15,15,15,15,15,15,35,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], TixModule "Main" 2188218545 12 [1,1,1,1,1,1,1,1,1,1,1,1]]

• cksum .mutants/*hs:

867631615 708 .mutants/x3007446877821632866.hs
1189363600 708 .mutants/x5803651688145767119.hs
1664748536 706 .mutants/x6417717130319991471.hs
728947275 708 .mutants/x6873935390713083844.hs
1973016361 693 .mutants/x8411020987074220058.hs
2647258150 708 .mutants/x8498849837389728181.hs
3970233645 607 .mutants/x8682569215982724986.hs
2413550981 706 .mutants/y3077196271526140747.hs
2966836406 707 .mutants/y4839556262729671373.hs
3261745637 707 .mutants/y674744731577824642.hs
708547651 707 .mutants/y781387083301850131.hs
429732091 707 .mutants/y7838082332529536864.hs
2699229846 707 .mutants/y8219306549412207072.hs

All *.log files are empty. Let me apply those changes you proposed and I'll get back to you.

[vrthra]

This happens because mucheck is unable to find the Test.MuCheck.TestAdapter.AssertCheck module because it is not in the search path. For now, you can work around this by cabal install once, then cabal clean cabal install --only-dependencies --enable-tests etc. as in readme.md . Let me know if this works.

[PiotrJustyna]

Looks like you're right. This is the output when I enable debugging:

Running mucheck...
Total mutants: 19 (basis for %)
    Covered: 13
    Sampled: 13
[WontCompile [GhcError {errMsg = ".mutants/y674744731577824642.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x8682569215982724986.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x8411020987074220058.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/y7838082332529536864.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x5803651688145767119.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x8498849837389728181.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x3007446877821632866.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x6873935390713083844.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/y3077196271526140747.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/x6417717130319991471.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/y8219306549412207072.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/y4839556262729671373.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}],WontCompile [GhcError {errMsg = ".mutants/y781387083301850131.hs:2:8:\n    Could not find module \8216Test.MuCheck.TestAdapter.AssertCheck\8217\n    Use -v to see a list of the files searched for."}]]
    Errors: 13  (68%)
    Alive: 0/6
    Killed: 0/6 (0%)

Let me clear the sandbox once again and repeat the installation process. Maybe I messed something up.

[PiotrJustyna]

Nope, no joy...

Let me experiment with building the mutants tomorrow. Thanks for your suggestions, though.

[vrthra]

Try running this on a fresh copy make install build hpcex. This should do the steps in the correct order.

[PiotrJustyna]

Tried make, but unfortunately getting the same results as yesterday. Don't have too much time to experiment with building the mutants today. Let me give you an update if I manage to fix the problem.