Merging of partially hit lines in separate reports

[Blacksmoke16]

I'm working on implementing compile time code coverage for macros in Crystal using the custom codecov format. This is so far working out quite well, however one question I have is how are partially hit lines merged?

For example say you have code like, keeping in mind this is compile time logic:

{{ some_conditional ? 1 : raise "Oh noes" }}

In the case of the truthy path it would be reported as 1/2. However because this happens at compile time, the falsey path of the ternary needs to be executed in its own sub-process since it results in a compile time error. I.e. both paths can't be tested at once. Because of this, two separate reports need to be generated; both of which would have this line as a partial hit of 1/2.

How would codecov handle this? There's not really a way for it to know what branch the 1/2 references. So I imagine it would continue to treat this line as 1/2? Is there maybe a different JSON format that would better handle this case?

[drazisil-codecov]

That's a good question. Offhand, I'm not sure how we would be able to confidently say anything other then 1/2, as you described.

Does Crystal have a native report format, or are you building the coverage instrumentator and reporter yourself?

[Blacksmoke16]

Crystal, at the moment, doesn't have it's own native format/instrumentator. The current best option is to use https://github.com/SimonKagstrom/kcov to generate a cobertura report for the runtime spec coverage, which works well for the most part. kcov also generates a sonarqube report, but from what I can tell both of these formats would suffer from the same problem.

However, kcov is only able to handle the runtime side of things. The feature I'm working on here is for a compile (build) time report, so I'm needing to manually instrument the compiler to expose the required information to generate the report. So changing the output structure of the JSON is totally within the realm of possibility as I have full control over it and just chose codecov format since it was documented and fairly simple.

It would be nice if there was a good way to handle this, but at the end of the day it's not that big of a problem as we can just suggest people try to avoid doing things on 1 line to get the most accurate report. Especially when it comes to raising compile time exceptions like this.

[drazisil-codecov]

The closest I can think of right now, would be try and output to a forward where you can tell Codecov that a partial should be counted as a hit https://docs.codecov.com/docs/codecovyml-reference#parsers

That would be more work then Codecov's custom format, but it would be more flexible. Otherwise you may have to do the merging yourself since you would in theory know it's different branches.

Similer to https://coverage.readthedocs.io/en/latest/cmd.html#cmd-combine

[Blacksmoke16]

I think for now I'll focus on getting my prototype working. Then once that is ready, we can revisit this to see if it's a common occurrence, or if it's more of an edge case.

Thanks for your help!