Fixed conformance reporting

[Razican]

We received reports in our Discord saying that our conformance representation was wrong.

We count a test that needs to be run both in strict and non-strict mode as 2 tests, and if it passes in both modes, we count 2 tests as passing, while if only one of them passes, we count one as passed, one as failed.

The correct interpretation is that a test is only considered passing if both strict and no-strict modes pass (tc39/test262#3871).

It changes the following:

• TestResult now contains 2 fields: strict and no_strict. strict is serialized as s, and no_strict is serialized as r (from "result")
• We don't store result strings in the TestResult structure, since it was being skipped and never used
• We no longer store feature information, since we were not using it, and the idea behind it was to get conformance per feature, not to have a feature list.
• Minified output by using test / suite names as keys
• Properly calculate conformance & comparisons

This requires a change in the conformance page to adhere to the new result format, and we need to migrate the current conformance results. I'll open a new PR for that.

This PR will fail when doing the result comparison, thoguh, so we might need to first merge the new result format, then this.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 51 lines in your changes are missing coverage. Please review.

Project coverage is 45.42%. Comparing base (6ddc2b4) to head (b68e588).
Report is 43 commits behind head on main.

❗ Current head b68e588 differs from pull request most recent head dca72d4. Consider uploading reports for the commit dca72d4 to get more accurate results

Files	Patch %	Lines
boa_tester/src/exec/mod.rs	0.00%	25 Missing ⚠️
boa_tester/src/results.rs	0.00%	25 Missing ⚠️
boa_tester/src/main.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3348      +/-   ##
==========================================
- Coverage   47.24%   45.42%   -1.82%     
==========================================
  Files         476      483       +7     
  Lines       46892    49768    +2876     
==========================================
+ Hits        22154    22609     +455     
- Misses      24738    27159    +2421     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jedel1043]

We could simplify this by making TestOutcomeResult something like:

#[serde(tag = "t", content = "d")
enum TestOutcomeResult {
    #[serde(rename = "O")]
    Passed,
    #[serde(rename = "I")]
    Ignored,
    #[serde(rename = "F")]
    Failed {
        strict: bool,
        reason: String
    },
    #[serde(rename = "P")]
    Panic {
        strict: bool,
        reason: String,
    }
}

Then, we won't need separate fields for strict and no_strict, just a single result.

[Razican]

I was thinking about that, and also about the file size & JS performance in the browser. I think that having dictionaries with test/suite names as keys would decrease the size of the JSON around a 15%.

Booleans occupy a bit more. Maybe adding more variants, such as "StrictFailed" and so on could be more performant.

[jedel1043]

Maybe it's time to transition to a binary serialization format such as MessagePack?

Another idea I had was to use Postcard and a small wasm glue code to deserialize the postcard and serialize it to JSON, then we would call that function from the conformance page.

[Razican]

I'll play a bit with those

[Razican]

I've given this a thought, and it could be a nice enhancement in the future, but I would for now just use JSON until the new website is up and running. What do you think?

[jedel1043]

Yep, we can revisit this when we finish the site redesign.

Putting that aside, I still think it would be better to have FailedStrict and PanicStrict for simplicity, if we do use that info. If we don't, maybe we can obviate the need for those variants and just have a single result field?

[Razican]

I converted it back to draft, because I'm still missing:

• ensure that the output result is the same as the new result in the website
• ensure that "short" results only contain the last value (except for the main branch)

[jedel1043]

Since we will probably do a release before this is finished, and we need to fix our tester before then, I'll open a PR with a simpler version of this that just fixes the bug without changing the data representation too much. We can apply the changes of this PR after that.

[Razican]

This has been rebased. It's still missing:

• ensure that the output result is the same as the new result in the website
• ensure that "short" results only contain the last value (except for the main branch)