Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom CulmulativeReporterBase receives odd children order when dealing with Skipped tests #2928

Open
tenpn opened this issue Nov 6, 2024 · 0 comments

Comments

@tenpn
Copy link

tenpn commented Nov 6, 2024

Describe the bug
I'm building a reporter from CulmulativeReporterBase. I'm iterating the children of the m_testRun, and walking through each child SectionNode.

void LogSectionNode(const Catch::CumulativeReporterBase::SectionNode& node, const std::string& prefix)
{
    std::cout << prefix << "Section: " << node.stats.sectionInfo.name <<
        " (child count: " << node.childSections.size() << " passed:" << node.stats.assertions.passed << " skipped:" << node.stats.assertions.skipped << ")\n";
    for(const auto& childSection : node.childSections)
    {
        LogSectionNode(*childSection, prefix + " ");
    }
}

void ReproReporter::testRunEndedCumulative()
{
    for (const auto& childNode : m_testRun->children)
    {
        std::cout << "test node: " << childNode->value.testInfo->name << '\n';
        for (const auto& childSection : childNode->children)
        {
            LogSectionNode(*childSection, " ");
        }
    }
}

I'm running it against a Scenario such as this:

SCENARIO("Repro")
{
    GIVEN("ReproGiven")
    {
        WHEN("ReproWhenA")
        {
            THEN("ReproThenA1")
            {
                SUCCEED();
            }
            THEN("ReproThenA2")
            {
                SUCCEED();
            }
        }
        WHEN("ReproWhenB")
        {
            //SKIP("skip");
            SUCCEED();
        }
    }
}

When I uncomment the skip, I get very odd output:

test node: Scenario: Repro
 Section: Scenario: Repro (child count: 2 passed:0 skipped:0)
  Section:      When: ReproWhenB (child count: 2 passed:0 skipped:1)
   Section:      When: ReproWhenA (child count: 2 passed:1 skipped:0)
    Section:      Then: ReproThenA1 (child count: 0 passed:1 skipped:0)
    Section:      Then: ReproThenA2 (child count: 0 passed:1 skipped:0)
   Section:     Given: ReproGiven (child count: 0 passed:0 skipped:1)
  Section:     Given: ReproGiven (child count: 0 passed:0 skipped:0)

Note that the scenario has two children, once called ReproWhenB and another called ReproGiven. There's a second ReproGIven under ReproWhenB, that contains the actual skipped test.

Expected behavior
Compare that to the output when I don't skip:

test node: Scenario: Repro
 Section: Scenario: Repro (child count: 1 passed:1 skipped:0)
  Section:     Given: ReproGiven (child count: 2 passed:1 skipped:0)
   Section:      When: ReproWhenA (child count: 2 passed:1 skipped:0)
    Section:      Then: ReproThenA1 (child count: 0 passed:1 skipped:0)
    Section:      Then: ReproThenA2 (child count: 0 passed:1 skipped:0)
   Section:      When: ReproWhenB (child count: 0 passed:1 skipped:0)

Now the scenario has one child, ReproGIven, which has two children, WhenA and WhenB. Nothing here looks weird.

Reproduction steps
https://github.com/tenpn/catch2-skipreporter-repro
download, build with xbuild build, comment/uncomment the skip in src/ReproScenario.cpp, run the tests with xbuild run binary -r repro.

Platform information:

  • OS: Windows 10
  • Compiler+version: VS 2022 Community
  • Catch version: v3.7.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant