[BUG]: VSTest@2 MSTest data-driven tests are all rerun when any subcase fails

[brandondong]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

VSTest

Task version

2

Issue Description

Is this expected behavior?

[TestMethod]
[DataRow(true, true)]
[DataRow(true, false)]
[DataRow(false, true)]
[DataRow(false, false)]
[TestCategory("UnitTests_PipelineTest")]
public void DataRowBoolTest(bool b1, bool b2)
{
    Console.WriteLine($"Starting DataRow bool test with {b1}, {b2}");
    if (b1 && b2)
    {
        Console.WriteLine($"Failing DataRow bool test with {b1}, {b2}");
        Assert.Fail("DataRow bool test fail");
    }
    Console.WriteLine($"Passing DataRow bool test with {b1}, {b2}");
}

With rerunFailedTests: true, all 4 test cases are rerun even though only (true, true) is failing. I would personally expect only the (true, true) to be rerun.

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

N/A

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps