Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failures in v3 theory tests using MemberData are reported against the first test value on Azure DevOps #424

Open
ewnh opened this issue Nov 11, 2024 · 5 comments
Open

Failures in v3 theory tests using MemberData are reported against the first test value on Azure DevOps #424

ewnh opened this issue Nov 11, 2024 · 5 comments
Labels
status: Waiting for Repro/Response

Comments

@ewnh
Copy link

ewnh commented Nov 11, 2024

Full disclosure this could be an issue with how DevOps handles test results, but as this issue only seems to occur on xUnit v3 (vs. v2, which behaves as expected) this feels like the place to start.

We have a theory test that uses test data generated in a method (via MemberData) - simplified reproduction:

[Theory]
[MemberData(nameof(GetValue))]
public void TestTheory(int value)
{
    Assert.True(value % 2 == 0);
}

public static TheoryData<int> GetValue()
{
    return new TheoryData<int>(new List<int>() { 6, 8, 9 });
}

When using xUnit v2 (2.9.2 to be precise) the results are as expected on DevOps:

image

On xUnit v3 (0.6.0-pre.7) however the failure is instead reported against the first test value:

image

This seems to happen no matter what test case actually fails - in our real tests it always seems to be reported against the first case.

Both sets of tests were run using version 3.0.0-pre.49 of this runner. Our DevOps pipeline is essentially just dotnet test followed by the PublishTestResults@2 task on Windows. This issue does not occur locally in the VS test explorer (also running in a Windows environment).

I've attached the v2 and v3 results files generated by dotnet test from our build server (these are the original .trx files renamed so GitHub will accept them). These files are slightly different but there's nothing obvious to me that'd explain this.

v2.txt
v3.txt
@bradwilson
Copy link
Member

Can you provide the exact azure-pipelines.yml tasks so I'm not incorrectly guessing what you're doing?

Thanks!

@bradwilson
Copy link
Member

bradwilson commented Nov 11, 2024 •
edited
Loading

I definitely see differences in the TRX. For v2:

  <TestDefinitions>
    <UnitTest name="v2.UnitTest1.TestTheory(value: 9)" storage="d:\a\1\s\v2\bin\debug\net8.0\v2.dll" id="208f1a4c-69c6-48b2-505f-71dfd5659147">
      <Execution id="aa14697a-f3b4-4cf3-bf72-b975cfdcb645" />
      <TestMethod codeBase="D:\a\1\s\v2\bin\Debug\net8.0\v2.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v2.UnitTest1" name="TestTheory" />
    </UnitTest>
    <UnitTest name="v2.UnitTest1.TestTheory(value: 6)" storage="d:\a\1\s\v2\bin\debug\net8.0\v2.dll" id="bda2b711-9a03-5639-8c10-504a01e9b920">
      <Execution id="251d7347-9f4f-4a97-9967-cc01eea78cd0" />
      <TestMethod codeBase="D:\a\1\s\v2\bin\Debug\net8.0\v2.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v2.UnitTest1" name="TestTheory" />
    </UnitTest>
    <UnitTest name="v2.UnitTest1.TestTheory(value: 8)" storage="d:\a\1\s\v2\bin\debug\net8.0\v2.dll" id="120edb30-e23c-a079-63ea-f7e6cfb065bd">
      <Execution id="7f136710-a848-47eb-bc40-28ff099ecd33" />
      <TestMethod codeBase="D:\a\1\s\v2\bin\Debug\net8.0\v2.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v2.UnitTest1" name="TestTheory" />
    </UnitTest>
  </TestDefinitions>

And for v3:

  <TestDefinitions>
    <UnitTest name="v3.UnitTest1.TestTheory" storage="d:\a\1\s\v3\bin\debug\net8.0\v3.dll" id="f1c0902d-858a-ac06-9a1f-f06ceaf476cd">
      <Execution id="51248fea-bf60-4379-b212-6b71068514ae" />
      <TestMethod codeBase="D:\a\1\s\v3\bin\Debug\net8.0\v3.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v3.UnitTest1" name="TestTheory" />
    </UnitTest>
  </TestDefinitions>

For v2, it appears that each theory row is an individual test, and in v3, it appears that all theory rows are grouped under a single test. Leaving aside the question of "why" for the moment, it's important to understand that both of these are legitimate (and legal) interpretations, as test cases in VSTest are allowed to report multiple results.

I have no idea where or why it chose to show v3.UnitTest1.TestTheory(value: 6) as the failing test. I'm guessing it either chose one at random, or perhaps chose one pseudo-randomly (the first one it saw, the last one it saw, etc.). Either way, it's definitely a bug in Azure Pipelines, because the display name it should be showing is the one from the TestDefinition; namely, v3.UnitTest1.TestTheory.

I would start with the VSTest team and ask them if they understand why the improper name is being chosen here, though I suspect the bug is actually in Pipelines.

@bradwilson
Copy link
Member

For completion, here is the full v2 TRX:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="2192c326-fd08-4a21-86bf-b22efec4ab3c" name="VssAdministrator@fv-az130-126 2024-11-11 21:25:51" runUser="fv-az130-126\VssAdministrator" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2024-11-11T21:25:51.9886809+00:00" queuing="2024-11-11T21:25:51.9886812+00:00" start="2024-11-11T21:25:49.6265529+00:00" finish="2024-11-11T21:25:52.0889044+00:00" />
  <TestSettings name="default" id="75436809-c7ae-495a-a652-527c62acf8f8">
    <Deployment runDeploymentRoot="VssAdministrator_fv-az130-126_2024-11-11_21_25_51" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="7f136710-a848-47eb-bc40-28ff099ecd33" testId="120edb30-e23c-a079-63ea-f7e6cfb065bd" testName="v2.UnitTest1.TestTheory(value: 8)" computerName="fv-az130-126" duration="00:00:00.0000110" startTime="2024-11-11T21:25:51.9233320+00:00" endTime="2024-11-11T21:25:51.9235839+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="7f136710-a848-47eb-bc40-28ff099ecd33" />
    <UnitTestResult executionId="aa14697a-f3b4-4cf3-bf72-b975cfdcb645" testId="208f1a4c-69c6-48b2-505f-71dfd5659147" testName="v2.UnitTest1.TestTheory(value: 9)" computerName="fv-az130-126" duration="00:00:00.0053112" startTime="2024-11-11T21:25:51.8041956+00:00" endTime="2024-11-11T21:25:51.8406333+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Failed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="aa14697a-f3b4-4cf3-bf72-b975cfdcb645">
      <Output>
        <ErrorInfo>
          <Message>Assert.True() Failure&#xD;
Expected: True&#xD;
Actual:   False</Message>
          <StackTrace>   at v2.UnitTest1.TestTheory(Int32 value) in D:\a\1\s\v2\UnitTest1.cs:line 9&#xD;
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)&#xD;
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
    <UnitTestResult executionId="251d7347-9f4f-4a97-9967-cc01eea78cd0" testId="bda2b711-9a03-5639-8c10-504a01e9b920" testName="v2.UnitTest1.TestTheory(value: 6)" computerName="fv-az130-126" duration="00:00:00.0015264" startTime="2024-11-11T21:25:51.9196849+00:00" endTime="2024-11-11T21:25:51.9203693+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="251d7347-9f4f-4a97-9967-cc01eea78cd0" />
  </Results>
  <TestDefinitions>
    <UnitTest name="v2.UnitTest1.TestTheory(value: 9)" storage="d:\a\1\s\v2\bin\debug\net8.0\v2.dll" id="208f1a4c-69c6-48b2-505f-71dfd5659147">
      <Execution id="aa14697a-f3b4-4cf3-bf72-b975cfdcb645" />
      <TestMethod codeBase="D:\a\1\s\v2\bin\Debug\net8.0\v2.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v2.UnitTest1" name="TestTheory" />
    </UnitTest>
    <UnitTest name="v2.UnitTest1.TestTheory(value: 6)" storage="d:\a\1\s\v2\bin\debug\net8.0\v2.dll" id="bda2b711-9a03-5639-8c10-504a01e9b920">
      <Execution id="251d7347-9f4f-4a97-9967-cc01eea78cd0" />
      <TestMethod codeBase="D:\a\1\s\v2\bin\Debug\net8.0\v2.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v2.UnitTest1" name="TestTheory" />
    </UnitTest>
    <UnitTest name="v2.UnitTest1.TestTheory(value: 8)" storage="d:\a\1\s\v2\bin\debug\net8.0\v2.dll" id="120edb30-e23c-a079-63ea-f7e6cfb065bd">
      <Execution id="7f136710-a848-47eb-bc40-28ff099ecd33" />
      <TestMethod codeBase="D:\a\1\s\v2\bin\Debug\net8.0\v2.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v2.UnitTest1" name="TestTheory" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="120edb30-e23c-a079-63ea-f7e6cfb065bd" executionId="7f136710-a848-47eb-bc40-28ff099ecd33" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="208f1a4c-69c6-48b2-505f-71dfd5659147" executionId="aa14697a-f3b4-4cf3-bf72-b975cfdcb645" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="bda2b711-9a03-5639-8c10-504a01e9b920" executionId="251d7347-9f4f-4a97-9967-cc01eea78cd0" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="3" executed="3" passed="2" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.0-pre.49+a7f5dc0cba (64-bit .NET 8.0.10)&#xD;
[xUnit.net 00:00:00.14]   Discovering: v2&#xD;
[xUnit.net 00:00:01.24]   Discovered:  v2&#xD;
[xUnit.net 00:00:01.28]   Starting:    v2&#xD;
[xUnit.net 00:00:01.37]       Assert.True() Failure&#xD;
[xUnit.net 00:00:01.37]       Expected: True&#xD;
[xUnit.net 00:00:01.37]       Actual:   False&#xD;
[xUnit.net 00:00:01.37]       Stack Trace:&#xD;
[xUnit.net 00:00:01.37]         D:\a\1\s\v2\UnitTest1.cs(9,0): at v2.UnitTest1.TestTheory(Int32 value)&#xD;
[xUnit.net 00:00:01.37]            at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)&#xD;
[xUnit.net 00:00:01.37]            at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)&#xD;
[xUnit.net 00:00:01.45]   Finished:    v2&#xD;
</StdOut>
    </Output>
    <RunInfos>
      <RunInfo computerName="fv-az130-126" outcome="Error" timestamp="2024-11-11T21:25:51.8559788+00:00">
        <Text>[xUnit.net 00:00:01.37]     v2.UnitTest1.TestTheory(value: 9) [FAIL]</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>

And the v3 TRX:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="e41d3653-3766-41d0-98e5-bc4e41527bbf" name="VssAdministrator@fv-az130-126 2024-11-11 21:25:52" runUser="fv-az130-126\VssAdministrator" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2024-11-11T21:25:52.3645389+00:00" queuing="2024-11-11T21:25:52.3645392+00:00" start="2024-11-11T21:25:49.6266293+00:00" finish="2024-11-11T21:25:52.3728545+00:00" />
  <TestSettings name="default" id="055cd6e2-4381-4f98-b8d1-fdee84c92e6f">
    <Deployment runDeploymentRoot="VssAdministrator_fv-az130-126_2024-11-11_21_25_52" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="654ecf7e-dee3-4e7f-a1cc-94b420932d69" testId="f1c0902d-858a-ac06-9a1f-f06ceaf476cd" testName="v3.UnitTest1.TestTheory(value: 8)" computerName="fv-az130-126" duration="00:00:00.0002605" startTime="2024-11-11T21:25:52.1976097+00:00" endTime="2024-11-11T21:25:52.1987308+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="654ecf7e-dee3-4e7f-a1cc-94b420932d69" />
    <UnitTestResult executionId="1b74c09d-b083-4b39-84c4-f0d5288e7c84" testId="f1c0902d-858a-ac06-9a1f-f06ceaf476cd" testName="v3.UnitTest1.TestTheory(value: 9)" computerName="fv-az130-126" duration="00:00:00.0024759" startTime="2024-11-11T21:25:52.1988792+00:00" endTime="2024-11-11T21:25:52.2217701+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Failed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="1b74c09d-b083-4b39-84c4-f0d5288e7c84">
      <Output>
        <ErrorInfo>
          <Message>Assert.True() Failure&#xD;
Expected: True&#xD;
Actual:   False</Message>
          <StackTrace>   at v3.UnitTest1.TestTheory(Int32 value) in D:\a\1\s\v3\UnitTest1.cs:line 9&#xD;
   at InvokeStub_UnitTest1.TestTheory(Object, Span`1)&#xD;
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
    <UnitTestResult executionId="51248fea-bf60-4379-b212-6b71068514ae" testId="f1c0902d-858a-ac06-9a1f-f06ceaf476cd" testName="v3.UnitTest1.TestTheory(value: 6)" computerName="fv-az130-126" duration="00:00:00.0207434" startTime="2024-11-11T21:25:52.1507968+00:00" endTime="2024-11-11T21:25:52.1947458+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="51248fea-bf60-4379-b212-6b71068514ae" />
  </Results>
  <TestDefinitions>
    <UnitTest name="v3.UnitTest1.TestTheory" storage="d:\a\1\s\v3\bin\debug\net8.0\v3.dll" id="f1c0902d-858a-ac06-9a1f-f06ceaf476cd">
      <Execution id="51248fea-bf60-4379-b212-6b71068514ae" />
      <TestMethod codeBase="D:\a\1\s\v3\bin\Debug\net8.0\v3.dll" adapterTypeName="executor://xunit/VsTestRunner3/netcore/" className="v3.UnitTest1" name="TestTheory" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="f1c0902d-858a-ac06-9a1f-f06ceaf476cd" executionId="654ecf7e-dee3-4e7f-a1cc-94b420932d69" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="f1c0902d-858a-ac06-9a1f-f06ceaf476cd" executionId="1b74c09d-b083-4b39-84c4-f0d5288e7c84" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="f1c0902d-858a-ac06-9a1f-f06ceaf476cd" executionId="51248fea-bf60-4379-b212-6b71068514ae" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="3" executed="3" passed="2" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.0-pre.49+a7f5dc0cba (64-bit .NET 8.0.10)&#xD;
[xUnit.net 00:00:00.28]   Discovering: v3&#xD;
[xUnit.net 00:00:01.10]   Discovered:  v3&#xD;
[xUnit.net 00:00:01.47]   Starting:    v3&#xD;
[xUnit.net 00:00:01.68]       Assert.True() Failure&#xD;
[xUnit.net 00:00:01.68]       Expected: True&#xD;
[xUnit.net 00:00:01.68]       Actual:   False&#xD;
[xUnit.net 00:00:01.68]       Stack Trace:&#xD;
[xUnit.net 00:00:01.68]         D:\a\1\s\v3\UnitTest1.cs(9,0): at v3.UnitTest1.TestTheory(Int32 value)&#xD;
[xUnit.net 00:00:01.69]            at InvokeStub_UnitTest1.TestTheory(Object, Span`1)&#xD;
[xUnit.net 00:00:01.69]            at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)&#xD;
[xUnit.net 00:00:01.70]   Finished:    v3&#xD;
</StdOut>
    </Output>
    <RunInfos>
      <RunInfo computerName="fv-az130-126" outcome="Error" timestamp="2024-11-11T21:25:52.2311655+00:00">
        <Text>[xUnit.net 00:00:01.68]     v3.UnitTest1.TestTheory(value: 9) [FAIL]</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>

@bradwilson
Copy link
Member

Additional data point: the TRX generated in Azure Pipelines is identical to the TRX generated locally.

Opening the v3 TRX in Visual Studio 2022 behaves exactly as expected as well:

image

@bradwilson
Copy link
Member

I will await your azure-pipelines.yaml just to be sure I'm not doing something differently from you, but at this point I believe with 95% confidence that this is a Pipelines bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: Waiting for Repro/Response
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bradwilson @ewnh