Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assert.Multiple doesn't provide stacktrace for found failures #1107

Open
jochenwezel opened this issue Jun 22, 2023 · 10 comments
Open

Assert.Multiple doesn't provide stacktrace for found failures #1107

jochenwezel opened this issue Jun 22, 2023 · 10 comments
Labels
Milestone

Comments

@jochenwezel
Copy link

Current results

The current output for my test in Visual Studio is as following and is missing the stacktrace line for the indiviual failed assertion:
image

The same output is shown at dotnet test, except that stack trace lines are not clickable

Requested results

Each asserition should

  1. show the missing stacktrace (at positions A in screenshot) from the initial stack of the multiple (show in screenshot at position B) to the individual assertion line, something like following
    HHErgebnisrechnungenBpVsAwpTest.ValidateBunch1() Zeile 304
    HHErgebnisrechnungenBpVsAwpTest.AssertEqualErgebnis.<Multi>b__1_1() Zeile 103
  1. in Visual Studio window, the stacktrace at positions A should be clickable (as at position B)
    image

Current code snippets

Assert.Multiple is called as following:

Public Sub ErgebnisrechnungAWPvsBP(id_planjahr as Int64, id_bereich as Int64)()
    Assert.Multiple(Sub()
                   ValidateBunch1()
                   ValidateBunch2()
              End Sub)
End Sub

Private Sub ValidateBunch1()
                    '...
                    Assert.AreEqual(Me.SummenTableBP(SumName, "rIstTMinus1"), System.Math.Round(Me.SummenTableAWP(SumName, "IstTMinus1"), 0), "BP vs AWP: " & SumName & " >> IstTMinus1")
                    Assert.AreEqual(Me.SummenTableBP(SumName, "rPlanT"), System.Math.Round(Me.SummenTableAWP(SumName, "PlanT"), 0), "BP vs AWP: " & SumName & " >> PlanT")
                    Assert.AreEqual(Me.SummenTableBP(SumName, "rHrT"), System.Math.Round(Me.SummenTableAWP(SumName, 
                    '...
End Sub

Private Sub ValidateBunch2()
                    '...
                    Assert.AreEqual(1, 2)
                    '...
End Sub

PackageReferences are as following:

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
@stevenaw
Copy link
Member

stevenaw commented Jul 2, 2023

Thanks for reporting this @jochenwezel
I haven't had the chance to dig into this myself yet but there may be two separate issues here.

For your first point, I'm reminded of nunit/nunit#4242 . Do you feel like that one captures what you're looking for?

For your second point, I suspect that it may actually be an adapter issue.

@stevenaw
Copy link
Member

stevenaw commented Jul 2, 2023

Also, thank you for the clean repro and package version info. Can you also please confirm which version of Visual Studio you're using?

@jochenwezel
Copy link
Author

jochenwezel commented Jul 3, 2023

For your first point, I'm reminded of nunit/nunit#4242 . Do you feel like that one captures what you're looking for?

The ticket of nunit/nunit#4242 describes an issue, that stack trace information is available for the multiple assertions, but with wrong line numbers.
In this (bug) tickete here, there is no stacktrace at all.

Can you also please confirm which version of Visual Studio you're using?

Microsoft Visual Studio Community 2022 (64-Bit), v17.6

Please note for point 2 of my request: the stacktrace line numbers are clickable usually. My intention to write down point 2 is just for presenting a complete "feature request description" ;-) Maybe the code for point 2 is already there and active as soon as the stacktrace (re-)appears in point 1.

@OsirisTerje
Copy link
Member

OsirisTerje commented Jul 3, 2023

The stacktrace is there, but there is something wrong either in the adapter code or in the way that the Test Explorer interprets the stacktrace.
I have added a repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4406
and the dump file contains the following block:

<assertion result='Failed'>
         <message><![CDATA[  Expected: 4
  But was:  42
]]></message>
         <stack-trace><![CDATA[   at Issue4406.Tests.<>c__DisplayClass2_0.<Test1>b__0() in C:\repos\nunit\nunit.issues\Issue4406\UnitTest1.cs:line 29
   at NUnit.Framework.Assert.Multiple(TestDelegate testDelegate)
   at Issue4406.Tests.Test1() in C:\repos\nunit\nunit.issues\Issue4406\UnitTest1.cs:line 24
]]></stack-trace>
      </assertion>

This comes for each assert that fails, and contains the correct line numbers.
But it also outputs a stack trace in the failure section,

<failure>
      <message><![CDATA[Multiple failures or warnings in test:
  1)   Expected: 2
  But was:  42

  2)   Expected: 3
  But was:  42

  3)   Expected: 4
  But was:  42

  4)   Expected: 55
  But was:  5

]]></message>
      <stack-trace><![CDATA[   at Issue4406.Tests.Test1() in C:\repos\nunit\nunit.issues\Issue4406\UnitTest1.cs:line 24
]]></stack-trace>
   </failure>

It can be that this is what is being sent to the Test Explorer.

So this looks like an adapter bug, and I'll move it there.

@OsirisTerje OsirisTerje transferred this issue from nunit/nunit Jul 3, 2023
@OsirisTerje OsirisTerje added this to the 4.6 milestone Jul 3, 2023
@jochenwezel
Copy link
Author

Partially fixed after latest update

Sorry that I haven't done it before: I recently upgraded dependencies today to

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.3" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
    <PackageReference Include="NUnit.Analyzers" Version="3.6.1">

and I've got stack traces for each of the several assertions which failed :-) :
image

New situation

  1. Stack trace is present for each failing assertion
  2. Code line seems to be correct
  3. BUT: the stack trace is not clickable in Visual Studio at the relevant position (see e.g. failure 1 with shown stacktrace to HHErgebnisrechnungenBpVsAwpTest.vb:line 380 instead of providing a clickable link)

@OsirisTerje
Copy link
Member

That is pretty interesting, because I don't ;-) Hmm.. until right now, I did get a stack trace too.... this is weird.
Can you check the repro at https://github.com/nunit/nunit.issues/tree/main/Issue4406 and see if that also works for you?

But I think the reason for 3) is the way we send the stack trace and how Test Explorer expects it to be. I'll see if I can change them to be what I assume TE expects, a single stack trace line.

@jochenwezel
Copy link
Author

Can you check the repro at https://github.com/nunit/nunit.issues/tree/main/Issue4406 and see if that also works for you?

There are at least 2 issues with the project in Issue4406:

  1. nuget.config contains <add key="Local" value="c:\nuget" /> which doesn't exist (ok, solved simply by dropping this line)
  2. NU1102 Package "NUnit3TestAdapter" with version (>= 4.6.0-alpha.3) not found
    • enabling beta releases in nuget still doesn't show up with this alpha asssembly
    • maybe you have to put and reference this assembly as part of this project in a subfolder of Issue4406 to allow full test of this project?

@OsirisTerje
Copy link
Member

OsirisTerje commented Jul 18, 2023

@jochenwezel Sorry about being to quick pushing up there. Just set the version number to 4.5.0, I didn't do any code changes, so that should work.

[Update] Just pushed up the changes, so just pull it down and it should work.

@jochenwezel
Copy link
Author

These are the test results

image

SingleTest

image

Test1

image

@OsirisTerje
Copy link
Member

Thanks! This is exactly the same as I see here. That's good.
The correct line numbers are in the 2nd line of each stacktrace. Test Explorer picks up the first line, so the way to fix this is to reduce the stacktrace to a single line identical to the 2nd line.

@OsirisTerje OsirisTerje modified the milestones: 4.6, 4.7 Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants