Microsoft Testing Platform for xUnit

[Evangelink]

Dear xUnit team,

As per our previous discussions, we would like to help you onboard xUnit to the new self-contained testing platform that powers MSTest Runner.

We are currently working on the technical documentation for framework authors and extensions authors so sadly for now there isn't much available for you to browse.

The new platform is already OSS and available inside of MSTest repository (at the moment).

Related issue about support NUnit on MSTest repo: microsoft/testfx#2164

MSTest runner blogpost announcement: https://devblogs.microsoft.com/dotnet/introducing-ms-test-runner/
MSTest runner doc: https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-intro?tabs=dotnetcli
Platform code: https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform
Sample code: https://github.com/microsoft/testfx/tree/main/samples/mstest-runner

[bradwilson]

How has user feedback been on the extremely abbreviated output from dotnet test? I would expect users to push back on the complete lack of any reporting information; notably, the lack of failed test information seems excessively terse by default.

• Interactive usage means dumping the log file as a separate manual step when tests fail
• If you're relying on the raw build logs to see test failure in CI, you'll either need to dump the log file on failure and/or save the log file and/or save a TRX output file.

[bradwilson]

Old output:

New output: (note that I had to cat the log file to find the failure)

[bradwilson]

Does enabling the new system preclude usage in older clients, like older VS Code/Visual Studio versions? Or does enabling the new system provide a "best of both worlds" by allowing the system using the older APIs to still function?

[MartyIX]

Out of curiosity: Does this issue and the corresponding PR (#403) represent a fix for xunit/xunit#2337?

Or is xunit/xunit#2337 about something else?

[bradwilson]

That issue is technically something else, though I did chat with the VSTest team about whether the protocol they're using here might be something I could re-use, so it would end up perhaps allowing me to reuse some of their code for my v3 cross-process protocol. It's definitely something I'm going to be investigating. 😄

[bradwilson]

save a TRX output file

Actually, this doesn't appear to be an option. The --logger switch to dotnet test appears to be entirely ignored in the new system, so I can't customize the console logger nor use the TRX logger. Is this an oversight? How are users expected to be able to get machine parseable logs from the new system?

[Evangelink]

How has user feedback been on the extremely abbreviated output from dotnet test? I would expect users to push back on the complete lack of any reporting information; notably, the lack of failed test information seems excessively terse by default.

Not liked by everyone microsoft/testfx#2162. We are working on some "interactive" mode where more things would be printed.

This default is more performant and is aligned with how we usually define things internally (for example this is matching the default output of arcade infra). We believe this is a good default for CI and we want to keep it but we understand that this is not what everyone wants and we are working on improving this.

We are also working on an improved terminal experience for terminals that supports better outputing.

Does enabling the new system preclude usage in older clients, like older VS Code/Visual Studio versions? Or does enabling the new system provide a "best of both worlds" by allowing the system using the older APIs to still function?

Thanks to the bridge, the old versions of Test Explorer (VS and VS Code) would communicate using VSTest and the new versions have some setting (default is using the runner directly through the new json protocol) to switch between modes.

Actually, this doesn't appear to be an option.

We will be updating the dotnet test documentation page but you should consider that all the options displayed there are VSTest related options. To be flowing down options to the runner you need to use TestingPlatformCommandLineArguments (as described here https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-integrations#dotnet-test---mstest-runner-mode)

so I can't customize the console logger

Correct, at the moment this is not supported but we have some infra (internal for now) to support this in the future. We still need some design and testing to ensure it's all good and can be open.

On the new platform we are making a distingo between output and logging (although we won't block if someone creates some specific logger that redirects to the output service) because they are fundamentaly different and this aggregation was causing lots of pain in VSTest platform.

nor use the TRX logger

This isn't true. You can install and use the trx extension (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-extensions#visual-studio-test-reports).

[bradwilson]

This isn't true. You can install and use the trx extension

How is this different from --logger trx (aka, why are there two ways to do it)?

[Evangelink]

Because all commands of dotnet test are VSTest oriented. Besides VSTest was using the logger concept for everything which was leading to many issues and confusions. On the new platform we are trying to have clear concepts: logger is like any regular app, console falls under an output device concept which will allow us to have different reporting (remote call, live web pages for CI, handling of devices...) and finally extensions that are subscribing to events (this is the case for TRX report).

[jkone27]

this is super useful aslo for debugging tests also cross languages in the .NET ecosystem, i hope this is available soon for Xunit

[bradwilson]

@jkone27 What is provided here that you don't already get with xUnit.net's VSTest integration? You can already debug tests in VS and VS Code. To be clear, what I see being offered here is only a potential performance advantage.

[hotchkj]

I wanted to say thank you @bradwilson for investigating this, and pushing back on logging & output concerns, while trying to wrangle V3 and have the test runner communication make sense. It's very much appreciated!

Coming from long usage and hassles of getting dotnet test to execute xUnit properly and produce meaningful output for CI, I'm glad it's well understood as a need.

[bradwilson]

@hotchkj The integration with v3 is rather non-trivial, because we have competing concerns with the fact that their adapter want to convert DLL projects into EXE projects, but we're already an EXE project. What their adapter does is rather under-documented for my tastes (and at the moment includes non-open source code, which I'd want to get addressed). Trying to get them to document things has been quite difficult.

I have a branch with some experimentation on how this might work for v3, but I've yet to be convinced that this is actually what people want. I believe the usability is significantly lower from a command line perspective (being forced to find and inspect a log file if tests fail is just a very strange decision, to pick one example). The VSTest argument is that "command line builds are for CI", which doesn't at all match with my experience. When trying to discuss the xUnit.net end user value with the VSTest team (starting here), the best answer I could get from them included new project types (that we don't support) as well as fixing bugs that I've never seen reported by our users.

I'm not against the idea of integrating, but if I'm going to write and maintain yet another adapter path, I'd like to understand what my end users are getting out of it, and right now the answer appears to be "nothing". 🤷🏼

[hotchkj]

Oh, quite understood! I perhaps phrased the second part poorly - I meant that if a new adapter is proposed, it needs to at least exceed what we have today (and just getting dotnet test reliable has been a journey, so lets' not repeat that if not needed). Plus I'm very glad to see your commenting e.g.:

How has user feedback been on the extremely abbreviated output from dotnet test? I would expect users to push back on the complete lack of any reporting information; notably, the lack of failed test information seems excessively terse by default.

For my part, detailed information about what tests are doing, as well as extracting that for various formats needed by tooling (ranging from CI-CD graphing through to code coverage to static analysis), is a critical part of the picture. I confess I don't really buy 'I don't want to see lots of console data' as an argument; the whole point of CI-CD is to throw automation at a problem to get devs focused on productive tasks, and developers need visual certainty that the tests they think they wrote actually executed. I've lost count of the times where we've had PRs of the form 'you added a test file but there are in fact no tests in it'.

The VSTest argument is that "command line builds are for CI", which doesn't at all match with my experience

Completely agreed. For what it's worth, we have a standard approach of a one-liner command line 'build, run, test' being a requirement for all repos, and it's expected that that's what devs run locally if they want to validate the build. No offence intended to IDE devs, but frankly I've had enough variance between what a command line solidly and reliably reproduces vs what e.g. Visual Studio claims to find and run, that I know which I'm going to rely on.

[julealgon]

but I've yet to be convinced that this is actually what people want.

@bradwilson I can only speak for myself here, but after reading their blogs about the new infrastructure and new SDK-based project, and seeing the reports of massive startup and execution time improvements, substantialy improved parallelism controls and efficiency, trimming support and future AOT compatibility I almost instantly considered moving from xUnit back to MSTest to see how much of a gain it was on our solution.

To see this thread in its current state makes me sad, as I really wanted to have this new mechanism available for xUnit as well, but apparently you guys are not playing well together at this point.

From my perspective, xUnit "doing its own thing" with v3 while Microsoft creates a brand new architecture that is supposed to be extensible is a huge mistake that I'm hoping is resolved soon.

I really hope to avoid having to make the xUnit -> MSTest transition but I'll definitely propose that to our team if xUnit continues to stay out of this game. A non-trivial speed-up in test execution time would make a huge difference for productivity on our team as I'm sure would be the case for many other companies.

[bradwilson]

@julealgon I am still working on integration support for v3, but I'm having to do it on my own since I don't fit the mold that their adapter is designed for. I have a prototype right now that works, I'm just trying to find holes in it before I ship it.

[bradwilson]

This is available with v3 0.4.0-pre.10 https://xunit.net/docs/using-ci-builds

Documentation: https://xunit.net/docs/getting-started/v3/microsoft-testing-platform

Note that the documentation is not yet linked from the home page, because I'm waiting for a review by the Microsoft Testing Platform team to see if they want to tweak any of the language.

[julealgon]

This is available with v3 0.4.0-pre.10 https://xunit.net/docs/using-ci-builds

@bradwilson do you have any estimates on when v3 non-preview will be available?

[bradwilson]

@julealgon I assume you're asking about a 1.0 release time frame. If so, I don't have a set date in mind yet, though it's probably not far away given my progress on the roadmap. I'd say the big issues I'd like to get resolved (or at least planned out) before 1.0 are the query filter (#1833), parallelization improvements (#1986 #2055), and custom reporters (#1874). I don't believe anything else on the roadmap would require any fundamental architectural changes to v3.

There's also an open issue to support MTP in v2 through the VSTest bridge (vs#416), which is separate and being driven by the VSTest team, and therefore may land before v3 1.0.

If your question was actually "when will 0.4.0 be on NuGet?", then the answer to that is that I want to finish the query filter first, which is work that's already underway. That could happen as soon as this weekend, but more likely sometime next week.