Allow xunit, nunit and other frameworks to run with the new runner

[nohwnd]

Summary

MSTest runner is a new lightweight runner for MSTest tests. Please vote below if you'd like the same experience for other frameworks. And optionally share why, and which framework, please.

https://devblogs.microsoft.com/dotnet/introducing-ms-test-runner

[Rand-Random]

Maybe I am missing something, but why not?
Sure it’s more work, but isn’t it worth the additional work?

Currently, it feels like if Entity Framework only supported MSSQL and is asking do you want support for eg MySQL?

[baronfel]

Big thumbs-up here for XUnit support - this is going to be necessary to start porting large chunks of the dotnet/* repositories onto this new runner.

[Evangelink]

@baronfel We just need to provide enough new nice features for you to want to migrate to MSTest 😉

[neilcawse]

Would be very useful to support xunit for us and many customers - it would save a lot on wasted CPU cycles in CI/CD

[Daniel-Svensson]

Would love to see if it can give a similar boost to nunit tests

[pieteckhart]

+1 for XUnit

[nicolasmohamed]

+1 NUnit

[SymbioticKilla]

@baronfel We just need to provide enough new nice features for you to want to migrate to MSTest 😉

Totally agree! The migration from xUnit to MSTest wasn't a hard job. There are just some nice-to-have features in xUnit that save some code lines.

[thomhurst]

I'm in the process of writing my own framework currently.

So is the new runner locked down to just MSTest at the present time?

I'd definitely like to make use of the new runner, especially with things like performance improvements over VSTest.

[Evangelink]

So is the new runner locked down to just MSTest at the present time?

It is advertised only as MSTest runner but the code is open to any custom framework.

The only issues for non-MSTest framework at the moment are:

• runner/platform code lives in MSTest (specifically this folder: https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform)
• there isn't yet any technical documentation available for "extensions"/"framework" authors.

For this last point, if your project is OSS I can definitely help you and you can already start getting some inspiration from there draft PRs I have open:

• Initial support of Microsoft.Testing.Platform nunit/nunit3-vs-adapter#1153
• Add support for Microsoft.Testing.Platform YoloDev/YoloDev.Expecto.TestSdk#152

[thomhurst]

Thanks @Evangelink ! Those PRs are very helpful.

Question, they both seem to be a VS bridge (which makes sense if we're trying to keep backwards compatibility.)

Since my framework is still a WIP, I don't care much about this.

So, is there a non-bridged mode? If so I guess I'll have to wait for documentation, as I'd have to rewrite the VSTest stuff that scans assemblies and generates test cases.

And if there is a non-bridged mode, does that, or will it, come with support for dotnet test and test explorer? I assume it would because these are quite core to the testing experience currently, but those PRs had comments that those features were only made available by that bridging package.

Thanks!

[Evangelink]

Yes there is a non-bridged mode, this is actually the default but to move the masses (MSTest, xUnit and NUnit) we had to bring this compat layer :)

If so I guess I'll have to wait for documentation,

We will craft a dummy sample (tomorrow) that show how to create and plug a custom framework and how to return some state. This can be easily done and would be useful for when we build the technical doc.

And if there is a non-bridged mode, does that, or will it, come with support for dotnet test and test explorer? I assume it would because these are quite core to the testing experience currently, but those PRs had comments that those features were only made available by that bridging package.

Yes definitely! The support of dotnet test will be brought by Microsoft.Testing.Platform.MSBuild package and the support of VS and VS Code Test Explorers is coming. For VS Code Test Explorer, it's already available publicly in the last preview version of C# Dev Kit. For VS Test Explorer, there was a missed part so it will come in VS 17.10 Preview 2.

[thomhurst]

Amazing! Thanks for the prompt replies and look forward to any coming samples and docs!

From what I could see when navigating the code base, this looks like a very welcome improvement.

From recently creating a VSTest adapter, the code felt very legacy and convoluted, and very temperamental around "FullyQualifiedName"s or specific TestCaseProperty's!

Glad to see there's a push for a better testing experience!

[Evangelink]

Dear .NET community, thank you all for your interest and feedback on our new platform!

I'll move forward by closing this ticket and invite you to track the respective test framework tickets:

• For xUnit: Microsoft Testing Platform for xUnit xunit/visualstudio.xunit#402 (related draft PR: Add initial support for Microsoft.Testing.Platform xunit/visualstudio.xunit#403)
• For NUnit: Microsoft Testing Platform for NUnit nunit/nunit3-vs-adapter#1152 (related draft PR: Initial support of Microsoft.Testing.Platform nunit/nunit3-vs-adapter#1153)
• For Expecto: Add support for Microsoft.Testing.Platform YoloDev/YoloDev.Expecto.TestSdk#152

[thomhurst]

Can I ask where to look for the samples for a non-bridged runner when they're out? The samples folder in this repository?

[nohwnd]

@thomhurst I am adding an example that is doing the minimum amount of work to "run a test" here: #2446 There is test framework implementation, and a project using it. In reality the framework would come from a nuget package, and that nuget package would probably generate program.cs via build targets.

All the details of how to actually discover or run a test are up to your implementation. :) I've commented on the parts that did not seem clear, let me know if (ideally in a new issue), if there are more explanations needed. We will eventually document all that api, and add more examples.

[bradwilson]

@nohwnd If I'm reading the code correctly, it seems like the bridge that's being replaced here is just the translation of old ObjectModel types into the new types.

How about for developers who want to control the console output when being run via dotnet run (or even dotnet test)? Is that possible in the new system? I'm thinking about how this system gets merged with my current v3 runners which are already stand-alone executables with their own command lines and behavior that I wouldn't want to lose when moving to the new system. Also, will users be able to pass custom command line arguments to our runners when running via dotnet test?

[thomhurst]

Thanks @nohwnd I've got my runner working! (With dotnet run, and manually hooking it up. I can't get it to automatically hook up with the msbuild package)

My next thing id like to ask is how to get test filters, like how VSTest can filter on properties, but I'll raise an issue for that

[MarcoRossignoli]

@nohwnd If I'm reading the code correctly, it seems like the bridge that's being replaced here is just the translation of old ObjectModel types into the new types.

Yes. And it's needed only when you run in "runner mode"(self-contained) and you don't want to change the "current" VSTest adapter implementation. So it's done for a smooth/quick/co-existence onboarding but it's not mandatory. Pay attention that when you run in "runner mode" you cannot use anymore VSTest extensions like VSTest loggers, datacollectors, in-process datacollectors. The new platform implements a completely different extensibility points and expose a protocol for "remote" interaction (like IDEs VS/VSCode).

How about for developers who want to control the console output when being run via dotnet run (or even dotnet test)? Is that possible in the new system?

The "output device" is abstracted in the new platform. So it's possible plug a completely custom UX. By default we provide a "console" one. The extension point at the moment is "internal" because we're following the "hardening" principle. We will open when we're "good" with the design and all the expected usage. So we can "easily" support the "dotnet run" output customization.

For the "dotnet test" is a bit more tricky.

dotnet test is part of the SDK that contains the infrastructure to translate the "dotnet test" gesture in MSBuild tasks that will be invoked in a "demultiplexer" mode by "every project in a solution or directly to the project self if specified". To support it we shipped an extension point https://nuget.info/packages/Microsoft.Testing.Platform.MSBuild that plug the runner to the SDK infrastructure. The implementation we did changed completely the UX because was done mainly for CI with the "arcade" pattern where you simply see running module and result (we forward the output to a file in the result directory for further analysis). This is improving the understanding and avoid the unclear overlap of the parallel executions, but you can opt-in the console output and it's done using MSBuild loggers (that anyway is changed in 9.0 where we use the new terminal logger).
Here you can find the information about the "dotnet test" support for the new runner https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-runner-integrations#dotnet-test---mstest-runner-mode

Another tricky part are the parameters. Historically the SDK(dotnet test) supports "only" the VSTest platform with fixed options. The new platform is extensible in it and so we cannot anymore have a "fixed" set of options. For this reason as explained in the guide above you have to pass the new custom options(they depends on extensions added) using an MSBuild property -p:TestingPlatformCommandLineArguments=" --option value ".

Anyway this is the first version. We're planning to make it "first citizen" for the SDK and so move back to dotnet test --option value. But for that we need to add some code inside the SDK to fold the "different underneath test applications could support different options so how we show the help? How we show the invalid options per test application?" and so on.

I'm thinking about how this system gets merged with my current v3 runners which are already stand-alone executables with their own command lines and behavior that I wouldn't want to lose when moving to the new system.

As explained above the new platform allows custom options so it can accommodate your scenario. Here what we support https://github.com/microsoft/testfx/blob/main/test/UnitTests/Microsoft.Testing.Platform.UnitTests/CommandLine/CommandLineTests.cs and here the parser https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/CommandLine/Parser.cs

[MarcoRossignoli]

My next thing id like to ask is how to get test filters, like how VSTest can filter on properties, but I'll raise an issue for that

@thomhurst ideally the platform should not define "how to filter tests" because the adapters could express it in different way. The new platform introduces the concept of "test node with attached properties" so your tests could be a tree where every node is not related to a "method".
For this reason we've added a new "filter" options called treenode-filter https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/CommandLine/TreeNodeFilterCommandLineOptionsProvider.cs#L13 where you can express the query using a Graph Query Filtering syntax (i.e. --treenode-filter /*/*/*/*MyTest*. At the moment the filtering infrastructure is "internal" and used by our "custom adapter" for dogfooding. If you're interested in it pls open an issue and we can discuss the usage. Here you can find the draft https://github.com/microsoft/testfx/blob/main/docs/mstest-runner-graphqueryfiltering/graph-query-filtering.md

Anyway you can implement "your custom" filtering system for your tests and expose it with an custom option.

[julealgon]

@nohwnd @Evangelink as part of this new runner architecture, have you considered having tests be a first-class thing in conjunction with the hosting model of Microsoft.Extensions.Hosting? For instance, could we eventually have tests run as, perhaps, a hosted service, and control other aspects of the runs using normal hosting abstractions like DI, logging, configuration, etc? In such environment, configuring a specific test library would look similar to how libraries and frameworks are configured in worker and web projects. Maybe MSTest would expose an AddMSTest(Action<MSTestOptions>) extension on IServiceCollection, while other frameworks would have their own like AddxUnit or AddNUnit.

For one, I'd love to be able to configure OpenTelemetry on some of our testing projects to track down and optimize execution times, push test-specific telemetry information to Datadog, etc. However, the entire test framework architecture today is a "walled garden" that doesn't integrate with anything that other "normal" projects do.

If there is an issue tracking that, please let me know. Otherwise, assuming you'd agree this makes sense, I can open an issue myself documenting the request.

[Evangelink]

@julealgon I'd say it depends how far you are seeing things. Here is the technical doc of the new platform: https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Index.md

In this folder https://github.com/microsoft/testfx/blob/main/samples/public/TestingPlatformExamples/ you will have some guided examples. Looking at Program.cs you can see that the platform can be extended.

As for different hosting capabilities, it's technically possible and we already have a few but we don't yet have anything open-telemetry based. OTel is definitely an essential addition we would like to provide but it's hard to make the progresses we want (too few people and too many things to handle). We don't have a public ticket for what we want to do with OTel but I have found this #1680 one that could maybe overlap with some of your requirements.

[julealgon]

@julealgon I'd say it depends how far you are seeing things. Here is the technical doc of the new platform: https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Index.md

In this folder https://github.com/microsoft/testfx/blob/main/samples/public/TestingPlatformExamples/ you will have some guided examples. Looking at Program.cs you can see that the platform can be extended.

Ok, I really like the way that is shaping up, thanks for sharing @Evangelink . It would be perfect if it was made to work on top of all the Microsoft.Extensions.Hosting model and used the generic host, however.

I see it is doing some of that API shape, but kind of just replicating it, in a sense, in a custom fashion.

If this was ever adapted to work on top of the hosting model, adding open telemetry support would become trivial as one could just tap into the existing OpenTelemetry.Hosting.Extensions package and start integrating with any extension points you provide in the test framework.

[Evangelink]

We had a long look at ASP.NET for the design and on the first prototype we really reused all principles but we realized some cases were either more specific in "web" wolrd or in "testing" world.

As for the explanation of not directly depending on the package, one of the most common issues with VSTest is that there was always a problem with the deps it has (newtonsoft, nuget framework...) where either the version was conflicting with the one wanted by user (and the deps is not declared as package deps but is force copied to the output - breaking user execution), some update issue or this was preventing some scenario to happen because one of the deps was not allowing it.
All in all, we decided to start the new platform with the pillar of having the core platform deps free and to have everything modular so that it's always possible to disable one part.

This might not seem super intuitive but this is the reality of what has been observed through VSTest life.

I would need to make some experiment but I'm wondering if we could have some extension that would allow this bridging.

[julealgon]

I would need to make some experiment but I'm wondering if we could have some extension that would allow this bridging.

One suggestion I'd share @Evangelink is precisely what the OpenTelemetry team did: they have this "raw" version of OTEL, called the SDK, that is basically dependency-free as well, and relies on some static calls and instantiations etc.

Then, they have the OpenTelemetry.Extensions.Hosting package which is basically the "bridge" you are alluding to: a package that seamlessly integrates the lower-level building blocks from the SDK package into the modern Microsoft.Extensions.Hosting model.

This basically allows for completely dependency-free use cases by using only the SDK directly, or the full integration mode for consumers that can opt for that using the hosting extensions.

Something like this seems like it would fit your constraints well, from what you described there.

[Evangelink]

@julealgon Thanks for the pointer! I will check that out (probably next week).