Execution of tests inside a WASM runtime

[idg10]

I maintain an open source library (Rx.NET) and am starting to see interest in using this library inside Blazor WebAssembly-based projects.

To support this, I would want to be able to run our complete test suite inside the target execution environment. We know that there are some differences in runtime capability that cause problems for the library in WebAssembly. (E.g., the inability to create threads is something we can work around, but don't currently automatically detect.) We'd be able to discover a lot of potential problems by executing the test suite inside a WebAssembly host.

As far as I can tell, bUnit isn't designed to be used this way. It seems that tests sit on the outside of Blazor components. It looks like test projects run in a normal .NET runtime environment, and bUnit provides the means for a test to spin up instance of Blazor components and poke at them from the outside.

We have for years already been doing something very similar to verify that Rx.NET works inside a UWP environment. From a testing perspective it's quite similar: it's a framework for building UI applications, and .NET code running on the inside of such an app often behaves quite differently from the same code running in other .NET runtimes. So to verify that Rx.NET behaves correctly when used from the inside of a UWP app, we spin up an actual interactive app (you see a window open during test execution because that's the only way to get the UWP runtime environment to host your code). And critically, we execute the entire test suite completely inside that application host. For Blazor WebAssembly, I was envisaging launching a browser and running the entire test suite inside a WASM-based .NET runtime environment running inside that browser, in a similar way.

We have over 3700 tests, and we're able to execute exactly the same tests across all runtime environments we support. So although I know it would be technically possible to take a different approach, testing Rx.NET by writing a series of Blazor components and then writing specialized tests that use bUnit to drive those components, that would be extremely painful given the number of tests.

Is this something that has been considered for this project? (Is there some other way to do this that I'm unaware of?)

[egil]

I maintain an open source library (Rx.NET) and am starting to see interest in using this library inside Blazor WebAssembly-based projects.

To support this, I would want to be able to run our complete test suite inside the target execution environment. We know that there are some differences in runtime capability that cause problems for the library in WebAssembly. (E.g., the inability to create threads is something we can work around, but don't currently automatically detect.) We'd be able to discover a lot of potential problems by executing the test suite inside a WebAssembly host.

As far as I can tell, bUnit isn't designed to be used this way. It seems that tests sit on the outside of Blazor components. It looks like test projects run in a normal .NET runtime environment, and bUnit provides the means for a test to spin up instance of Blazor components and poke at them from the outside.

This is correct. bUnit is closer to Blazor Server or Blazor Web Apps in that all code is being executed in a standard .NET runtime on your machine, in bUnits case, typically a unit testing project, e.g. xUnit.

We have for years already been doing something very similar to verify that Rx.NET works inside a UWP environment. From a testing perspective it's quite similar: it's a framework for building UI applications, and .NET code running on the inside of such an app often behaves quite differently from the same code running in other .NET runtimes. So to verify that Rx.NET behaves correctly when used from the inside of a UWP app, we spin up an actual interactive app (you see a window open during test execution because that's the only way to get the UWP runtime environment to host your code). And critically, we execute the entire test suite completely inside that application host. For Blazor WebAssembly, I was envisaging launching a browser and running the entire test suite inside a WASM-based .NET runtime environment running inside that browser, in a similar way.

We have over 3700 tests, and we're able to execute exactly the same tests across all runtime environments we support. So although I know it would be technically possible to take a different approach, testing Rx.NET by writing a series of Blazor components and then writing specialized tests that use bUnit to drive those components, that would be extremely painful given the number of tests.

Using bUnit to drive the tests would not change the environment anyway, it would still be running in full .NET runtime, not the Blazor WASM runtime.

Is this something that has been considered for this project? (Is there some other way to do this that I'm unaware of?)

Not directly. We did consider providing a bunit.e2e package that would build on top of selenium or playwright and enable some of that. However, it never made sense to go there since Playwright provides most of what you need.

There is a demo of testing with Playwright over here - https://github.com/egil/BlazorTestingAZ - and the related presentation is here - https://www.youtube.com/watch?v=aorfcDeHUpw.

What you would probably want to do is to create a Blazor Wasm solution, then invoke the xunit test runner from a Blazor component, capture the output of the xunit test run, and report that back somehow. But it would be running in a browser so sandbox and all that stuff would have to be considered.

Another approach would be to use WASI (https://devblogs.microsoft.com/dotnet/extending-web-assembly-to-the-cloud/). You would compile rx.net and test suite to wasm, and run the test project using wasmtime. In theory, since it is all WASM, that should verify that things work there too, even though it is not using the browser implementation of the wasm runtime.

[idg10]

What you would probably want to do is to create a Blazor Wasm solution, then invoke the xunit test runner from a Blazor component

It would be MSTest (because we continue to support UWP, and xunit has effectively dropped support for UWP, so we had to convert all the xunit tests into MSTest tests earlier this year to enable them to continue to run), but otherwise, yes, this is what I feared would be necessary.

One significant problem with this approach is that if you're driving the test framework yourself directly you're going to lose all tooling integration—stuff like dotnet test isn't going to work, and perhaps more significantly for us, we'd lose the automatic collection of test results in our CI builds. The 'obvious' solution would be to write a test adapter, but having written one of those in the past (https://github.com/interactsw/fuchu-adapter) I know how much work that is, and don't relish the prospect.

So I guess this means that nobody's running any .NET unit tests inside WASM runtime environments today. I would hope that if Microsoft is serious about WASM as a target (particularly if WASI gains momentum) then they will have to plug this gap at some point, so I'd rather they did it than me...

[egil]

Are you doing anything you think won't work in wasm. If things break, it's likely a dotnet wasm issue and not a problem you can do much about.

Compared to other platforms you are supporting the wasm runtime is modern in that it supports modern versions of dotnet and c#.

So I would not prioritize running tests if I was in your shoes.

If you must, you can probably skip creating a custom test runner. Instead, I would:

• create a Blazor Wamn app that reference your test project.
• in the app, create a RunTest.razor page with a querystring parameter which points to a test from the test project. When navigated to, the page will run the test it is pointed at and print result to screen.
• run the app using playwright from MSTest. It could be a data driven test such that a single invocation points to a single test from the real test project. It would use playwright to spin up the blazor wasm app, go to the RunTest and have it run the test and read the result and then throw if the test fails in the browser.

[egil]

Btw. I am a huge fan of Rx and its great that you and the team have brought it back to life.

[ScarletKuro]

Necro posting, but can’t you just use the dotnet wasi sdk https://github.com/dotnet/runtime/blob/main/src/mono/wasi/README.md to run in WASM runtime? You can also look at https://github.com/SteveSandersonMS/DotNetIsolator (but it's missing a lot of features). You don’t need a browser for that. Obviously this won't bring out of the box integration with unit tests :(