Refactor/rewrite this repository's test suite #1161
Comments
|
Thanks for the write up and taking the initiative. Composing runner and test components totally makes sense to me and I think this would be great for https://github.com/bytecodealliance/go-modules as well since it's currently lacking the |
As I've read more and more of the Rust async runtime support and other various bits and pieces I've wanted more and more the ability to easily write tests for guest interactions with the host. While I don't think it's feasible to generate arbitrary hosts I do think it's possible to do this much more easily than is done today with the testing support in this repository. In essence this commit is an implementation of bytecodealliance#1161. The goal of this commit is to add a `wit-bindgen test` test suite runner. This test suite will be used to migrate all existing tests in this repository to this new framework. In the limit this is expected to make it easier to write tests (no Rust knowledge necessary), make it more flexible to write tests (now you can use raw `*.wat`), and additionally improve the quality of the test suite by making it more reusable. The reusability isn't the highest priority at this time as it's not clear what else would want to reuse this, but my hope is that this refactoring is at least a large-ish leap forward towards having a component model test suite of some kind eventually.
|
#1192 has some initial work towards this |
|
Hooray, thanks Alex! Would the test suite be broken out into its own repo that we can submodule into https://github.com/bytecodealliance/go-modules or live here in this repo? |
|
Heh I honestly hadn't thought that far forward really, so for now my plan was to leave the test suite here. One thing this makes me think of though is the possible use of "git subtrees" as a way of vendoring the test suite. That might make it easy to make local edits (e.g. annotate tests as "this is expected to fail" or something like that). Nevertheless I suspect there's still going to be integration problems for repositories outside this location (e.g. go-modules), but I'd consider those integration pain points as things to fix for sure. |
|
We can just as easily submodule this entire repo and harness tests in a subdirectory. |
As I've read more and more of the Rust async runtime support and other various bits and pieces I've wanted more and more the ability to easily write tests for guest interactions with the host. While I don't think it's feasible to generate arbitrary hosts I do think it's possible to do this much more easily than is done today with the testing support in this repository. In essence this commit is an implementation of bytecodealliance#1161. The goal of this commit is to add a `wit-bindgen test` test suite runner. This test suite will be used to migrate all existing tests in this repository to this new framework. In the limit this is expected to make it easier to write tests (no Rust knowledge necessary), make it more flexible to write tests (now you can use raw `*.wat`), and additionally improve the quality of the test suite by making it more reusable. The reusability isn't the highest priority at this time as it's not clear what else would want to reuse this, but my hope is that this refactoring is at least a large-ish leap forward towards having a component model test suite of some kind eventually.
As I've read more and more of the Rust async runtime support and other various bits and pieces I've wanted more and more the ability to easily write tests for guest interactions with the host. While I don't think it's feasible to generate arbitrary hosts I do think it's possible to do this much more easily than is done today with the testing support in this repository. In essence this commit is an implementation of bytecodealliance#1161. The goal of this commit is to add a `wit-bindgen test` test suite runner. This test suite will be used to migrate all existing tests in this repository to this new framework. In the limit this is expected to make it easier to write tests (no Rust knowledge necessary), make it more flexible to write tests (now you can use raw `*.wat`), and additionally improve the quality of the test suite by making it more reusable. The reusability isn't the highest priority at this time as it's not clear what else would want to reuse this, but my hope is that this refactoring is at least a large-ish leap forward towards having a component model test suite of some kind eventually.
|
I thinking more about #1192 I'm starting to wonder about next steps moving forward. I'd like to sunset
That would only leave the grunt work of actually porting the |
I'd be fine with removing the
You can run them on Mac and Linux with some effort and hacks, and I think there are CI-generated Linux builds of the NativeAOT-LLVM packages available now, so I think we could make that work out-of-the box with a bit of effort. |
|
For c#, It shouldn't be to much effort to enable running the tests for Linux. Mac would require some more work upstream in the runtimelabs repository. I can see if I can get it running quickly on Linux in the current setup, but if you need to move forward with this I will help migrate the runtime tests as a follow up. |
|
Just took a look and the runtime tests already pass on Linux with |
|
I think that "passes" due to the (this is a fork of #1192 with some C# support) |
Got it, I didn't see that just the test passing. Getting these running was pretty straightforward: #1200 We can either add that or do it as part of the migration. |
I've personally long lamented tests in this repository. Currently tests fall into roughly three categories:
codegentests (Rust) - Rust codegen tests use some macro-magic to run agenerate!macro invocation with various options over all the tests intests/codegen/*.wit. This then compiles the output for the native host (not wasm) and then nothing actually happens at runtime as nothing is actually run, things are only generated.codegentests (not Rust) - other languages typically have a#[test]-per-generate!also using some macro-magic. Each test typically spawns some external process and ensures it passes, for example a C compiler is spawned for thewit-bindgen-ccrate.runtimetests - Intests/runtime/*there are a number of test that are organized as#[test]. The idea is that there's one monolithic host runner based on a custom Wasmtime embedding. Each runtime test can be implemented in an array of languages and each language gets run. Compilation happens in an external process and, like before, compilation of Rust binaries is "special" compared to other languages.There are a lot of downsides to this, including:
runtimetests takes forever because you have to compile all of Wasmtime and its embedding.runtimetests take forever because it takes awhile to compile.Overall I would like to move in a new direction that solves all of these problems with a new foundation for running tests. Specifically what I'm imagining is something like this:
runnerimports thetest.runnercomponent exports thewasi:cli/runinterface and looks like a command line executable. In other words it's a "command" in terms of WASI.testcomponent looks like a "reactor" in terms of WASI.runnerandtestcan import WASI functions, for example to print to the screen, read environment variables, etc.runnerwith thetest, producing a final component that's a "command" and only imports WASI.wasmtimeCLI, critically not the embedding API.This scheme solves many of the above concerns (except maybe the "Rust is special" part through perhaps infrastructure refactoring). Compilation of tests should be much quicker as it's just one test at a time and each test case compiles quickly. Furthermore any host can be relatively easily slotted in to reuse this test suite by swapping out the
wasmtimeCLI for something else. This additionally avoids the need to define wonky host APIs that are only visible in the embedding, instead it's a pair ofrunnerandtestcomponents that work with each other. No loss of test coverage should happen because imported functions are tested by implementing arunnerand exported functions are tested by being atest. Both therunnerand thetestcould be implemented in any language which exercises various interesting cross-products of language pairs.This refactoring should also ideally put this on a path towards making it more reasonable to be a sort of "reference test suite" because standalone wasm binaries are produced from source which can be executed anywhere on a WASI runtime. That should make it much easier to, for example, in the future publish these as a separate test suite somewhere like the wasi-testsuite repository.
The text was updated successfully, but these errors were encountered: