Skip to content

Commit

Permalink
docs: new testing guide (#6519)
Browse files Browse the repository at this point in the history
* docs: new testing guide

* docs: new testing guide

* docs: new testing guide

* docs: new testing guide

* docs: new testing guide

---------

Co-authored-by: neverland <[email protected]>
  • Loading branch information
LingyuCoder and chenjiahan authored May 17, 2024
1 parent 96d98de commit be7b382
Show file tree
Hide file tree
Showing 11 changed files with 1,121 additions and 164 deletions.
2 changes: 0 additions & 2 deletions packages/rspack-test-tools/etc/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -643,8 +643,6 @@ export interface IStatsAPITaskProcessorOptions<T extends ECompilerType> {
// (undocumented)
compilerType: T;
// (undocumented)
cwd?: string;
// (undocumented)
name: string;
// (undocumented)
options?: (context: ITestContext) => TCompilerOptions<T>;
Expand Down
1 change: 0 additions & 1 deletion packages/rspack-test-tools/src/processor/stats-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ const serializer = require("jest-serializer-path");
export interface IStatsAPITaskProcessorOptions<T extends ECompilerType> {
options?: (context: ITestContext) => TCompilerOptions<T>;
name: string;
cwd?: string;
compilerType: T;
compiler?: (context: ITestContext, compiler: TCompiler<T>) => Promise<void>;
build?: (context: ITestContext, compiler: TCompiler<T>) => Promise<void>;
Expand Down
37 changes: 26 additions & 11 deletions webpack-test/README.md
Original file line number Diff line number Diff line change
@@ -1,75 +1,89 @@
>**Note**
>**Note**
> This package is heavily based on [webpack/test](https://github.com/webpack/webpack/tree/main/test)
# Progressively migrate webpack test
# Progressively migrate webpack test

Originally, we use this formula to calculate the compatibility `passedTestCaseCount / totalTestCount`, totalTestCount = passedTestCaseCount + failedTestCount + skippedTestCount , but sometimes it maybe hard to compatible with all webpack test cases for some reasons (e.g. performance, legacy feature that we don't want to support), we need a method to skip these tests that we will not support. Thus, we adjust the original formula to `(passedTestCaseCount + willNotSupportTestCount) / totalTestCount`.
Originally, we use this formula to calculate the compatibility `passedTestCaseCount / totalTestCount`, totalTestCount = passedTestCaseCount + failedTestCount + skippedTestCount , but sometimes it maybe hard to compatible with all webpack test cases for some reasons (e.g. performance, legacy feature that we don't want to support), we need a method to skip these tests that we will not support. Thus, we adjust the original formula to `(passedTestCaseCount + willNotSupportTestCount) / totalTestCount`.

Currently, we use a `test.filter.js` under each failed test case directory to skip failed test case, using this method could let us migrate webpack test case progressively without affect the real compatibility (Because this method will not influence the real `passedTestCaseCount`).
Currently, we use a `test.filter.js` under each failed test case directory to skip failed test case, using this method could let us migrate webpack test case progressively without affect the real compatibility (Because this method will not influence the real `passedTestCaseCount`).
e.g.

```js
// test.fitler.js
// test.filter.js
module.exports = () => {
return false; // false means this testcase is skipped for now, but maybe we will support in the future, `-1` means this test case we don't want to compatible with, this related to `willNotSupportTest`.
}

```
When you find that we have passed some failed testcases which is skipped for now, you could change the `test.filter.js` to

When you find that we have passed some failed testcases which is skipped for now, you could change the `test.filter.js` to

```js
module.exports = () => {
return true
}

```

or delete the `test.filter.js`

# Welcome to the webpack test suite

# Welcome to the webpack test suite!!!!
Every pull request that you submit to webpack (besides README and spelling corrections in comments) requires tests that are created.

But don't give up hope!!! Although our tests may appear complex and overwhelming, once you become familiar with the test suite and structure, adding and creating tests will be fun and beneficial as you work inside the codebase! ❤

## tl;dr

Run all tests (this automatically runs the setup):

```sh
pnpm test
```

Run an individual suite:

```sh
pnpm jest ConfigTestCases
```

Watch mode:

```sh
pnpm jest --watch ConfigTestCases
```

See also: [Jest CLI docs](https://jestjs.io/docs/cli)

## Test suite overview

We use Jest for our tests. For more information on Jest you can visit their [homepage](https://jestjs.io/)!

### Class Tests

All test files can be found in *.test.js. There are many tests that simply test APIs of a specific class/file (such as `Compiler`, `Errors`, Integration, `Parser`, `RuleSet`, Validation).
If the feature you are contributing involves one of those classes, then best to start there to understand the structure.

### xCases

In addition to Class specific tests, there are also directories that end in "Cases". The suites for these cases also have corresponding *.test.js files.

#### cases (`TestCases.test.js`) <sup>1</sup>

Cases are a set of general purpose tests that will run against a variety of permutations of webpack configurations. When you are making a general purpose change that doesn't require you to have a special configuration, you would likely add your tests here. Inside of the `./test/cases` directory you will find tests are broken into thematic sub directories. Take a moment to explore the different options.

To add a new case, create a new directory inside of the top level test groups, and then add an `index.js` file (and any other supporting files).

By default this file will be the entry point for the test suite and you can add your `it()`'s there. This will also become bundled so that node env support happens as well.

#### configCases (`ConfigTestCases.basictest.js`) <sup>1</sup>

If you are trying to solve a bug which is reproducible when x and y properties are used together in a config, then configCases is the place to be!!!!

In addition to an `index.js`, these configCases require a `webpack.config.js` is located inside of your test suite. This will run this specific config through `webpack` just as you were building individually. They will use the same loading/bundling technique of your `it()` tests, however you now have a more specific config use cases that you can write even before you start coding.

#### statsCases (`StatsTestCases.basictest.js`)

Stats cases are similar to configCases except specifically focusing on the `expected` output of your stats. Instead of writing to the console, however the output of stats will be written to disk.

By default, the "expected" outcome is a pain to write by hand so instead when statsCases are run, runner is checking output using jest's awesome snapshot functionality.
Expand All @@ -83,6 +97,7 @@ Please follow the approach described below:
```javascript
import("./someModule");
```

* don't forget the `webpack.config.js`
* run the test
* jest will automatically add the output from your test code to `StatsTestCases.test.js.snap` and you can always check your results there
Expand All @@ -91,16 +106,16 @@ import("./someModule");
You can read more about SnapShot testing [right here](https://jestjs.io/docs/snapshot-testing)

## Questions? Comments?
If you are still nervous or don't quite understand, please submit an issue and tag us in it, and provide a relevant PR while working on!

If you are still nervous or don't quite understand, please submit an issue and tag us in it, and provide a relevant PR while working on!

## Footnotes
<sup>1</sup> webpack's parser supports the use of ES2015 features like arrow functions, harmony exports, etc. However as a library we follow Node.js' timeline for dropping older versions of node. Because of this we expect your tests on GitHub Actions to pass all the way back to NodeJS v10; Therefore if you would like specific tests that use these features to be ignored if they are not supported, then you should add a `test.filter.js` file. This allows you to import the syntax needed for that test, meanwhile ignoring it on node versions (during CI) that don't support it. webpack has a variety of helpful examples you can refer to if you are just starting out. See the `./helpers` folder to find a list of the versions.

<sup>1</sup> webpack's parser supports the use of ES2015 features like arrow functions, harmony exports, etc. However as a library we follow Node.js' timeline for dropping older versions of node. Because of this we expect your tests on GitHub Actions to pass all the way back to NodeJS v10; Therefore if you would like specific tests that use these features to be ignored if they are not supported, then you should add a `test.filter.js` file. This allows you to import the syntax needed for that test, meanwhile ignoring it on node versions (during CI) that don't support it. webpack has a variety of helpful examples you can refer to if you are just starting out. See the `./helpers` folder to find a list of the versions.

## Credits

Thanks to:

- [The webpack team and community](https://webpack.js.org/) for creating a great bundler and ecosystem from which we draw a lot of inspiration.
- [@sokra](https://github.com/sokra) for the great work on the [webpack](https://github.com/webpack/webpack) project.
* [The webpack team and community](https://webpack.js.org/) for creating a great bundler and ecosystem from which we draw a lot of inspiration.
* [@sokra](https://github.com/sokra) for the great work on the [webpack](https://github.com/webpack/webpack) project.
11 changes: 10 additions & 1 deletion website/docs/en/contribute/development/_meta.json
Original file line number Diff line number Diff line change
@@ -1 +1,10 @@
["prerequisites", "building", "testing", "debugging", "profiling", "releasing"]
[
"prerequisites",
"building",
"testing",
"testing-webpack",
"testing-rspack",
"debugging",
"profiling",
"releasing"
]
Loading

2 comments on commit be7b382

@rspack-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Benchmark detail: Open

Name Base (2024-05-17 96d98de) Current Change
10000_development-mode + exec 2.59 s ± 24 ms 2.59 s ± 18 ms +0.17 %
10000_development-mode_hmr + exec 699 ms ± 14 ms 702 ms ± 8.8 ms +0.38 %
10000_production-mode + exec 2.46 s ± 48 ms 2.5 s ± 24 ms +1.55 %
arco-pro_development-mode + exec 2.43 s ± 65 ms 2.45 s ± 80 ms +0.60 %
arco-pro_development-mode_hmr + exec 432 ms ± 1.5 ms 433 ms ± 2.6 ms +0.29 %
arco-pro_development-mode_hmr_intercept-plugin + exec 440 ms ± 3.3 ms 441 ms ± 1.6 ms +0.24 %
arco-pro_development-mode_intercept-plugin + exec 3.28 s ± 58 ms 3.29 s ± 71 ms +0.30 %
arco-pro_production-mode + exec 4.04 s ± 64 ms 4.05 s ± 56 ms +0.09 %
arco-pro_production-mode_intercept-plugin + exec 4.88 s ± 89 ms 4.87 s ± 58 ms -0.17 %
threejs_development-mode_10x + exec 1.97 s ± 28 ms 1.96 s ± 20 ms -0.14 %
threejs_development-mode_10x_hmr + exec 761 ms ± 21 ms 749 ms ± 2.6 ms -1.55 %
threejs_production-mode_10x + exec 5.21 s ± 29 ms 5.19 s ± 24 ms -0.26 %

@rspack-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Ran ecosystem CI: Open

suite result
modernjs, self-hosted, Linux, ci ❌ failure
_selftest, ubuntu-latest ✅ success
nx, ubuntu-latest ✅ success
rspress, ubuntu-latest ✅ success
rsbuild, ubuntu-latest ✅ success
compat, ubuntu-latest ✅ success
examples, ubuntu-latest ✅ success

Please sign in to comment.