Skip to content

Commit

Permalink
Merge pull request #48 from woocommerce/24-11/orchestration-docs
Browse files Browse the repository at this point in the history
Update Custom Tests Documentation with Advanced Scaffolding and Lifecycle Changes
  • Loading branch information
Luc45 authored Dec 4, 2024
2 parents e1b5efc + d13770b commit c7c7911
Show file tree
Hide file tree
Showing 11 changed files with 191 additions and 255 deletions.
39 changes: 33 additions & 6 deletions docs/custom-tests/01-generating-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,15 @@ This will create a basic E2E test in the `e2e` directory with essentially a `exa

```
bootstrap (Optional)
bootstrap.sh
bootstrap.php
setup.js
setup.sh
dependencies.json
mu-plugin.php
example.spec.js
```

:::tip
Bootstrap files are optional and can be removed if not needed. [Learn more about bootstrapping](/docs/custom-tests/bootstrap-and-test-phases) and its use cases.
Bootstrap files are optional and can be removed if not needed. [Learn more about bootstrapping](/docs/custom-tests/understanding-lifecycle) and its use cases.
:::

You can run your first test locally with:
Expand All @@ -36,6 +37,33 @@ qit run:e2e <your-plugin> ./e2e

You can then expand it with more tests, or even generate tests with Playwright Codegen.

## Advanced Scaffolding

When scaffolding, you can also include shared setups and teardowns. Refer to the [Understanding Lifecycle](/docs/custom-tests/understanding-lifecycle) documentation for more information.

```qitbash
qit scaffold:e2e ./e2e --with-shared --with-teardown
```

This will give you the following structure:

```
bootstrap
setup.js (Playwright file that runs in isolation before your tests)
setup.sh (Bash file that runs in isolation before your tests)
shared-setup.js (Playwright file that runs before all tests in a compatibility test)
shared-setup.sh (Bash file that runs before all tests in a compatibility test)
shared-teardown.js (Playwright file that runs after all tests in a compatibility test)
shared-teardown.sh (Bash file that runs after all tests in a compatibility test)
teardown.js (Playwright file that runs in isolation after your tests)
teardown.sh (Bash file that runs in isolation after your tests)
example.spec.js
```

When you run a compatibility test (eg: `qit run:e2e example-plugin --plugin example-plugin:test`), a database snapshot is taken after the shared setup, and restored for each plugin test. The shared teardown is run after all tests are done.

Anything you do in your isolated setup and teardown files will only affect your plugin's test environment. Anything you do in the shared setup and teardown files will affect all tests in the compatibility test.

## Codegen

Codegen is a helpful Playwright feature for generating most of the E2E test code for you.
Expand All @@ -54,12 +82,11 @@ It essentially records your interactions with the browser and generates the code

When you generate tests with `--codegen`, they will be generated with the URLs you visited during the recording, eg:


#### How Codegen generates it:

```js
await page.goto('http://localhost:32456');
await page.goto('http://localhost:32456?cat=pictures');
await page.goto('http://localhost:32456?action=foo');
await page.goto('http://localhost:32456/my-page');
await page.goto('http://localhost:32456/wp-admin');
```
Expand All @@ -70,7 +97,7 @@ After pasting it in a test file, remove the URLs, as the test run uses a `baseUR

```js
await page.goto('/');
await page.goto('/?cat=pictures');
await page.goto('/?action=foo');
await page.goto('/my-page');
await page.goto('/wp-admin');
```
Expand Down
24 changes: 13 additions & 11 deletions docs/custom-tests/02-tagging-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ The custom E2E tests feature is available as early-access.

## Introduction

You can use test tags to run specific tests from a plugin/theme.
You can publish your tests to QIT so that other developers can use it to run compatibility tests with their own extensions.

Each extension can have one or multiple test tags, and we also offer some generic tests that can be used by any plugin/theme.
Similarly, you can run other developers' tests to ensure that your extension is compatible with theirs.

One plugin can have multiple test tags, which can be used to run different sets of tests.

## Listing available test tags

Expand All @@ -21,61 +23,61 @@ qit tag:list
To list the test tags for a specific plugin/theme:

```qitbash
qit tag:list qit-beaver
qit tag:list example-plugin
```

## Uploading tests

You can upload your tests and make them available as a tag with the command:

```qitbash
qit tag:upload qit-beaver /path/to/tests
qit tag:upload example-plugin /path/to/tests
```

By default, the test will be uploaded as the `default` tag.

If you want to specify a test tag, you can add the tag in this format: `extension:tag`:

```qitbash
qit tag:upload qit-beaver:my-tag /path/to/tests
qit tag:upload example-plugin:my-tag /path/to/tests
```

## Running test tags

Now you can run your test both locally and in CI using the `default` tag:

```qitbash
qit run:e2e qit-beaver
qit run:e2e example-plugin
```

Or, if it's a specific tag:

```qitbash
qit run:e2e qit-beaver my-tag
qit run:e2e example-plugin my-tag
```

## Running test tags from other plugins

Other developers that have access to your extension can also use your tests for compatibility testing.

Let's suppose that `qit-dog` has published their tests. You can run your tests and theirs with:
Let's suppose that `example-plugin-2` has published their tests. You can run your tests and theirs with:

```qitbash
qit run:e2e qit-beaver --plugin qit-dog:test
qit run:e2e example-plugin --plugin example-plugin-2:test
```

## Running multiple tags

You can also compose multiple tags by passing a comma-separated list of test tags:

```qitbash
qit run:e2e qit-beaver default,rc --plugin qit-dog:test:feature-dog-pictures
qit run:e2e example-plugin default,rc --plugin example-plugin-2:test:some-feature
```

## Deleting test tags

You can delete test tags that you have previously published:

```qitbash
qit tag:delete qit-beaver:my-tag
qit tag:delete example-plugin:my-tag
```
68 changes: 34 additions & 34 deletions docs/custom-tests/03-running-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,22 @@ The custom E2E tests feature is available as early-access.
Assuming you have generated and uploaded a E2E test, the basic syntax for running a test is:

```qitbash
qit run:e2e qit-beaver
qit run:e2e example-plugin
```

If you haven't uploaded a test to QIT, you can run a local test:

```qitbash
qit run:e2e qit-beaver ~/my-plugins/qit-beaver/tests
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests
```

:::tip
Replace "qit-beaver" with the slug of an extension you own.
Replace "example-plugin" with the slug of an extension you own.
:::

## Using a config file

Place a `qit-env.json` or `qit-env.yml` file in the directory you run `qit run:e2e` from.
Place a `qit.json` or `qit.yml` file in the directory you run `qit run:e2e example-plugin` from.

Here's an example of a complex config file:

Expand All @@ -45,11 +45,11 @@ values={[
"php_version": "8.3",
"object_cache": true,
"plugins": {
"qit-beaver": {
"example-plugin": {
"action": "test",
"source": "~/my-plugins/qit-beaver",
"source": "~/my-plugins/example-plugin",
"test_tags": [
"~/my-plugins/qit-beaver/tests"
"~/my-plugins/example-plugin/tests"
]
},
"woocommerce": {
Expand All @@ -61,12 +61,12 @@ values={[
"foo-plugin": {
"action": "install"
},
"qit-dog": {
"example-plugin-2": {
"action": "test",
"source": "https://github.com/qit-plugins/qit-dog/releases/tag/nightly.zip",
"source": "https://github.com/qit-plugins/example-plugin-2/releases/tag/nightly.zip",
"test_tags": [
"nightly",
"feature-dog-pictures"
"some-feature"
]
}
}
Expand All @@ -80,23 +80,23 @@ wordpress_version: nightly
php_version: 8.3
object_cache: true
plugins:
qit-beaver:
example-plugin:
action: test
source: ~/.qit/plugins/qit-beaver
source: ~/.qit/plugins/example-plugin
test_tags:
- ~/my-plugins/qit-beaver/tests
- ~/my-plugins/example-plugin/tests
woocommerce:
action: bootstrap
test_tags:
- default
foo-plugin:
action: install
qit-dog:
example-plugin-2:
action: test
source: https://github.com/qit-plugins/qit-dog/releases/tag/nightly.zip
source: https://github.com/qit-plugins/example-plugin-2/releases/tag/nightly.zip
test_tags:
- nightly
- feature-dog-pictures
- some-feature
```
</TabItem>
Expand All @@ -109,13 +109,13 @@ You can also mimick this entire config file using only runtime parameters.
Suppose you are scripting a test run and want to pass everything as parameters:
```qitbash
qit run:e2e qit-the-beaver ~/my-plugins/qit-beaver/tests --source ~/.qit/plugins/qit-beaver \
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests --source ~/.qit/plugins/example-plugin \
--wp nightly \
--php-version 8.3 \
--object-cache \
--plugin woocommerce:bootstrap \
--plugin foo-plugin:activate \
--plugin https://github.com/qit-plugins/qit-dog/releases/tag/nightly.zip:test:nightly,feature-dog-pictures:qit-dog
--plugin https://github.com/qit-plugins/example-plugin-2/releases/tag/nightly.zip:test:nightly,some-feature:example-plugin-2
```

## The plugin syntax
Expand Down Expand Up @@ -144,9 +144,9 @@ Examples:
```
# Scenarios wehere inferred `slug` is `extension2`:
qit run:e2e qit-beaver --plugin extension2
qit run:e2e qit-beaver --plugin ~/my-plugins/qit-beaver/extension2.zip
qit run:e2e qit-beaver --plugin https://github.com/woocommerce/qit-beaver/releases/tag/extension2.zip
qit run:e2e example-plugin --plugin extension2
qit run:e2e example-plugin --plugin ~/my-plugins/example-plugin/extension2.zip
qit run:e2e example-plugin --plugin https://github.com/woocommerce/example-plugin/releases/tag/extension2.zip
```

## Examples:
Expand All @@ -156,19 +156,19 @@ qit run:e2e qit-beaver --plugin https://github.com/woocommerce/qit-beaver/releas
Directory as `source`:

```qitbash
qit run:e2e qit-the-beaver --source ~/my-plugins/qit-beaver
qit run:e2e example-plugin --source ~/my-plugins/example-plugin
```

Zip file as `source`:

```qitbash
qit run:e2e qit-the-beaver --source ~/my-plugins/qit-beaver/qit-beaver.zip
qit run:e2e example-plugin --source ~/my-plugins/example-plugin/example-plugin.zip
```

URL as `source`:

```qitbash
qit run:e2e qit-the-beaver --source https://github.com/woocommerce/qit-beaver/releases/tag/qit-beaver.zip
qit run:e2e example-plugin --source https://github.com/woocommerce/example-plugin/releases/tag/example-plugin.zip
```


Expand All @@ -179,59 +179,59 @@ If you have a test that you haven't uploaded yet, you can run it directly:
Slug as `source`, directory as `test-tags`:

```qitbash
qit run:e2e qit-the-beaver ~/my-plugins/qit-beaver/tests
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests
```

Directory as `source`, directory as `test-tags`:

```qitbash
qit run:e2e qit-the-beaver ~/my-plugins/qit-beaver/tests --source ~/my-plugins/qit-beaver
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests --source ~/my-plugins/example-plugin
```

Zip file as `source`, directory as `test-tags`:

```qitbash
qit run:e2e qit-the-beaver ~/my-plugins/qit-beaver/tests --source ~/my-plugins/qit-beaver/build.zip
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests --source ~/my-plugins/example-plugin/build.zip
```

URL as `source`, directory as `test-tags`:

```qitbash
qit run:e2e qit-the-beaver ~/my-plugins/qit-beaver/tests --source https://github.com/woocommerce/qit-beaver/releases/tag/nightly.zip
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests --source https://github.com/woocommerce/example-plugin/releases/tag/nightly.zip
```

## Using test tags

If you have uploaded a test with a tag different than the default one, you can run it with:

```qitbash
qit run:e2e qit-beaver nightly
qit run:e2e example-plugin nightly
```

You can also run multiple tags:

```qitbash
qit run:e2e qit-beaver nightly,foo-feature
qit run:e2e example-plugin nightly,foo-feature
```

And you can run a test with a tag from a local directory, file, or URL:

```qitbash
qit run:e2e qit-the-beaver nightly,foo-feature --source ~/my-plugins/qit-beaver
qit run:e2e example-plugin nightly,foo-feature --source ~/my-plugins/example-plugin
```

```qitbash
qit run:e2e qit-the-beaver nightly,foo-feature --source ~/my-plugins/qit-beaver/qit-beaver.zip
qit run:e2e example-plugin nightly,foo-feature --source ~/my-plugins/example-plugin/example-plugin.zip
```

```qitbash
qit run:e2e qit-the-beaver nightly,foo-feature --source https://github.com/woocommerce/qit-beaver/releases/tag/qit-beaver.zip
qit run:e2e example-plugin nightly,foo-feature --source https://github.com/woocommerce/example-plugin/releases/tag/example-plugin.zip
```

And even run a local test in the mix:

```qitbash
qit run:e2e qit-the-beaver ~/my-plugins/qit-beaver/tests,nightly,foo-feature --source ~/my-plugins/qit-beaver
qit run:e2e example-plugin ~/my-plugins/example-plugin/tests,nightly,foo-feature --source ~/my-plugins/example-plugin
```

## Testing plugins that require a live site
Expand Down
Loading

0 comments on commit c7c7911

Please sign in to comment.