diff --git a/docs/snippets/common/chromatic-storybook-add.npm.js.mdx b/docs/snippets/common/chromatic-storybook-add.npm.js.mdx new file mode 100644 index 000000000000..e2acf17f17e3 --- /dev/null +++ b/docs/snippets/common/chromatic-storybook-add.npm.js.mdx @@ -0,0 +1,3 @@ +```shell +npx storybook@latest add @chromatic-com/storybook +``` diff --git a/docs/snippets/common/chromatic-storybook-add.pnpm.js.mdx b/docs/snippets/common/chromatic-storybook-add.pnpm.js.mdx new file mode 100644 index 000000000000..eec9409ebf86 --- /dev/null +++ b/docs/snippets/common/chromatic-storybook-add.pnpm.js.mdx @@ -0,0 +1,3 @@ +```shell +pnpm dlx storybook@latest add @chromatic-com/storybook +``` diff --git a/docs/snippets/common/chromatic-storybook-add.yarn.js.mdx b/docs/snippets/common/chromatic-storybook-add.yarn.js.mdx new file mode 100644 index 000000000000..e105517209d9 --- /dev/null +++ b/docs/snippets/common/chromatic-storybook-add.yarn.js.mdx @@ -0,0 +1,3 @@ +```shell +yarn dlx storybook@latest add @chromatic-com/storybook +``` diff --git a/docs/writing-tests/accessibility-testing.md b/docs/writing-tests/accessibility-testing.md index 7151a43e8b73..8c29a2a2c6b5 100644 --- a/docs/writing-tests/accessibility-testing.md +++ b/docs/writing-tests/accessibility-testing.md @@ -252,7 +252,7 @@ Additionally, if you have already [disabled accessibility](#how-to-disable-a11y- Browser-based accessibility tests, like those found in Storybook, evaluate the rendered DOM because that gives you the highest accuracy. Auditing code that hasn't been compiled yet is one step removed from the real thing, so you won't catch everything the user might experience. -#### Learn about other UI tests +**Learn about other UI tests** - [Test runner](./test-runner.md) to automate test execution - [Visual tests](./visual-testing.md) for appearance diff --git a/docs/writing-tests/chromatic-first-build-optimized.png b/docs/writing-tests/chromatic-first-build-optimized.png deleted file mode 100644 index a325b18fd148..000000000000 Binary files a/docs/writing-tests/chromatic-first-build-optimized.png and /dev/null differ diff --git a/docs/writing-tests/chromatic-second-build-optimized.png b/docs/writing-tests/chromatic-second-build-optimized.png deleted file mode 100644 index c751c0547c13..000000000000 Binary files a/docs/writing-tests/chromatic-second-build-optimized.png and /dev/null differ diff --git a/docs/writing-tests/component-visual-testing-optimized.mp4 b/docs/writing-tests/component-visual-testing-optimized.mp4 index 0f5491b70550..34ce04dd0962 100644 Binary files a/docs/writing-tests/component-visual-testing-optimized.mp4 and b/docs/writing-tests/component-visual-testing-optimized.mp4 differ diff --git a/docs/writing-tests/interaction-testing.md b/docs/writing-tests/interaction-testing.md index 4cc94872b15f..b949aa284a8a 100644 --- a/docs/writing-tests/interaction-testing.md +++ b/docs/writing-tests/interaction-testing.md @@ -224,7 +224,7 @@ Interaction tests can be expensive to maintain when applied wholesale to every c Interaction tests integrate Jest and Testing Library into Storybook. The biggest benefit is the ability to view the component you're testing in a real browser. That helps you debug visually, instead of getting a dump of the (fake) DOM in the command line or hitting the limitations of how JSDOM mocks browser functionality. It's also more convenient to keep stories and tests together in one file than having them spread across files. -#### Learn about other UI tests +**Learn about other UI tests** - [Test runner](./test-runner.md) to automate test execution - [Visual tests](./visual-testing.md) for appearance diff --git a/docs/writing-tests/stories-in-unit-tests.md b/docs/writing-tests/stories-in-unit-tests.md index 146bae494833..0b7989bf0d12 100644 --- a/docs/writing-tests/stories-in-unit-tests.md +++ b/docs/writing-tests/stories-in-unit-tests.md @@ -182,7 +182,7 @@ When using the `composeStories` or `composeStory` functions, the components bein -#### Learn about other UI tests +**Learn about other UI tests** - [Test runner](./test-runner.md) to automate test execution - [Visual tests](./visual-testing.md) for appearance diff --git a/docs/writing-tests/test-coverage.md b/docs/writing-tests/test-coverage.md index de8b5e3cbf20..07dd1d17bcfe 100644 --- a/docs/writing-tests/test-coverage.md +++ b/docs/writing-tests/test-coverage.md @@ -193,7 +193,7 @@ If you generated a production build optimized for performance with the [`--test` As the [coverage addon](https://storybook.js.org/addons/@storybook/addon-coverage) is based on Webpack5 loaders and Vite plugins for code instrumentation, frameworks that don't rely upon these libraries (e.g., Angular configured with Webpack), will require additional configuration to enable code instrumentation. In that case, you can refer to the following [repository](https://github.com/yannbf/storybook-coverage-recipes) for more information. -#### Learn about other UI tests +**Learn about other UI tests** - [Test runner](./test-runner.md) to automate test execution - [Visual tests](./visual-testing.md) for appearance diff --git a/docs/writing-tests/test-runner.md b/docs/writing-tests/test-runner.md index 9bd1a8f5f0d8..d83ae2af1143 100644 --- a/docs/writing-tests/test-runner.md +++ b/docs/writing-tests/test-runner.md @@ -465,7 +465,7 @@ As the test runner is based on Playwright, you might need to use specific docker If you've enabled filtering tests with tags and provided similar tags to the `include` and `exclude` lists, the test-runner will execute the tests based on the `exclude` list and ignore the `include` list. To avoid this, make sure the tags provided to the `include` and `exclude` lists differ. -#### Learn about other UI tests +**Learn about other UI tests** - Test runner to automate test execution - [Visual tests](./visual-testing.md) for appearance diff --git a/docs/writing-tests/visual-testing.md b/docs/writing-tests/visual-testing.md index c4e2f66f68ac..a848ada9eb60 100644 --- a/docs/writing-tests/visual-testing.md +++ b/docs/writing-tests/visual-testing.md @@ -1,10 +1,11 @@ --- -title: 'Visual tests' +title: Visual tests +hideRendererSelector: true --- -Visual tests, also called visual regression tests, catch bugs in UI appearance. They work by taking screenshots of every story and comparing them commit-to-commit to identify changes. +Visual tests catch bugs in UI appearance. They work by taking screenshots of every story and comparing them to previous versions to identify visual changes. This is ideal for verifying layout, color, size, contrast, and any other visual aspect of your UI. -Ideal for verifying what the user sees: layout, color, size, and contrast. Storybook is a fantastic tool for visual testing because every story is essentially a test specification. Any time you write or update a story, you get a spec for free. +Storybook supports cross-browser visual testing natively using [Chromatic](https://www.chromatic.com/storybook/?ref=storybook_site), a cloud service made by the Storybook team. When you enable visual testing, every story is automatically turned into a test. This gives you instant feedback on UI bugs directly in Storybook. -There are [many tools](https://github.com/mojoaxel/awesome-regression-testing) for visual testing. We recommend [Chromatic](https://www.chromatic.com?utm_source=storybook_website&utm_medium=link&utm_campaign=storybook) by Storybook maintainers to run visual tests in a lightning-fast cloud browser environment. +### Install the addon -For a self-managed alternative to Chromatic, we offer [test runner](./test-runner.md). It allows you to run visual tests on stories by integrating with [Jest](https://jestjs.io/) and [Playwright](https://playwright.dev/). Here's an example [recipe for visual testing stories](https://github.com/storybookjs/test-runner#image-snapshot-recipe). - -## Setup Chromatic addon - -Chromatic is a cloud service built for Storybook. It allows you to run visual tests with zero-config. - -To get started, sign up with your [GitHub](https://github.com/), [GitLab](https://about.gitlab.com/), [Bitbucket](https://bitbucket.org/), or email and generate a unique `` for your Storybook. - -Next, install the [chromatic](https://www.npmjs.com/package/chromatic) CLI package from npm: +Add visual tests to your project by installing `@chromatic-com/storybook`, the official addon by Storybook maintainers: -Run the following command after the package finishes installing: + -```shell -npx chromatic --project-token -``` +Storybook 7.4 or higher is required. Read the [migration guide](../migration-guide.md) to upgrade your project. - - - Don't forget to replace `your-project-token` with the one provided by Chromatic. - -```shell -Build 1 published. +### Enable visual tests + +When you start Storybook, you'll see a new addon panel for Visual Tests where you can run tests and view results. + +![Visual Tests addon enabled](./vta-enable.png) + +To enable visual testing, sign up for [Chromatic](https://www.chromatic.com/start?startWithSignup=true&ref=storybook_site) and create a project. This will give you access to a fleet of cloud browsers. + +![Visual Tests addon project selection](./vta-select-project.png) + +Select a project from your project list to finish setup. If you're setting up the addon for the first time, the configuration files and necessary project identifiers will be added for you automatically. + +### Configure -View it online at https://www.chromatic.com/build?appId=...&number=1. +The addon includes configuration options covering most use cases by default. You can also fine-tune the addon configuration to match your project's requirements via the [`./chromatic.config.json`](https://www.chromatic.com/docs/cli#configuration) file. Below are the available options and examples of how to use them. + +| Option | Description | +| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| `projectId` | Automatically configured. Sets the value for the project identifier
`options: { projectId: 'Project:64cbcde96f99841e8b007d75' }` | +| `buildScriptName` | Optional. Defines the custom Storybook build script
`options: { buildScriptName: 'deploy-storybook' }` | +| `debug` | Optional. Output verbose debugging information to the console.
`options: { debug: true }` | +| `zip` | Optional. Recommended for large projects. Configures the addon to deploy your Storybook to Chromatic as a zip file.
`options: { zip: true }` | + +```jsonc +// .storybook/chromatic.config.json +{ + "buildScriptName": "deploy-storybook", + "debug": true, + "projectId": "Project:64cbcde96f99841e8b007d75", + "zip": true +} ``` - +### Run visual tests -Before running Chromatic's CLI ensure you have at least two commits added to the repository to prevent build failures, as Chromatic relies on a full Git history graph to establish the baselines. Read more about baselines in Chromatic's [documentation](https://www.chromatic.com/docs/branching-and-baselines?utm_source=storybook_website&utm_medium=link&utm_campaign=storybook). +Click the ▶️ Play button in the Storybook sidebar to run visual tests. This will send your stories to the cloud to take snapshots and detect visual changes. - +![Storybook running visual tests with the addon](./vta-run-tests.png) + +### Review changes + +If there are visual changes in your stories, they will be 🟡 highlighted in the sidebar. Click the story and go to the Visual Tests addon panel to see which pixels changed. + +If the changes are intentional, ✅ accept them as baselines locally. If the changes aren't intentional, fix the story and rerun the tests using the ▶️ Play button. -When Chromatic finishes, it should have successfully deployed your Storybook and established the baselines, that is to say, the starting point for all your component's stories. Additionally, providing you with a link to the published Storybook that you can share with your team to gather feedback. +![Confirm UI changes in Storybook](./vta-changes-found.png) -![Chromatic project first build](./chromatic-first-build-optimized.png) +When you finish accepting changes as baselines in the addon, you're ready to push the code to your remote repository. This will sync baselines to the cloud for anyone who checks out your branch. -## Catching UI changes +![Accept UI changes in Storybook](./vta-changes-accepted.png) -Each time you run Chromatic, it will generate new snapshots and compare them against the existing baselines. That’s ideal for detecting UI changes and preventing potential UI regressions. +### Automate with CI -For example, let's assume you're working on a component and you tweak the styling. When Chromatic is re-run, it will highlight the difference between the baseline and the updated component. +The addon is designed to be used in tandem with CI. We recommend using the addon to check for changes during development and then running visual tests in CI as you get ready to merge. -![Chromatic project second build](./chromatic-second-build-optimized.png) +Changes you accept as baselines in the addon will get auto-accepted as baselines in CI so you don’t have to review twice. -If the changes are intentional, accept them as baselines. Otherwise, deny them to prevent UI regressions. +1. Add a step to your CI workflow to run Chromatic. -Learn how to [integrate Chromatic UI Tests](https://www.chromatic.com/docs/?utm_source=storybook_website&utm_medium=link&utm_campaign=storybook) into your CI pipeline. + - [GitHub Actions](https://chromatic.com/docs/github-actions?ref=storybook_docs) + - [GitLab Pipelines](https://chromatic.com/docs/gitlab?ref=storybook_docs) + - [Bitbucket Pipelines](https://chromatic.com/docs/bitbucket-pipelines?ref=storybook_docs) + - [CircleCI](https://chromatic.com/docs/circleci?ref=storybook_docs) + - [Travis CI](https://chromatic.com/docs/travisci?ref=storybook_docs) + - [Jenkins](https://chromatic.com/docs/jenkins?ref=storybook_docs) + - [Azure Pipelines](https://chromatic.com/docs/azure-pipelines?ref=storybook_docs) + - [Custom CI provider](https://chromatic.com/docs/custom-ci-provider?ref=storybook_docs) + +2. Configure your CI to include environment variables to authenticate with Chromatic (project token). + +#### PR checks + +Once you successfully set up Chromatic in CI, your pull/merge requests will be badged with a UI Tests check. The badge notifies you of test errors or UI changes that need to be verified by your team. Make the check required in your Git provider to prevent accidental UI bugs from being merged. + +![PR badge for visual tests](./vta-prbadge-test.png) --- -#### What’s the difference between visual tests and snapshot tests? +### What’s the difference between visual tests and snapshot tests? + +[Snapshot tests](./snapshot-testing.md) compare the rendered markup of every story against known baselines. This means the test compares blobs of HTML and not what the user actually sees. Which in turn, can lead to an increase in false positives as code changes don’t always yield visual changes in the component. -Snapshot tests compare the rendered markup of every story against known baselines. This means the test compares blobs of HTML and not what the user actually sees. Which in turn, can lead to an increase in false positives as code changes don’t always yield visual changes in the component. +Visual tests compare the rendered pixels of every story against known baselines. Because you're testing the same thing your users actually experience, your tests will be richer and easier to maintain. -#### Learn about other UI tests +**Learn about other UI tests** - [Test runner](./test-runner.md) to automate test execution - Visual tests for appearance diff --git a/docs/writing-tests/vta-changes-accepted.png b/docs/writing-tests/vta-changes-accepted.png new file mode 100644 index 000000000000..543ed220c5a8 Binary files /dev/null and b/docs/writing-tests/vta-changes-accepted.png differ diff --git a/docs/writing-tests/vta-changes-found.png b/docs/writing-tests/vta-changes-found.png new file mode 100644 index 000000000000..9f9785b59baf Binary files /dev/null and b/docs/writing-tests/vta-changes-found.png differ diff --git a/docs/writing-tests/vta-enable.png b/docs/writing-tests/vta-enable.png new file mode 100644 index 000000000000..cbac2eea2b80 Binary files /dev/null and b/docs/writing-tests/vta-enable.png differ diff --git a/docs/writing-tests/vta-prbadge-test.png b/docs/writing-tests/vta-prbadge-test.png new file mode 100644 index 000000000000..ed87813a2f96 Binary files /dev/null and b/docs/writing-tests/vta-prbadge-test.png differ diff --git a/docs/writing-tests/vta-run-tests.png b/docs/writing-tests/vta-run-tests.png new file mode 100644 index 000000000000..f44f98dd6b29 Binary files /dev/null and b/docs/writing-tests/vta-run-tests.png differ diff --git a/docs/writing-tests/vta-select-project.png b/docs/writing-tests/vta-select-project.png new file mode 100644 index 000000000000..182a50baea9c Binary files /dev/null and b/docs/writing-tests/vta-select-project.png differ