Introduce Functional Tests using Playwright and Behave #31

* Initial locally working example tests

Add dependencies

* Run in CI Github Action

Add missing dependency

Ignore pylint false negative

Playwright install in action and try to understand if we have pg_dump available

Try to understand if we have pg_dump available

Set paths to PG client tools, remove debugging outputs

Update the action runner postgres client

apt get update first

Try the postgres package instead

Try brew installation

Use path to brew

Add brew link

Try pointing the pg_dump/restore to the brew install location

* Matrix the functional tests on browser

* Record and save traces on test failures

Remove step name that may be bad syntax

Update step 'if' syntax

Fix traces path and ensure traces are enabled

Try shortening step name

Fail a step intentionally to get a trace

Add step name

Try removing the new step

Re-add step

* Refactor feature steps into a better example structure, add info page feature

Remove commented code

Neaten up code and add comments

* Add make targets

Add make file comments

* Remove browsers with issues from the actions run

* Use docker compose instead of manual configured postgres

* Use DSLR instead of postgres client

* Add basic README documentation on running the test

* Add scenario for release page

Stop the markdown linter complaining

Fail a test on purpose

Fix intentional failure

* Add more docs

Update README

* Make compatible with multi-db configuration

Format

* Small structural refactor, fix factory imports, update Docs

* Set the settings module variable in environment.py

* Format

* Move environment variable setting to the module level

* Add missing README links

* Fix README wording

* Fix PR issues, make the test development app a separate compose file

* First round or PR comments addressed, WIP on fixing issues

* Improve code example

* Address more PR comments, tests fix still WIP

* Fix markdown that prettier broke

* Fix code example

* Format

* Remove checks for missing titles

* Use behave verbose mode to see error logs

* Add a sleep to check if there's a timing issue

* Check if the issue is related to removing the dev app

* Try preparing the test database with migrations

* Fix headings, fix test env initialisation issues, utilise compose extends to reduce duplication

* Use existing factory classes, remove duplication

* README updates and fixes

* Update functional_tests/steps/home_page.py

Make pylint comments more readable

Co-authored-by: Dan Braghiș <[email protected]>

* Update functional_tests/steps/release_page.py

Make pylint comments more readble

Co-authored-by: Dan Braghiș <[email protected]>

* Use readable pylint names instead of codes

* Remove verbose flag from CI

* Intentionally fail a scenario for demonstration purposes

* Revert intentional fail

* Fix incorrect ports in dev app compose

* Update functional_tests/README.md

Co-authored-by: Mebin Abraham <[email protected]>

* Update functional_tests/README.md

Co-authored-by: Mebin Abraham <[email protected]>

* Address review comments

* Streamline CI steps with make command

* Address review comments, update make targets and comments

* More consistent use of settings flag

* Fix broken auto-format

* Attempt to fix formatting

* Format

* Resolve merge conflicts

* Fix table of contents formatting

* Fix test link location

---------

Co-authored-by: Dan Braghiș <[email protected]>
Co-authored-by: Mebin Abraham <[email protected]>
Co-authored-by: Dan Braghiș <[email protected]>

.github/workflows/ci.yml

@@ -147,3 +147,53 @@ jobs:
       - uses: actions/checkout@v4
       - name: Build Docker Image
         run: docker build --target web -t ons .
+
+  functional-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        browser:
+          - chromium
+    needs:
+      - lint
+      - lint-front-end
+      - compile_static
+
+    env:
+      DJANGO_SETTINGS_MODULE: cms.settings.functional_test
+      SECRET_KEY: fake_secret_key_to_run_tests # pragma: allowlist secret
+      PLAYWRIGHT_BROWSER: ${{ matrix.browser }}
+      PLAYWRIGHT_TRACES_DIR: playwright_traces
+      PLAYWRIGHT_TRACE: 'true'
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Poetry
+        run: pipx install poetry==${{ env.POETRY_VERSION }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: .python-version
+          cache: poetry
+
+      - name: Install dependencies
+        run: make install-dev
+
+      - name: Playwright Install
+        run: poetry run python -m playwright install --with-deps ${{ matrix.browser }}
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: static
+          path: cms/static_compiled/
+
+      - name: Run Functional Tests
+        run: make functional-tests
+
+      - name: Upload Failure Traces
+        uses: actions/upload-artifact@v4
+        if: failure()
+        with:
+          name: playwright-traces-${{ matrix.browser }}
+          path: playwright_traces/

.gitignore

@@ -196,3 +196,6 @@ static/
 
 # bak files
 *.bak
+
+# Playwright Traces
+/tmp_traces

Makefile

@@ -152,3 +152,28 @@ runserver: ## Run the Django application locally
 
 .PHONY: dev-init
 dev-init: load-design-system-templates collectstatic makemigrations migrate createsuperuser ## Run the pre-run setup scripts
+
+.PHONY: functional-tests-up
+functional-tests-up:  ## Start the functional tests docker compose dependencies
+	docker compose -f functional_tests/docker-compose.yml up -d
+
+.PHONY: functional-tests-dev-up
+functional-tests-dev-up:  ## Start the functional tests docker compose dependencies and dev app
+	docker compose -f functional_tests/docker-compose-dev.yml up -d
+
+.PHONY: functional-tests-down
+functional-tests-down:  ## Stop the functional tests docker compose dependencies (and dev app if running)
+	docker compose -f functional_tests/docker-compose-dev.yml down
+
+.PHONY: functional-tests-run
+functional-tests-run: load-design-system-templates collectstatic ## Only run the functional tests (dependencies must be run separately)
+	# Run migrations to work around Django bug (#35967)
+	poetry run ./manage.py migrate --noinput --settings cms.settings.functional_test
+	poetry run behave functional_tests
+
+.PHONY: functional-tests
+functional-tests: functional-tests-up functional-tests-run functional-tests-down  ## Run the functional tests with dependencies (all in one)
+
+.PHONY: playwright-install
+playwright-install:  ## Install Playwright dependencies
+	poetry run playwright install --with-deps

README.md

@@ -28,6 +28,7 @@ The Wagtail CMS for managing and publishing content for the Office for National
     - [Front-end tooling](#front-end-tooling)
     - [Adding Python packages](#adding-python-packages)
     - [Run Tests with Coverage](#run-tests-with-coverage)
+    - [Functional Tests](#functional-tests)
     - [Linting and Formatting](#linting-and-formatting)
         - [Python](#python)
         - [Front-end](#front-end)
@@ -245,6 +246,67 @@ make test
 During tests, the `cms.settings.test` settings module is used. When running test without using `make test`, ensure this
 settings module is used.
 
+### Functional Tests
+
+Our suite of functional browser driven tests uses [Behave](https://behave.readthedocs.io/en/latest/),
+[Playwright](https://playwright.dev/python/docs/intro) and
+[Django Live Server Test Cases](https://docs.djangoproject.com/en/stable/topics/testing/tools/#liveservertestcase) to
+run BDD Cucumber feature tests against the app from a browser.
+
+#### Installation
+
+Install the Playwright dependencies (including its browser drivers) with:
+
+```shell
+make playwright-install
+```
+
+#### Run the Functional Tests
+
+You can run the tests as an all-in-one command with:
+
+```shell
+make functional-tests
+```
+
+This will start and stop the docker compose services with the relevant tests.
+
+To run the docker compose dependencies (database and redis) separately, e.g. if you want to run individual functional
+tests yourself for development, start the docker compose dependencies with:
+
+```shell
+make functional-tests-up
+```
+
+This will start the dependent services in the background, allowing you to then run the tests separately.
+
+Then once you are finished testing, stop the dependencies with:
+
+```shell
+make functional-tests-down
+```
+
+#### Showing the Tests Browser
+
+By default, the tests will run in headless mode with no visible browser window.
+
+To disable headless mode and show the browser, set `PLAYWRIGHT_HEADLESS=False` in the environment from which you are
+running the tests. In this circumstance, you will probably also find it helpful to enable "slow mo" mode, which slows
+down the automated browser interactions to make it possible to follow what the tests are doing. You can configure it
+using the `PLAYWRIGHT_SLOW_MO` environment variable, passing it a value of milliseconds by which to slow each
+interaction, e.g. `PLAYWRIGHT_SLOW_MO=1000` will cause each individual browser interaction from the tests to be delayed
+by 1 second.
+
+For example, you can run the tests with visible browser and each interaction slowed by a second by running:
+
+```shell
+PLAYWRIGHT_HEADLESS=False PLAYWRIGHT_SLOW_MO=1000 make functional-tests
+```
+
+#### Developing Functional Tests
+
+Refer to the detailed [functional tests development docs](./functional_tests/README.md)
+
 ### Linting and Formatting
 
 Various tools are used to lint and format the code in this project.

cms/settings/functional_test.py

@@ -0,0 +1,10 @@
+import copy
+
+from .test import *  # noqa: F403  # pylint: disable=wildcard-import,unused-wildcard-import
+
+DATABASES = {
+    "default": dj_database_url.config(default="postgres://ons:ons@localhost:15432/ons"),  # noqa: F405
+}
+DATABASES["read_replica"] = copy.deepcopy(DATABASES["default"])
+
+REDIS_URL = "redis://localhost:16379"

cms/users/tests/factories.py

@@ -13,7 +13,7 @@ class Meta:
 
     @factory.post_generation
     def access_admin(self, create, extracted, **kwargs):
-        """Creates BundlePage instances for the bundle.
+        """Allows the group access to the Wagtail Admin interface.
 
         Usage:
             # Create a Django generic_user group
@@ -42,13 +42,13 @@ class Meta:
 
     @factory.post_generation
     def access_admin(self, create, extracted, **kwargs):
-        """Creates BundlePage instances for the bundle.
+        """Allows the user access to the Wagtail Admin interface.
 
         Usage:
-            # Create a Django generic_user group
+            # Create a Django generic_user user
             generic_user = UserFactory()
 
-            # Create a Django generic_user group with Wagtail admin access
+            # Create a Django generic_user user with Wagtail admin access
             generic_user = UserFactory(access_admin=True)
         """
         if not create: