Skip to content

Commit

Permalink
ci: Streamline CI browser playwright tests (#13276)
Browse files Browse the repository at this point in the history
This PR streamlines our browser integration tests a bit more:

1. We now only install chromium for playwright tests in most places.
This may help cut down test run time a bit, as we do not need to install
native dependencies for webkit and firefox everywhere.
2. I changed how we split the matrix jobs for the browser integration
tests. Previously, we sharded this via playwright. Now, we only do this
for esm tests, which we now only run in chromium, not all browsers. Only
the full bundle tests are run in all browsers (they are considerably
faster than the esm tests...), and there we now do not use sharding, but
have one matrix job for chromium (default), one for webkit and one for
firefox (where each can only install the native dependencies it needs).
3. I renamed the jobs a bit so that we can see a bit more of the job
name in the Github UI... it still truncates but you see more than
before, at least...
4. For all other playwright tests (e.g. e2e tests etc) we only install
chromium, AFAIK we do not have any non-chromium tests anywhere there.

Some data points:
* E2E tests seem to be about 10-20% faster than before, by looking
through traces here:
https://sentry.sentry.io/performance/trace/c5c1c643f1db0e8dd97df658405c2035/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&node=txn-8892bc908791478aba163dd3edd373b4&project=5899451&query=&sort=-timestamp&source=discover&statsPeriod=1h&timestamp=1723118364&yAxis=count%28%29
* For the playwright steps it's a bit harder to say as they are renamed,
also they are not super consistent in execution time, we'll have to
monitor it a bit afterwards, but subjectively it appears a bit shorter
in most cases too.
  • Loading branch information
mydea authored Aug 8, 2024
1 parent adf7b40 commit 21830b1
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 18 deletions.
7 changes: 6 additions & 1 deletion .github/actions/install-playwright/action.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
name: "Install Playwright dependencies"
description: "Installs Playwright dependencies and caches them."
inputs:
browsers:
description: 'What browsers to install.'
default: 'chromium webkit firefox'

runs:
using: "composite"
Expand All @@ -17,12 +21,13 @@ runs:
~/.cache/ms-playwright
key: playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}

# We always install all browsers, if uncached
- name: Install Playwright dependencies (uncached)
run: npx playwright install chromium webkit firefox --with-deps
if: steps.playwright-cache.outputs.cache-hit != 'true'
shell: bash

- name: Install Playwright system dependencies only (cached)
run: npx playwright install-deps chromium webkit firefox
run: npx playwright install-deps ${{ inputs.browsers || 'chromium webkit firefox' }}
if: steps.playwright-cache.outputs.cache-hit == 'true'
shell: bash
48 changes: 31 additions & 17 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -528,7 +528,7 @@ jobs:
run: yarn lerna run test --scope @sentry/profiling-node

job_browser_playwright_tests:
name: Playwright (${{ matrix.bundle }}${{ matrix.shard && format(' {0}/{1}', matrix.shard, matrix.shards) || ''}}) Tests
name: Playwright ${{ matrix.bundle }}${{ matrix.project && matrix.project != 'chromium' && format(' {0}', matrix.project) || ''}}${{ matrix.shard && format(' ({0}/{1})', matrix.shard, matrix.shards) || ''}} Tests
needs: [job_get_metadata, job_build]
if: needs.job_build.outputs.changed_browser_integration == 'true' || github.event_name != 'pull_request'
runs-on: ubuntu-20.04-large-js
Expand All @@ -548,31 +548,30 @@ jobs:
project:
- chromium
include:
# Only check all projects for esm & full bundle
# Only check all projects for full bundle
# We also shard the tests as they take the longest
- bundle: bundle_tracing_replay_feedback_min
project: ''
shard: 1
shards: 2
project: 'webkit'
- bundle: bundle_tracing_replay_feedback_min
project: ''
shard: 2
shards: 2
project: 'firefox'
- bundle: esm
project: ''
project: chromium
shard: 1
shards: 3
shards: 4
- bundle: esm
project: chromium
shard: 2
shards: 3
shards: 4
- bundle: esm
project: ''
project: chromium
shard: 3
shards: 3
shards: 4
- bundle: esm
project: chromium
shard: 4
shards: 4
exclude:
# Do not run the default chromium-only tests
- bundle: bundle_tracing_replay_feedback_min
project: 'chromium'
# Do not run the un-sharded esm tests
- bundle: esm
project: 'chromium'

Expand All @@ -592,12 +591,15 @@ jobs:

- name: Install Playwright
uses: ./.github/actions/install-playwright
with:
browsers: ${{ matrix.project }}

- name: Run Playwright tests
env:
PW_BUNDLE: ${{ matrix.bundle }}
working-directory: dev-packages/browser-integration-tests
run: yarn test:ci${{ matrix.project && format(' --project={0}', matrix.project) || '' }}${{ matrix.shard && format(' --shard={0}/{1}', matrix.shard, matrix.shards) || '' }}

- name: Upload Playwright Traces
uses: actions/upload-artifact@v3
if: always()
Expand All @@ -606,7 +608,7 @@ jobs:
path: dev-packages/browser-integration-tests/test-results

job_browser_loader_tests:
name: Playwright Loader (${{ matrix.bundle }}) Tests
name: PW ${{ matrix.bundle }} Tests
needs: [job_get_metadata, job_build]
if: needs.job_build.outputs.changed_browser_integration == 'true' || github.event_name != 'pull_request'
runs-on: ubuntu-20.04
Expand Down Expand Up @@ -639,6 +641,8 @@ jobs:

- name: Install Playwright
uses: ./.github/actions/install-playwright
with:
browsers: chromium

- name: Run Playwright Loader tests
env:
Expand Down Expand Up @@ -750,8 +754,12 @@ jobs:
uses: ./.github/actions/restore-cache
env:
DEPENDENCY_CACHE_KEY: ${{ needs.job_build.outputs.dependency_cache_key }}

- name: Install Playwright
uses: ./.github/actions/install-playwright
with:
browsers: chromium

- name: Run integration tests
env:
NODE_VERSION: ${{ matrix.node }}
Expand Down Expand Up @@ -953,6 +961,8 @@ jobs:

- name: Install Playwright
uses: ./.github/actions/install-playwright
with:
browsers: chromium

- name: Get node version
id: versions
Expand Down Expand Up @@ -1050,6 +1060,8 @@ jobs:

- name: Install Playwright
uses: ./.github/actions/install-playwright
with:
browsers: chromium

- name: Get node version
id: versions
Expand Down Expand Up @@ -1150,6 +1162,8 @@ jobs:

- name: Install Playwright
uses: ./.github/actions/install-playwright
with:
browsers: chromium

- name: Get node version
id: versions
Expand Down

0 comments on commit 21830b1

Please sign in to comment.