build: Add codecov test analytics for playwright tests #13654

AbhiPrasad · 2024-09-10T15:05:52Z

https://docs.codecov.com/docs/test-result-ingestion-beta

https://playwright.dev/docs/test-reporters

Adds codecov test analytics to the repo, specifically for our playwright tests.

This works by using the junit reporter with playwright, and then uploading that via the codecov/test-results-action@v1 GitHub Action.

mydea · 2024-09-10T15:12:03Z

.github/workflows/build.yml

@@ -610,6 +610,13 @@ jobs:
          overwrite: true
          retention-days: 7

+      - name: Upload test results to Codecov
+        if: ${{ !cancelled() }}


Suggested change

if: ${{ !cancelled() }}

if: cancelled() == false

I think that's enough here?

I just copied the snippet from https://docs.codecov.com/docs/test-result-ingestion-beta#2-add-the-script-codecovtest-results-actionv1-to-your-ci-yaml-file

but I think you're right, this feels more correct to me. Will switch!

AbhiPrasad · 2024-09-10T15:13:43Z

Looks like it works! https://app.codecov.io/gh/getsentry/sentry-javascript/tests/abhi-playwright-codecov-test-analytics (make sure you're logged in)

codecov

The code contains Git diff changes which primarily aim at integrating codecov test results actions in GitHub Workflows, and modifying test reporters for CI. The changes are well implemented, but are redundant and hard-coded, which might raise maintenance issues in the future. Moreover, the usage of tokens can be risky and should be carefully verified.

codecov · 2024-09-10T15:37:02Z

.github/workflows/build.yml

+        uses: codecov/test-results-action@v1
+        with:
+          directory: dev-packages/browser-integration-tests
+          token: ${{ secrets.CODECOV_TOKEN }}


This code creates a potential problem where if the Codecov token is not set, the action may fail without a clear error. Consider adding error handling to handle the situation when secrets.CODECOV_TOKEN is not available.

codecov · 2024-09-10T15:37:02Z

.github/workflows/build.yml

+      - name: Upload test results to Codecov
+        if: cancelled() == 'false'
+        uses: codecov/test-results-action@v1
+        with:


Repeating the same actions in each workflow can be both inefficient and error-prone. Consider creating a reusable workflow to maintain this action in a single place.

codecov · 2024-09-10T15:37:02Z

dev-packages/e2e-tests/test-applications/ember-classic/playwright.config.ts

@@ -35,7 +35,7 @@ const config: PlaywrightTestConfig = {
  forbidOnly: !!process.env.CI,
  retries: 0,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
-  reporter: 'list',
+  reporter: process.env.CI ? [['line'], ['junit', { outputFile: 'results.junit.xml' }]] : 'list',


The same note for the CI environment variable applies here and for all other similar changes.

codecov · 2024-09-10T15:37:03Z

dev-packages/e2e-tests/test-applications/ember-embroider/playwright.config.ts

@@ -35,7 +35,7 @@ const config: PlaywrightTestConfig = {
  forbidOnly: !!process.env.CI,
  retries: 0,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
-  reporter: 'list',
+  reporter: process.env.CI ? [['line'], ['junit', { outputFile: 'results.junit.xml' }]] : 'list',


The same reporter switching logic is repeated. You should factor out this piece of code into a reusable function or a separate configuration file.

codecov · 2024-09-10T15:37:03Z

dev-packages/e2e-tests/test-applications/react-send-to-sentry/playwright.config.mjs

@@ -23,7 +23,7 @@ const config = {
  /* Opt out of parallel tests on CI. */
  workers: 1,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
-  reporter: 'list',
+  reporter: process.env.CI ? [['line'], ['junit', { outputFile: 'results.junit.xml' }]] : 'list',


Similar repetition of the reporter assignment based on environment variable 'CI'. This should be refactored into a utility function.

codecov · 2024-09-10T15:37:03Z

dev-packages/test-utils/src/playwright-config.ts

@@ -37,7 +37,7 @@ export function getPlaywrightConfig(
    /* In dev mode some apps are flaky, so we allow retry there... */
    retries: testEnv === 'development' ? 3 : 0,
    /* Reporter to use. See https://playwright.dev/docs/test-reporters */
-    reporter: process.env.CI ? 'line' : 'list',
+    reporter: process.env.CI ? [['line'], ['junit', { outputFile: 'results.junit.xml' }]] : 'list',


Again, similar repetition of the reporter assignment based on environment variable 'CI' as in line 129. Please consider creating a dedicated function or configurating module for this logic.

codecov · 2024-09-10T15:37:03Z

packages/remix/package.json

@@ -106,7 +106,7 @@
    "test:integration:prepare": "(cd test/integration && yarn install)",
    "test:integration:clean": "(cd test/integration && rimraf .cache node_modules build)",
    "test:integration:client": "yarn playwright install-deps && yarn playwright test test/integration/test/client/ --project='chromium'",
-    "test:integration:client:ci": "yarn test:integration:client --reporter='line'",
+    "test:integration:client:ci": "yarn test:integration:client",


The removal of the reporter from the test script has been noted. This is now handled by the test configuration file.

codecov · 2024-09-10T15:37:03Z

packages/remix/playwright.config.ts

@@ -8,6 +8,7 @@ const config: PlaywrightTestConfig = {
  },
  // Run tests inside of a single file in parallel
  fullyParallel: true,
+  reporter: process.env.CI ? [['line'], ['junit', { outputFile: 'results.junit.xml' }]] : 'list',


The use of 'junit' reporter for CI environment is a good addition. It will allow tests to generate JUnit style XML files, which can be used by various CI tools for detailed test reports.

github-actions · 2024-09-10T15:37:07Z

size-limit report 📦

Path	Size	% Change	Change
@sentry/browser	22.52 KB	-	-
@sentry/browser - with treeshaking flags	21.3 KB	-	-
@sentry/browser (incl. Tracing)	34.79 KB	-	-
@sentry/browser (incl. Tracing, Replay)	71.25 KB	-	-
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags	61.69 KB	-	-
@sentry/browser (incl. Tracing, Replay with Canvas)	75.59 KB	-	-
@sentry/browser (incl. Tracing, Replay, Feedback)	88.32 KB	-	-
@sentry/browser (incl. Tracing, Replay, Feedback, metrics)	90.16 KB	-	-
@sentry/browser (incl. metrics)	26.83 KB	-	-
@sentry/browser (incl. Feedback)	39.6 KB	-	-
@sentry/browser (incl. sendFeedback)	27.19 KB	-	-
@sentry/browser (incl. FeedbackAsync)	31.9 KB	-	-
@sentry/react	25.28 KB	-	-
@sentry/react (incl. Tracing)	37.75 KB	-	-
@sentry/vue	26.72 KB	-	-
@sentry/vue (incl. Tracing)	36.66 KB	-	-
@sentry/svelte	22.66 KB	-	-
CDN Bundle	23.77 KB	-	-
CDN Bundle (incl. Tracing)	36.49 KB	-	-
CDN Bundle (incl. Tracing, Replay)	70.95 KB	-	-
CDN Bundle (incl. Tracing, Replay, Feedback)	76.26 KB	-	-
CDN Bundle - uncompressed	69.65 KB	-	-
CDN Bundle (incl. Tracing) - uncompressed	108.23 KB	-	-
CDN Bundle (incl. Tracing, Replay) - uncompressed	220.01 KB	-	-
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed	233.21 KB	-	-
@sentry/nextjs (client)	37.51 KB	-	-
@sentry/sveltekit (client)	35.36 KB	-	-
@sentry/node	120.5 KB	-0.01%	-1 B 🔽
@sentry/node - without tracing	92.78 KB	-	-
@sentry/aws-serverless	102.49 KB	-	-

View base workflow run

codecov

CodecovAI submitted a new review for b4228b8

codecov · 2024-09-13T08:35:29Z

dev-packages/browser-integration-tests/package.json

@@ -35,7 +35,7 @@
    "test:loader:replay_buffer": "PW_BUNDLE=loader_replay_buffer yarn test:loader",
    "test:loader:full": "PW_BUNDLE=loader_tracing_replay yarn test:loader",
    "test:loader:debug": "PW_BUNDLE=loader_debug yarn test:loader",
-    "test:ci": "yarn test:all --reporter='line'",
+    "test:ci": "yarn test:all",


The change from specifying the reporter in the test script to be handled by the test configuration file has been noted. Please ensure the 'line' reporter is the intended default for local testing.

codecov · 2024-09-13T08:35:29Z

dev-packages/browser-integration-tests/playwright.config.ts

@@ -30,6 +30,8 @@ const config: PlaywrightTestConfig = {
    },
  ],

+  reporter: process.env.CI ? [['line'], ['junit', { outputFile: 'results.junit.xml' }]] : 'list',


If the CI environment variable is not set, this will default to 'list' reporter. Is this the intended behavior? If not, consider adding a default reporter.

codecov · 2024-09-13T08:46:53Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
396	1	395	17

View the top 1 failed tests by shortest run time

tracing/metrics/handlers-lcp/test.ts should capture metrics for LCP instrumentation handlers

Stack Traces | 10.6s run time

test.ts:10:11 should capture metrics for LCP instrumentation handlers

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

build: Add codecov test analytics for playwright tests

b98bc08

AbhiPrasad requested review from mydea, a team and Lms24 and removed request for a team September 10, 2024 15:05

mydea reviewed Sep 10, 2024

View reviewed changes

mydea approved these changes Sep 10, 2024

View reviewed changes

AbhiPrasad added 2 commits September 10, 2024 17:18

Make sure reporter is not being overriden

2a010d1

use easier to read condition

c7b39f2

AbhiPrasad added the ci-codecov-ai-review AI Review by Codecov label Sep 10, 2024

codecov bot reviewed Sep 10, 2024

View reviewed changes

fix condition

b4228b8

codecov bot reviewed Sep 13, 2024

View reviewed changes

AbhiPrasad merged commit f44785c into develop Sep 13, 2024
130 checks passed

AbhiPrasad deleted the abhi-playwright-codecov-test-analytics branch September 13, 2024 09:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: Add codecov test analytics for playwright tests #13654

build: Add codecov test analytics for playwright tests #13654

AbhiPrasad commented Sep 10, 2024

mydea Sep 10, 2024

AbhiPrasad Sep 10, 2024

AbhiPrasad commented Sep 10, 2024

codecov bot left a comment

codecov bot Sep 10, 2024 •

edited

Loading

codecov bot Sep 10, 2024

codecov bot Sep 10, 2024 •

edited

Loading

codecov bot Sep 10, 2024

codecov bot Sep 10, 2024

codecov bot Sep 10, 2024

codecov bot Sep 10, 2024 •

edited

Loading

codecov bot Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

codecov bot left a comment

codecov bot Sep 13, 2024

codecov bot Sep 13, 2024

codecov bot commented Sep 13, 2024

build: Add codecov test analytics for playwright tests #13654

build: Add codecov test analytics for playwright tests #13654

Conversation

AbhiPrasad commented Sep 10, 2024

mydea Sep 10, 2024

Choose a reason for hiding this comment

AbhiPrasad Sep 10, 2024

Choose a reason for hiding this comment

AbhiPrasad commented Sep 10, 2024

codecov bot left a comment

Choose a reason for hiding this comment

codecov bot Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot Sep 10, 2024

Choose a reason for hiding this comment

codecov bot Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot Sep 10, 2024

Choose a reason for hiding this comment

codecov bot Sep 10, 2024

Choose a reason for hiding this comment

codecov bot Sep 10, 2024

Choose a reason for hiding this comment

codecov bot Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 10, 2024 • edited Loading

size-limit report 📦

codecov bot left a comment

Choose a reason for hiding this comment

codecov bot Sep 13, 2024

Choose a reason for hiding this comment

codecov bot Sep 13, 2024

Choose a reason for hiding this comment

codecov bot commented Sep 13, 2024

❌ 1 Tests Failed:

codecov bot Sep 10, 2024 •

edited

Loading

codecov bot Sep 10, 2024 •

edited

Loading

codecov bot Sep 10, 2024 •

edited

Loading

codecov bot Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading