Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate CI e2e test flakiness #9087

Closed
mnholtz opened this issue Sep 3, 2024 · 3 comments
Closed

Investigate CI e2e test flakiness #9087

mnholtz opened this issue Sep 3, 2024 · 3 comments
Assignees
@mnholtz
Labels
testing

Comments

@mnholtz
Copy link
Collaborator

mnholtz commented Sep 3, 2024 •
edited
Loading

We recently started seeing an uptick in text flakiness in CI, e.g. https://pixiebrix.slack.com/archives/C06RG13FBRS/p1725369771614789

Some of the flakiness has to due with failing Sidebar pings. Possibly related to #9011
@mnholtz mnholtz added the testing label Sep 3, 2024
@mnholtz mnholtz self-assigned this Sep 3, 2024
@mnholtz mnholtz mentioned this issue Sep 4, 2024
Merged
mnholtz added a commit that referenced this issue Sep 4, 2024
@mnholtz
#9087 Wait for built-in integration before activating mod in e2e test (…
ad8d224 
…#9088)

* address test flakiness in copy mod test

* refactor extract logic into helper method on activatemodpage pom
@mnholtz
Copy link
Collaborator Author

mnholtz commented Sep 5, 2024

Observed issues (that I'm having trouble consistently replicating locally):

  1. Sidebar is not opening with the message "The Sidebar is taking longer than expected to load. Retrying..."
    • Affected tests: tests/pageEditor/addStarterBrick.spec.ts:138 › Add starter brick to mod, tests/pageEditor/liveEditing.spec.ts:24 › live editing behavior
  2. Missing/slow-appearing FAB
    • Affected tests: Sometimes on activation.spec.ts tests, but I haven't found any recent failures with non-expired artifacts today for these

Other observations:

  • Running tests via command line (vs. ui mode) seems to produce failures more reliably, but I am still struggling to get enough consistency to have confidence in a fix

@mnholtz
Copy link
Collaborator Author

mnholtz commented Sep 5, 2024 •
edited
Loading

I took inventory on failing tests in the Pre-release Browser workflow since last week (all of which are failing on msedge/msedge-beta:

  • tests/regressions/sidebarLinks.spec.ts:76:5 › #8206: clicking links from the sidebar doesn't crash browser › Clicking extension console link
  • tests/pageEditor/liveEditing.spec.ts:24:5 › live editing behavior › Verify render panel functionality
  • tests/pageEditor/addStarterBrick.spec.ts:138:5 › Add starter brick to mod › Add Trigger starter brick to mod
  • tests/regressions/welcomeStarterBricks.spec.ts:24:5 › #8740: can view the starter mods on the [pixiebrix.com/welcome](http://pixiebrix.com/welcome) page
  • tests/pageEditor/copyMod.spec.ts:78:5 › run a copied mod with a built-in integration › Run the copied mod to verify that the copy activated correctly

I'm going to double-check that these failures don't seem especially concerning, and skip these tests for Edge.

This was referenced Sep 9, 2024
Closed
Merged
Open
@mnholtz
Copy link
Collaborator Author

mnholtz commented Sep 9, 2024

Closing in favor of #9125 and #9127

@mnholtz mnholtz closed this as completed Sep 9, 2024
mnholtz added a commit that referenced this issue Sep 9, 2024
@mnholtz
#9087 Skip tests for pre-release e2e workflow on msedge/msedge-beta (#…
edb2563 
…9126)

* skip sidebarLink test

* create utils for checking chromium channel

* skip remaining tets

* introduce new constant for pre-release browser workflow name

* replace test.skip with test.fixme

* remove unused chromium util

* replace logic with isMsEdge where appropriate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mnholtz mnholtz

Labels
testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mnholtz