Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the preferences test more robust #5015

Merged
merged 4 commits into from
Oct 9, 2024
Merged

Make the preferences test more robust #5015

merged 4 commits into from
Oct 9, 2024

Conversation

obulat
Copy link
Contributor

@obulat obulat commented Oct 2, 2024

Fixes

Fixes #5001 by @obulat

Description

This PR updates the preferences test to make it less flaky:

  • to find the switch, we use the getByRole Playwright locator with the checked property set to true or undefined. We add an extra expect, checking that the switch is enabled. This function is extracted to expectCheckboxState, which is added as a valid expect to the Openverse ESLint plugin.
  • instead of testing the saved cookie value that is not visible by the users, we reload the page and check the switch state. If the state was saved in a cookie, then the state on page load should be updated.
  • an extra check with the snapshot of all of the feature flag store state is added.

Testing Instructions

The Playwright e2e test should pass in the CI. I will re-run this step several times to make sure that the test is not flaky.

Checklist

  • My pull request has a descriptive title (not a vague title likeUpdate index.md).
  • My pull request targets the default branch of the repository (main) or a parent feature branch.
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no visible errors.
  • I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
    PRs) or the media properties generator (ov just catalog/generate-docs media-props
    for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@openverse-bot openverse-bot added 🧱 stack: frontend Related to the Nuxt frontend 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work labels Oct 2, 2024
Copy link

github-actions bot commented Oct 2, 2024

Latest k6 run output1

$     ✗ status was 200
      ↳  99% — ✓ 7999 / ✗ 1

     checks.........................: 99.98% ✓ 7999      ✗ 1   
     data_received..................: 1.8 GB 9.0 MB/s
     data_sent......................: 1.0 MB 5.1 kB/s
     http_req_blocked...............: avg=14.34µs  min=1.84µs  med=3.36µs   max=6.96ms  p(90)=4.78µs   p(95)=5.15µs  
     http_req_connecting............: avg=10.54µs  min=0s      med=0s       max=6.91ms  p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=531.19ms min=50.58ms med=475.54ms max=2.2s    p(90)=999.84ms p(95)=1.1s    
       { expected_response:true }...: avg=531.09ms min=50.58ms med=475.53ms max=2.2s    p(90)=999.83ms p(95)=1.1s    
   ✓ http_req_failed................: 0.01%  ✓ 1         ✗ 7999
     http_req_receiving.............: avg=139.07µs min=37.99µs med=102.33µs max=13.93ms p(90)=168.28µs p(95)=215.89µs
     http_req_sending...............: avg=19.94µs  min=6.1µs   med=17.45µs  max=2.53ms  p(90)=23.4µs   p(95)=28.75µs 
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=531.03ms min=50.5ms  med=475.35ms max=2.2s    p(90)=999.72ms p(95)=1.1s    
     http_reqs......................: 8000   39.029563/s
     iteration_duration.............: avg=3.54s    min=1.1s    med=2.96s    max=10.48s  p(90)=7.84s    p(95)=8.3s    
     iterations.....................: 1200   5.854435/s
     vus............................: 3      min=3       max=30
     vus_max........................: 30     min=30      max=30

Footnotes

  1. This comment will automatically update with new output each time k6 runs for this PR

@obulat obulat added 🟧 priority: high Stalls work on the project or its dependents 💻 aspect: code Concerns the software code in the repository 🛠 goal: fix Bug fix and removed 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work labels Oct 7, 2024
@obulat obulat force-pushed the fix/#4995-attempt2 branch 6 times, most recently from f61685b to 51f7669 Compare October 7, 2024 14:45
@obulat obulat self-assigned this Oct 7, 2024
@obulat obulat force-pushed the fix/#4995-attempt2 branch 3 times, most recently from ae75ded to 5344bfe Compare October 7, 2024 18:29
@obulat obulat changed the title Use only user-facing features for preferences test Make the preferences test more robust Oct 7, 2024
@obulat obulat marked this pull request as ready for review October 7, 2024 18:57
@obulat obulat requested a review from a team as a code owner October 7, 2024 18:57
@obulat obulat force-pushed the fix/#4995-attempt2 branch 7 times, most recently from 44b56eb to 81dc5f2 Compare October 8, 2024 08:49
@obulat obulat force-pushed the fix/#4995-attempt2 branch 3 times, most recently from cf13d4e to 574dd42 Compare October 8, 2024 10:29
@obulat obulat force-pushed the fix/#4995-attempt2 branch 2 times, most recently from ce24d94 to b5351eb Compare October 8, 2024 15:02
@obulat obulat merged commit 9c90950 into main Oct 9, 2024
46 checks passed
@obulat obulat deleted the fix/#4995-attempt2 branch October 9, 2024 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository 🛠 goal: fix Bug fix 🟧 priority: high Stalls work on the project or its dependents 🧱 stack: frontend Related to the Nuxt frontend
Projects
Archived in project
3 participants