Move all tests over to electron (#4484)

* Move all tests over to electron * Pass the correct param to playwright-electron.sh * Add shebang to script and add macos-14-large as a target * Get sketch-tests.spec.ts passing in electron * Try out 4 workers * Got testing-segment-overlays passing * Pass testing-selections.spec.ts * Go back to fix up sketch-tests test * Pass various.spec.ts, by far the hardest one * Pass can-sketch-on-all-planes... with ease * Pass command bar tests * fmt * Completely fix code mirror text navigating for tests * Pass debug pane tests * Pass desktop export tests * Pass editor tests * Pass file tree tests * Pass onboarding tests * Corrected a fixme in file-tree.spec! * Painfully fix hardcoded coordinates in point-click.spec * Pass machine.spec tests * Pass projects, fought hard with filechooser * Pass regresion-tests.spec tests * Pass network and connection tests * Pass camera-movement.spec tests * Extreme time eaten by gizmo test fixes. All passing now. * Merge main (tests changed x_x) and pass all constraints.spec tests (pain) * Pass another painful spec suite: testing-settings * Pass perspective-toggle, interesting note * Pass samples loading tests * Pass app header tests * Pass text-to-cad tests * Pass segment-overlays (minor ache) and ability to switch to web if needed :) * Fix a ton of syntax changes and deflake 2 more tests (pain) * Correct all tsc errors * Remove to-electron script * Add an f-ton of shit because playwright doesnt want S P R E A D * Try CI again * Stop snapshots of exports (already test in e2e) * Fix flake in double click editor * Hopefully help CI flake * Fixmes, fixmes everywhere * One more fixme to settings * Skip another code pane flake * Port jess's projects.spec tests * fixup * Reuse electron window; difficult task * Rebased and refixed * Remove duplicate cases * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * Reduce the workers to something CI can handle * Lower it further, we need to think about the others * Update package.json Co-authored-by: Pierre Jacquier <[email protected]> * Update package.json Co-authored-by: Pierre Jacquier <[email protected]> * Fix the last tests and tsc errors * Timeout to 120 and windows-2022-16core * Fix windows runner detection, enable concurrency temporarily * Hopefully this time fix windows runner detection * Comment out Vector, add back removed camera test code * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * Fix camera tests again * Massively deflake a whole class of tests * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * Try new CI and fix small onboarding test * Derp * No github tuning * Try mac * Add back all the OS * Lord, hallow be thy name * A snapshot a day keeps the bugs away! 📷🐛 (OS: namespace-profile-ubuntu-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: windows-latest-8-cores) * One last try with window-16-cores * Trigger CI * Try AWS Windows runner * Passing on windows locally with a few skips * Skip more win tests, add back all three oses * A snapshot a day keeps the bugs away! 📷🐛 (OS: windows-16-cores) * Add two more fixmes * 2 more fixmes * skip segment overlays on win32 * Another fixme * Trigger CI * Trigger CI * Quick clean up * Move all tests over to electron * Pass the correct param to playwright-electron.sh * Add shebang to script and add macos-14-large as a target * Get sketch-tests.spec.ts passing in electron * Try out 4 workers * Got testing-segment-overlays passing * Pass testing-selections.spec.ts * Go back to fix up sketch-tests test * Pass various.spec.ts, by far the hardest one * Pass can-sketch-on-all-planes... with ease * Pass command bar tests * fmt * Completely fix code mirror text navigating for tests * Pass debug pane tests * Pass desktop export tests * Pass editor tests * Pass file tree tests * Pass onboarding tests * Corrected a fixme in file-tree.spec! * Painfully fix hardcoded coordinates in point-click.spec * Pass machine.spec tests * Pass projects, fought hard with filechooser * Pass regresion-tests.spec tests * Pass network and connection tests * Pass camera-movement.spec tests * Extreme time eaten by gizmo test fixes. All passing now. * Merge main (tests changed x_x) and pass all constraints.spec tests (pain) * Pass another painful spec suite: testing-settings * Pass perspective-toggle, interesting note * Pass samples loading tests * Pass app header tests * Pass text-to-cad tests * Pass segment-overlays (minor ache) and ability to switch to web if needed :) * Fix a ton of syntax changes and deflake 2 more tests (pain) * Correct all tsc errors * Remove to-electron script * Add an f-ton of shit because playwright doesnt want S P R E A D * Try CI again * Stop snapshots of exports (already test in e2e) * Fix flake in double click editor * Hopefully help CI flake * Fixmes, fixmes everywhere * One more fixme to settings * Skip another code pane flake * Port jess's projects.spec tests * fixup * Reuse electron window; difficult task * Rebased and refixed * Remove duplicate cases * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * Reduce the workers to something CI can handle * Lower it further, we need to think about the others * Update package.json Co-authored-by: Pierre Jacquier <[email protected]> * Update package.json Co-authored-by: Pierre Jacquier <[email protected]> * Fix the last tests and tsc errors * Timeout to 120 and windows-2022-16core * Fix windows runner detection, enable concurrency temporarily * Hopefully this time fix windows runner detection * Comment out Vector, add back removed camera test code * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * Fix camera tests again * Massively deflake a whole class of tests * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * Try new CI and fix small onboarding test * Derp * No github tuning * Try mac * Add back all the OS * Lord, hallow be thy name * A snapshot a day keeps the bugs away! 📷🐛 (OS: namespace-profile-ubuntu-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: windows-latest-8-cores) * Try AWS Windows runner * Passing on windows locally with a few skips * Trigger CI * A snapshot a day keeps the bugs away! 📷🐛 (OS: namespace-profile-ubuntu-8-cores) * fmt, tsc, lint * Enable two fixmes again * Fix lint, codespell, fmt * Fix lint * Don't run e2e on draft, add back concurrency, clean up * One last windows skip --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Pierre Jacquier <[email protected]> Co-authored-by: Pierre Jacquier <[email protected]>
KittyCAD · Dec 18, 2024 · 02055a8 · 02055a8
1 parent 9389142
commit 02055a8
Show file tree

Hide file tree

Showing 83 changed files with 5,339 additions and 5,630 deletions.
diff --git a/.github/ci-cd-scripts/playwright-browser-chrome.sh b/.github/ci-cd-scripts/playwright-browser-chrome.sh
diff --git a/.github/ci-cd-scripts/playwright-electron.sh b/.github/ci-cd-scripts/playwright-electron.sh
@@ -1,15 +1,17 @@
+#!/bin/bash
+
 # bash strict mode
 set -euo pipefail
 
 if [[ ! -f "test-results/.last-run.json" ]]; then
     # if no last run artifact, than run plawright normally
     echo "run playwright normally"
-        if [[ "$1" == ubuntu-latest* ]]; then
-            xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" -- yarn test:playwright:electron:ubuntu || true
-        elif [[ "$1" == windows-latest* ]]; then
-            yarn test:playwright:electron:windows || true
-        elif [[ "$1" == macos-14* ]]; then
-            yarn test:playwright:electron:macos || true
+        if [[ "$3" == *ubuntu* ]]; then
+            xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" -- yarn test:playwright:electron:ubuntu -- --shard=$1/$2 || true
+        elif [[ "$3" == *windows* ]]; then
+            yarn test:playwright:electron:windows -- --shard=$1/$2 || true
+        elif [[ "$3" == *macos* ]]; then
+            yarn test:playwright:electron:macos  -- --shard=$1/$2 || true
         else
             echo "Do not run playwright. Unable to detect os runtime."
             exit 1
@@ -28,11 +30,11 @@ while [[ $retry -le $max_retrys ]]; do
         if [[ $failed_tests -gt 0 ]]; then
             echo "retried=true" >>$GITHUB_OUTPUT
             echo "run playwright with last failed tests and retry $retry"
-            if [[ "$1" == ubuntu-latest* ]]; then
+            if [[ "$3" == *ubuntu* ]]; then
                 xvfb-run --auto-servernum --server-args="-screen 0 1280x960x24" -- yarn test:playwright:electron:ubuntu -- --last-failed || true
-            elif [[ "$1" == windows-latest* ]]; then
+            elif [[ "$3" == *windows* ]]; then
                 yarn test:playwright:electron:windows -- --last-failed || true
-            elif [[ "$1" == macos-14* ]]; then
+            elif [[ "$3" == *macos* ]]; then
                 yarn test:playwright:electron:macos -- --last-failed || true
             else
                 echo "Do not run playwright. Unable to detect os runtime."

diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml
@@ -18,6 +18,7 @@ permissions:
 jobs:
 
   check-rust-changes:
+    if: github.event.pull_request.draft == false
     runs-on: ubuntu-latest
     outputs:
       rust-changed: ${{ steps.filter.outputs.rust }}
@@ -33,20 +34,20 @@ jobs:
             rust:
               - 'src/wasm-lib/**'
 
-  browser:
-    timeout-minutes: ${{ matrix.os == 'macos-14' && 60 || 50 }}
-    name: playwright:browser:${{ matrix.os }} ${{ matrix.shardIndex }} ${{ matrix.shardTotal }}
+  electron:
+    if: github.event.pull_request.draft == false
+    timeout-minutes: 60
+    name: playwright:electron:${{ matrix.os }} ${{ matrix.shardIndex }} ${{ matrix.shardTotal }}
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest-8-cores, windows-latest-8-cores]
+        # TODO: enable self-hosted-windows-8-cores once available
+        os: [namespace-profile-ubuntu-8-cores, namespace-profile-macos-8-cores, windows-16-cores]
         shardIndex: [1, 2, 3, 4]
         shardTotal: [4]
     runs-on: ${{ matrix.os }}
     needs: check-rust-changes
     steps:
-    - name: Tune GitHub-hosted runner network
-      uses: smorimoto/tune-github-hosted-runner-network@v1
     - uses: actions/checkout@v4
     - uses: actions/setup-node@v4
       with:
@@ -101,7 +102,8 @@ jobs:
         echo "/opt/homebrew/opt/gnu-sed/libexec/gnubin" >> $GITHUB_PATH
     - name: Install vector
       shell: bash
-      if:  ${{ !startsWith(matrix.os, 'windows') }}
+      # TODO: figure out what to do with this, it's failing
+      if: false
       run: |
         curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev > /tmp/vector.sh
         chmod +x /tmp/vector.sh
@@ -123,13 +125,13 @@ jobs:
       if: steps.download-wasm.outcome == 'failure'
       shell: bash
       run: yarn build:wasm
-    - name: build web
-      run: yarn build:local
+    - name: build electron
       shell: bash
+      run: yarn tron:package
     - name: Run ubuntu/chrome snapshots
       shell: bash
       run: |
-        yarn playwright test --project="Google Chrome" --config=playwright.ci.config.ts --retries="3" --update-snapshots --grep=@snapshot  --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
+        PLATFORM=web yarn playwright test --config=playwright.config.ts --retries="3" --update-snapshots --grep=@snapshot  --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
       env:
         CI: true
         NODE_ENV: development
@@ -186,24 +188,19 @@ jobs:
       with:
         name: test-results-${{ matrix.os }}-${{ matrix.shardIndex }}-${{ github.sha }}
         path: test-results/
-    - name: Run playwright/chrome flow (with retries)
+    - name: Run playwright/electron flow (with retries)
       id: retry
       if: ${{ !cancelled() && (success() || failure()) }}
       shell: bash
       run: |
-        .github/ci-cd-scripts/playwright-browser-chrome.sh ${{matrix.shardIndex}} ${{matrix.shardTotal}} ${{matrix.os}}
+        .github/ci-cd-scripts/playwright-electron.sh ${{matrix.shardIndex}} ${{matrix.shardTotal}} ${{matrix.os}}
       env:
         CI: true
         FAIL_ON_CONSOLE_ERRORS: true
         NODE_ENV: development
         VITE_KC_DEV_TOKEN: ${{ secrets.KITTYCAD_API_TOKEN_DEV }}
         VITE_KC_SKIP_AUTH: true
         token: ${{ secrets.KITTYCAD_API_TOKEN_DEV }}
-    - name: send to axiom
-      if: always()
-      shell: bash
-      run: |
-        node playwrightProcess.mjs | tee /tmp/github-actions.log
     - uses: actions/upload-artifact@v4
       if: always()
       with:
@@ -221,136 +218,3 @@ jobs:
         retention-days: 30
         overwrite: true
 
-
-  electron:
-    name: playwright:electron:${{matrix.os}}
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest-8-cores, windows-latest-8-cores, macos-14-large]
-    timeout-minutes: 60
-    runs-on: ${{ matrix.os }}
-    needs: check-rust-changes
-    steps:
-    - name: Tune GitHub-hosted runner network
-      uses: smorimoto/tune-github-hosted-runner-network@v1
-    - uses: actions/checkout@v4
-    - uses: actions/setup-node@v4
-      with:
-        node-version-file: '.nvmrc'
-        cache: 'yarn'
-    - uses: KittyCAD/action-install-cli@main
-    - name: Install dependencies
-      shell: bash
-      run: yarn
-    - name: Cache Playwright Browsers
-      uses: actions/cache@v4
-      with:
-        path: |
-          ~/.cache/ms-playwright/
-        key: ${{ runner.os }}-playwright-${{ hashFiles('yarn.lock') }}
-    - name: Install Playwright Browsers
-      shell: bash
-      run: yarn playwright install chromium --with-deps
-    - name: Download Wasm Cache
-      id: download-wasm
-      if: needs.check-rust-changes.outputs.rust-changed == 'false'
-      uses: dawidd6/action-download-artifact@v7
-      continue-on-error: true
-      with:
-        github_token: ${{secrets.GITHUB_TOKEN}}
-        name: wasm-bundle
-        workflow: build-and-store-wasm.yml
-        branch: main
-        path: src/wasm-lib/pkg
-    - name: copy wasm blob
-      if: needs.check-rust-changes.outputs.rust-changed == 'false'
-      shell: bash
-      run: cp src/wasm-lib/pkg/wasm_lib_bg.wasm public
-      continue-on-error: true
-    - name: Setup Rust
-      uses: dtolnay/rust-toolchain@stable
-    - name: Cache Wasm (because rust diff)
-      if: needs.check-rust-changes.outputs.rust-changed == 'true'
-      uses: Swatinem/rust-cache@v2
-      with:
-        workspaces: './src/wasm-lib'
-    - name: OR Cache Wasm (because wasm cache failed)
-      if: steps.download-wasm.outcome == 'failure'
-      uses: Swatinem/rust-cache@v2
-      with:
-        workspaces: './src/wasm-lib'
-    - name: install good sed
-      if:  ${{ startsWith(matrix.os, 'macos') }}
-      shell: bash
-      run: |
-        brew install gnu-sed
-        echo "/opt/homebrew/opt/gnu-sed/libexec/gnubin" >> $GITHUB_PATH
-    - name: Install vector
-      if:  ${{ startsWith(matrix.os, 'ubuntu') }}
-      shell: bash
-      run: |
-        curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev > /tmp/vector.sh
-        chmod +x /tmp/vector.sh
-        /tmp/vector.sh -y -no-modify-path
-        mkdir -p /tmp/vector
-        cp .github/workflows/vector.toml /tmp/vector.toml
-        sed -i "s#GITHUB_WORKFLOW#${GITHUB_WORKFLOW}#g" /tmp/vector.toml
-        sed -i "s#GITHUB_REPOSITORY#${GITHUB_REPOSITORY}#g" /tmp/vector.toml
-        sed -i "s#GITHUB_SHA#${GITHUB_SHA}#g" /tmp/vector.toml
-        sed -i "s#GITHUB_REF_NAME#${GITHUB_REF_NAME}#g" /tmp/vector.toml
-        sed -i "s#GH_ACTIONS_AXIOM_TOKEN#${{secrets.GH_ACTIONS_AXIOM_TOKEN}}#g" /tmp/vector.toml
-        cat /tmp/vector.toml
-        ${HOME}/.vector/bin/vector --config /tmp/vector.toml &
-    - name: Build Wasm (because rust diff)
-      if: needs.check-rust-changes.outputs.rust-changed == 'true'
-      shell: bash
-      run: yarn build:wasm
-    - name: OR Build Wasm (because wasm cache failed)
-      if: steps.download-wasm.outcome == 'failure'
-      shell: bash
-      run: yarn build:wasm
-    - name: build electron
-      shell: bash
-      run: yarn tron:package
-    - uses: actions/download-artifact@v4
-      if: ${{ !cancelled() && (success() || failure()) }}
-      continue-on-error: true
-      with:
-        name: test-results-electron-${{ matrix.os }}-${{ github.sha }}
-        path: test-results/
-    - name: Run electron tests (with retries)
-      id: retry
-      if: ${{ !cancelled() && (success() || failure()) }}
-      shell: bash
-      run: |
-        .github/ci-cd-scripts/playwright-electron.sh ${{ matrix.os }}
-      env:
-        CI: true
-        FAIL_ON_CONSOLE_ERRORS: true
-        NODE_ENV: development
-        VITE_KC_DEV_TOKEN: ${{ secrets.KITTYCAD_API_TOKEN_DEV }}
-        VITE_KC_SKIP_AUTH: true
-        IS_UBUNTU: ${{ startsWith(matrix.os, 'ubuntu') && 'true' || 'false' }}
-        #DEBUG: 'pw:browser*'
-    - name: send to axiom
-      if: ${{ !cancelled() && (success() || failure()) && !startsWith(matrix.os, 'windows') }}
-      shell: bash
-      run: |
-        node playwrightProcess.mjs | tee /tmp/github-actions.log
-    - uses: actions/upload-artifact@v4
-      if: ${{ !cancelled() && (success() || failure()) }}
-      with:
-        name: test-results-electron-${{ matrix.os }}-${{ github.sha }}
-        path: test-results/
-        include-hidden-files: true
-        retention-days: 30
-        overwrite: true
-    - uses: actions/upload-artifact@v4
-      if: ${{ !cancelled() && (success() || failure()) }}
-      with:
-        name: playwright-report-electron-${{ matrix.os }}-${{ github.sha }}
-        path: playwright-report/
-        include-hidden-files: true
-        retention-days: 30
-        overwrite: true
diff --git a/README.md b/README.md
@@ -388,23 +388,6 @@ yarn test:unit:local
 
 #### E2E Tests
 
-**Playwright Browser**
-
-These E2E tests run in a browser (without electron).
-There are tests that are skipped if they are ran in a windows OS or Linux OS. We can use playwright tags to implement test skipping.
-
-Breaking down the command `yarn test:playwright:browser:chrome:windows`
-- The application is `playwright`
-- The runtime is a `browser`
-- The specific `browser` is `chrome`
-- The test should run in a `windows` environment. It will skip tests that are broken or flaky in the windows OS.
-
-```
-yarn test:playwright:browser:chrome
-yarn test:playwright:browser:chrome:windows
-yarn test:playwright:browser:chrome:ubuntu
-```
-
 **Playwright Electron**
 
 These E2E tests run in electron. There are tests that are skipped if they are ran in a windows, linux, or macos environment. We can use playwright tags to implement test skipping.