experimental-multiple-windows fails with Error: WebSocket connection closed on completion of some tests

[keif888]

What is your Scenario?

Use native automation to open places page on PhotoPrism.
Validate that searching for a location will return the map, with a cluster of photos for that location.
Zoom out.
Select the cluster, and then open another window from the browse button shown in the UI.

FYI:
Maps fail in non native automation due to an issue in hammerhead (already logged), so we are attempting to use the --experimental-multiple-windows flag for tests that use multiple windows.

What is the Current behavior?

On successful completion of the test (as per the console.log messages), the an error is thrown as testcafe is tearing everything down. The test that fails with a WebSocket connection closed is unpredictable.

Execution Number 1

> photoprism@1 testcafe
> testcafe chrome --headless=new --skip-js-errors --experimental-multiple-windows ./tests/multitests.js

 Running tests in:
 - Chrome 133.0.0.0 / Ubuntu 24.10

 Test photos
  Start Test
  End Test
 ✓ Multi-Window: Navigate from card view to place
  Start Test
  End Test
 ✖ Multi-Window: Navigate from card view to photos taken at the same date

   1) Unhandled promise rejection:
      
      Error: WebSocket connection closed
      at /usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:94:35
      at Chrome._handleConnectionClose (/usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:256:13)
      at WebSocket.<anonymous> (/usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:243:22)
      at WebSocket.emit (node:events:524:28)
      at WebSocket.emit (node:domain:489:12)
      at WebSocket.emitClose (/usr/lib/node_modules/testcafe/node_modules/ws/lib/websocket.js:246:10)
      at Socket.socketOnClose (/usr/lib/node_modules/testcafe/node_modules/ws/lib/websocket.js:1150:15)
      at Socket.emit (node:events:524:28)
      at Socket.emit (node:domain:489:12)
      at TCP.<anonymous> (node:net:351:12)

      Browser: Chrome 133.0.0.0 / Ubuntu 24.10


 Search and open photo from places
  Start Test
  End Test
 ✓ Multi-Window: Test places


 1/3 failed (1m 37s)

What is the Expected behavior?

 Running tests in:
 - Chrome 133.0.0.0 / Ubuntu 24.10

 Test photos
  Start Test
  End Test
 ✓ Multi-Window: Navigate from card view to place
  Start Test
  End Test
 ✓ Multi-Window: Navigate from card view to photos taken at the same date
 Search and open photo from places
  Start Test
  End Test
 ✓ Test places


 3 passed (1m 45s)

What is the public URL of the test page? (attach your complete example)

The public demo of PhotoPrism can be used for this test.
https://demo.photoprism.app/library/login is the initial landing page (no credentials required as this is a public demo).
The test also navigates to the following pages:
https://demo.photoprism.app/library/places
https://demo.photoprism.app/library/browse

What is your TestCafe test code?

import { Selector } from "testcafe";

fixture`Test New Tab 1`.page`https://demo.photoprism.app/library/login`;

test.meta("testID", "testtab-001").meta({ mode: "public" })(
"Multi-Window: First Attempt",async (t) => {
    await generictester(t)
})

test.meta("testID", "testtab-002").meta({ mode: "public" })(
    "Multi-Window: Second Attempt",async (t) => {
        await generictester(t)
    })



fixture`Test New Tab 2`.page`https://demo.photoprism.app/library/login`;

test.meta("testID", "testtab-003").meta({ mode: "public" })(
"Multi-Window: Third Attempt",async (t) => {
    await generictester(t)
})
    
async function generictester(t){
    console.log("  Start Test");
    
    await t.expect(Selector(".nav-places").visible).ok()
    await t.expect(Selector("button.action-open-date", { timeout: 7000 }).visible).ok()
    await t.click(Selector("button.action-open-date").nth(0))
    await t.expect(Selector(".nav-places").visible).ok()

    console.log("  End Test");
}

Your complete configuration file

testcaferc.json

{
  "skipJsErrors": true,
  "quarantineMode": false,
  "selectorTimeout": 6000,
  "assertionTimeout": 7000,
  "hostname": "localhost",
  "screenshots": {
    "path": "tests/acceptance/screenshots",
    "takeOnFails": true
  }
}

Your complete test report

npm run testcafe -- "chrome --headless=new" --experimental-multiple-windows ./tests/testcafeissue8377.js 

> photoprism@1 testcafe
> testcafe chrome --headless=new --experimental-multiple-windows ./tests/testcafeissue8377.js

 Running tests in:
 - Chrome 133.0.0.0 / Ubuntu 24.10

 Test New Tab 1
  Start Test
  End Test
 ✖ Multi-Window: First Attempt

   1) Unhandled promise rejection:
      
      Error: WebSocket connection closed
      at /usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:94:35
      at Chrome._handleConnectionClose (/usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:256:13)
      at WebSocket.<anonymous> (/usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:243:22)
      at WebSocket.emit (node:events:524:28)
      at WebSocket.emit (node:domain:489:12)
      at WebSocket.emitClose (/usr/lib/node_modules/testcafe/node_modules/ws/lib/websocket.js:246:10)
      at Socket.socketOnClose (/usr/lib/node_modules/testcafe/node_modules/ws/lib/websocket.js:1150:15)
      at Socket.emit (node:events:524:28)
      at Socket.emit (node:domain:489:12)
      at TCP.<anonymous> (node:net:351:12)

      Browser: Chrome 133.0.0.0 / Ubuntu 24.10


  Start Test
  End Test
 ✓ Multi-Window: Second Attempt

 Test New Tab 2
  Start Test
  End Test
 ✖ Multi-Window: Third Attempt

   1) Unhandled promise rejection:
      
      Error: WebSocket connection closed
      at /usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:94:35
      at Chrome._handleConnectionClose (/usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:256:13)
      at WebSocket.<anonymous> (/usr/lib/node_modules/testcafe/node_modules/chrome-remote-interface/lib/chrome.js:243:22)
      at WebSocket.emit (node:events:524:28)
      at WebSocket.emit (node:domain:489:12)
      at WebSocket.emitClose (/usr/lib/node_modules/testcafe/node_modules/ws/lib/websocket.js:246:10)
      at Socket.socketOnClose (/usr/lib/node_modules/testcafe/node_modules/ws/lib/websocket.js:1150:15)
      at Socket.emit (node:events:524:28)
      at Socket.emit (node:domain:489:12)
      at TCP.<anonymous> (node:net:351:12)

      Browser: Chrome 133.0.0.0 / Ubuntu 24.10



 2/3 failed (13s)

Screenshots

No screen shots are taken as the test has already successfully completed, and testcafe fails as it tears the test down.

Steps to Reproduce

1. Open a terminal window and execute the following command:
2. for i in {1..10}; do echo "Execution Number" $i; npm run testcafe -- "chrome --headless=new" --experimental-multiple-windows ./tests/testcafeissue8377.js || break; done

This command will run up to 10 times, but stop when the 1st failure is encountered.
The error is not consistent and can take a couple of executions to fail.

TestCafe version

11.1.0

Node.js version

v22.13.1

Command-line arguments

-- "chrome --headless=new" --skip-js-errors --experimental-multiple-windows ./tests/testcafeissue8377.js

Browser name(s) and version(s)

Chrome headless 133

Platform(s) and version(s)

Linux Ubuntu 24.10

Other

The test script above has been created to show the issue. It is pared back to the minimum required to show the issue.
Non Native Automation of maps fail as per this issue: DevExpress/testcafe-hammerhead#3041
which is why we are attempting to use --experimental-multiple-windows

[keif888]

I updated the source code in the issue above to remove use of resizeWindow which is listed as not supported for --experimental-multiple-windows.

[keif888]

FYI:
As a workaround to this issue, execute only a single test per execution of testcafe with --experimental-multiple-windows.
I have executed 600 individual executions of a single test without a failure (the 3 different tests above executed 200 times).
BUT, if you execute multiple tests in a single execution of testcafe then there is a high probability of encountering this issue.

My workaround was to call testcafe using the --test "testname" parameter for each individual test that needs multiple windows.

npm run testcafe -- "chrome --headless=new" --experimental-multiple-windows --test-meta mode=public --test "Multi-Window: Navigate from card view to place" "tests/acceptance"
npm run testcafe -- "chrome --headless=new" --experimental-multiple-windows --test-meta mode=public --test "Multi-Window: Navigate from card view to photos taken at the same date" "tests/acceptance"
npm run testcafe -- "chrome --headless=new" --experimental-multiple-windows --test-meta mode=public --test "Multi-Window: Test places" "tests/acceptance"

I did this using find, perl, and xargs. Your milage may vary.

find ./tests/acceptance -type f -name "*.js" | xargs -i perl -0777 -ne 'while(/(?:mode: \"auth[^,]*\,)|(Multi-Window\:[A-Za-z 0-9\-_]*)/g){print "$1\n" if ($1);}' {} | xargs -I testname bash -c 'npm run testcafe -- "chrome --headless=new" --experimental-multiple-windows --test-meta mode=public --test "testname" "tests/acceptance"'

We have auth and public tests which need to be executed separately, which is why the regex in the perl single line matches but doesn't capture auth, so that it only finds the right tests.

[Bayheck]

Hello,

Thank you for reaching out to us.

I tried to run your example and encountered a different error related to invalid date creation.

After a bit of debugging, I found that the buttonDate is invalid.

The test also contains a significant amount of code that is unrelated to the issue.

Additionally, try to place the await before the Selector, like so:

if (await Selector('div[role="option"]').visible) {
   break;
}

Please remove any unnecessary code and provide a minimal, updated version of the test so I can further investigate the issue.

[keif888]

I have replaced the test script in the issue with a newly created minimal test script that shows the issue without extra code.
The missing await was only in my test script, not in the real code base.