connection_checker_test.js has flaky tests

[BenHenning]

Check for duplicates

• I have searched for similar issues before opening a new one.

Description

Sometimes when running npm run test locally, I see failures in connection_checker_test.js. I found this while investigating #8676 and it occurs to me that it could be environment specific.

The specific tests failing are:

• Connection checker Dragging Checks Stacks Do not splice into unmovable stack (about 3% flaky): test link.
• Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack (about 14% flaky): test link.

Reproduction steps

1. Run tests locally several times.
2. Observe that two specific tests seem to fail at a fairly high rate.

Some techniques that I find helpful:

• Run the tests in headless mode (seems to be better performance, and I'm developing over a remote shell): CI=true npm run test.
• Comment out all but mocha from tasks in scripts/gulpfiles/test_tasks.js (so that only Mocha tests run).
• Replace the top-level suite call with suite.only in tests/mocha/connection_checker_test.js (so that only this suite runs).
• Use one of the scripts in https://gist.github.com/BenHenning/a4daa87d7983026411ac12f3b357005b to run the tests multiple times to detect flakes (I prefer the Kotlin one since it handles output much better).

Stack trace

Output from running ``run_blockly_tests.main.kts`` per the Gist above:


Running Blockly tests in /usr/.../blockly 100 time(s) across X core(s):
Completed 10/100 (finished: 13.386s, avg: 1.3385999s, rem: ~120.474s)
Completed 20/100 (finished: 16.146s, avg: 1.4765999s, rem: ~118.128s)
Completed 30/100 (finished: 19.628s, avg: 1.6386666s, rem: ~114.706665s)
Completed 40/100 (finished: 20.929s, avg: 1.7522249s, rem: ~105.1335s)
Completed 50/100 (finished: 22.803s, avg: 1.85784s, rem: ~92.892s)
Completed 60/100 (finished: 25.131s, avg: 1.9670501s, rem: ~78.682s)
Completed 70/100 (finished: 27.237s, avg: 2.0751429s, rem: ~62.254284s)
Completed 80/100 (finished: 29.679s, avg: 2.1867375s, rem: ~43.73475s)
Completed 90/100 (finished: 31.924s, avg: 2.298478s, rem: ~22.984777s)
Completed 100/100 (finished: 22.88s, avg: 2.29743s, rem: ~0.0s)

Finished running tests in 34.086 seconds (total execution time: 229.743 seconds). Results:
83/100 runs passed.
17/100 runs failed.
Specific failures:
- 3 times: Connection checker Dragging Checks Stacks Do not splice into unmovable stack
- 14 times: Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack

The failure for the 'Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack' test was:

"Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack" failed: Should connect below an unmovable stack block: expected false to be true

I didn't collect the stack trace of the other test failure.

Screenshots

N/A since this is a test assertion flake.

Browsers

No response

[BenHenning]

Note that I just ran the entire Mocha test suite 100 times and these two tests were the only ones flaky for me (the most recent run showed the 'do not splice' test as 8% flaky and the 'connect to bottom of unmovable stack' test as 11% flaky).

I also did a run of just those two flaky tests in isolation over 1000 runs to get a better estimate of flakiness. Over 1000 runs:

• 9.3% flaky: Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack
• 9.2% flaky: Connection checker Dragging Checks Stacks Do not splice into unmovable stack

[BenHenning]

I'm not sure who has been affected by these flakes in CI, but I just ran into one of them in #8689.