progress update

got a first version of the script to run tests like in CI
but quite a lot of tests failed (usually around 17-19)

Reasons seemed to be:

• contention / deadlocks (tests never finishing)
• contention over shared OS resource (for instance “port already in use”, …)
• timeouts
• tests rely on time but time not simulated with tokio -> time -> start_paused
  • test relies on time passing and logic getting processed fast enough, asserts on end state
  • my (remote) machine might not process this fast enough

Tried:

• looking into actual CI runs’ logs, seeing if the tests that fail on my machine run/pass in CI
  • same tests seem to pass in CI
• splitting failing tests into simtests / non-sim tests
  • better pass rate but still some tests usually still failing / flaky
• running problematic (both simtests and nonsim-tests) tests with zero concurrency
  • this seems to give the best pass rate so far

Left to do:

• keep configuring nextest to run this test with no concurrency/ no other test threads interfering
• remove now-useless parts of the script

Other smaller side problems:
CI scripts rely on cargo +nightly … which only works when installed via rustup.
-> solution: emulate it without rustup via a cargo subcommand