Timeout functional tests

[kgaughan]

If the functional tests don't exist after five seconds of waiting on them, they should be killed. This also uses pytest-timeout to impose a maximum runtime of 30secs on any test in the test suite.

Hopefully this should prevent tests from hanging.

[kgaughan]

Closer. The functional tests look to be completing, but pytest (or tox) looks to get stuck, so the final step in the workflow doesn't exit. My best guess is that there's a wait or join being executed in the atexit handler that's preventing the process from exiting.

[kgaughan]

I just reran the failed workflows, and they succeeded. I guess it's at least better that the tests themselves aren't hanging and, but I'm not at all sure why the process sometimes doesn't exit like it should

[digitalresistor]

Not a fan of this change. I feel like it papers over waitress not existing cleanly. I want tests to hang.

The GitHub actions runners have been slow and having issues for the last couple of days, locally these tests always succeed correctly.

[kgaughan]

In that case, I can drop the changes to tests/test_functional.py and keep the timeout on the test: it should fail on the offending test while it's waiting on the join() and wait() calls to return. Sound good?

[digitalresistor]

What do we gain? I am not sure I fully understand what the point of it is, today with the current test suite. The additional dependency/package is an additional one to download/install, across all CI runs. While it may be small it adds up, and I try to be cognizant of the overhead of adding yet another thing that can cause issues because it is not maintained/not updated as quickly as other parts of the ecosystem.

If the functional tests hanging was a bigger issue and one that could be triggered regularly I wouldn't be opposed to adding extra security against it.

[kgaughan]

What I was originally trying to do was to see if it was reveal which of the functional tests was the most problematic when it came to hanging in the hopes of making the test suite more reliable. They seemed to most consistently fail on EchoTests and TooLargeTests with the timeout in place.

If you don't see it as worthwhile, I can close the PR though.

[digitalresistor]

Where do you see them hanging? I am trying to understand better because I too want the test suite to not fail/and understand the issues and fix them.

While working on some stuff lately to fix some of the security issues I have re-ran the test suite 10's of times, I haven't seen it hang at all, and my test suite runs have been ~14 seconds (with coverage enabled, couple seconds shorter without coverage).

[digitalresistor]

I will note that there seems to be an issue with the Github actions, the test suite there historically has ran in about ~60 seconds at most, and now I am seeing runs that are a couple of minutes a piece, or even longer.

I don't know how to account for those changes in runtime, it's the same code base, same tests.

[kgaughan]

Yep, it confused me too. It hangs for me randomly from time to time in the middle of the functional tests. I couldn't detect a pattern to it, hence why I tried cleaning up the fixture subprocesses more aggressively.