sound/pipewire: add truncated exp backoff to tests and fork them

[epilys]

Sometimes pw backend creation on CI fails with errors like for example:

thread 'audio_backends::pipewire::tests::test_pipewire_backend_invalid_stream' panicked at vhost-device-sound/src/audio_backends/pipewire.rs:98:42:
Failed to connect to core: CreationFailed

Add a simple kind of truncated exponential backoff retry wrapper to
PwBackend::new().

Also, run all tests that modify their process environment under a fork using the rusty_fork crate.

Signed-off-by: Manos Pitsidianakis [email protected]

Depends on #788 which fixes lint warnings

[epilys]

@stefano-garzarella can we retry the CI jobs a few times manually to see if they don't hangup indefinitely? (since there is no max try limit in this version of the code)

[stefano-garzarella]

@stefano-garzarella can we retry the CI jobs a few times manually to see if they don't hangup indefinitely? (since there is no max try limit in this version of the code)

sure, let me restart it!

[stefano-garzarella]

@epilys is there any log messages I should check if the retry worked? (not yet looked at the PR, sorry)

[epilys]

@stefano-garzarella the CI seems to capture test output by default, so it'd only show up if something failed unfortunately

[stefano-garzarella]

@stefano-garzarella the CI seems to capture test output by default, so it'd only show up if something failed unfortunately

I see, BTW 3 pipeline rebuilds so far without issues ;-) I'll arrive at least at 10 and report here!

[epilys]

Mentioning that this is related to #647, but since we cannot know for sure if this PR fixes it we cannot close it yet at least if this is merged.

[epilys]

Ok so the problem is not solved by retrying, since this job https://buildkite.com/rust-vmm/vhost-device-ci/builds/2868#019387cd-8337-4fc4-a24f-ab6d701d4bb4/46-726 managed to hang:

test audio_backends::pipewire::tests::test_pipewire_backend_invalid_stream has been running for over 60 seconds

Some other idea to debug this (posting them here as notes for the future):

PipewireTestHarness::new spawns a pipewire process, but just assumes it launches correctly. Never does a try_wait on the spawned child. Maybe we should do that and see what the failure is, possible put the retry/backoff approach there.

[stefano-garzarella]

Ok so the problem is not solved by retrying, since this job buildkite.com/rust-vmm/vhost-device-ci/builds/2868#019387cd-8337-4fc4-a24f-ab6d701d4bb4/46-726 managed to hang:

test audio_backends::pipewire::tests::test_pipewire_backend_invalid_stream has been running for over 60 seconds

Some other idea to debug this (posting them here as notes for the future):

PipewireTestHarness::new spawns a pipewire process, but just assumes it launches correctly. Never does a try_wait on the spawned child. Maybe we should do that and see what the failure is, possible put the retry/backoff approach there.

yeah, this is a good point! I agree on doing that.

[epilys]

Ok that's interesting, my latest change to panic if dbus/pipewire process don't exist triggered a new test failure:

     Running unittests src/lib.rs (target/debug/deps/vhost_device_sound-e293b9b9920f5bbe)
running 34 tests
error: test failed, to rerun pass `-p vhost-device-sound --lib`
Caused by:
  process didn't exit successfully: `/workdir/target/debug/deps/vhost_device_sound-e293b9b9920f5bbe` (signal: 11, SIGSEGV: invalid memory reference)

I'm beginning to suspect tests running in threads is at play here; setting environment variables is not thread safe. Perhaps this could be the source of errors... Maybe all tests using alsa/pipewire stuff should be run with the sealed_test crate.

[stefano-garzarella]

Ok that's interesting, my latest change to panic if dbus/pipewire process don't exist triggered a new test failure:

     Running unittests src/lib.rs (target/debug/deps/vhost_device_sound-e293b9b9920f5bbe)
running 34 tests
error: test failed, to rerun pass `-p vhost-device-sound --lib`
Caused by:
  process didn't exit successfully: `/workdir/target/debug/deps/vhost_device_sound-e293b9b9920f5bbe` (signal: 11, SIGSEGV: invalid memory reference)

I'm beginning to suspect tests running in threads is at play here; setting environment variables is not thread safe.

Goooood point!

Perhaps this could be the source of errors... Maybe all tests using alsa/pipewire stuff should be run with the sealed_test crate.

Cool, this can help. Can we also have other concurrency issues?
I had an issue with vhost-vdpa devices, where the only solution was to run tests sequentially to avoid issues (in that case a /dev/vhost-vdpa-0 device can be opened only by 1 thread).

[epilys]

Updated to run all process environment-modifying tests under forks using the rusty-fork crate like it's done in the vhost-device-gpu crate tests in staging.

[epilys]

@stefano-garzarella

Can we also have other concurrency issues?
I had an issue with vhost-vdpa devices, where the only solution was to run tests sequentially to avoid issues (in that case a /dev/vhost-vdpa-0 device can be opened only by 1 thread).

Not sure, we're not dealing with any /dev/ device files (I think). For alsa at least, the test harness uses null devices.

[stefano-garzarella]

Sorry, I just saw some really minor things, feel free to ignore them.