refactor: fix flaky unit tests #3338

jarhodes314 · 2025-01-17T17:56:27Z

Proposed changes

This fixes some flaky unit tests I observed locally. One was the mqtt_channel tests, which were failing due to a genuine bug where the connection was closed before all messages were published. The other was a mapper test, which implicitly assumed messages would be delivered in a fixed order.

A few other things that have changed in this PR:

I've added --status-level fail to cargo nextest run in just test. This stops cargo nextest listing every passing test name, which was previously obscuring the error output in my terminal.
I've removed serial-test from mqtt_channel in favour of using distinct topic/session names against a single broker in each test
Replaced some std::env::var calls with env! and some relative file paths with absolute paths so the test processes can be called without running them under cargo test. This is important for using tools like cargo stress

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
Documentation Update (if none of the other choices apply)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

I don't believe either of these had associated issues.

Checklist

I have read the CONTRIBUTING doc
I have signed the CLA (in all commits with git commit -s)
I ran cargo fmt as mentioned in CODING_GUIDELINES
I used cargo clippy as mentioned in CODING_GUIDELINES
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

Further comments

…ection gracefully

jarhodes314 · 2025-01-17T17:59:07Z

crates/tests/mqtt_tests/src/test_mqtt_server.rs

    let mut servers = HashMap::new();
    servers.insert("1".to_string(), server_config);

    rumqttd::Config {
        id: 0,
        router: router_config,
        cluster: None,
-        console: Some(console_settings),
+        console: None,


I don't know why this was ever enabled, I haven't observed us using it anywhere

It appears to be a defensive diagnostic mechanism kept in place to debug the server when some rare flaky failure occurs, as running the test again by turning it ON may not reproduce the failure.

We can safely ignore these console settings.

jarhodes314 · 2025-01-17T18:00:32Z

crates/extensions/c8y_mapper_ext/src/tests.rs


+    // The messages might get processed out of order, we don't care about the ordering of the messages
+    requests.sort();


This probably isn't technically necessary as I imagine the requests are always sent in the order from the original c8y message, but I think the point still stands that the ordering is irrelevant.

codecov · 2025-01-17T18:16:40Z

Codecov Report

Attention: Patch coverage is 93.41317% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/common/mqtt_channel/src/tests.rs	90.00%	0 Missing and 9 partials ⚠️
crates/common/mqtt_channel/src/connection.rs	94.59%	1 Missing and 1 partial ⚠️

Additional details and impacted files

📢 Thoughts on this report? Let us know!

github-actions · 2025-01-17T18:17:53Z

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
556	0	2	556	100	1h31m39.853377s

albinsuresh

The main fix that ensures that all the messages to be published are exhausted properly with acks looks fine. But, the test failures look concerning. Just some queries/comments on the other bits as well.

albinsuresh · 2025-01-20T06:24:17Z

crates/tests/mqtt_tests/src/test_mqtt_server.rs

+            Ok(Ok(())) => {
+                // I don't know why it happened, but I have observed this once while testing
+                // So just log the error and retry starting the broker on a new port
+                eprintln!("MQTT-TEST ERROR: `broker.start()` should not terminate until after `spawn_broker` returns")


But when this happens and when we retry the broker start on the next iteration, the client thread that's started (line. 137) in the last iteration also must be aborted somehow, right? Because once that client thread enters the loop, I don't see how it can break out even on a connection error because of the very specific if let check.

The client thread is not part of the loop, it's only started once we have a healthy broker.

albinsuresh · 2025-01-20T06:49:48Z

crates/tests/mqtt_tests/src/test_mqtt_server.rs

    let mut servers = HashMap::new();
    servers.insert("1".to_string(), server_config);

    rumqttd::Config {
        id: 0,
        router: router_config,
        cluster: None,
-        console: Some(console_settings),
+        console: None,


It appears to be a defensive diagnostic mechanism kept in place to debug the server when some rare flaky failure occurs, as running the test again by turning it ON may not reproduce the failure.

didier-wenzek

Thank you for finding and fixing this bug on closing the connection a bit too soon. However, one has to do the same for QoS 2.

crates/common/mqtt_channel/src/connection.rs

didier-wenzek · 2025-01-20T09:23:30Z

crates/common/mqtt_channel/src/connection.rs

+            if let Poll::Ready(None) = futures::poll!(awaiting_acks.as_mut().peek()) {
+                // If the channel is dropped, the sender loop has stopped
+                // and we've received an ack for every message published
+                mqtt_client.disconnect().await.unwrap();


I would rather ignore errors here.

Suggested change

mqtt_client.disconnect().await.unwrap();

let _ = mqtt_client.disconnect().await;

Co-authored-by: Didier Wenzek <[email protected]>

This should fix thin-edge#3021

crates/common/mqtt_channel/src/connection.rs

…ly closed

…ppens correctly

jarhodes314 added 3 commits January 17, 2025 17:44

Improve robustness of mqtt test broker startup

176b0a9

Ensure mqtt_channel waits for messages to publish before closing conn…

28f09e5

…ection gracefully

Fix flakiness from out of order messages

97c4550

jarhodes314 requested review from didier-wenzek, albinsuresh and rina23q as code owners January 17, 2025 17:56

jarhodes314 temporarily deployed to Test Pull Request January 17, 2025 17:56 — with GitHub Actions Inactive

jarhodes314 added the theme:testing label Jan 17, 2025

jarhodes314 commented Jan 17, 2025

View reviewed changes

jarhodes314 had a problem deploying to Test Auto January 17, 2025 18:02 — with GitHub Actions Failure

albinsuresh reviewed Jan 20, 2025

View reviewed changes

didier-wenzek reviewed Jan 20, 2025

View reviewed changes

Update crates/common/mqtt_channel/src/connection.rs

31d2bde

Co-authored-by: Didier Wenzek <[email protected]>

jarhodes314 temporarily deployed to Test Pull Request January 20, 2025 10:52 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 20, 2025 11:05 — with GitHub Actions Failure

Handle republishes and track last will publish

838f5aa

jarhodes314 temporarily deployed to Test Pull Request January 21, 2025 15:17 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 21, 2025 15:29 — with GitHub Actions Failure

jarhodes314 added 2 commits January 21, 2025 17:38

Avoid blocking on awaiting acks

ac28600

Don't risk blocking the mqtt-channel task

584daf2

jarhodes314 requested a deployment to Test Pull Request January 21, 2025 18:16 — with GitHub Actions Waiting

Ensure the bridge test proxy is always cleaned up

88c40c0

This should fix thin-edge#3021

jarhodes314 temporarily deployed to Test Pull Request January 21, 2025 18:19 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 21, 2025 18:28 — with GitHub Actions Failure

jarhodes314 commented Jan 22, 2025

View reviewed changes

crates/common/mqtt_channel/src/connection.rs Outdated Show resolved Hide resolved

jarhodes314 added 2 commits January 22, 2025 17:40

Explicitly close the connection to make sure the connection is actual…

69ae10a

…ly closed

Replace unneeded channel with simpler abstraction

c679907

jarhodes314 temporarily deployed to Test Pull Request January 23, 2025 10:07 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 23, 2025 11:03 — with GitHub Actions Failure

Merge remote-tracking branch 'upstream/main' into bug/flaky-unit-tests

2cebc89

jarhodes314 temporarily deployed to Test Pull Request January 23, 2025 14:25 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 23, 2025 14:31 — with GitHub Actions Error

jarhodes314 had a problem deploying to Test Auto January 23, 2025 15:04 — with GitHub Actions Failure

Replace ConnectionGuard with tokio::Semaphore to ensure waking ha…

608f20b

…ppens correctly

jarhodes314 requested review from a team as code owners January 24, 2025 16:22

jarhodes314 requested a deployment to Test Pull Request January 24, 2025 16:22 — with GitHub Actions Waiting

jarhodes314 added 3 commits January 24, 2025 16:29

Remove serial_test from mqtt-channel

bcceb23

Make tests more portable to allow them to run under cargo stress

ad6572e

Make cargo-nextest output less verbose

51408a0

jarhodes314 force-pushed the bug/flaky-unit-tests branch from f4bc3fd to 51408a0 Compare January 24, 2025 16:29

jarhodes314 requested a review from reubenmiller as a code owner January 24, 2025 16:29

jarhodes314 temporarily deployed to Test Pull Request January 24, 2025 16:30 — with GitHub Actions Inactive

jarhodes314 deployed to Test Auto January 24, 2025 16:37 — with GitHub Actions Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: fix flaky unit tests #3338

refactor: fix flaky unit tests #3338

jarhodes314 commented Jan 17, 2025 •

edited

Loading

jarhodes314 Jan 17, 2025

albinsuresh Jan 20, 2025 •

edited

Loading

didier-wenzek Jan 20, 2025

jarhodes314 Jan 17, 2025

codecov bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

albinsuresh left a comment •

edited

Loading

albinsuresh Jan 20, 2025

jarhodes314 Jan 24, 2025

albinsuresh Jan 20, 2025 •

edited

Loading

didier-wenzek left a comment

didier-wenzek Jan 20, 2025


		// The messages might get processed out of order, we don't care about the ordering of the messages
		requests.sort();

	mqtt_client.disconnect().await.unwrap();
	let _ = mqtt_client.disconnect().await;

refactor: fix flaky unit tests #3338

Are you sure you want to change the base?

refactor: fix flaky unit tests #3338

Conversation

jarhodes314 commented Jan 17, 2025 • edited Loading

Proposed changes

Types of changes

Paste Link to the issue

Checklist

Further comments

jarhodes314 Jan 17, 2025

Choose a reason for hiding this comment

albinsuresh Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

didier-wenzek Jan 20, 2025

Choose a reason for hiding this comment

jarhodes314 Jan 17, 2025

Choose a reason for hiding this comment

codecov bot commented Jan 17, 2025 • edited Loading

Codecov Report

github-actions bot commented Jan 17, 2025 • edited Loading

Robot Results

albinsuresh left a comment • edited Loading

Choose a reason for hiding this comment

albinsuresh Jan 20, 2025

Choose a reason for hiding this comment

jarhodes314 Jan 24, 2025

Choose a reason for hiding this comment

albinsuresh Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

didier-wenzek left a comment

Choose a reason for hiding this comment

didier-wenzek Jan 20, 2025

Choose a reason for hiding this comment

jarhodes314 commented Jan 17, 2025 •

edited

Loading

albinsuresh Jan 20, 2025 •

edited

Loading

codecov bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

albinsuresh left a comment •

edited

Loading

albinsuresh Jan 20, 2025 •

edited

Loading