test-flake: integration_tests::updates::test_update_uninitialized

[rcgoodfellow]

This test passed in a PR, then failed on main once the PR landed.

See

• buildomat log
• test_all-177f78ec8b61999f-test_update_end_to_end.104806.1.log
• test_all-177f78ec8b61999f-test_update_end_to_end.104806.3.log
• test_all-177f78ec8b61999f-test_update_uninitialized.104806.0.log

[iliana]

Error: repository fetch should have failed with 500 error

Caused by:
    0: making request to server
    1: connection closed before message completed

Yet the server seemed to do the right thing?

[rcgoodfellow]

It was the Linux job that had this failure. The helios job passed on the first go. The test passed when re-running the Linux CI job.

[smklein]

Saw this same flake here: https://github.com/oxidecomputer/omicron/pull/4936/checks?check_run_id=21134488445

[sunshowers]

I wrote this test, will look tomorrow.

[smklein]

One more data point: #5246 , still flaking

[smklein]

This was on buildomat / build-and-test (helios) , which is not a Linux job, fwiw

[sunshowers]

Started looking at this a couple weeks ago but got preempted :(

Wondering if we should just throw a retry on the test for now.

[elaine-oxide]

I encountered the same issue (in buildomat / build-and-test (ubuntu-22.04), FAIL for omicron-nexus::test_all integration_tests::updates::test_update_uninitialized), where the test passed in a PR, then failed on main once the PR landed.

3d3f6d7

https://github.com/oxidecomputer/omicron/runs/29801528621

• buildomat log
• /var/tmp/omicron_tmp/test_all-65dbc55f09e622ff-test_update_uninitialized.176014.0.log
• /var/tmp/omicron_tmp/test_all-65dbc55f09e622ff-test_update_end_to_end.176014.1.log
• /var/tmp/omicron_tmp/test_all-65dbc55f09e622ff-test_update_end_to_end.176014.3.log
• /var/tmp/omicron_tmp/test_all-65dbc55f09e622ff-test_update_end_to_end.176014.4.log

[elaine-oxide]

The previously failing test passed on re-run.
https://github.com/oxidecomputer/omicron/runs/29808706363

[andrewjstone]

Looks like we hit this again: https://buildomat.eng.oxide.computer/wg/0/details/01J83YQ7AHHW6QKHJDPJ1VNTQZ/TO4wpZD6Wkn0cqGZqM1XfG8KqjefjStmDUPQq3Xtnnd1ZgiZ/01J83YQVGZ2RM05A2M6PA2SF1J

[andrewjstone]

On the latest run, there are a few odd failures. I'm not actually sure this is the test's fault in particular. It may be a symptom of some other issue. I'm seeing issues with DNS and a network unreachable error. I'm not sure if these are things we would ever expect to see.

https://buildomat.eng.oxide.computer/wg/0/artefact/01J83YQ7AHHW6QKHJDPJ1VNTQZ/TO4wpZD6Wkn0cqGZqM1XfG8KqjefjStmDUPQq3Xtnnd1ZgiZ/01J83YQVGZ2RM05A2M6PA2SF1J/01J841N63E4WRZMTEKP3AK76NY/test_all-d3f0fb0580db6861-test_update_end_to_end.176436.1.log?format=x-bunyan#L704
https://buildomat.eng.oxide.computer/wg/0/artefact/01J83YQ7AHHW6QKHJDPJ1VNTQZ/TO4wpZD6Wkn0cqGZqM1XfG8KqjefjStmDUPQq3Xtnnd1ZgiZ/01J83YQVGZ2RM05A2M6PA2SF1J/01J841N63E4WRZMTEKP3AK76NY/test_all-d3f0fb0580db6861-test_update_end_to_end.176436.1.log?format=x-bunyan#L3058

CC @davepacheco @rcgoodfellow

[FelixMcFelix]

This one's come up again today for me: https://buildomat.eng.oxide.computer/wg/0/details/01JADDKWGJPE0W6GWXXKVACPCS/tntBJzr7RPDoVr0k1n2H2XbfwB7E1I31lHt7USHEwUNUteb7/01JADDM60TPPKHJFF6GYQ0E9QW.

[andrewjstone]

This one's come up again today for me: https://buildomat.eng.oxide.computer/wg/0/details/01JADDKWGJPE0W6GWXXKVACPCS/tntBJzr7RPDoVr0k1n2H2XbfwB7E1I31lHt7USHEwUNUteb7/01JADDM60TPPKHJFF6GYQ0E9QW.

I think you actually hit this error: #6771 but in a test.