The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2718498	dotnet-runtime	Initialize containers	Log
2718053	dotnet-arcade-services	Run E2E tests	Log
2718453	dotnet-performance	Send job to Helix (Unix)	Log
2718416	dotnet-runtime	Send job to Helix (Unix)	Log
2718399	dotnet-runtime	Publish Build Assets	Log
1052621	dotnet/runtime	Send to Helix	Log
1052722	dotnet/runtime	Send to Helix	Log
2718276	dotnet-runtime	Publish Browser Wasm Artifacts	Log
1052701	dotnet/runtime	AOT compile CoreCLR tests	Log
1052705	dotnet/roslyn	Run Integration Tests	Log
1052688	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#78724
1052356	dotnet/runtime	Send to Helix	Log
1052560	dotnet/runtime	Send to Helix	Log
1052406	dotnet/runtime	Send to Helix	Log
2718156	dotnet-dotnet	Build (centOSStreamContainer)	Log
2718160	dotnet-aspire	Build	Log	#50415
2718142	dotnet-runtime	Run performance job script	Log
1052408	dotnet/runtime	Send to Helix	Log
2718027	dotnet-aspnetcore	SDL Binary Analysis Checks (1ES PT)	Log
2718076	dotnet-dotnet	Build	Log
1052367	dotnet/runtime	AOT compile CoreCLR tests	Log
1052357	dotnet/runtime	AOT compile CoreCLR tests	Log
1052353	dotnet/runtime	Send to Helix	Log
1052308	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116018
1052275	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115937
2717986	dotnet-docker-tools	🟪 Build Images (Authenticated)	Log
2717974	dotnet-dotnet-docker	Start Network Isolation	Log
1052152	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116025
2717914	dotnet-runtime	Send job to Helix (Unix)	Log
1050671	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115976
2717734	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
1050662	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115975
1051748	dotnet/runtime	Send to Helix	Log
2717800	dotnet-runtime	Run performance job script	Log
1050951	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1419
1050896	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1418
2717748	dotnet-dotnet	Build (centOSStreamContainer)	Log
1051774	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115937
1051750	dotnet/runtime	Send to Helix	Log
2717735	dotnet-dotnet	Download Blob Artifacts	Log
1051749	dotnet/runtime	Send job to Helix (Unix)	Log
1051730	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115983
1051722	dotnet/runtime	Send to Helix	Log
1051720	dotnet/runtime	AOT compile CoreCLR tests	Log
1051684	dotnet/runtime	Send tests to Helix (Unix)	Log
2717656	dotnet-runtime	Run performance job script	Log
1050441	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115964
2717537	dotnet-runtime	Run performance job script	Log
1051573	dotnet/arcade	XHarness Android Helix Testing (Linux)	Log	dotnet/arcade#15853
1051501	dotnet/runtime	Send to Helix	Log
1051480	dotnet/runtime	AOT compile CoreCLR tests	Log
2717391	dotnet-runtime	Run performance job script	Log
1051387	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115937
1051118	dotnet/runtime	Send to Helix	Log
2717151	dotnet-runtime	Send UTF8Fuzzer to OneFuzz	Log
2717200	dotnet-sdk	🟣 Run TestBuild Tests	Log	#50392
2717188	dotnet-sdk	🛡 Guardian: CredScan (Binary)	Log	#50392
2717147	dotnet-runtime	Run performance job script	Log
1051098	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115985/merge to s	Log	dotnet/runtime#115985
1051121	dotnet/runtime	Send job to Helix (Unix)	Log
1049688	dotnet/arcade	XHarness Android Helix Testing (Linux)	Log	dotnet/arcade#15858
2717021	dotnet-runtime	Run performance job script	Log
1050984	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115589
1050867	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115800
1050786	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115704
1050822	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115019
1050760	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115980
2716937	dotnet-runtime	Run performance job script	Log
1050813	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115981
1050821	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#115957
1050818	dotnet/runtime	Send job to Helix (Unix)	Log
1050725	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115947
1050458	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115542
1050770	dotnet/runtime	Send to Helix	Log
2716864	dotnet-runtime	Run performance job script	Log
1050754	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115453
1050750	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115957
1050749	dotnet/runtime	Send tests to Helix (Windows)	Log
1050747	dotnet/runtime	AOT compile CoreCLR tests	Log
1050720	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115977
1050715	dotnet/runtime	Send job to Helix (Unix)	Log
2716799	dotnet-runtime	Run performance job script	Log
1050710	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115974
1050691	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115658
1050686	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115948
1050675	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115959
2716656	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
1050613	dotnet/runtime	Send to Helix	Log
1050592	dotnet/runtime	AOT compile CoreCLR tests	Log
1050594	dotnet/runtime	Send to Helix	Log
1050583	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115972
1050568	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115704
1050430	dotnet/runtime	Send to Helix	Log
1050551	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115856
1050542	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115937
1050518	dotnet/arcade	XHarness Android Helix Testing (Linux)	Log	dotnet/arcade#15853
1049740	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115914
1050496	dotnet/runtime	Build product	Log
1050477	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115388
2716439	dotnet-install-scripts	Run Tests	Log
Displaying 100 of 1038 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
36	290	1038

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)