slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2829151	dotnet-performance	Performance powershell windows Win11 x64 Viper 8.0	Log
1190831	dotnet/arcade	Android_Devices Build_Release	Log	dotnet/arcade#16256
1190689	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#119432
1190414	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121186
2828480	dotnet-performance	Performance powershell windows Win11 x64 Viper main	Log
1190441	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51512
2828507	dotnet-maui	Blazor macOS	Log
1190131	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1190095	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#120887
1190101	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log
1190086	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50468
2828085	dotnet-performance	Performance scenarios Affinity_85 windows Win11 x64 Tiger 8.0	Log
2827986	dotnet-performance	Performance nativeaot_scenarios windows Win11 x64 Viper 9.0	Log
1189840	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#117788
2828150	dotnet-sdk	TestBuild: macOS (x64)	Log
2828031	dotnet-performance	Performance nativeaot_scenarios windows Win11 x64 Viper main	Log
2827747	dotnet-performance	Performance bepuphysics windows Win11 x64 Viper 8.0	Log
2827904	dotnet-performance	Performance nativeaot_scenarios windows Win11 x64 Tiger 8.0	Log
1188793	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log
1188690	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2827296	dotnet-install-scripts	MacOSTests	Log
1188640	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51462
1188509	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121164
1188466	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121175
1188458	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121110
1188470	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120976
2827083	dotnet-performance	Performance roslyn windows Win11 x64 Tiger main	Log
2826596	dotnet-performance	Performance nativeaot_scenarios windows Win11 x64 Viper 9.0	Log
2826695	dotnet-maui	Blazor macOS	Log
2826495	dotnet-maui	Build macOS	Log
1187787	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2826619	dotnet-razor	macOS release	Log
2826510	dotnet-performance	Performance scenarios windows Win11 x64 Viper main	Log
2826419	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2826338	dotnet-maui	Blazor macOS	Log
1187347	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1187222	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50685
1186956	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1186904	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121124
2825591	dotnet-maui	Samples macOS	Log
2825602	dotnet-machinelearning	MacOS	Log
1186386	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2825001	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2824643	dotnet-install-scripts	MacOSTests	Log
1185374	dotnet/roslyn	Build_macOS_Debug
2824061	dotnet-machinelearning	MacOS	Log
2824037	dotnet-machinelearning	MacOS_Apple_Silicon	Log
1184241	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1184076	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#16241
1183918	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121051
1183911	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120640
1183956	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#121030
1183924	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
2823277	dotnet-performance	Performance roslyn ubuntu 2204 arm64 Ampere 8.0	Log
1183742	dotnet/performance	Performance powershell windows 24H2 x64 Open 8.0	Log
1183711	dotnet/performance	Performance powershell windows 24H2 x64 Open 8.0	Log
2823136	dotnet-install-scripts	MacOSTests	Log
2823105	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2822972	dotnet-machinelearning	MacOS_Apple_Silicon	Log
1183391	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#120303
1183077	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1182956	dotnet/runtime	osx-x64 release Runtime_Release	Log
1182175	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#120958
1182128	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#118932
2822163	dotnet-machinelearning	MacOS_Apple_Silicon	Log
2821855	dotnet-sdk	Darwin	Log	#54569
2821780	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2821721	dotnet-machinelearning	MacOS_Apple_Silicon	Log
2821714	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1181505	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#120885
2821555	dotnet-machinelearning	MacOS	Log
1181457	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT_RuntimeTests_AppSandbox	Log
2821405	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1180939	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1180693	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#120926
1180676	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51382
1180611	dotnet/runtime	osx-x64 Debug PALTests	Log
2820698	dotnet-aspire	osx_arm64	Log
1180297	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2820521	dotnet-aspire	osx_x64	Log
1179506	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2819746	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2819649	dotnet-performance	Performance sdk_scenarios windows Win11 x86 Viper main	Log
1179335	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2819529	dotnet-sdk	TestBuild: macOS (x64)	Log	#54383
2819428	dotnet-install-scripts	MacOSTests	Log
2819249	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2819151	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Release	Log
1178966	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2818877	dotnet-sdk	AoT: macOS (x64)	Log	#54417
1178788	dotnet/runtime	osx-x64 Debug PALTests	Log
2818939	dotnet-aspire	osx_x64	Log
1178614	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#119432
1178554	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120853
1178493	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50749
1178488	dotnet/runtime	osx-x64 Debug PALTests	Log
1177919	dotnet/runtime	osx-x64 Debug PALTests	Log
1178298	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120281
1178307	dotnet/runtime	osx-x64 Debug PALTests	Log
2818614	dotnet-sdk	Darwin	Log	#54377
Displaying 100 of 432 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	44	432

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.