slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2826619	dotnet-razor	macOS release	Log
2826495	dotnet-maui	Blazor macOS	Log
2826419	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2826338	dotnet-maui	Blazor macOS	Log
1187347	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1187222	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50685
1186956	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1186904	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121124
2825591	dotnet-maui	Samples macOS	Log
2825602	dotnet-machinelearning	MacOS	Log
1186386	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2825001	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2824643	dotnet-install-scripts	MacOSTests	Log
1185374	dotnet/roslyn	Build_macOS_Debug
2824061	dotnet-machinelearning	MacOS	Log
2824037	dotnet-machinelearning	MacOS_Apple_Silicon	Log
1184241	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1184076	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#16241
1183918	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121051
1183911	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120640
1183956	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#121030
1183924	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
2823277	dotnet-performance	Performance roslyn ubuntu 2204 arm64 Ampere 8.0	Log
1183742	dotnet/performance	Performance powershell windows 24H2 x64 Open 8.0	Log
1183711	dotnet/performance	Performance powershell windows 24H2 x64 Open 8.0	Log
2823136	dotnet-install-scripts	MacOSTests	Log
2823105	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2822972	dotnet-machinelearning	MacOS_Apple_Silicon	Log
1183391	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#120303
1183077	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1182956	dotnet/runtime	osx-x64 release Runtime_Release	Log
1182175	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#120958
1182128	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#118932
2822163	dotnet-machinelearning	MacOS_Apple_Silicon	Log
2821855	dotnet-sdk	Darwin	Log	#54569
2821780	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2821721	dotnet-machinelearning	MacOS_Apple_Silicon	Log
2821714	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1181505	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#120885
2821555	dotnet-machinelearning	MacOS	Log
1181457	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT_RuntimeTests_AppSandbox	Log
2821405	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1180939	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1180693	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#120926
1180676	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51382
1180611	dotnet/runtime	osx-x64 Debug PALTests	Log
2820698	dotnet-aspire	osx_arm64	Log
1180297	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2820521	dotnet-aspire	osx_x64	Log
1179506	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2819746	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2819649	dotnet-performance	Performance sdk_scenarios windows Win11 x86 Viper main	Log
1179335	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2819529	dotnet-sdk	TestBuild: macOS (x64)	Log	#54383
2819428	dotnet-install-scripts	MacOSTests	Log
2819249	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2819151	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Release	Log
1178966	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2818877	dotnet-sdk	AoT: macOS (x64)	Log	#54417
1178788	dotnet/runtime	osx-x64 Debug PALTests	Log
2818939	dotnet-aspire	osx_x64	Log
1178614	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#119432
1178554	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120853
1178493	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50749
1178488	dotnet/runtime	osx-x64 Debug PALTests	Log
1177919	dotnet/runtime	osx-x64 Debug PALTests	Log
1178298	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120281
1178307	dotnet/runtime	osx-x64 Debug PALTests	Log
2818614	dotnet-sdk	Darwin	Log	#54377
2818612	dotnet-sdk	Darwin	Log	#54379
1177647	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120623
1177918	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51331
1177823	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120812
1177851	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#120812
1177820	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120787
1177741	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120810
1177779	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120411
1177945	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120694
1177938	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120281
1177925	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#118270
1177930	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#120516
1177920	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Release	Log
1177923	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1177908	dotnet/runtime	osx-x64 Release CoreCLR_Release	Log
1177901	dotnet/runtime	osx-x64 Release CoreCLR_Release	Log
1177848	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#120840
1177753	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#51277
1177792	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#119492
1177809	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log
1177840	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1177836	dotnet/runtime	osx-x64 Debug PALTests	Log
1177839	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1177832	dotnet/xharness	Helix Tests Build_Debug	Log
1177759	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#120838
1177755	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#118570
1177818	dotnet/sdk	Darwin Build_Release	Log
1177717	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R_CG2 osx x64 checked	Log
1177699	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51319
1177737	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#119432
1177692	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
Displaying 100 of 439 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	38	439

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.