slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2840054	dotnet-install-scripts	MacOSTests	Log
1204493	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51747
1204086	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121652
1204146	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#121656
1204083	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#121653
1204029	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#119432
1204077	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
2839504	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#55295
1204014	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121650
1203955	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121648
1203974	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121457
2839448	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1203995	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#121649
1203929	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121640
1203973	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#121563
1203968	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#121621
1203910	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121646
1203882	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121643
2839349	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1203863	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1203906	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT_RuntimeTests_AppSandbox	Log
1203804	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1203764	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121637
1203851	dotnet/runtime	ios-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#121598
1203811	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1203760	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121603
1203770	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121639
1203814	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono_AppSandbox	Log
1203586	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121508
1203607	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#116203
1203597	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121633
1203641	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#121417
1203539	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121634
1203410	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121630
2839094	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1203360	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121341
1203337	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121625
1203340	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121626
1203379	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#121628
1203317	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121622
1203305	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#120330
2839007	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1203272	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119203
1203263	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121621
1203192	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121406
1203210	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121366
1203198	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121583
1203171	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#120667
1203229	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#121270
1202368	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1492
1203238	dotnet/runtime	osx-x64 Release CoreCLR_Release	Log
1203153	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121616
1202054	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121407
1203188	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#121442
1203149	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121380
1203126	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121614
1203111	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121498
1203117	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121610
1203104	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121613
1203160	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#51736
1203102	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log
1203151	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121380
1203144	dotnet/runtime	osx-x64 Debug PALTests	Log
1203141	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono	Log
1203139	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1203090	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121609
1203029	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51734
1203133	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51397
1202955	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#51702
1203071	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#121612
1202952	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51705
1203009	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#51730
1202950	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51707
1203063	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#121366
2838725	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1203023	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#121609
1202993	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log
1203008	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#51704
1203007	dotnet/sdk	Darwin Build_Release	Log
1202725	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#121366
1202963	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#116203
1202996	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono_RuntimeTests	Log
2838640	dotnet-dotnet-monitor	Test MacOS arm64 Release	Log
1202920	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#121606
1202736	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#121334
1202908	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121515
1202959	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#51714
1202956	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51633
1202866	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121605
1202929	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51733
1202916	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51732
1202907	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51731
1202902	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51565
1202903	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#51729
1202855	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#121566
1202758	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121373
1202788	dotnet/runtime	ios-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#121603
2838517	dotnet-performance	Performance imagesharp windows Win11 x64 Viper main	Log
1201261	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#121186
1202703	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#121583
Displaying 100 of 525 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	324	525

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.