The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1118719	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#50091
2768182	dotnet-orleans	Test	Log
1118362	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79844
2767941	dotnet-dotnet	Build (centOSStreamContainer)	Log
2768113	dotnet-dotnet	Build (azurelinuxX64CrossContainer)	Log
2768078	dotnet-dotnet-docker	🟦 Set Image Info Path Vars	Log
1115895	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#50091
2767995	dotnet-dotnet-docker	🛡 Guardian: CodeQL - TypeScript and JavaScript	Log
2767981	dotnet-dotnet-docker	🛡 Guardian: Code Inspector	Log
1118071	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118507
2767885	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
2767914	dotnet-dotnet-docker	🛡 Guardian: CodeQL - TypeScript and JavaScript	Log
2767894	dotnet-performance	Download PDN	Log
2767816	dotnet-runtime	Run performance job script	Log
2767776	dotnet-dotnet	Build (centOSStreamContainer)	Log
1117610	dotnet/runtime	Send to Helix	Log
1117687	dotnet/runtime	Send to Helix	Log
1117683	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118525
1117686	dotnet/runtime	Send to Helix	Log
1117647	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118523
1117671	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118524
1117650	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118523
1117696	dotnet/runtime	Send to Helix	Log
1117612	dotnet/runtime	Send tests to Helix (Unix)	Log
1117655	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117593
2767693	dotnet-dotnet	Build	Log
1117611	dotnet/runtime	Send tests to Helix (Windows)	Log
1117472	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117502
1117517	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116419
1117484	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118507
1117495	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#50131
2767439	dotnet-installer	Build	Log	#52077
1117273	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79837
2767372	dotnet-orleans	Test	Log
2767300	dotnet-diagnostics	Package, Sign, and Publish	Log
1117124	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79834
1117103	dotnet/roslyn	Run Unit Tests	Log
2767024	dotnet-dotnet	Auto-baselining mode and SDL pre-checks (1ES PT)	Log
1116733	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118434
1116603	dotnet/runtime	Send to Helix	Log
2766911	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
2766861	dotnet-runtime	Run performance job script	Log
2766790	dotnet-dotnet	Build	Log
2766832	dotnet-dotnet	Agent Pool Validation	Log
2766439	dotnet-dotnet	Build	Log
2766711	dotnet-dotnet	Build	Log
1116417	dotnet/roslyn	Run Unit Tests	Log
2766589	dotnet-aspnetcore	🛡 Generate SBoM Manifest	Log
2766692	dotnet-dotnet-monitor	Build	Log
2766678	dotnet-dotnet-monitor	Build	Log
2766637	dotnet-sdk	🛡 Guardian: Post Analysis	Log	#51985
2766661	dotnet-sdk	Run AoT Tests in Helix	Log	#52072
2766543	dotnet-vscode-csharp	🛡 Component Governance	Log
2766485	dotnet-diagnostics	Package, Sign, and Publish	Log
1116108	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#118464
2766500	dotnet-roslyn	Test	Log
2766502	dotnet-arcade	🔒 Start Network Isolation	Log
2766473	dotnet-sdk	Run AoT Tests in Helix	Log	#52072
2766474	dotnet-sdk	Run Tests in Helix	Log	#52071
2766443	dotnet-roslyn	Test	Log
2766397	dotnet-sdk	Run AoT Tests	Log	#51985
2766398	dotnet-sdk	Run TestBuild Tests	Log	#52073
2766356	dotnet-dotnet-monitor	Build	Log
2766317	dotnet-sdk	Run AoT Tests in Helix	Log	#52072
2766222	dotnet-dotnet-monitor	Get Release Version	Log
2766169	dotnet-dotnet	Prepare BuildLogs staging directory (centOSStreamContainer)	Log
2766156	dotnet-dotnet	🛡 Guardian: Post Analysis	Log
2766131	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
1115541	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118161
1115512	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49728
1115359	dotnet/runtime	Send tests to Helix (Unix)	Log
1115281	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62903
1115332	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117593
1115290	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118432
1115257	dotnet/runtime	Send to Helix	Log
1115300	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118434
2766018	dotnet-dotnet	Build (centOSStreamContainer)	Log
2766008	dotnet-sdk	Run AoT Tests in Helix	Log	#52072
2766006	dotnet-sdk	Run Tests in Helix	Log	#52070
1115100	dotnet/aspnetcore	Build JS	Log	dotnet/aspnetcore#62885
2765810	dotnet-roslyn	Run Unit Tests	Log
1115069	dotnet/roslyn	Run Integration Tests	Log
1114914	dotnet/runtime	Send to Helix	Log	dotnet/runtime#111178
2765731	dotnet-dotnet-monitor	Get Release Version	Log
2765668	dotnet-orleans	Test	Log
1114835	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#118415
1114823	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#49942
1114687	dotnet/runtime	Send to Helix	Log
1114865	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118338
2765665	dotnet-aspnetcore	Run build.cmd helix target	Log
1114850	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118418
1114858	dotnet/runtime	Send to Helix	Log
1114800	dotnet/aspnetcore	Run Quarantined E2E tests	Log	dotnet/aspnetcore#63135
2765656	dotnet-performance	Send job to Helix (Unix)	Log
1114825	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118181
1114878	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50068
1114809	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118416
2765504	dotnet-roslyn	Run Unit Tests	Log
1114621	dotnet/runtime	Send tests to Helix (Unix)	Log
2755501	dotnet-installer	Generate SBOM manifest	Log
Displaying 100 of 1270 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	176	1270

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez