The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2814131	dotnet-runtime	Send job to Helix (Unix)	Log
2814087	dotnet-runtime	Send tests to Helix (Unix)	Log
2814072	dotnet-runtime	Send job to Helix (Unix)	Log
1171929	dotnet/roslyn	Test	Log	dotnet/roslyn#80669
1171922	dotnet/roslyn	Test	Log	dotnet/roslyn#80668
1171915	dotnet/roslyn	Test	Log	dotnet/roslyn#80667
1171841	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120633
1171659	dotnet/runtime	Send to Helix	Log
2813992	dotnet-aspnetcore	Run build.cmd helix target	Log
2813996	dotnet-aspnetcore	Run build.cmd helix target	Log
2813991	dotnet-performance	Send job to Helix (Unix)	Log
2813988	dotnet-aspnetcore	Run build.cmd helix target	Log
1171735	dotnet/runtime	Send to Helix	Log
1171741	dotnet/runtime	Send to Helix	Log
2813946	dotnet-runtime	Send job to Helix (Unix)	Log
1171684	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120633
2813882	dotnet-runtime	Send job to Helix (Unix)	Log
1171118	dotnet/runtime	Send to Helix	Log
2813764	dotnet-runtime	Send job to Helix (Windows)	Log
1170844	dotnet/runtime	Send tests to Helix (Unix)	Log
2813763	dotnet-runtime	Send job to Helix (Unix)	Log
2813748	dotnet-aspnetcore	Run build.cmd helix target	Log
2813736	dotnet-aspnetcore	Run build.cmd helix target	Log
2813740	dotnet-performance	Send job to Helix (Unix)	Log
2813742	dotnet-install-scripts	Run Tests	Log
1171424	dotnet/runtime	Send to Helix	Log
1171026	dotnet/runtime	Send tests to Helix (Unix)	Log
1171078	dotnet/runtime	Send to Helix	Log
1171074	dotnet/runtime	Send to Helix	Log
1170856	dotnet/runtime	Send to Helix	Log
2813498	dotnet-runtime	Send to Helix	Log
1171012	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120041
1170977	dotnet/aspnetcore	Run build.cmd helix target	Log
2813416	dotnet-sdk	Run AoT Tests	Log	#54241
1170786	dotnet/runtime	Build product	Log	dotnet/runtime#119432
1170846	dotnet/runtime	Send to Helix	Log
1170840	dotnet/runtime	Send tests to Helix (Unix)	Log
2813263	dotnet-sdk	Run Tests in Helix	Log	#54233
2813264	dotnet-sdk	Run Tests in Helix	Log	#54231
2813261	dotnet-roslyn	Run Unit Tests	Log
2813262	dotnet-sdk	Run Tests in Helix	Log	#54230
2813240	dotnet-sdk	Run AoT Tests	Log	#54236
2813245	dotnet-sdk	Run AoT Tests	Log	#54241
1170680	dotnet/runtime	Send to Helix	Log
2813239	dotnet-sdk	Build	Log	#54237
2813222	dotnet-sdk	Run Tests in Helix	Log	#54230
2813224	dotnet-sdk	Run Tests in Helix	Log	#54233
2813223	dotnet-sdk	Run Tests in Helix	Log	#54231
1170695	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#120118
1170379	dotnet/runtime	Send tests to Helix (Unix)	Log
1170679	dotnet/runtime	Send to Helix	Log
2813154	dotnet-sdk	Run TestBuild Tests	Log	#53989
2813153	dotnet-sdk	Run AoT Tests	Log	#54089
2813165	dotnet-efcore	Send job to helix	Log
2813149	dotnet-sdk	Run Tests in Helix	Log	#53890
2813148	dotnet-sdk	Run Tests in Helix	Log	#53891
2813152	dotnet-sdk	Build	Log	#54089
2813147	dotnet-sdk	Run Tests in Helix	Log	#53892
1170634	dotnet/runtime	Build	Log	dotnet/runtime#119432
2813051	dotnet-dotnet	Build (centOSStreamContainer)	Log
1170515	dotnet/runtime	Send tests to Helix (Unix)	Log
1170538	dotnet/runtime	Send tests to Helix (Unix)	Log
2813058	dotnet-aspnetcore	Generate delegation SAS Token for dotnetbuilds/internal	Log	#54011
1170518	dotnet/runtime	Send tests to Helix (Unix)	Log
1165389	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120396
1170356	dotnet/runtime	Send to Helix	Log
2812888	dotnet-dotnet	Checkout dotnet-dotnet@dev/mthalman/2xx-sbrp to s	Log
2812880	dotnet-dotnet	Auto-baselining mode and SDL pre-checks (1ES PT)	Log
2812870	dotnet-dotnet	Build (centOSStreamContainer)	Log
1170355	dotnet/runtime	Build product	Log
2812820	dotnet-source-indexer	Build source index	Log
1170163	dotnet/roslyn	Publish BuildValidator debug outputs	Log	dotnet/roslyn#80631
1170206	dotnet/runtime	Send to Helix	Log
1170211	dotnet/runtime	Send to Helix	Log
1170233	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120462/merge to s	Log	dotnet/runtime#120462
1170232	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120462/merge to s	Log	dotnet/runtime#120462
1170195	dotnet/runtime	Build product	Log	dotnet/runtime#120330
1170109	dotnet/runtime	Send to Helix	Log
2812537	dotnet-arcade-validation	Run Tests (Windows)	Log
2812450	dotnet-roslyn	Run Unit Tests	Log
2812442	dotnet-roslyn	Run Unit Tests	Log
1169720	dotnet/runtime	Send to Helix	Log
1169765	dotnet/runtime	Build product	Log	dotnet/runtime#119432
1169766	dotnet/runtime	Build	Log	dotnet/runtime#119432
2812321	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
2812346	dotnet-sdk	Run TestBuild Tests	Log	#54189
1169719	dotnet/runtime	AOT compile CoreCLR tests	Log
1169718	dotnet/runtime	Send to Helix	Log
2812297	dotnet-efcore	Send job to helix	Log
1169618	dotnet/runtime	Send to Helix	Log
1169629	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120409
1169598	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120408
2812201	dotnet-arcade-services	Build docker image	Log
2811972	dotnet-dotnet	🛡 Guardian: BinSkim 4.4.6	Log
1169451	dotnet/runtime	Build product	Log	dotnet/runtime#119432
1169448	dotnet/runtime	AOT compile CoreCLR tests	Log
1169393	dotnet/runtime	Send tests to Helix (Unix)	Log
2812088	dotnet-efcore	Send job to helix	Log
2812072	dotnet-efcore	Build	Log
1169450	dotnet/runtime	Build product	Log	dotnet/runtime#119432
Displaying 100 of 1138 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
17	208	1138

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez