The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2688901	dotnet-dotnet	Find associated build	Log
2688832	dotnet-aspnetcore	🛡 Generate SBoM Manifest	Log
2688756	dotnet-runtime	Build product	Log
2688720	dotnet-arcade-services	Run E2E tests	Log
2688644	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
1018143	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114734
1018093	dotnet/runtime	Send to Helix	Log	dotnet/runtime#110472
2688618	dotnet-runtime	Run performance job script	Log
2688539	dotnet-runtime	Send job to Helix (Windows)	Log
1018005	dotnet/runtime	AOT compile CoreCLR tests	Log
2688012	dotnet-arcade-services	Run E2E tests	Log
2687885	dotnet-arcade-services	Run E2E tests	Log
2688359	dotnet-runtime	Send job to Helix (Windows)	Log
1017759	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114048
1017754	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114675
1017742	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114609
1017726	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114725
1017731	dotnet/runtime	Send job to Helix (Unix)	Log
2688257	dotnet-runtime	Send job to Helix (Unix)	Log
1017647	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114719
1017703	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114649
1017682	dotnet/roslyn	Run Integration Tests	Log
1017573	dotnet/runtime	Publish	Log	dotnet/runtime#114285
1017611	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114718
1015815	dotnet/runtime	Publish	Log	dotnet/runtime#113907
1017655	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114585
1017651	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114722
1017492	dotnet/runtime	Send tests to Helix (Unix)	Log
1017534	dotnet/runtime	Publish	Log	dotnet/runtime#114688
1017606	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114716
2688224	dotnet-runtime	Publish Build Assets	Log
1017528	dotnet/runtime	Publish	Log	dotnet/runtime#113956
2688205	dotnet-runtime	Publish Browser Wasm Artifacts	Log
1017603	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#4835
2688196	dotnet-dotnet	Download Blob Artifacts	Log
1017554	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113988
2688191	dotnet-sdk	Build (centosStream9)	Log
2688169	dotnet-sdk	🛡 Guardian: Security Analysis Report	Log	#49223
1017549	dotnet/roslyn	Run Integration Tests	Log
2687909	dotnet-runtime	Publish Upload artifacts SuperPMI benchmarks-run_pgo collection	Log
1017473	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114711
1017445	dotnet/runtime	Send to Helix	Log	dotnet/runtime#112129
1017446	dotnet/runtime	Send to Helix	Log	dotnet/runtime#112129
1017171	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114048
1017429	dotnet/runtime	Send to Helix	Log
1017297	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114610
1017421	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114710
1017344	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114495
1017426	dotnet/runtime	AOT compile CoreCLR tests	Log
2687944	dotnet-runtime	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2688090	dotnet-runtime	Run performance job script	Log
1017376	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114707
2688053	dotnet-runtime	Send job to Helix (Windows)	Log
1017370	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114521
1017367	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114523
1017302	dotnet/runtime	Publish	Log	dotnet/runtime#114056
2687996	dotnet-dotnet	Build	Log
1017355	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114660
1017383	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#48488
1017267	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114702
2688044	dotnet-runtime	Publish Build Assets	Log
1017243	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114700
2687701	dotnet-dotnet	Download Blob Artifacts	Log
1017380	dotnet/aspire	Build	Log
2687943	dotnet-runtime	🔒 Publish intermediate artifacts	Log
1017320	dotnet/runtime	Build	Log	dotnet/runtime#113948
1017319	dotnet/runtime	Build	Log	dotnet/runtime#114675
1017276	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#114649
1017307	dotnet/runtime	Publish Build Assets	Log
1017254	dotnet/runtime	Send to Helix	Log
1017227	dotnet/runtime	Send to Helix	Log
2687995	dotnet-dotnet	Find associated build	Log
1017253	dotnet/runtime	Send to Helix	Log
1017151	dotnet/runtime	Publish	Log	dotnet/runtime#114630
1017225	dotnet/runtime	Send to Helix	Log
2687913	dotnet-runtime	Publish Browser Wasm Artifacts	Log
1017144	dotnet/runtime	Publish	Log	dotnet/runtime#114692
2687872	dotnet-arcade-services	Run E2E tests	Log
1017110	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#114687
2687843	dotnet-runtime	Component Detection	Log
2687801	dotnet-runtime	Send job to Helix (Windows)	Log
2687788	dotnet-performance	Send job to Helix (Windows)	Log
1016978	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114682
1016939	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114676
2687690	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
2687696	dotnet-performance	Run performance job script	Log
1016829	dotnet/runtime	Download unified artifacts	Log	dotnet/runtime#114643
1016775	dotnet/runtime	AOT compile CoreCLR tests	Log
2687534	dotnet-runtime	Send job to Helix (Windows)	Log
2687335	dotnet-runtime	🛡 Guardian: 1ES Secret Scanning (SPMI)	Log
1016457	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#114667
1016456	dotnet/runtime	Send to Helix	Log
1016453	dotnet/runtime	Send job to Helix (Unix)	Log
1016432	dotnet/runtime	Send tests to Helix (Unix)	Log
1016396	dotnet/runtime	Send to Helix	Log
1016410	dotnet/runtime	Send job to Helix (Unix)	Log
2687338	dotnet-runtime	Send job to Helix (Windows)	Log
1016395	dotnet/aspire	Run helix tests	Log
1016398	dotnet/runtime	Send job to Helix (Unix)	Log
1016360	dotnet/roslyn	Run Integration Tests	Log
Displaying 100 of 1152 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
65	334	1152

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)