Read E2E logs and fail the job if no tests matched the filter

[ilonatommy]

Trying to avoid the situation we faced with #62223

Description

For a few days we were merging PRs that did not run through all the tests. If we had a mechanism that would check the test output for "No test matches the given testcase filter" and would fail the job, we would have caught it earlier.

I am aware that not only E2E tests were affected, template tests or generally, tests running on Helix also had this problem, I don't see an easy way how to use a similar mechanism there.

[ilonatommy]

The alternative way of running triggered the test process. We got an error informing about missing xunit.runner.console package:
artifacts

The application to execute does not exist: '/home/vsts/work/1/s/.packages/xunit.runner.console/2.9.2/tools/netcoreapp2.0/xunit.console.dll'
=== COMMAND LINE ===
"/home/vsts/work/1/s/.dotnet/dotnet" exec --depsfile "/home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.deps.json" --runtimeconfig "/home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.runtimeconfig.json"  "/home/vsts/work/1/s/.packages/xunit.runner.console/2.9.2/tools/netcoreapp2.0/xunit.console.dll" "/home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.dll" -noautoreporters -xml "/home/vsts/work/1/s/artifacts/TestResults/Release/Microsoft.AspNetCore.Components.E2ETests_net10.0_x64.xml" -html "/home/vsts/work/1/s/artifacts/TestResults/Release/Microsoft.AspNetCore.Components.E2ETests_net10.0_x64.html" -notrait "Quarantined=true"  -nocolor > "/home/vsts/work/1/s/artifacts/log/Release/Microsoft.AspNetCore.Components.E2ETests_net10.0_x64.log" 2>&1

Locally we don't experience these problems using eng/build.sh command.

[ilonatommy]

The binlog of dotnet test does not present any abnormalities in comparison the the passing binlog.
The test results contain:
[xUnit.net 00:00:00.17] Skipping: Microsoft.AspNetCore.Components.E2ETests (no reference to xUnit.net)

artifacts

[ilonatommy]

This PR's CI is supposed to fail, to show that the mechanism is working. After merging #62250 it should go green.

[Youssef1313]

Or, migrate to xunit.v3+MTP where zero tests ran is error by default :)

[ilonatommy]

Or, migrate to xunit.v3+MTP where zero tests ran is error by default :)

This is an interesting point. Is #62250 doing that? Does "arcade's default" mean v3?

[ilonatommy]

Or, migrate to xunit.v3+MTP where zero tests ran is error by default :)

This is an interesting point. Is #62250 doing that? Does "arcade's default" mean v3?

Yes, it does:
https://github.com/dotnet/arcade/blob/d3a9e0bf843815be1a2b30895e2060ef73735772/Directory.Packages.props#L20

in this situation, I don't see a point in reading the logs if the newer version will fail it.

[Youssef1313]

Or, migrate to xunit.v3+MTP where zero tests ran is error by default :)

This is an interesting point. Is #62250 doing that? Does "arcade's default" mean v3?

Nope. My PR only uses xunit.runner.visualstudio 3.x, but it's still xunit 2.x.
The update to xunit.v3 is usually more involved.

But happy to help you with the migration. It was done for Aspire (dotnet/aspire#8403 and dotnet/aspire#8498), and in progress for winforms (dotnet/winforms#13540). We also did it for Uno Platform (https://platform.uno/blog/microsoft-contributes-to-uno-platform-dramatically-reducing-testing-time/), and many more internal (and external) repos.

[DeagleGross]

This PR's CI is supposed to fail, to show that the mechanism is working. After merging #62250 it should go green.

it works!

[ilonatommy]

@lewing, what are your thoughts about xunit upgrade? Should we go for it right now or rather grep for now and log that as a future work?

[ilonatommy]

Threshold working as well:

Setting auto merge.