[CI] Avoid generic build failures for reported test failures

[mdh1418]

Summary

Make published Azure Pipelines test results the authoritative Build Analysis signal for ordinary xUnit test failures.

• Suppress Arcade's generic MSBuild error when xUnit successfully produces test results.
• Publish failed-test metadata without making  PublishTestResults  fail merely because tests failed.
• Continue failing for missing, malformed, or assembly-level test results.
• Publish partial test results when an earlier build or test-infrastructure step fails.
• Explicitly mark the Windows ARM64 cross-build as a configuration that does not run tests.

Motivation

Build Analysis can match individual test failures against known issues, but diagnostics currently also reports the enclosing test step as a generic  Build failed with exit code 1  or  Bash exited with code 1  failure. Making  PublishTestResults  fail on failed tests similarly adds the generic  There are one or more test failures detected in result files  error. Those generic failures cannot safely be matched to a known test issue because they can represent unrelated failures.

The runtime pipeline leaves individual published test results for Build Analysis to evaluate instead of requiring a second generic failure from the test runner. This change applies equivalent behavior to diagnostics' Arcade-based local test execution while preserving real publication failures.

Behavior

Ordinary failed tests now:

1. Produce xUnit XML without failing the MSBuild invocation.
2. Are published by  PublishTestResults@2  without making that task fail solely because tests failed.
3. Are evaluated individually by Build Analysis, which rejects unmatched failures and can accept failures matched to known issues.

Missing results and genuine result-publication failures still fail  PublishTestResults . Runner crashes, timeouts, malformed XML, assembly-level xUnit errors, and genuine build/script failures continue to fail independently.

[Copilot]

Pull request overview

This PR adjusts CI/test signaling so that, for ordinary xUnit failures, Azure Pipelines test-result publishing becomes the authoritative failure mechanism (instead of also reporting a generic build/script failure), while still failing the build for runner/result corruption scenarios.

Changes:

• Adds an MSBuild post-test validation target to ensure suppressed test failures still produce valid/usable xUnit result XML and to fail for missing/assembly-level error conditions.
• Updates the Azure Pipelines build template to (a) suppress “error on test failure” during the build/test step and (b) fail via PublishTestResults@2 for test failures, while still publishing partial results on build/test step failures.
• Marks the Windows ARM64 cross-build leg as not running tests (runTests: false).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
eng/Tests.targets	Adds MSBuild validation logic intended to catch missing/malformed/assembly-error test results when test-failure suppression is enabled.
eng/pipelines/build.yml	Adjusts CI step env/conditions and splits test publishing into “partial on failed build” vs “authoritative on succeeded build”.
diagnostics.yml	Disables test execution for Windows ARM64 Release cross-build configuration.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[mdh1418]

Testing by forcing a known test error - Build Analysis passed

[mdh1418]

This doesn't quite work.

With failTaskOnFailedTests: false, the underlying AzDO build is successful.
Build Insights currently preserves that successful build status even when it discovers an unmatched test failure. Therefore:

┌───────────────────────────────┬─────────┬────────────────────────────────────────────────────────┐
│ Test result                   │ Desired │ Current PR                                             │
├───────────────────────────────┼─────────┼────────────────────────────────────────────────────────┤
│ All tests pass                │ Green   │ Green                                                  │
├───────────────────────────────┼─────────┼────────────────────────────────────────────────────────┤
│ Only known failures           │ Green   │ Green                                                  │
├───────────────────────────────┼─────────┼────────────────────────────────────────────────────────┤
│ Any unknown failure           │ Red     │ Green, although Build Analysis reports the failure     │
├───────────────────────────────┼─────────┼────────────────────────────────────────────────────────┤
│ Runner/infrastructure failure │ Red     │ Red                                                    │
└───────────────────────────────┴─────────┴────────────────────────────────────────────────────────┘

If we restore failTaskOnFailedTests: true, unknown failures become red again, but known failures also remain red because the generic PublishTestResults error cannot be correlated with the known test. Thus, diagnostics currently has an unavoidable choice between false positives and false negatives.

Two likely solutions

1. Migrate diagnostics tests to Helix. Helix separates reported test failures from process failures and supplies a test-aware failed-build carrier that Build Analysis knows how to correlate. Aligns diagnostics with runtime. Convert all the test legs to Helix #165
2. Modify Build Insights. Allow an unmatched published test to turn an otherwise-successful build analysis red, while leaving builds with only known test failures green. Diagnostics could continue running tests locally.