[dfrankland]

Stacked on #1254, which is stacked on #1236.

upload and test now take --summary-output-file <path> (or TRUNK_SUMMARY_OUTPUT_FILE). The summary is also written to that file as JSON, for a step that comments on the pull request. The terminal output doesn't change. The RSpec gem writes the same file when TRUNK_SUMMARY_OUTPUT_FILE is set. trunk-analytics-cli summary-output-schema prints the file's JSON Schema. This replaces #1122.

What changes.

• The flag is in UploadArgs, so both commands get it.
• The file holds the UploadReport from Build the upload summary as data, separate from rendering #1236 with schemaVersion: 1 and type: "UPLOAD". When the run fails outright, it holds the ErrorSummary instead, with type: "ERROR" (report/json.rs).
• Each entry in validations has Build the validate summary as data, separate from rendering #1254's shape: parseError is {"error": …, "causes": […]}, and testSuiteCount and testCaseCount are included (null when the file couldn't be parsed).
• Enum values are SCREAMING_SNAKE_CASE (type, outcome, kind, quarantining), and keys are camelCase, as in protobuf's JSON mapping. A group's key is {"type": "FILE", "name": …}, {"type": "SUITE", "name": …} or {"type": "OTHER"}.
• The JSON is written to a temporary file next to the target, which is then renamed into place. Missing parent directories are created.
• dryRun says the run uploaded nothing (--dry-run). A dry run still checks quarantining, so its summary has quarantine results but no upload ID or upload link.
• quarantining says whether failed tests could be checked for quarantining, and if not, why: CHECKED, NOT_NEEDED, DISABLED_VIA_CLI (--disable-quarantining), DISABLED_VIA_API (turned off for the repo in Trunk) or LOOKUP_FAILED. Telemetry's quarantine query result now comes from the same status.
• On success, "Summary written to " follows the summary on stderr. If the write fails, a warning goes there instead and the exit code doesn't change. The warning is also logged, but logs only show with -v.
• summary-output-schema prints a JSON Schema that schemars generates from the report types. Doc comments on the report fields become its descriptions. It's generated for serialization, so fields that are always written, even as null, are required.
• The RSpec gem (test_report's publish) writes the file after uploading. Its messages go to stderr, because the gem's own logs only reach the console at error level.
  • The gem's file has no exitCode, because RSpec decides the exit code. UploadCommand::RSpec leaves it unset, and the field is skipped when unset.
  • The upload trusts the gem's quarantine results without looking them up again, so quarantining comes from the gem's own lookup.
  • ErrorReport::unlogged builds the error object without logging it, since the gem already sends that error to Sentry.

Shape (shortened):

{
  "schemaVersion": 1,
  "type": "UPLOAD",
  "error": null,
  "dryRun": false,
  "bundleUploadId": "4b1d2c3e-…",
  "uploadUrl": null,
  "validations": [{ "file": "junit.xml", "parseError": null, "errors": [], "warnings": ["…"] }],
  "counts": { "total": 3, "passed": 1, "failed": 2, "quarantined": 0, "passRatio": 0.3333333333333333 },
  "quarantining": "CHECKED",
  "quarantined": [],
  "failures": [
    { "key": { "type": "FILE", "name": "src/math.rs" },
      "tests": [{ "name": "subtracts", "suite": "MathTests", "className": "MathTests", "file": "src/math.rs",
                  "id": "1a028561-…", "variant": "", "failureMessage": "assertion failed",
                  "quarantined": false, "link": "https://app.trunk.io/…/flaky-tests/test/1a028561-…?repo=…" }] }
  ],
  "outcome": "FAILURES_NOT_QUARANTINED",
  "exitCode": 1,
  "notices": []
}

Differences from #1122.

• No separate summary math. The counts, groups and links are the ones the terminal shows.
• Each failure has only the bundle::Test fields plus its link: no line number, duration or Bazel target.
• Field names follow the report (counts.passed, link, suite, quarantined), and failures are grouped, with quarantined ones in their own list.

Worth a look.

• The RSpec gem doesn't honor TRUNK_DISABLE_QUARANTINING during the run. It still checks and quarantines failures, and only the upload sees the setting. That's existing behavior and isn't changed here; the gem's quarantining reports what it actually did.
• variant is "" rather than null when a run has no variant, because that's what bundle::Test holds.
• On an error, exitCode is the code the error calls for, and null means a run that fails outright exits 0. The schema describes this.

Testing.

• cargo test -p trunk-analytics-cli --lib --test summary --test test --test validate: all pass. New tests check the exact JSON for an upload report and for an error, test writing the file via the env var, and summary-output-schema printing the schema.
• cargo test -p trunk-analytics-cli --test upload: all pass except the three Swift xunit tests, which need a Swift toolchain and fail the same way on the base branch locally. New tests check that the file is written (to a nested path), that a dry run checks quarantining and writes dryRun: true without uploading, that an outright failure writes the error object, that --disable-quarantining writes DISABLED_VIA_CLI, and that a failed write keeps the exit code and leaves no temporary file. A summary test covers every quarantining value.
• Files from hand runs, one with failures and one from a failed run, validate against the printed schema with check-jsonschema.
• cargo test -p test_report: all pass. New tests check that the gem writes the summary without exitCode, that TRUNK_DRY_RUN writes dryRun: true, writes the error object when the run fails outright, and reports LOOKUP_FAILED and DISABLED_VIA_API from its own lookup.
• A real RSpec run through the built gem, with one passing and one failing example, wrote the file and printed "Summary written to …" on stderr. The gem's own suite (rake test) passes.
• cargo clippy -p trunk-analytics-cli -p test_report --all-targets: no new warnings.

🤖 Generated with Claude Code

[trunk-staging-io]

   

Failed Test	Failure Summary	Logs
pending_quarantine_test should be quarantined when run with variant	A test marked as pending was expected to fail but unexpectedly passed.	Logs ↗︎
variant_quarantine_test should be quarantined when run with variant	The quarantined variant test failed because it expected 2 + 2 to equal 5, but the result was 4.	Logs ↗︎
multiple_exception_error_test nested context with multiple failures multiple expectations that fail	The failure aggregation test reports three failed expectations and an additional RuntimeError raised inside the aggregation block.	Logs ↗︎
multiple_exception_error_test with before and after hook failures test execution with hook errors	The test failed because both its before hook and after hook raised errors, preventing RSpec from matching the failure to a backtrace line.	Logs ↗︎

... and 3 more

View Full Report ↗︎ ⋅ Docs

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.78947% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.45%. Comparing base (a408da5) to head (ad00eed).

Files with missing lines	Patch %	Lines
test_report/src/report.rs	84.37%	5 Missing ⚠️
cli/src/summary_output/text.rs	92.85%	2 Missing ⚠️
cli/src/summary_output/json.rs	96.66%	1 Missing ⚠️

Additional details and impacted files

@@                                               Coverage Diff                                                @@
##           dylan/trunk-22402-build-the-validate-summary-as-data-separate-from-rendering    #1246      +/-   ##
================================================================================================================
+ Coverage                                                                         82.12%   85.45%   +3.33%     
================================================================================================================
  Files                                                                                83       84       +1     
  Lines                                                                             18654    18927     +273     
================================================================================================================
+ Hits                                                                              15320    16175     +855     
+ Misses                                                                             3334     2752     -582     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[trunk-io]

This pull request was merged into main as part of stacked PR 1265.