Skip to content

Say when no tests were found but the exit code isn't 0 - #1253

Merged
trunk-io[bot] merged 1 commit into
dylan/trunk-22762-define-each-cli-exit-code-in-one-place-with-itsfrom
dylan/trunk-19074-no-tests-exit-code
Oct 29, 2026
Merged

trunk-io[bot] merged 1 commit into
dylan/trunk-22762-define-each-cli-exit-code-in-one-place-with-itsfrom
dylan/trunk-19074-no-tests-exit-code

Conversation

@dfrankland

@dfrankland dfrankland commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1255.

Part of TRUNK-19074. When the test reports have no tests, the summary printed "No tests were found in the provided test results" and nothing else, even when the run then exits with the test step's non-zero code. A test step that failed before running any tests looked like an empty report, not a failure. In the ticket's example, a Playwright global setup timed out.

What changes.

  • When no tests were found and the exit code isn't 0, the summary ends with:
    ⚠️  No tests were found in the provided test results, but non zero exit code provided: 1
    The test step appears to have failed without running tests -- see above for the original failure.
    
    This follows the wording of the "No test failures found, but non zero exit code provided" case next to it.
  • With exit code 0, the single "No tests were found" line is unchanged.
  • The JSON summary doesn't change. It already has outcome: "NO_TESTS_FOUND" and exitCode. The planned hidden github-annotations command will read those from the summary file and use the same wording, so the analytics-uploader action never parses the JSON itself.

Which exit code. The new line uses report.cli_exit_code(), like the neighbouring outcomes, and checks it with #1255's ExitCode::is_success(). Wherever the outcome is printed, that's the test run's code, for both upload and test:

  • An error that carries its own exit code (unauthorized, git repository, interrupted, all 70) blocks quarantining, so render_upload stops before the outcome.
  • An error without one leaves upload's exit code as the test run's.

Testing.

  • cargo nextest run -p trunk-analytics-cli: 146 of 149 pass. The three Swift xunit tests in --test upload fail the same way on the base branch locally ("helloworld() got no file from its declaration"), as noted on Add --summary-output-file to upload and test #1246.
  • New summary test for zero tests with exit code 1. The existing zero-tests test still covers exit code 0.
  • test's exit 128 end-to-end test now checks for the new lines. upload's zero-tests test (exit 0) checks they're absent.
  • cargo fmt --check passes. cargo clippy -p trunk-analytics-cli --tests: no warnings in the changed code.

🤖 Generated with Claude Code

@codecov-commenter

codecov-commenter commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.63636% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.50%. Comparing base (3dea134) to head (46db108).

Files with missing lines Patch % Lines
cli/src/summary_output/text.rs 63.63% 4 Missing ⚠️
Additional details and impacted files
@@                                          Coverage Diff                                          @@
##           dylan/trunk-22762-define-each-cli-exit-code-in-one-place-with-its    #1253      +/-   ##
=====================================================================================================
- Coverage                                                              85.52%   85.50%   -0.02%     
=====================================================================================================
  Files                                                                     85       85              
  Lines                                                                  19037    19046       +9     
=====================================================================================================
+ Hits                                                                   16282    16286       +4     
- Misses                                                                  2755     2760       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@trunk-staging-io

trunk-staging-io Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

Failed Test Failure Summary Logs
pending_quarantine_test should be quarantined when run with variant A test marked as pending was expected to fail but unexpectedly passed. Logs ↗︎
variant_quarantine_test should be quarantined when run with variant The quarantined variant test failed because it expected 2 + 2 to equal 5, but the result was 4. Logs ↗︎
multiple_exception_error_test nested context with multiple failures multiple expectations that fail The failure aggregation test reports three failed expectations and an additional RuntimeError raised inside the aggregation block. Logs ↗︎
multiple_exception_error_test with before and after hook failures test execution with hook errors The test failed because both its before hook and after hook raised errors, preventing RSpec from matching the failure to a backtrace line. Logs ↗︎

... and 3 more

View Full Report ↗︎ ⋅ Docs

@trunk-io

trunk-io Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@dfrankland
dfrankland force-pushed the dylan/trunk-19074-no-tests-exit-code branch from 1d03ea0 to 9be903e Compare October 27, 2026 20:42
@dfrankland
dfrankland changed the base branch from dylan/trunk-22403-add-summary-output-file-to-upload-and-test to dylan/trunk-22762-define-each-cli-exit-code-in-one-place-with-its October 27, 2026 20:43
@dfrankland
dfrankland added this pull request to stack #1268 October 28, 2026 15:54
When the test reports have no tests, the summary only said "No tests
were found in the provided test results", even when the run exits with
the test step's non-zero code. A test step that failed before running
any tests, such as a Playwright global setup that timed out, looked
like an empty report rather than a failure.

The summary now names the exit code and points at the step's own
output:

  ⚠️  No tests were found in the provided test results, but non zero exit code provided: 1
  The test step appears to have failed without running tests -- see above for the original failure.

With exit code 0, the single line is unchanged. The code is the one
the CLI exits with, which is the test run's code wherever the outcome
is printed: an error with its own exit code stops the summary before
the outcome. The JSON summary already has `outcome: NO_TESTS_FOUND`
and `exitCode`, so it doesn't change.

Refs TRUNK-19074

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfrankland
dfrankland force-pushed the dylan/trunk-19074-no-tests-exit-code branch from 9be903e to 46db108 Compare October 28, 2026 21:28
@trunk-io
trunk-io Bot merged commit efce013 into main Oct 29, 2026
74 of 86 checks passed
@trunk-io

trunk-io Bot commented Oct 29, 2026

Copy link
Copy Markdown

This pull request was merged into main as part of stacked PR 1265.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants