Skip to content

test(server): show what the browser and the failed server printed when the flaky startup checks fail - #223

Merged
GoDiao merged 2 commits into
mainfrom
fix/flaky-server-tests
Oct 31, 2026
Merged

GoDiao merged 2 commits into
mainfrom
fix/flaky-server-tests

Conversation

@GoDiao

@GoDiao GoDiao commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Part of #209.

This does not fix a root cause. I could not reproduce either failure, so I changed the two tests so that the next failure shows what actually happened.

What I established

  • DiagnosticsService test. In run 40896064562 (attempt 1) the second server's stderr was a complete Error\n at startServer (...) stack. The first stack line had no message, although startServer always passes a non-empty message to new Error. So the read was not truncated, and the cause of the empty first line is not established. The test asserted on that first line. The issue says the test failed in 6 of 7 failed runs; it failed once. In the other four runs listed in the issue (39715621460, 39713535549, 39381580281, 39144653837) it passed, and other tests failed.
  • ShowUI browser hook. In run 40905539947 (attempt 1) the hook took 21.2 s: the 20 s timeout plus setup. The hook discarded what Chrome printed, and it had no 'error' listener on the child, so a failed spawn would also surface only as "browser did not start". Which of these happened is not established.

What changed

  • diagnostics-service.test.ts: the test reads the server_uncaught_exception event from diagnostics.jsonl and asserts its summary is Failed to start server. Is port <port> in use?, plus exit code 1, empty stdout and the [Server] Uncaught exception: label on stderr. The assertion is one toMatchObject on { exitCode, stdout, stderr, summary }, so a failure prints stderr. The test no longer requires the message on stderr, because the stack's first line is not a reliable place for it.
  • show-ui-isolation.test.ts: the "browser did not start" and "browser exited" errors include Chrome's stderr (and exit code and signal), and a spawn error rejects the hook at once.

How I tested

  • With the server's uncaught-exception writer temporarily changed to print a stack whose first line has no message, the old test fails with the same stderr as CI (Error\n at startServer ...) and the new test passes. With summary temporarily hardcoded, the new test fails and prints the received object. Both changes were reverted.
  • With DREAMCODER_TEST_CHROME pointing at scripts that (a) print a line and hang, (b) print and exit 3, (c) are not executable, the hook errors now show stderr: "fake chrome says hi\n", exited with code 3 ... stderr: "boom\n" and failed to spawn: EACCES. Before, all three showed browser did not start with nothing else.
  • bun test --randomize --seed=1 on both files: 10 pass. 8 more randomized runs of both files on macOS: 0 failures.
  • On Linux (a temporary workflow on ubuntu-latest, Bun 1.4.2, Chrome 156, 4 CPUs; the branch and workflow are deleted): 0 failures in 100 standalone duplicate-server runs, 100 runs of the diagnostics test, 100 standalone Chrome launches, 60 runs of the ShowUI test file, and 32 full server suite runs over 8 runners.

Not verified

  • Neither failure reproduced, so neither cause is confirmed and I do not know that the flakes are gone. Whether the empty first stack line also happens in a running server (a real user would see Error with no message on stderr) is open.
  • CHANGELOG: no entry, as in test(desktop): make MermaidRenderer tests independent of order #210 for a test-only change.

@GoDiao
GoDiao merged commit 9b54a56 into main Oct 31, 2026
4 checks passed
@GoDiao
GoDiao deleted the fix/flaky-server-tests branch October 31, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant