Skip to content

fix(runner): give tier-2 build-failure envelopes one fingerprint (Sentry DEMOS-4G) - #270

Merged
demtario merged 1 commit into
masterfrom
fix/sentry-demos-stderr-fingerprint
Aug 31, 2026
Merged

fix(runner): give tier-2 build-failure envelopes one fingerprint (Sentry DEMOS-4G)#270
demtario merged 1 commit into
masterfrom
fix/sentry-demos-stderr-fingerprint

Conversation

@demtario

@demtario demtario commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

MONITOR_EVENT_CEILING is 20 and shared across every monitor kind on a page, including the in-page runtime reporter. Because Tier-2 build-failure envelopes never deduped (below), a visitor whose Angular demo fails to build ~20 times in one session burns the entire budget on repeats of the same non-diagnostic line, and any genuine runtime DemoError after that is silently never reported. That is the reason this matters — the tidy Sentry issue list is secondary.

Scale. As of this PR, 27 of the 49 unresolved tier:2 issues in handsoncode/demos are Application bundle generation failed. [N seconds] - <ISO>, each at exactly one event. It is an unbounded population: one new Sentry issue per Tier-2 build failure, forever, not a spike.

Cause 1 — the fingerprint. normalizeMonitorMessage (runner/packages/runtime/src/monitor.ts) collapses numbers with /\b\d+(\.\d+)*\b/g. The trailing \b fails to match where a digit run abuts a letter, so an ISO timestamp's milliseconds and Z survive normalisation, and every rebuild mints a fresh fingerprint. Verified on the two real titles (Sentry: DEMOS-4G, DEMOS-4V):

before: Application bundle generation failed. [<n> seconds] - <n>-<n>-25T18:<n>:<n>.937Z
before: Application bundle generation failed. [<n> seconds] - <n>-<n>-27T14:<n>:<n>.952Z   (not equal)

after:  Application bundle generation failed. [<n> seconds] - <ts>
after:  Application bundle generation failed. [<n> seconds] - <ts>                          (equal)

The fix is a dedicated ISO-8601 rule (/\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?/g<ts>) inserted ahead of the number rule. I deliberately did not take the one-character fix of just dropping the number rule's \b boundaries (/\d+(\.\d+)*/g). That also collapses the timestamp, and it also passes a naive "TS1005 vs TS2304" inequality check — but only because the surrounding prose happens to differ, not because the codes are distinguished on purpose. It silently rewrites every TypeScript diagnostic code to TS<n>, so two codes that ever share prose would merge into one issue. The boundaries are load-bearing in the other direction from the timestamp fix, and this PR's test suite includes a constructed TS2554/TS2555 pair (both "Expected N arguments, but got M") specifically to catch a regression back to that shortcut.

I also verified no live TS-code issue regroups: I ran both the old and the new normalizer over the real DEMOS-3K title (TS1005) and three other TS codes, byte for byte, and every output is identical before and after — nothing under the fix collapses that wasn't already collapsed.

Cause 2 — the relay budget leak. ContainerRuntime.relayStderr (runner/packages/runtime/src/container.ts) dedupes with stderrSeen, keyed on the raw stderr line, before sentry.ts ever fingerprints it. The TS diagnostic line is byte-identical across rebuilds of the same broken code, so it already deduped. The envelope line carries a live clock, so it never did — every failed edit relayed a fresh copy and spent a relay slot that carried no new diagnostic, which is the mechanism behind the budget-exhaustion consequence above. stderrSeen now keys on normalizeMonitorMessage(message), the same fingerprint sentry.ts groups the Sentry issue by, while the relayed payload itself stays the raw, truncated line — the diagnostic reaching Sentry is still verbatim compiler output.

The honest trade, in the same terms as the design write-up behind this fix: a coarser key means only the first variant of a class now ever leaves the page. Cannot find module 'foo' and Cannot find module 'bar' normalize identically, so today both relay and one Sentry issue holds two inspectable samples; after this change it holds one. No issue disappears and no text is truncated — only sample diversity within an issue narrows, which is the price of not spending a second relay slot on an event that was always going to land in the same group the parent had already decided was one fault.

Tests, red before green (runner/pipeline/monitor-inject.test.mjs, runner/pipeline/monitor-stderr-relay.test.mjs, new). Against unmodified code:

not ok - one build failure is one fingerprint, whatever the clock said
not ok - the other timestamp shapes a dev server prints collapse too
ok     - a compiler diagnostic code survives normalisation
ok     - two codes with identical prose stay two issues
not ok - re-running one broken build does not spend a second relay slot
        ["...TS1005...", "...bundle...2026-08-25...", "...bundle...2026-08-27..."]
        3 !== 2
ok     - a genuinely different fault still gets through

After the fix, all six pass. Full suite from runner/: pnpm build && pnpm typecheck && pnpm test — clean build, clean typecheck across all four packages, 901 tests / 899 pass / 0 fail / 2 pre-existing todo. The pinned normalizer test at monitor-inject.test.mjs (the one existing assertion using a message with no timestamp) is unmodified and still passes. REPORTER_SOURCE, the hand-written ES5 in-page reporter, does not duplicate normalizeMonitorMessage — confirmed unaffected, so pipeline/monitor-inject.test.mjs's ES5-parse tests needed no changes.

Rebase note. This branch was rebased onto current master (which had picked up PR #263's edge-block handling in the same file) with no conflicts — the two changes land in disjoint regions of container.ts. Re-ran the full sequence after rebasing and re-confirmed the discriminating tests still go red with just the ISO rule reverted.

Out of scope, each considered and rejected for this change: constant-titling the envelope issue (Sentry still re-derives the issue title from its newest event, so post-fix titles will show whichever timestamp arrived last — bounded and harmless, a follow-up); collapsing the 13 TS-code issues into one umbrella issue (would regroup live, correctly-grouped issues, and the brief requires distinct TS codes to stay distinct); keying demoRelayBudget.admit on the normalized message in sentry.ts (would change budget semantics for every monitor kind, not just stderr); widening CAUSE_LINE / routing relayStderr through failureDetail() (shared with the boot-failure and Worker share paths, its own change); Sentry-side cleanup of the 27 existing envelope issues (a console action after this ships, not a code change).

Sentry: DEMOS-4G (https://handsoncode.sentry.io/issues/DEMOS-4G), DEMOS-4V (https://handsoncode.sentry.io/issues/DEMOS-4V), DEMOS-3K (https://handsoncode.sentry.io/issues/DEMOS-3K). No ClickUp ticket.

🤖 Generated with Claude Code


Note

Medium Risk
Changes demo-runtime error fingerprinting and the shared 20-event monitor ceiling; mis-tuning could merge distinct faults or drop relay samples, though regression tests target TS codes and relay dedupe.

Overview
Fixes Sentry issue sprawl and monitor budget exhaustion when Tier-2 Angular (and similar) builds fail repeatedly: the "Application bundle generation failed. [N seconds] - <ISO>" envelope used to get a new fingerprint and a new stderr relay on every rebuild because ISO timestamps survived normalizeMonitorMessage.

normalizeMonitorMessage now strips ISO-8601 datetimes to <ts> before the existing number rule, so identical build failures group regardless of clock. TypeScript codes like TS1005 stay distinct (tests guard against loosening word boundaries on the number rule).

ContainerRuntime.relayStderr dedupes with normalizeMonitorMessage(truncateMessage(line)) instead of the raw line, aligned with sentry.ts fingerprinting, while callbacks still receive the verbatim truncated stderr. Trade-off: only the first variant of messages that normalize the same way is relayed (e.g. different missing module names).

New pipeline tests cover normalization edge cases and keepalive-driven stderr relay behavior.

Reviewed by Cursor Bugbot for commit 2464f33. Bugbot is set up for automated code reviews on this repo. Configure here.

…try DEMOS-4G)

`normalizeMonitorMessage`'s number rule (`/\b\d+(\.\d+)*\b/g`) fails to match
where a digit run abuts a letter, so an ISO timestamp's milliseconds and `Z`
survived normalisation. Every Tier-2 "Application bundle generation failed.
[N seconds] - <ISO>" envelope therefore fingerprinted uniquely — 28 issues in
`handsoncode/demos`, one event each, an unbounded population that grows one
issue per build failure forever. Fixed with a dedicated ISO-8601 rule inserted
ahead of the number rule, emitting `<ts>`. The number rule's word boundaries
stay untouched: they're what keeps a diagnostic code like `TS1005` from
collapsing into `TS<n>`, which is what a naive `\b`-dropping fix would do and
still pass a "TS1005 != TS2304" check purely because the surrounding prose
differs.

Second cause, same fix: `relayStderr`'s `stderrSeen` dedupe keyed on the raw
stderr line, so the timestamped envelope never deduped there either — every
failed edit relayed a fresh copy and spent a `MONITOR_EVENT_CEILING` slot on a
line with no new diagnostic. After ~20 such relays a visitor's genuine runtime
`DemoError`s were silently never reported, since the budget is shared across
monitor kinds. `stderrSeen` now keys on `normalizeMonitorMessage(message)` —
the same fingerprint `sentry.ts` groups the Sentry issue by — so the relay and
the parent's grouping agree; the relayed payload itself stays the raw line.

Both changes verified against the real DEMOS-4G/4V titles and the 13
correctly-grouped TS-code issues, which keep their fingerprints byte-for-byte.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@demtario
demtario merged commit 02d6877 into master Aug 31, 2026
6 checks passed
@demtario
demtario deleted the fix/sentry-demos-stderr-fingerprint branch August 31, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant