fix(runner): give tier-2 build-failure envelopes one fingerprint (Sentry DEMOS-4G) - #270
Merged
Merged
Conversation
…try DEMOS-4G) `normalizeMonitorMessage`'s number rule (`/\b\d+(\.\d+)*\b/g`) fails to match where a digit run abuts a letter, so an ISO timestamp's milliseconds and `Z` survived normalisation. Every Tier-2 "Application bundle generation failed. [N seconds] - <ISO>" envelope therefore fingerprinted uniquely — 28 issues in `handsoncode/demos`, one event each, an unbounded population that grows one issue per build failure forever. Fixed with a dedicated ISO-8601 rule inserted ahead of the number rule, emitting `<ts>`. The number rule's word boundaries stay untouched: they're what keeps a diagnostic code like `TS1005` from collapsing into `TS<n>`, which is what a naive `\b`-dropping fix would do and still pass a "TS1005 != TS2304" check purely because the surrounding prose differs. Second cause, same fix: `relayStderr`'s `stderrSeen` dedupe keyed on the raw stderr line, so the timestamped envelope never deduped there either — every failed edit relayed a fresh copy and spent a `MONITOR_EVENT_CEILING` slot on a line with no new diagnostic. After ~20 such relays a visitor's genuine runtime `DemoError`s were silently never reported, since the budget is shared across monitor kinds. `stderrSeen` now keys on `normalizeMonitorMessage(message)` — the same fingerprint `sentry.ts` groups the Sentry issue by — so the relay and the parent's grouping agree; the relayed payload itself stays the raw line. Both changes verified against the real DEMOS-4G/4V titles and the 13 correctly-grouped TS-code issues, which keep their fingerprints byte-for-byte. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MONITOR_EVENT_CEILINGis 20 and shared across every monitor kind on a page, including the in-page runtime reporter. Because Tier-2 build-failure envelopes never deduped (below), a visitor whose Angular demo fails to build ~20 times in one session burns the entire budget on repeats of the same non-diagnostic line, and any genuine runtimeDemoErrorafter that is silently never reported. That is the reason this matters — the tidy Sentry issue list is secondary.Scale. As of this PR, 27 of the 49 unresolved
tier:2issues inhandsoncode/demosareApplication bundle generation failed. [N seconds] - <ISO>, each at exactly one event. It is an unbounded population: one new Sentry issue per Tier-2 build failure, forever, not a spike.Cause 1 — the fingerprint.
normalizeMonitorMessage(runner/packages/runtime/src/monitor.ts) collapses numbers with/\b\d+(\.\d+)*\b/g. The trailing\bfails to match where a digit run abuts a letter, so an ISO timestamp's milliseconds andZsurvive normalisation, and every rebuild mints a fresh fingerprint. Verified on the two real titles (Sentry: DEMOS-4G, DEMOS-4V):The fix is a dedicated ISO-8601 rule (
/\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?/g→<ts>) inserted ahead of the number rule. I deliberately did not take the one-character fix of just dropping the number rule's\bboundaries (/\d+(\.\d+)*/g). That also collapses the timestamp, and it also passes a naive "TS1005 vs TS2304" inequality check — but only because the surrounding prose happens to differ, not because the codes are distinguished on purpose. It silently rewrites every TypeScript diagnostic code toTS<n>, so two codes that ever share prose would merge into one issue. The boundaries are load-bearing in the other direction from the timestamp fix, and this PR's test suite includes a constructed TS2554/TS2555 pair (both "Expected N arguments, but got M") specifically to catch a regression back to that shortcut.I also verified no live TS-code issue regroups: I ran both the old and the new normalizer over the real DEMOS-3K title (
TS1005) and three other TS codes, byte for byte, and every output is identical before and after — nothing under the fix collapses that wasn't already collapsed.Cause 2 — the relay budget leak.
ContainerRuntime.relayStderr(runner/packages/runtime/src/container.ts) dedupes withstderrSeen, keyed on the raw stderr line, beforesentry.tsever fingerprints it. The TS diagnostic line is byte-identical across rebuilds of the same broken code, so it already deduped. The envelope line carries a live clock, so it never did — every failed edit relayed a fresh copy and spent a relay slot that carried no new diagnostic, which is the mechanism behind the budget-exhaustion consequence above.stderrSeennow keys onnormalizeMonitorMessage(message), the same fingerprintsentry.tsgroups the Sentry issue by, while the relayed payload itself stays the raw, truncated line — the diagnostic reaching Sentry is still verbatim compiler output.The honest trade, in the same terms as the design write-up behind this fix: a coarser key means only the first variant of a class now ever leaves the page.
Cannot find module 'foo'andCannot find module 'bar'normalize identically, so today both relay and one Sentry issue holds two inspectable samples; after this change it holds one. No issue disappears and no text is truncated — only sample diversity within an issue narrows, which is the price of not spending a second relay slot on an event that was always going to land in the same group the parent had already decided was one fault.Tests, red before green (
runner/pipeline/monitor-inject.test.mjs,runner/pipeline/monitor-stderr-relay.test.mjs, new). Against unmodified code:After the fix, all six pass. Full suite from
runner/:pnpm build && pnpm typecheck && pnpm test— clean build, clean typecheck across all four packages, 901 tests / 899 pass / 0 fail / 2 pre-existing todo. The pinned normalizer test atmonitor-inject.test.mjs(the one existing assertion using a message with no timestamp) is unmodified and still passes.REPORTER_SOURCE, the hand-written ES5 in-page reporter, does not duplicatenormalizeMonitorMessage— confirmed unaffected, sopipeline/monitor-inject.test.mjs's ES5-parse tests needed no changes.Rebase note. This branch was rebased onto current
master(which had picked up PR #263's edge-block handling in the same file) with no conflicts — the two changes land in disjoint regions ofcontainer.ts. Re-ran the full sequence after rebasing and re-confirmed the discriminating tests still go red with just the ISO rule reverted.Out of scope, each considered and rejected for this change: constant-titling the envelope issue (Sentry still re-derives the issue title from its newest event, so post-fix titles will show whichever timestamp arrived last — bounded and harmless, a follow-up); collapsing the 13 TS-code issues into one umbrella issue (would regroup live, correctly-grouped issues, and the brief requires distinct TS codes to stay distinct); keying
demoRelayBudget.admiton the normalized message insentry.ts(would change budget semantics for every monitor kind, not just stderr); wideningCAUSE_LINE/ routingrelayStderrthroughfailureDetail()(shared with the boot-failure and Worker share paths, its own change); Sentry-side cleanup of the 27 existing envelope issues (a console action after this ships, not a code change).Sentry: DEMOS-4G (https://handsoncode.sentry.io/issues/DEMOS-4G), DEMOS-4V (https://handsoncode.sentry.io/issues/DEMOS-4V), DEMOS-3K (https://handsoncode.sentry.io/issues/DEMOS-3K). No ClickUp ticket.
🤖 Generated with Claude Code
Note
Medium Risk
Changes demo-runtime error fingerprinting and the shared 20-event monitor ceiling; mis-tuning could merge distinct faults or drop relay samples, though regression tests target TS codes and relay dedupe.
Overview
Fixes Sentry issue sprawl and monitor budget exhaustion when Tier-2 Angular (and similar) builds fail repeatedly: the
"Application bundle generation failed. [N seconds] - <ISO>"envelope used to get a new fingerprint and a new stderr relay on every rebuild because ISO timestamps survivednormalizeMonitorMessage.normalizeMonitorMessagenow strips ISO-8601 datetimes to<ts>before the existing number rule, so identical build failures group regardless of clock. TypeScript codes likeTS1005stay distinct (tests guard against loosening word boundaries on the number rule).ContainerRuntime.relayStderrdedupes withnormalizeMonitorMessage(truncateMessage(line))instead of the raw line, aligned withsentry.tsfingerprinting, while callbacks still receive the verbatim truncated stderr. Trade-off: only the first variant of messages that normalize the same way is relayed (e.g. different missing module names).New pipeline tests cover normalization edge cases and keepalive-driven stderr relay behavior.
Reviewed by Cursor Bugbot for commit 2464f33. Bugbot is set up for automated code reviews on this repo. Configure here.