feat(backend): report the backend's memory footprint as Sentry gauges - #249
Open
gmaclennan wants to merge 3 commits into
Open
feat(backend): report the backend's memory footprint as Sentry gauges#249gmaclennan wants to merge 3 commits into
gmaclennan wants to merge 3 commits into
Conversation
Member
Author
|
Verified end to end on a local Release build (Pixel_7a_API_34, The boot log line the local harness reads also renders as intended: |
The `:ComapeoCore` process is the one Android kills first when memory is short — it runs in the background with `oom_score_adj 200` and the kernel's `oom_score` scales with footprint — so "what does this change cost in memory" is worth being able to answer cheaply and repeatedly. Until now there was no answer: the only memory signal was a `heapUsed` gauge every 60s, and no process-level number at all. `backend/lib/memory-snapshot.js` reads `/proc/self/status` and `v8.getHeapStatistics()`. Three new gauges join `heap_used_bytes`: `heap_physical_bytes` (heap committed and touched — the figure that tracks anonymous RSS, where `heap_used_bytes` tracks the live object graph), and on Android `rss_bytes` and `rss_peak_bytes`. The peak is `VmHWM`, which is effectively what the low-memory killer scores the process on, and being monotonic it needs no high-rate sampling to find. `/proc` does not exist on iOS, where node shares the app process and an rss would describe the UI too, so the reader returns null there and the pair is simply not emitted — the platform gate is the filesystem, not a flag that can drift. All four carry `runtime` (the nodejs-mobile revision, so two runtime builds can be compared in a staged rollout) plus `device_class` / `os_major`. Memory is the metric whose whole point is the cheap device, and `heap_used_bytes` shipping without them means its 16k samples to date cannot be attributed to a device class at all. They sit at the diagnostic tier, matching `heap_used_bytes`. They describe the process's own resource use at a fixed cadence that is deliberately not activity-triggered, name nothing the user did, and at boot are overwhelmingly a property of the build and device rather than the data. Free *device* memory stays usage-tier where it already is. Reasoning in docs/BENCHMARKING.md so it can be argued with later. An extra sample fires 3s after `ready`: a 60s-only sampler silently biases the data towards processes that survived, and 88 of the FGS exits reported from production in the last 90 days sit in the `<10s` uptime bucket. One snapshot measured ~28 us on an arm64 emulator, so this is roughly 0.3 ms of CPU across a ten-minute session. The boot sample also logs one `[comapeo.memory] boot` line per launch, unconditionally: no PII, and it lets a build be measured on a device with telemetry switched off.
gmaclennan
force-pushed
the
feat/backend-memory-telemetry
branch
from
August 22, 2026 13:17
3e20895 to
0d525f4
Compare
Review follow-ups on the memory gauges. The four gauges now carry `sample` (`boot` or `interval`). Without it the boot sample joins the same series as the 60s sampler, so a release that changes how long processes live moves the percentiles on its own and reads as a footprint change. `runtime` was derived twice — once for the snapshot, once for the `nodejs_mobile` event tag — and threaded through every snapshot argument. It now comes from one exported `runtimeVersion()` and from the metrics config, alongside its `platform` / `device_class` / `os_major` siblings, so events and metrics cannot drift apart. The snapshot keeps its own `runtime` for the `[comapeo.memory] boot` log line. The four open-coded gauge emissions fold into one name-to-value table, and the comment blocks that restated docs/BENCHMARKING.md point at it instead. The reasoning they carried that the doc did not already have — the `<10s`-uptime figure behind the boot sample and the cardinality argument for `runtime` — moved into the doc.
Pure processMemoryFrom/heapStatsFrom carry the mapping tests on fixtures; memorySnapshot() takes no parameters and a smoke test runs the real path, which exercises both sides of the /proc platform gate for real (Linux CI has /proc, macOS dev machines don't).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
:ComapeoCoreprocess is the one Android kills first when memory is short — it runs in the background withoom_score_adj 200for life, and the kernel'soom_scorescales with footprint. So "what does this change cost in memory" is worth being able to answer cheaply and repeatedly. Until now there was no way to: the only memory signal was aheapUsedgauge every 60s, with no process-level number at all.backend/lib/memory-snapshot.jsreads/proc/self/statusandv8.getHeapStatistics(). Three gauges joinheap_used_bytes:heap_physical_bytes(heap committed and touched — the figure that tracks anonymous RSS, whereheap_used_bytestracks the live object graph), and on Androidrss_bytesandrss_peak_bytes. The peak isVmHWM, effectively what the low-memory killer scores the process on, and being monotonic it needs no high-rate sampling to find./procdoes not exist on iOS, where node shares the app process and an rss would describe the UI too, so the reader returns null there and the pair is simply not emitted. The platform gate is the filesystem, not a flag that can drift out of sync.All four carry
runtime(the nodejs-mobile revision, so two runtime builds can be compared during a staged rollout) plusdevice_class/os_major. Memory is the metric whose whole point is the cheap device, andheap_used_bytesshipped without them — its ~16k samples to date cannot be attributed to a device class at all.runtimecomes from one sharedruntimeVersion()helper, the same value as thenodejs_mobileevent tag, so events and metrics join on it; the metrics layer picks it up from its config alongsideplatform/device_class/os_major.An extra sample fires 3s after
ready. A 60s-only sampler silently biases the data towards processes that survived, and 88 of the FGS exits reported from production in the last 90 days sit in the<10suptime bucket. The boot sample also writes one[comapeo.memory] bootline per launch, unconditionally — no PII, and it is what lets a build be measured on a device with telemetry switched off entirely (the harness in #253 reads it). The metric emission is still gated.The gauges therefore also carry
sample(bootorinterval), naming which call site emitted them. Without it the two populations are one series, and a release that changes how long processes live moves the percentiles on its own — a pre/post comparison would read that as a footprint change.Consent tier. All four sit at the diagnostic tier, matching
heap_used_bytes, which has been diagnostic since it was added. They describe the process's own resource use at a fixed cadence that is deliberately not activity-triggered, they name nothing the user did, and at boot they are overwhelmingly a property of the build and the device rather than of the data. Free device memory stays usage-tier where it already lives in thenode_resourcescontext. The full reasoning is written up indocs/BENCHMARKING.mdso it can be argued with later — this is the part of the PR most worth a second opinion.Cost. One snapshot measured ~28 µs on an arm64 emulator over 500 iterations in a Release build: a 1.3 KB
/procread plusv8.getHeapStatistics(). Two calls in the first minute, one a minute after — about 0.3 ms of CPU across a ten-minute session.Validation
npm run lint,npm test(78) andnpm run backend:test(114) pass. The new unit tests cover the snapshot reader against fixture/proc/self/statuscontent including the iOS (no/proc) path, and the metrics wrappers for attribute injection, thesampleattribute from both call sites, and the Android-only gauge pair.The local A/B benchmark harness that also consumes this snapshot was split out into #253, which stacks on this branch.