Skip to content

feat(backend): report the backend's memory footprint as Sentry gauges - #249

Open
gmaclennan wants to merge 3 commits into
mainfrom
feat/backend-memory-telemetry
Open

feat(backend): report the backend's memory footprint as Sentry gauges#249
gmaclennan wants to merge 3 commits into
mainfrom
feat/backend-memory-telemetry

Conversation

@gmaclennan

@gmaclennan gmaclennan commented Aug 21, 2026

Copy link
Copy Markdown
Member

The :ComapeoCore process is the one Android kills first when memory is short — it runs in the background with oom_score_adj 200 for life, and the kernel's oom_score scales with footprint. So "what does this change cost in memory" is worth being able to answer cheaply and repeatedly. Until now there was no way to: the only memory signal was a heapUsed gauge every 60s, with no process-level number at all.

backend/lib/memory-snapshot.js reads /proc/self/status and v8.getHeapStatistics(). Three gauges join heap_used_bytes: heap_physical_bytes (heap committed and touched — the figure that tracks anonymous RSS, where heap_used_bytes tracks the live object graph), and on Android rss_bytes and rss_peak_bytes. The peak is VmHWM, effectively what the low-memory killer scores the process on, and being monotonic it needs no high-rate sampling to find.

/proc does not exist on iOS, where node shares the app process and an rss would describe the UI too, so the reader returns null there and the pair is simply not emitted. The platform gate is the filesystem, not a flag that can drift out of sync.

All four carry runtime (the nodejs-mobile revision, so two runtime builds can be compared during a staged rollout) plus device_class / os_major. Memory is the metric whose whole point is the cheap device, and heap_used_bytes shipped without them — its ~16k samples to date cannot be attributed to a device class at all. runtime comes from one shared runtimeVersion() helper, the same value as the nodejs_mobile event tag, so events and metrics join on it; the metrics layer picks it up from its config alongside platform / device_class / os_major.

An extra sample fires 3s after ready. A 60s-only sampler silently biases the data towards processes that survived, and 88 of the FGS exits reported from production in the last 90 days sit in the <10s uptime bucket. The boot sample also writes one [comapeo.memory] boot line per launch, unconditionally — no PII, and it is what lets a build be measured on a device with telemetry switched off entirely (the harness in #253 reads it). The metric emission is still gated.

The gauges therefore also carry sample (boot or interval), naming which call site emitted them. Without it the two populations are one series, and a release that changes how long processes live moves the percentiles on its own — a pre/post comparison would read that as a footprint change.

Consent tier. All four sit at the diagnostic tier, matching heap_used_bytes, which has been diagnostic since it was added. They describe the process's own resource use at a fixed cadence that is deliberately not activity-triggered, they name nothing the user did, and at boot they are overwhelmingly a property of the build and the device rather than of the data. Free device memory stays usage-tier where it already lives in the node_resources context. The full reasoning is written up in docs/BENCHMARKING.md so it can be argued with later — this is the part of the PR most worth a second opinion.

Cost. One snapshot measured ~28 µs on an arm64 emulator over 500 iterations in a Release build: a 1.3 KB /proc read plus v8.getHeapStatistics(). Two calls in the first minute, one a minute after — about 0.3 ms of CPU across a ten-minute session.

Validation

npm run lint, npm test (78) and npm run backend:test (114) pass. The new unit tests cover the snapshot reader against fixture /proc/self/status content including the iOS (no /proc) path, and the metrics wrappers for attribute injection, the sample attribute from both call sites, and the Android-only gauge pair.

The local A/B benchmark harness that also consumes this snapshot was split out into #253, which stacks on this branch.

@gmaclennan

Copy link
Copy Markdown
Member Author

Verified end to end on a local Release build (Pixel_7a_API_34, device_class: low), three launch cycles. All four gauges arrive and slice on every attribute:

heap_used_bytes        n=3  max= 25.2 MB
heap_physical_bytes    n=3  max= 37.4 MB
rss_bytes              n=3  max=172.0 MB
rss_peak_bytes         n=3  max=255.8 MB

rss_peak_bytes by attribute:
device_class=low  os_major=android.14  runtime=24.19.0-0  platform=android  →  268.2MB

The boot log line the local harness reads also renders as intended:

[comapeo.memory] boot {"runtime":"24.19.0-0","heap":{"usedBytes":26470400,"physicalBytes":38756352,
"totalBytes":43417600,"limitBytes":569114624,"externalBytes":10018067},"process":{"rssBytes":180359168,
"peakRssBytes":249913344,"anonBytes":83300352,"fileBytes":96313344,"swapBytes":27844608}}

@gmaclennan
gmaclennan added this pull request to the merge queue Aug 21, 2026
@gmaclennan
gmaclennan removed this pull request from the merge queue due to a manual request Aug 21, 2026
The `:ComapeoCore` process is the one Android kills first when memory is
short — it runs in the background with `oom_score_adj 200` and the kernel's
`oom_score` scales with footprint — so "what does this change cost in
memory" is worth being able to answer cheaply and repeatedly. Until now
there was no answer: the only memory signal was a `heapUsed` gauge every
60s, and no process-level number at all.

`backend/lib/memory-snapshot.js` reads `/proc/self/status` and
`v8.getHeapStatistics()`. Three new gauges join `heap_used_bytes`:
`heap_physical_bytes` (heap committed and touched — the figure that tracks
anonymous RSS, where `heap_used_bytes` tracks the live object graph), and
on Android `rss_bytes` and `rss_peak_bytes`. The peak is `VmHWM`, which is
effectively what the low-memory killer scores the process on, and being
monotonic it needs no high-rate sampling to find. `/proc` does not exist on
iOS, where node shares the app process and an rss would describe the UI
too, so the reader returns null there and the pair is simply not emitted —
the platform gate is the filesystem, not a flag that can drift.

All four carry `runtime` (the nodejs-mobile revision, so two runtime builds
can be compared in a staged rollout) plus `device_class` / `os_major`.
Memory is the metric whose whole point is the cheap device, and
`heap_used_bytes` shipping without them means its 16k samples to date
cannot be attributed to a device class at all.

They sit at the diagnostic tier, matching `heap_used_bytes`. They describe
the process's own resource use at a fixed cadence that is deliberately not
activity-triggered, name nothing the user did, and at boot are
overwhelmingly a property of the build and device rather than the data.
Free *device* memory stays usage-tier where it already is. Reasoning in
docs/BENCHMARKING.md so it can be argued with later.

An extra sample fires 3s after `ready`: a 60s-only sampler silently biases
the data towards processes that survived, and 88 of the FGS exits reported
from production in the last 90 days sit in the `<10s` uptime bucket. One
snapshot measured ~28 us on an arm64 emulator, so this is roughly 0.3 ms of
CPU across a ten-minute session. The boot sample also logs one
`[comapeo.memory] boot` line per launch, unconditionally: no PII, and it
lets a build be measured on a device with telemetry switched off.
@gmaclennan
gmaclennan force-pushed the feat/backend-memory-telemetry branch from 3e20895 to 0d525f4 Compare August 22, 2026 13:17
@gmaclennan gmaclennan changed the title feat(backend): measure the backend's memory footprint, locally and in the field feat(backend): report the backend's memory footprint as Sentry gauges Aug 22, 2026
Review follow-ups on the memory gauges.

The four gauges now carry `sample` (`boot` or `interval`). Without it the
boot sample joins the same series as the 60s sampler, so a release that
changes how long processes live moves the percentiles on its own and reads
as a footprint change.

`runtime` was derived twice — once for the snapshot, once for the
`nodejs_mobile` event tag — and threaded through every snapshot argument.
It now comes from one exported `runtimeVersion()` and from the metrics
config, alongside its `platform` / `device_class` / `os_major` siblings, so
events and metrics cannot drift apart. The snapshot keeps its own `runtime`
for the `[comapeo.memory] boot` log line.

The four open-coded gauge emissions fold into one name-to-value table, and
the comment blocks that restated docs/BENCHMARKING.md point at it instead.
The reasoning they carried that the doc did not already have — the
`<10s`-uptime figure behind the boot sample and the cardinality argument for
`runtime` — moved into the doc.
Pure processMemoryFrom/heapStatsFrom carry the mapping tests on fixtures;
memorySnapshot() takes no parameters and a smoke test runs the real path,
which exercises both sides of the /proc platform gate for real (Linux CI
has /proc, macOS dev machines don't).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature (changelog)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant