Skip to content

feat(native): ARM32 registers and both frame-record shapes in the crash daemon - #2053

Open
GLinnik21 wants to merge 2 commits into
getsentry:masterfrom
GLinnik21:feat/arm32-crash-daemon
Open

feat(native): ARM32 registers and both frame-record shapes in the crash daemon#2053
GLinnik21 wants to merge 2 commits into
getsentry:masterfrom
GLinnik21:feat/arm32-crash-daemon

Conversation

@GLinnik21

@GLinnik21 GLinnik21 commented Sep 2, 2026

Copy link
Copy Markdown

Summary

Two scoped changes to the Linux crash daemon for 32-bit ARM, which #1659 made a supported target:

  1. Register serialisation lacked ARM32. build_registers_from_ctx covered x86_64 and aarch64 only, so ARM32 events shipped an empty registers object. This adds an __arm__ branch (r0–r10, fp, ip, sp, lr, pc, cpsr from the saved ucontext).
  2. The frame-pointer walk assumed one frame-record shape. The generic walker already runs on ARM32 (ip/fp/sp extraction exists), but it reads [fp] as the saved fp and [fp + 4] as the return address. That is clang's layout; GCC's is the other way round, and one ARM32 process routinely contains both. This reads both candidate records and keeps the one shaped like a frame.

Why both shapes

Both compilers emit push {.., fp, lr} and then point fp at a different slot of the same push:

gcc    (glibc, most distro code):   add fp, sp, #20   → fp on the LR slot:       [fp-4] = saved fp, [fp] = return address
clang / rustc:                      add fp, sp, #16   → fp on the saved-fp slot: [fp] = saved fp, [fp+4] = return address

With a single assumed shape the walk misreads the first record of the other shape it meets — reporting a saved fp as a return address, or stopping. Both candidates are judged against the crashed process's mappings, which the daemon records with their permission bits (including anonymous and special mappings, in a dynamically sized daemon-local cache; ARM32 only, so no other architecture pays for it) while it parses /proc/<pid>/maps for modules. A candidate is accepted only if its return address lies in an executable, non-writable mapping — code, as opposed to a stack (writable, and on some systems executable too) or the heap — and its saved fp is zero (outermost frame) or lies above the current frame in a writable mapping. That rejects the other compiler's record shape, whose "saved fp" is really a return address, even when code sits numerically below the stack. Nothing is bounded by the captured window (stack_end is where the capture stops, not where the stack does), and the existing post-read checks (is_valid_code_addr, monotonic fp) still apply after selection.

Two things the walk does not do, stated so nobody reads more into it: it reads frame records out of the one captured stack window, so a chain is followed only while it stays inside that window — a signal stack or a split stack that links to a lower or non-contiguous segment ends the walk (as it did before this PR); and JIT code in writable mappings is not treated as code.

The daemon stops only the crashing thread, so the mappings can change under it. The walk therefore re-reads /proc/<pid>/maps once the stack bytes have been copied and compares every mapping with the recorded one, field by field and in order; if the snapshot is incomplete (a line the parser cannot read, an allocation failure, a read error) or differs from the first read, the ARM32 walk stops rather than guess a shape. Maps lines are read as logical lines (an over-long pathname is truncated and the remainder consumed), so a long path can neither disable the walk nor be parsed as a mapping of its own; that reader also serves the existing module parser.

What that does not cover, stated plainly: state that changed between the crash and the first read. The two reads narrow that window; they do not close it. Nor is it specific to this PR — the stack bytes the frame-pointer walk reads, and the module inventory, are captured after the crash on every platform while peer threads keep running, and a peer thread can in principle write to, unmap or remap the stopped thread's stack before the copy is taken. The shape selection here consumes those same post-crash inputs and inherits that exposure; it does not add a new one, but it also cannot promise more than the generic walk does. Freezing the whole thread group for the duration of capture would close the window for the generic walk as well; that is a daemon-wide change I'd propose separately rather than fold into this PR.

Not covered: Thumb code that uses r7 as its frame pointer (the walk still seeds from r11 / arm_fp, as before this PR — the changelog says r11-based), and JIT code in writable mappings.

Verification (ARM32 hardware, isolated from #2052)

LG webOS 4.x television, Cortex-A9, glibc 2.24, in-process libunwind disabled so the FP fallback is exercised. Two daemons were built from the same tree: #2052 only, and #2052 + this PR.

  • GCC-built test program (-O2 -fno-omit-frame-pointer, noinline f1 → f2 → f3 → leaf, null write in leaf):
  • rustc-built application (LLVM shape, which matches the existing assumption): fix(native): read frame records at pointer width in the crash daemon's FP walk #2052 only and fix(native): read frame records at pointer width in the crash daemon's FP walk #2052 + this PR both give gsignal ← plex_run ← __libc_start_main for a SIGABRT — i.e. the second shape only changes the result when GCC-built frame-pointer frames are in the chain.
  • Registers: 17 ARM registers on the crashed thread's stacktrace; none before this PR.
  • Re-verified after the review changes that alter selection (permission-bit validation, fail-closed on incomplete mappings): daemon log Captured 11 modules and 66 mappings (complete); GCC chain leaf ← f2 ← f1 ← __libc_start_main; rustc application crash_on_purpose ← plex_run ← __libc_start_main; 17 registers. This target's main-thread [stack] mapping is rwxp, so the "executable but not writable" rule for return addresses is exercised, not theoretical.
  • The revision at the current head (exact tuple comparison on the second maps read, logical-line reader, ARM32-only gating) was run on the same hardware: Captured 66 mappings (complete); GCC chain leaf ← f2 ← f1 ← __libc_start_main; rustc application crash_on_purpose ← plex_run ← __libc_start_main; 17 registers.
  • sentry-crash cross-compiles for arm-linux-gnueabi (GCC 12); non-ARM paths are unchanged (#if defined(__arm__) / #else); macOS arm64 native-backend build and clang-format clean.
  • No automated ARM32 coverage of the native backend runs in CI today (the ARM32 job uses TEST_QEMU, which skips the native integration suite), and the walker is daemon-internal with no unit-test seam, so this PR adds no test; happy to add one in whatever shape you prefer.

Stacked on #2052, which it needs for pointer-width stack reads on ARM32.

Note: this PR is stacked on #2052 and therefore includes its commit; it will rebase to a single commit once #2052 lands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

read_stack_value still uses overflow-prone unsigned addition in its bounds check, which can allow out-of-range reads when addr is corrupted (common in crash contexts).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Improves Linux crash-daemon native backtraces on 32-bit ARM by (1) serializing ARM32 register state into events and (2) making the frame-pointer fallback unwinder handle both GCC and clang/LLVM ARM32 frame-record layouts (plus the stacked pointer-width stack reads prerequisite).

Changes:

  • Add ARM32 (__arm__) register serialization from ucontext_t (r0–r10, fp/ip/sp/lr/pc/cpsr).
  • Teach the FP walker on ARM32 to probe both possible (saved_fp, return_addr) frame-record shapes and select a plausible candidate.
  • Switch stack reads/offsets in the FP walker to use sizeof(uintptr_t) (stacked from #2052).
File summaries
File Description
src/backends/native/sentry_crash_daemon.c Adds ARM32 register reporting and dual-shape ARM32 FP frame-record decoding; updates stack value reads to pointer width.
CHANGELOG.md Documents the ARM32 registers + frame-record handling and the pointer-width read fix.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 628 to 632
if (addr < stack_start
|| addr + sizeof(uint64_t) > stack_start + stack_size) {
|| addr + sizeof(uintptr_t) > stack_start + stack_size) {
return false;
}
uint64_t offset = addr - stack_start;
Comment thread src/backends/native/sentry_crash_daemon.c Outdated
`read_stack_value` bounds-checked and copied `sizeof(uint64_t)` bytes
regardless of the target's pointer size, and the walk stepped to the
return address with the same constant. On a 32-bit target every read
therefore took two stack slots per pointer: the high half of each
`saved_fp` and `return_addr` was the neighbouring word, the return
address was read from the wrong slot, and the last legitimate frame
record failed the bounds check. Read a `uintptr_t` and widen it, and
step by pointer size; 64-bit targets are unchanged.

While here, range-check by subtraction rather than `addr + size`: the
address comes from a frame pointer in a crashed process, and a corrupted
value near the top of the address space wraps the sum past the naive
comparison.
@GLinnik21
GLinnik21 force-pushed the feat/arm32-crash-daemon branch from aaa8863 to 6f1940f Compare September 2, 2026 21:45
@GLinnik21

Copy link
Copy Markdown
Author

Addressed both reviews:

  • The overflow-prone bounds check in read_stack_value is fixed in fix(native): read frame records at pointer width in the crash daemon's FP walk #2052 (subtraction-based range check, uintptr_t read then widened); this branch is rebased on it.
  • The capture-window bound on saved_fp is gone. arm_frame_record_plausible no longer compares the saved fp against stack_end (which is the captured window, not the thread's stack) — only saved_fp == 0 || saved_fp > current_fp, so a real frame just past the window is kept and its own read fails cleanly on the next iteration. The "is this a return address" half is now a module-range test (address inside a module captured from the crashed process), falling back to is_valid_code_addr + outside-the-window only when no modules were captured; that closes the case where is_valid_code_addr (no upper bound on ARM32) would have let a stack address through as instruction_addr.

Re-verified on ARM32 hardware with the updated daemon: GCC-built chain leaf ← f2 ← f1 ← __libc_start_main, rustc application crash_on_purpose ← plex_run ← __libc_start_main, 17 registers.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6f1940f. Configure here.

Comment thread src/backends/native/sentry_crash_daemon.c
@GLinnik21
GLinnik21 force-pushed the feat/arm32-crash-daemon branch from 6f1940f to f4fb76f Compare September 2, 2026 22:40
@GLinnik21

Copy link
Copy Markdown
Author

Second pass on the reviews (Cursor's "walker can pick the wrong frame shape" and the follow-ups it implies):

  • The selection no longer trusts numeric order alone. The daemon now records every mapping of the crashed process with its permission bits (ARM32 only — a dynamically sized daemon-local cache, first allocation failure latched, capacity overflow-checked) while parsing /proc/<pid>/maps. A return address must be in an executable, non-writable mapping; a saved fp must be above the current frame in a writable mapping. A GCC record examined with the clang shape puts its real return address in the "saved fp" slot, and a code address is not writable, so that candidate is rejected even when the stack sits below the module. No stack-identity or capture-window bound is used.
  • Fail closed: if the mappings could not be read completely, or the second read (taken after the stack bytes are copied, compared field by field and in order) differs from the first, the ARM32 walk stops rather than guess a shape.
  • Maps lines are read as logical lines, so an over-long pathname is truncated and consumed rather than parsed as a mapping of its own; the module parser uses the same reader.
  • Scope stated in the body: r11-based records only (Thumb r7 chains are not walked, and the changelog says so); JIT code in writable mappings is not treated as code; a chain is followed only within the captured window. What this does not close is the post-crash capture window itself — the stack bytes and module inventory are read after the crash on every platform while peer threads keep running. That is inherited, not introduced, and closing it means freezing the thread group during capture, which I'd propose as a separate change.

Hardware: the permission-bit selection was verified on the ARM32 set (66 mappings (complete); GCC chain leaf ← f2 ← f1 ← __libc_start_main; rustc app crash_on_purpose ← plex_run ← __libc_start_main; 17 registers). The latest revision changes only the gate's bookkeeping; its hardware run is queued behind a shared device and I'll confirm it here.

@GLinnik21

Copy link
Copy Markdown
Author

Hardware confirmation for the current head (f4fb76f): daemon log Captured 66 mappings (complete); GCC-built chain leaf ← f2 ← f1 ← __libc_start_main; rustc application crash_on_purpose ← plex_run ← __libc_start_main; 17 registers; the second maps read matched the first, so the walk ran rather than fail-closed.

Comment thread src/backends/native/sentry_crash_daemon.c
…sh daemon

Two scoped changes for 32-bit ARM on Linux, a supported target since getsentry#1659:

- `build_registers_from_ctx` covered x86_64 and aarch64 only, so ARM32
  events shipped an empty `registers` object. Add an `__arm__` branch
  (r0-r10, fp, ip, sp, lr, pc, cpsr from the saved ucontext).
- The generic frame-pointer walk already runs on ARM32 but assumes one
  frame-record shape: [fp] = saved fp, [fp+4] = return address. That is
  clang's layout. GCC's `push {.., fp, lr}; add fp, sp, #N` leaves fp on
  the LR slot instead ([fp-4] = saved fp, [fp] = return address), and one
  ARM32 process routinely contains both (an application built by one
  compiler, a libc or shim built by the other). Read both candidate
  records and keep the one whose return address is outside the stack and
  whose saved fp is zero or further up the stack. Both are judged
  against the crashed process's mappings, recorded with their permission
  bits while /proc/<pid>/maps is parsed for modules: a return address must
  be in an executable, non-writable mapping (code, as opposed to a stack -
  writable, and on some systems executable too - or the heap), and a saved
  fp must be in a writable mapping above the current frame. That rejects
  the other shape's record, whose "saved fp" is really a return address,
  even when code sits numerically below the stack. The walk reads records
  out of the one captured stack window, so a chain is followed only while
  it stays inside it; signal stacks and split stacks end the walk. Only
  the crashing thread is stopped while the daemon works, so the mappings
  are recorded (ARM32 only) and re-read, field by field, once the stack
  bytes are copied; if they are incomplete or differ, the ARM32 walk
  stops rather than guess a shape. That narrows, but does not close, the
  window every post-crash capture in this daemon has: the stack bytes and
  the module inventory are read after the crash on every platform, while
  peer threads keep running and could in principle modify or remap the
  stopped thread's stack. Freezing the thread group during capture would
  close it for the generic walk too; that is a separate change.

Thumb code using r7 as its frame pointer is not covered; the walk still
seeds from r11. JIT code in writable mappings is not walked either.

Measured on an LG webOS 4.x television (Cortex-A9, glibc 2.24, in-process
libunwind disabled so the FP fallback runs), with daemons built from the
same tree with and without this change, both including the pointer-width
fix this is stacked on. A GCC-built chain (-O2 -fno-omit-frame-pointer,
noinline f1 -> f2 -> f3 -> leaf, null write in leaf) goes from `leaf` plus
a stack address to `leaf <- f2 <- f1 <- __libc_start_main`; a rustc-built
application is unchanged (`gsignal <- plex_run <- __libc_start_main`),
since its frames already match the assumed shape.
@GLinnik21

Copy link
Copy Markdown
Author

Good catch on g_vma_alloc_failed — it is now reset at the top of capture_modules_from_proc_maps with the other per-capture state, so an allocation failure disables the walk for that capture only.

@GLinnik21
GLinnik21 force-pushed the feat/arm32-crash-daemon branch from f4fb76f to 0e66ba2 Compare September 3, 2026 07:41
GLinnik21 added a commit to GLinnik21/plx-native that referenced this pull request Sep 3, 2026
…pture (#65)

sentry[bot] on getsentry/sentry-native#2053: `g_vma_alloc_failed` was never
reset in `capture_modules_from_proc_maps`, so one allocation failure would
disable the ARM32 frame-pointer walk for every later crash the same daemon
instance handled. Reset it with the other per-capture state. Mirrors the
same one-line change on the upstream PR.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants