Skip to content

fix(callgrind): correct V8 call graphs (Sparkplug returns, whole-stack contexts, pop-on-jump) - #63

Draft
not-matthias wants to merge 3 commits into
masterfrom
cod-4155-remove-obj-skip-for-codspeed-node
Draft

not-matthias wants to merge 3 commits into
masterfrom
cod-4155-remove-obj-skip-for-codspeed-node

Conversation

@not-matthias

@not-matthias not-matthias commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

TLDR: Three changes that make callgrind call graphs of the node binary usable. (1) A callee that returns with SP below its frame is now treated as returning, instead of as calling back into its caller. (2) A new --separate-callers=all separates every function by its whole call stack, so shared V8 builtins stop joining unrelated JS functions. Meant to be combined with --pop-on-jump, which gets two bug fixes. (3) With --pop-on-jump, a callee that pops its return address before a tail jump no longer drops its caller. (amd64 tests; arm64 uses the same paths but isn't covered by CI.)

1. End a call that returns with SP below its frame

  • Why: V8's Sparkplug prologue (Builtins_BaselineOutOfLinePrologue) builds its caller's frame and returns on top of it. Callgrind recorded that as a call back into the caller, so every JS function's callees ended up under the shared prologue.
  • Fix: setup_bbcc gets a third "real return" rule: SP below the top frame and target == that frame's ret_addr → pop that one frame. Equal and higher SP cases are unchanged.
  • Instruction totals unchanged: ret_below_frame gives 291,775 Ir on master and with this change.
  • Tests:
    • ret_below_frame: hand-written amd64 prologue. On master: frame_prologue -> with_prologue_a'2 -> leaf_a. Now: with_prologue_a -> leaf_a.
    • ret_below_frame_node: node --always-sparkplug; the prologue must not call into JIT code. On master: 66 such callees. Now: 0.

2. --separate-callers=all and --pop-on-jump fixes

  • Why: V8 builtins (Call*, CallFunction*, CompileLazy, bytecode handlers) sit between most pairs of JS functions. As single nodes they connect every JS caller to every JS callee, which merges call chains and forms large false cycles.
  • What: all as a value for --separate-callers: each function context is its whole call stack, named in the usual fn'caller'caller… form.
  • --pop-on-jump fixes (needed so tail-jump chains such as bytecode handler dispatch don't grow the stack):
    • After a pop on jump, the new call took the SP of the frame below the replaced one, so its return was often missed and the caller nested again (OPENSSL_LH_doall 179 times in one chain in node -e 0). It now keeps the replaced call's SP.
    • A jump from a stack entry without a call (skipped or seeded frames) hit CLG_ASSERT(nonskipped != 0). It now nests like a normal call.
  • Cost (node -e 0): =all alone: depth 367, 704 MB .out. With --pop-on-jump: depth 90, 309 MB. No depth cap yet.
  • Tests: separate_callers_all_node (callerA → calleeX, callerB → calleeY stay in separate contexts) and pop_on_jump_node (OPENSSL_LH_doall at most once per context).

3. --pop-on-jump: keep the caller of a callee that popped its return address

  • Why: Builtins_InterpreterPushUndefinedAndArgsThenCall (every JS call from Ignition bytecode) pops its return address, pushes arguments and the return address back, then tail-jumps to the callee. The SP rise ends its frame first, so its jump replaced the JS caller's frame and the callee was recorded under the caller's caller.
  • Fix: when an SP rise unwinds a frame while execution stays in that callee, the exec state remembers it. A jump from it nests under the current top frame at the current SP.
  • Test: pop_ra_tail_jump: native amd64 repro. Without the fix: main -> target. With it: caller -> target.

Verification

  • Node flamegraph harness (4 fixtures × Sparkplug/Ignition, JS edges checked against the source): 8/8.
  • Callgrind regtests: 33/34; bug497723 needs an unbuilt memcheck test binary.
  • The vitest e2e numbers from the earlier --separate-callers-jit version no longer apply and have not been re-measured.

Consumers: the runner has to pass --separate-callers=all --pop-on-jump, and the backend has to strip 'caller context suffixes from names (it only strips 'N recursion levels today).

Part of COD-4155. Related: CodSpeedHQ/codspeed-node#121, CodSpeedHQ/codspeed#617

Callgrind only treated a ret as a return when SP moved back above the
frame of the top call, or stayed equal and hit the recorded return address.
A callee that returns with SP lower than it was called with, because it
builds a stack frame for its caller, was turned into a 'RET w/o CALL' jump
and then emulated as a call back into its caller. Everything the caller did
afterwards was recorded under the callee.

V8's Sparkplug prologue (Builtins_BaselineOutOfLinePrologue) works this way.
With the node binary instrumented, every JS function's callees ended up
under the shared prologue, which tied all JS functions into one cycle and
showed non-recursive functions as recursive.

A ret to the return address recorded for the top call now pops that call
even when SP is still below its frame. Instruction totals are unchanged.

Tests: ret_below_frame reproduces the pattern with a hand-written amd64
prologue, ret_below_frame_node checks that the Sparkplug prologue never
calls into JIT code under node --always-sparkplug.
@codspeed

codspeed Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 84 untouched benchmarks
⏩ 60 skipped benchmarks1


Comparing cod-4155-remove-obj-skip-for-codspeed-node (1a437dc) with master (f2aa7a9)

Open in CodSpeed

Footnotes

  1. 60 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@not-matthias not-matthias changed the title fix(callgrind): end a call that returns with SP below its frame fix(callgrind): correct V8 call graphs (Sparkplug returns, builtin contexts) Nov 9, 2026
@not-matthias
not-matthias force-pushed the cod-4155-remove-obj-skip-for-codspeed-node branch from c08aa27 to 93a11f2 Compare November 9, 2026 13:47
--separate-callers=all names every function context by its whole call
stack. With the node binary instrumented, V8 builtins and trampolines sit
between most pairs of JS functions; as single nodes they link every JS
caller to every JS callee and form false cycles. Whole-stack contexts
keep unrelated call chains apart.

Full-stack contexts make the --pop-on-jump bookkeeping visible, so fix
two bugs there:
- After a pop on jump, the delayed push reused the SP of the frame below
  the replaced call. Keep the replaced call's SP, so tail-jump chains
  (V8 bytecode handlers, OpenSSL doall callbacks) no longer nest.
- A jump from a stack entry without a call (skipped or seeded frames)
  hit an assert. It now nests like a normal call.

Without --pop-on-jump, node -e 0 reaches depth 367 and a 704 MB dump;
with it, depth 90 and 309 MB. The docs recommend combining both.

Tests: separate_callers_all_node and pop_on_jump_node, which checks
OPENSSL_LH_doall appears once per context.
A callee that pops its return address and then tail-jumps loses its frame
to the SP rise before the jump: callgrind unwinds it as a return. With
--pop-on-jump, the jump then replaced the caller's frame instead, so the
jump target was recorded under the caller's caller. V8 does this for
every JS call from Ignition bytecode
(Builtins_InterpreterPushUndefinedAndArgsThenCall), which hung JS callees
under the wrong JS function.

When an SP rise unwinds a frame while execution stays in that callee,
remember the callee in the exec state. A jump from it nests under the
current top frame at the current SP, so the target's return is matched.

pop_ra_tail_jump reproduces the pattern in native code: without the fix
it reports main -> target, with it caller -> target.
@not-matthias
not-matthias force-pushed the cod-4155-remove-obj-skip-for-codspeed-node branch from 93a11f2 to 1a437dc Compare November 9, 2026 13:47
@not-matthias not-matthias changed the title fix(callgrind): correct V8 call graphs (Sparkplug returns, builtin contexts) fix(callgrind): correct V8 call graphs (Sparkplug returns, whole-stack contexts, pop-on-jump) Nov 9, 2026
not-matthias added a commit to CodSpeedHQ/codspeed that referenced this pull request Nov 9, 2026
Pass --separate-callers=all --pop-on-jump to callgrind. With the node
executable instrumented, V8 builtins sit between most pairs of JS
functions; as single nodes they join every JS caller to every JS callee
and form false cycles. Whole-stack contexts keep the chains apart, and
--pop-on-jump keeps tail-jump chains (bytecode handler dispatch) from
nesting.

Requires a valgrind-codspeed release with --separate-callers=all
(CodSpeedHQ/valgrind-codspeed#63).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant