From 50a55103ee6be9f45794d9a2e76284e192e3fddc Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Tue, 27 Oct 2026 23:37:21 +0100 Subject: [PATCH 1/7] test(profiler): share virtual-thread remount chaos coverage --- AGENTS.md | 4 + ddprof-stresstest/build.gradle.kts | 9 +- ddprof-stresstest/src/chaos/README.md | 55 +++ .../com/datadoghq/profiler/chaos/Main.java | 2 + .../chaos/VirtualThreadRemountAntagonist.java | 178 +++++++++ ddprof-test/build.gradle.kts | 1 + .../profiler/ContinuationStackChecks.java | 197 ++++++++++ .../profiler/ContinuationStackChecksTest.java | 371 ++++++++++++++++++ .../VirtualThreadCarrierChaosTest.java | 80 ++++ doc/README.md | 2 +- doc/building/TestingGuide.md | 18 + 11 files changed, 915 insertions(+), 2 deletions(-) create mode 100644 ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/VirtualThreadRemountAntagonist.java create mode 100644 ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecks.java create mode 100644 ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecksTest.java create mode 100644 ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadCarrierChaosTest.java diff --git a/AGENTS.md b/AGENTS.md index e62173a1de..05b6dcbc19 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -129,6 +129,10 @@ Two-JDK pattern: `JAVA_HOME` (build, JDK 17+, Gradle runs on 21 LTS) runs Gradle `JAVA_TEST_HOME` runs tests against the target JDK; target bytecode is Java 8 via `--release 8`. +The `ddprof-stresstest:chaosElements` outgoing configuration shares the chaos JAR +with `ddprof-test` stack-correctness tests. See +[TestingGuide.md](doc/building/TestingGuide.md#virtual-thread-remount-correctness). + Details and troubleshooting: [build-logic/README.md](build-logic/README.md), [doc/building/NativeBuildPlugin.md](doc/building/NativeBuildPlugin.md) (task DSL, debug-symbol diff --git a/ddprof-stresstest/build.gradle.kts b/ddprof-stresstest/build.gradle.kts index 67896f2b7d..c774484333 100644 --- a/ddprof-stresstest/build.gradle.kts +++ b/ddprof-stresstest/build.gradle.kts @@ -87,7 +87,7 @@ dependencies { "chaosCompileOnly"(project(mapOf("path" to ":ddprof-lib", "configuration" to "debug"))) } -tasks.register("chaosJar") { +val chaosJar = tasks.register("chaosJar") { group = "build" description = "Fat jar of the chaos reliability harness" archiveFileName.set("chaos.jar") @@ -101,6 +101,13 @@ tasks.register("chaosJar") { } } +// Reuse the same chaos workloads in the profiler's JFR correctness tests. +configurations.create("chaosElements") { + isCanBeConsumed = true + isCanBeResolved = false + outgoing.artifact(chaosJar) +} + // --- reference-chains repro app --------------------------------------------- // Plain, dependency-free demo app for manually reproducing the reference-chains // feature: launched directly with `-agentpath:libjavaProfiler.so=start,...` diff --git a/ddprof-stresstest/src/chaos/README.md b/ddprof-stresstest/src/chaos/README.md index 74e5f19d4b..e04ba20cee 100644 --- a/ddprof-stresstest/src/chaos/README.md +++ b/ddprof-stresstest/src/chaos/README.md @@ -1,3 +1,5 @@ + + # Chaos harness Long-running antagonist workload driving the reliability CI cell. Runs as a @@ -12,11 +14,64 @@ the runner script. | ----------------- | ------------------------------------------------------------------------ | | `thread-churn` | signal-vs-teardown races, slot reuse, calltrace storage put/get | | `vthread-churn` | virtual thread mount/unmount, carrier-thread context, `ProfiledThread` | +| `vthread-remount` | bounded workers parking under deep callers, post-remount CPU work, stack arguments and JIT transitions | | `classloader-churn` | class unload racing stack walk, `CodeCache`/`Symbols` invalidation | | `alloc-storm` | Java alloc engine + GOT-patched libc malloc/free | | `trace-context` | `setTraceContext`/`clearTraceContext` racing signals, span ID propagation | | `reapply-context-value` | concurrent per-slot `setContextValue`/`clearContextValue` churn interleaved with span activation, racing the all-native record's write/read window | +## Virtual-thread carrier recovery + +`vthread-remount` keeps four virtual threads alive and alternates register-only +and stack-passed arguments under 128–224 recursive callers. Each leaf parks, +then burns CPU after remounting. Worker failures and shutdown timeouts propagate +to the harness. It can be selected by name through `run-chaos-harness.sh`'s +fourth argument, including alongside other antagonists. + +`VirtualThreadCarrierChaosTest` profiles this same workload with native CPU and +wall sampling in both `cstack=vmx` and `cstack=vm`. It requires return-barrier +hits, progress in both argument layouts, and at least ten samples with missing +outer callers from each engine and argument layout. It checks carrier presence, +uniqueness and ordering, and requires missing outer callers to remain marked +truncated. Each engine must reach a continuation boundary in at least 99% of +classified workload walks, both per argument layout and overall. Generic walk +errors on carrier OS threads discovered from either engine count toward the +overall failure budget even when no workload frame was decoded. Generic failures +with carrier ancestry still count as failures. Continuation-pointer errors, +invalid ancestry and false completeness fail unconditionally. Completeness checks +apply to workload samples missing a continuation boundary or outer caller. A native +gap alongside complete workload ancestry still consumes the generic failure budget +but does not itself imply missing Java ancestors. A skipped native gap on a known +carrier is excluded from the continuation budget only when its Java frames form +an ordered `Unsafe.park` → optional `LockSupport.park*` → `ForkJoinPool.awaitWork` +→ `ForkJoinPool.runWorker` → `ForkJoinWorkerThread.run` idle chain, with no other +Java frames, continuation roots or other walk errors. These gaps remain visible +in the separate `idleNativeGaps` counter and do not increase successful-walk +counts. Ambiguous carrier samples remain in the failure budget. Error markers +do not bypass carrier uniqueness or ordering checks. Errors on unrelated +threads do not enter the generic failure budget. Profile events are streamed +rather than retained for the whole run. + +The ordinary `testDebug` suite runs six seconds per mode on supported HotSpot +JDK 21+ runtimes. For a longer run, select the build and test JDKs with +`JAVA_HOME` and `JAVA_TEST_HOME`, then run: + +```sh +DDPROF_VTHREAD_STRESS_SECONDS=120 JAVA_TOOL_OPTIONS=-XX:-PreserveFramePointer \ + ./.claude/commands/build-and-summarize :ddprof-test:testDebug \ + -Ptests=VirtualThreadCarrierChaosTest -Pskip-gtest -PkeepJFRs +``` + +Repeat with `JAVA_TOOL_OPTIONS=-XX:+PreserveFramePointer` as a control. +The duration accepts 2–3600 seconds. Test XML and retained JFR recordings include +both the CPU and wall-clock checks, even if one engine fails. Diagnostics include +per-layout coverage, the distribution of failure reasons and the first failing +sample's event type, thread name, OS ID, truncation flag and up to 16 frames. +`ContinuationStackChecksTest` exercises the acceptance checks with controlled +resolved JFR samples, including excessive generic failures and malformed carrier +ancestry. Passing process survival in the agent chaos harness alone does not +establish stack correctness. + ## Deferred - **`dlopen`/`dlclose` churn** — load and unload throwaway native libraries to diff --git a/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/Main.java b/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/Main.java index a8e51e8666..155b6f6d93 100644 --- a/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/Main.java +++ b/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/Main.java @@ -74,6 +74,8 @@ private static Antagonist create(String name) { return new AllocStormAntagonist(); case "vthread-churn": return new VirtualThreadChurnAntagonist(); + case "vthread-remount": + return new VirtualThreadRemountAntagonist(); case "classloader-churn": return new ClassLoaderChurnAntagonist(); case "trace-context": diff --git a/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/VirtualThreadRemountAntagonist.java b/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/VirtualThreadRemountAntagonist.java new file mode 100644 index 0000000000..2c2b272739 --- /dev/null +++ b/ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/VirtualThreadRemountAntagonist.java @@ -0,0 +1,178 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ +package com.datadoghq.profiler.chaos; + +import java.lang.reflect.Method; +import java.time.Duration; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.LockSupport; + +/** + * Repeatedly parks a bounded set of virtual threads under deep Java callers, then burns CPU after + * remounting. Alternating argument layouts and recursion depths exercise partial thaw while the + * recursive methods progress through interpreted, compiled and OSR execution. + */ +public final class VirtualThreadRemountAntagonist implements Antagonist { + private static final Method START_VIRTUAL_THREAD = resolveStartVirtualThread(); + private final Worker[] workers; + private final AtomicReference failure = new AtomicReference<>(); + private volatile boolean running; + private boolean started; + + /** Creates four workers, with no thread or allocation churn in their recursive hot loops. */ + public VirtualThreadRemountAntagonist() { + this(4); + } + + /** Creates between one and sixteen workers; each alternates both argument layouts. */ + public VirtualThreadRemountAntagonist(int workerCount) { + if (workerCount < 1 || workerCount > 16) throw new IllegalArgumentException("workerCount must be 1..16"); + workers = new Worker[workerCount]; + for (int i = 0; i < workers.length; i++) workers[i] = new Worker(i); + } + + /** Name accepted by the chaos harness. */ + @Override + public String name() { + return "vthread-remount"; + } + + /** Starts the workers once; gracefully skips runtimes without virtual threads. */ + @Override + public void start() { + if (started) throw new IllegalStateException("Already started"); + started = true; + if (START_VIRTUAL_THREAD == null) { + System.out.println("[chaos] vthread-remount: skipping (virtual threads unavailable)"); + return; + } + running = true; + try { + for (Worker worker : workers) { + worker.thread = (Thread) START_VIRTUAL_THREAD.invoke(null, worker); + } + } catch (ReflectiveOperationException e) { + failure.compareAndSet(null, e); + stopGracefully(Duration.ofSeconds(10)); + } + } + + /** Unparks and joins every worker within one shared deadline; propagates worker failures. */ + @Override + public void stopGracefully(Duration timeout) { + running = false; + for (Worker worker : workers) { + if (worker.thread != null) worker.thread.interrupt(); + } + long deadline = System.nanoTime() + timeout.toNanos(); + boolean interrupted = false; + for (Worker worker : workers) { + if (worker.thread == null) continue; + long remaining = deadline - System.nanoTime(); + if (remaining <= 0) break; + try { + worker.thread.join(Math.max(1, TimeUnit.NANOSECONDS.toMillis(remaining))); + } catch (InterruptedException e) { + interrupted = true; + break; + } + } + if (interrupted) Thread.currentThread().interrupt(); + for (Worker worker : workers) { + if (worker.thread != null && worker.thread.isAlive()) { + throw new IllegalStateException("vthread-remount worker did not stop"); + } + } + if (failure.get() != null) throw new IllegalStateException("vthread-remount worker failed", failure.get()); + } + + /** Number of completed register-argument park/remount cycles across all workers. */ + public long registerCycles() { + long count = 0; + for (Worker worker : workers) count += worker.registerCycles; + return count; + } + + /** Number of completed stack-argument park/remount cycles across all workers. */ + public long stackCycles() { + long count = 0; + for (Worker worker : workers) count += worker.stackCycles; + return count; + } + + private final class Worker implements Runnable { + private final int id; + private Thread thread; + private volatile long registerCycles; + private volatile long stackCycles; + private volatile long checksum; + + private Worker(int id) { + this.id = id; + } + + /** Alternates layouts until stopped, retaining any failure for the owner. */ + @Override + public void run() { + try { + long cycle = id; + while (running) { + int depth = 128 + 32 * (int) (cycle & 3); + registerWorkload(this, depth, cycle); + if (!running) break; + stackWorkload(this, depth, cycle); + cycle++; + } + } catch (Throwable t) { + failure.compareAndSet(null, t); + running = false; + } + } + } + + /** This outer caller always exists logically, even if it remains frozen at sampling time. */ + private void registerWorkload(Worker worker, int depth, long seed) { + worker.checksum = registerFrames(depth, seed); + worker.registerCycles++; + } + + /** Keeps recursive callers live while work in the leaf is sampled after a park. */ + private long registerFrames(int depth, long seed) { + if (depth > 0) return registerFrames(depth - 1, seed) + depth; + LockSupport.parkNanos(500_000L); + long deadline = System.nanoTime() + 2_000_000L; + do { + for (int i = 0; i < 256; i++) seed = seed * 6364136223846793005L + 1442695040888963407L; + } while (running && System.nanoTime() < deadline); + return seed; + } + + /** Establishes an outer caller for the stack-passed argument variant. */ + private void stackWorkload(Worker worker, int depth, long seed) { + worker.checksum = stackFrames(depth, seed, 2, 3, 4, 5, 6, 7, 8, 9); + worker.stackCycles++; + } + + /** Keeps nine long arguments live across recursive calls, partial thaw and compilation. */ + private long stackFrames(int depth, long a, long b, long c, long d, long e, long f, long g, long h, long i) { + if (depth > 0) return stackFrames(depth - 1, a, b, c, d, e, f, g, h, i) + depth + b + c + d + e + f + g + h + i; + LockSupport.parkNanos(500_000L); + long deadline = System.nanoTime() + 2_000_000L; + do { + for (int n = 0; n < 256; n++) a = a * 6364136223846793005L + 1442695040888963407L; + } while (running && System.nanoTime() < deadline); + return a + b + c + d + e + f + g + h + i; + } + + /** Resolves the Java 21 factory while keeping the harness compatible with Java 8. */ + private static Method resolveStartVirtualThread() { + try { + return Thread.class.getMethod("startVirtualThread", Runnable.class); + } catch (NoSuchMethodException e) { + return null; + } + } +} diff --git a/ddprof-test/build.gradle.kts b/ddprof-test/build.gradle.kts index e0fa1e7a23..d19b88a452 100644 --- a/ddprof-test/build.gradle.kts +++ b/ddprof-test/build.gradle.kts @@ -54,6 +54,7 @@ dependencies { "testCommon"(libs.bundles.profiler.runtime) "testCommon"(libs.jafar.parser) "testCommon"(libs.asm) + "testCommon"(project(mapOf("path" to ":ddprof-stresstest", "configuration" to "chaosElements"))) // Main/application dependencies "mainCommon"(libs.slf4j.simple) diff --git a/ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecks.java b/ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecks.java new file mode 100644 index 0000000000..e1b509ae59 --- /dev/null +++ b/ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecks.java @@ -0,0 +1,197 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ +package com.datadoghq.profiler; + +import com.datadoghq.profiler.chaos.VirtualThreadRemountAntagonist; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.junit.jupiter.api.function.Executable; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Checks continuation ancestry and bounds failed walks in a remount workload recording. */ +public final class ContinuationStackChecks { + private static final String WORKLOAD_CLASS = VirtualThreadRemountAntagonist.class.getName(); + private static final int MIN_SAMPLES = 10; + private static final int MIN_SUCCESS_PERCENT = 99; + private final boolean carrierFrames; + private final Set carrierIds; + private final long[] eligible = new long[2]; + private final long[] partialThaws = new long[2]; + private final long[] otherStops = new long[2]; + private final Map failureReasons = new LinkedHashMap<>(); + private long carrierOnlyFailures; + private long idleNativeGaps; + private long continuationErrors; + private long invalidAncestry; + private long falseComplete; + private String firstFailure; + + /** Takes carrier OS IDs discovered from workload samples in either sampling engine. */ + public ContinuationStackChecks(boolean carrierFrames, Set carrierIds) { + this.carrierFrames = carrierFrames; + this.carrierIds = new HashSet<>(carrierIds); + } + + /** Discovers carriers before checking failures that contain no application frames. */ + public static void collectCarriers(Set ids, JfrEvent event) { + for (JfrFrame frame : event.getStackTrace().frames()) { + if (WORKLOAD_CLASS.equals(frame.className()) && ("registerFrames".equals(frame.methodName()) + || "stackFrames".equals(frame.methodName()))) { + long id = carrierId(event); + assertTrue(id > 0, "Workload sample has no carrier OS ID: " + event); + ids.add(id); + return; + } + } + } + + private static long carrierId(JfrEvent event) { + Object thread = event.get("eventThread"); + Object id = thread instanceof Map ? ((Map) thread).get("osThreadId") : null; + return id instanceof Number ? ((Number) id).longValue() : -1; + } + + /** Classifies one sample; keeps counts and only the first failing stack. */ + public void accept(JfrEvent event) { + JfrStackTrace trace = event.getStackTrace(); + boolean register = false, stack = false, outer = false; + String walkError = null; + boolean continuationError = false; + int carriers = 0, carrierIndex = -1, lastApplicationIndex = -1, roots = 0, index = 0; + for (JfrFrame frame : trace.frames()) { + String method = frame.methodName(); + if (WORKLOAD_CLASS.equals(frame.className())) { + register |= "registerFrames".equals(method); + stack |= "stackFrames".equals(method); + outer |= "registerWorkload".equals(method) || "stackWorkload".equals(method); + lastApplicationIndex = index; + } + if ("java.util.concurrent.ForkJoinWorkerThread".equals(frame.className()) && "run".equals(method)) { + carriers++; + carrierIndex = index; + } + if ("JVM Continuation".equals(method)) roots++; + if (method != null) { + continuationError |= method.startsWith("break_cont_"); + if (walkError == null && (method.startsWith("break_") || "unknown_nmethod".equals(method) + || "no_Java_frame".equals(method) || "[skipped frames]".equals(method))) walkError = method; + } + index++; + } + boolean workload = register || stack; + boolean knownCarrier = carrierIds.contains(carrierId(event)); + if (continuationError) continuationErrors++; + if (trace.isEmpty() && knownCarrier) walkError = "empty stack"; + boolean failedWalk = walkError != null && (workload || knownCarrier || continuationError); + // Only a complete idle scheduler chain establishes that no continuation walk was attempted. + // Ambiguous carrier samples and all other error markers keep consuming the failure budget. + boolean idleNativeGap = knownCarrier && !workload && "[skipped frames]".equals(walkError) && isIdleCarrier(trace); + if (idleNativeGap) idleNativeGaps++; + else if (failedWalk) { + if (workload) otherStops[stack ? 1 : 0]++; + else if (knownCarrier) carrierOnlyFailures++; + recordFailure(event, walkError); + } + if (!workload) return; + int layout = stack ? 1 : 0; + // A skipped native region can coexist with complete Java ancestry. Require + // truncation only when this workload's continuation boundary or outer caller is missing. + if (!trace.isTruncated() && (carriers == 0 && roots == 0 || !outer)) { + falseComplete++; + recordFailure(event, "missing Java ancestor reported complete"); + } + if (carriers == 0 && roots == 0) { + if (!failedWalk) { + otherStops[layout]++; + recordFailure(event, "missing continuation boundary"); + } + return; + } + if (register && stack || (carrierFrames + ? carriers != 1 || roots != 0 || carrierIndex <= lastApplicationIndex + : carriers != 0 || roots != 1)) { + invalidAncestry++; + recordFailure(event, "invalid carrier ancestry"); + } else if (!failedWalk && !outer && trace.isTruncated()) partialThaws[layout]++; + if (!failedWalk) eligible[layout]++; + } + + /** Requires an ordered park/idle-worker chain with no application frames or continuation evidence. */ + private static boolean isIdleCarrier(JfrStackTrace trace) { + int phase = 0; + for (JfrFrame frame : trace.frames()) { + String owner = frame.className(), method = frame.methodName(); + if (owner == null || method == null) return false; + if (owner.isEmpty()) { + // Check every marker: [skipped frames] can precede a more specific failure. + if (method.startsWith("break_") || "unknown_nmethod".equals(method) + || "no_Java_frame".equals(method) || "JVM Continuation".equals(method)) return false; + continue; + } + if (phase == 0 && "jdk.internal.misc.Unsafe".equals(owner) && "park".equals(method)) phase = 1; + else if (phase == 1 && "java.util.concurrent.locks.LockSupport".equals(owner) + && ("park".equals(method) || "parkUntil".equals(method) || "parkNanos".equals(method))) continue; + else if (phase == 1 && "java.util.concurrent.ForkJoinPool".equals(owner) && "awaitWork".equals(method)) phase = 2; + else if (phase == 2 && "java.util.concurrent.ForkJoinPool".equals(owner) && "runWorker".equals(method)) phase = 3; + else if (phase == 3 && "java.util.concurrent.ForkJoinWorkerThread".equals(owner) && "run".equals(method)) phase = 4; + else return false; + } + return phase == 4; + } + + private void recordFailure(JfrEvent event, String reason) { + failureReasons.merge(reason, 1L, Long::sum); + if (firstFailure != null) return; + StringBuilder text = new StringBuilder(event.typeName()).append(" thread=") + .append(event.getThreadName("eventThread")).append(" osId=").append(carrierId(event)) + .append(" truncated=").append(event.getStackTrace().isTruncated()).append(" reason=").append(reason); + List frames = event.getStackTrace().frames(); + for (int i = 0; i < Math.min(16, frames.size()); i++) { + JfrFrame frame = frames.get(i); + text.append("\n ").append(frame.className()).append('.').append(frame.methodName()); + } + firstFailure = text.toString(); + } + + /** Requires coverage in both layouts and at least 99% successful boundary walks. */ + public void assertValid(String eventType) { + List checks = new ArrayList<>(); + checks.add(() -> assertFalse(carrierIds.isEmpty(), "No workload carriers identified")); + checks.add(() -> assertEquals(0, continuationErrors, "Continuation recovery errors")); + checks.add(() -> assertEquals(0, invalidAncestry, "Missing, duplicated or misplaced carrier frames")); + checks.add(() -> assertEquals(0, falseComplete, "Incomplete walks reported complete")); + for (int layout = 0; layout < 2; layout++) { + final int i = layout; + checks.add(() -> assertTrue(eligible[i] >= MIN_SAMPLES, "Insufficient boundary samples for layout " + i)); + checks.add(() -> assertTrue(partialThaws[i] >= MIN_SAMPLES, "Insufficient partial-thaw samples for layout " + i)); + checks.add(() -> assertTrue(eligible[i] * 100 >= (eligible[i] + otherStops[i]) * MIN_SUCCESS_PERCENT, + "Fewer than 99% successful boundary walks for layout " + i)); + } + long successful = eligible[0] + eligible[1]; + long attempts = successful + otherStops[0] + otherStops[1] + carrierOnlyFailures; + checks.add(() -> assertTrue(successful * 100 >= attempts * MIN_SUCCESS_PERCENT, + "Fewer than 99% successful boundary walks including carrier-only failures")); + assertAll(eventType + " carrier=" + carrierFrames + " " + this, checks); + } + + /** Reports coverage, failure reasons and the first offending thread and stack. */ + @Override + public String toString() { + return "eligible=" + Arrays.toString(eligible) + " partialThaws=" + Arrays.toString(partialThaws) + + " otherStops=" + Arrays.toString(otherStops) + " carrierOnlyFailures=" + carrierOnlyFailures + + " idleNativeGaps=" + idleNativeGaps + " continuationErrors=" + continuationErrors + " invalidAncestry=" + invalidAncestry + + " falseComplete=" + falseComplete + " failureReasons=" + failureReasons + + (firstFailure == null ? "" : "\nFirst failure: " + firstFailure); + } +} diff --git a/ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecksTest.java b/ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecksTest.java new file mode 100644 index 0000000000..7cf003dfe6 --- /dev/null +++ b/ddprof-test/src/test/java/com/datadoghq/profiler/ContinuationStackChecksTest.java @@ -0,0 +1,371 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ +package com.datadoghq.profiler; + +import com.datadoghq.profiler.chaos.VirtualThreadRemountAntagonist; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import org.junit.jupiter.api.Test; +import org.opentest4j.MultipleFailuresError; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Exercises the chaos recording oracle with resolved JFR samples, including failure substitutions. */ +public class ContinuationStackChecksTest { + private static final String WORKLOAD = VirtualThreadRemountAntagonist.class.getName(); + private static final String CARRIER = "java.util.concurrent.ForkJoinWorkerThread#run"; + + private static Map fields(Object... pairs) { + Map values = new HashMap<>(); + for (int i = 0; i < pairs.length; i += 2) values.put((String) pairs[i], pairs[i + 1]); + return values; + } + + /** Builds the same resolved field layout consumed from an actual profiler recording. */ + private static JfrEvent sample(long carrier, boolean truncated, String... frames) { + Object[] stack = Arrays.stream(frames).map(frame -> { + String[] name = frame.split("#", -1); + return fields("method", fields("type", fields("name", name[0]), "name", name[1])); + }).toArray(); + return new JfrEvent("datadog.ExecutionSample", fields( + "eventThread", fields("osThreadId", carrier, "osName", "carrier-" + carrier), + "stackTrace", fields("frames", stack, "truncated", truncated))); + } + + private static ContinuationStackChecks covered(boolean carriers, int samplesPerLayout) { + ContinuationStackChecks checks = new ContinuationStackChecks(carriers, Collections.singleton(7L)); + for (int i = 0; i < samplesPerLayout; i++) { + for (String method : new String[]{"registerFrames", "stackFrames"}) { + checks.accept(sample(7, true, WORKLOAD + "#" + method, carriers ? CARRIER : "#JVM Continuation")); + } + } + return checks; + } + + /** Both VM modes must accept valid, truncated partial walks in both argument layouts. */ + @Test + public void acceptsCoveredPartialWalksInBothModes() { + covered(false, 10).assertValid("wall"); + covered(true, 10).assertValid("cpu"); + } + + /** A complete idle scheduler chain identifies a parked carrier, even with a native gap. */ + private static JfrEvent idleSample(long carrier, boolean truncated, String parkMethod, String... prefix) { + String[] frames = Arrays.copyOf(prefix, prefix.length + 5); + frames[prefix.length] = "jdk.internal.misc.Unsafe#park"; + frames[prefix.length + 1] = "java.util.concurrent.locks.LockSupport#" + parkMethod; + frames[prefix.length + 2] = "java.util.concurrent.ForkJoinPool#awaitWork"; + frames[prefix.length + 3] = "java.util.concurrent.ForkJoinPool#runWorker"; + frames[prefix.length + 4] = CARRIER; + return sample(carrier, truncated, frames); + } + + /** Confirmed idle gaps remain visible but do not consume continuation coverage or its budget. */ + @Test + public void excludesConfirmedIdleNativeGapBurstsFromContinuationBudget() { + for (boolean carriers : new boolean[]{false, true}) { + for (boolean truncated : new boolean[]{false, true}) { + for (String park : new String[]{"park", "parkUntil", "parkNanos"}) { + ContinuationStackChecks checks = covered(carriers, 10); + for (int i = 0; i < 1000; i++) { + checks.accept(idleSample(7, truncated, park, "#__syscall_cancel_arch_start", "#[skipped frames]")); + } + checks.assertValid("wall"); + assertTrue(checks.toString().contains("idleNativeGaps=1000")); + assertTrue(checks.toString().contains("carrierOnlyFailures=0")); + assertTrue(checks.toString().contains("failureReasons={}")); + } + } + } + } + + /** No workload frames is insufficient evidence of idleness; all ordered scheduler frames are required. */ + @Test + public void rejectsIncompleteMisorderedOrDuplicatedIdleSignatures() { + String unsafe = "jdk.internal.misc.Unsafe#park"; + String await = "java.util.concurrent.ForkJoinPool#awaitWork"; + String worker = "java.util.concurrent.ForkJoinPool#runWorker"; + for (String[] frames : new String[][]{ + {"#[skipped frames]", await, worker, CARRIER}, + {"#[skipped frames]", unsafe, worker, CARRIER}, + {"#[skipped frames]", unsafe, await, CARRIER}, + {"#[skipped frames]", unsafe, await, worker}, + {"#[skipped frames]", unsafe, worker, await, CARRIER}, + {"#[skipped frames]", CARRIER, unsafe, await, worker}, + {"#[skipped frames]", unsafe, await, worker, CARRIER, CARRIER}}) { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(sample(7, false, frames)); + assertThrows(AssertionError.class, () -> checks.assertValid("wall")); + assertTrue(checks.toString().contains("idleNativeGaps=0")); + assertTrue(checks.toString().contains("carrierOnlyFailures=1")); + } + } + + /** Application or continuation evidence prevents an idle-looking suffix from hiding a failed walk. */ + @Test + public void rejectsIdleLookingStacksWithApplicationOrContinuationFrames() { + for (String frame : new String[]{WORKLOAD + "#registerFrames", WORKLOAD + "#registerWorkload", + "example.Task#run", "jdk.internal.vm.Continuation#run", "java.lang.VirtualThread#runContinuation", + "#JVM Continuation"}) { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(idleSample(7, true, "park", "#[skipped frames]", frame)); + assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + assertTrue(checks.toString().contains("idleNativeGaps=0")); + } + } + + /** A skipped marker must not conceal a second, more specific walk error on an idle-looking stack. */ + @Test + public void rejectsOtherWalkErrorsAlongsideIdleNativeGaps() { + for (String error : new String[]{"break_unwind_stub_failed", "break_not_walkable", "unknown_nmethod", "no_Java_frame"}) { + for (boolean first : new boolean[]{false, true}) { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(idleSample(7, false, "park", first ? "#" + error : "#[skipped frames]", + first ? "#[skipped frames]" : "#" + error)); + assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + assertTrue(checks.toString().contains("idleNativeGaps=0")); + assertTrue(checks.toString().contains("carrierOnlyFailures=1")); + } + } + } + + /** Continuation errors remain unconditional failures on both known and unidentified idle-looking threads. */ + @Test + public void rejectsContinuationErrorsAlongsideIdleNativeGaps() { + for (long carrier : new long[]{7, 99}) { + ContinuationStackChecks checks = covered(true, 100); + checks.accept(idleSample(carrier, false, "park", "#[skipped frames]", "#break_cont_entry_fp")); + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, () -> checks.assertValid("wall")); + assertEquals(1, failure.getFailures().size()); + assertTrue(failure.getFailures().get(0).getMessage().startsWith("Continuation recovery errors")); + assertTrue(checks.toString().contains("idleNativeGaps=0")); + } + } + + /** Excluded idle samples cannot improve the success ratio or hide ambiguous carrier failures. */ + @Test + public void idleGapsCannotDiluteFailureBudget() { + ContinuationStackChecks checks = covered(true, 10); + for (int i = 0; i < 1000; i++) checks.accept(idleSample(7, false, "park", "#[skipped frames]")); + checks.accept(sample(7, true, "#[skipped frames]", CARRIER)); + assertThrows(AssertionError.class, () -> checks.assertValid("wall")); + assertTrue(checks.toString().contains("eligible=[10, 10]")); + assertTrue(checks.toString().contains("carrierOnlyFailures=1")); + assertTrue(checks.toString().contains("idleNativeGaps=1000")); + } + + /** Complete workload ancestry can coexist with a skipped native region below the sampled leaf. */ + @Test + public void acceptsNontruncatedNativeGapWithCompleteJavaAncestryWithinFailureBudget() { + for (boolean carriers : new boolean[]{false, true}) { + for (String layout : new String[]{"register", "stack"}) { + ContinuationStackChecks checks = covered(carriers, 100); + checks.accept(sample(7, false, "#native_leaf", "#[skipped frames]", + WORKLOAD + "#" + layout + "Frames", WORKLOAD + "#" + layout + "Workload", + carriers ? CARRIER : "#JVM Continuation")); + checks.assertValid("cpu"); + } + } + } + + /** Workload gaps and ambiguous parked samples keep consuming the unchanged failure budget. */ + @Test + public void rejectsNativeGapBurstsWithoutClaimingMissingJavaAncestry() { + for (boolean carriers : new boolean[]{false, true}) { + for (boolean workload : new boolean[]{false, true}) { + ContinuationStackChecks checks = covered(carriers, 100); + for (int i = 0; i < 10; i++) { + checks.accept(workload + ? sample(7, false, "#[skipped frames]", WORKLOAD + "#registerFrames", + WORKLOAD + "#registerWorkload", carriers ? CARRIER : "#JVM Continuation") + : sample(7, false, "#[skipped frames]", "jdk.internal.misc.Unsafe#park", CARRIER)); + } + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, + () -> checks.assertValid("wall")); + assertEquals(workload ? 2 : 1, failure.getFailures().size()); + assertTrue(failure.getFailures().stream() + .allMatch(error -> error.getMessage().startsWith("Fewer than 99%"))); + assertTrue(failure.getMessage().contains("[skipped frames]=10")); + } + } + } + + /** A native gap must not hide a missing caller or boundary that was reported complete. */ + @Test + public void rejectsMissingJavaAncestorsWithOrWithoutNativeGap() { + for (boolean carriers : new boolean[]{false, true}) { + for (String layout : new String[]{"register", "stack"}) { + for (boolean nativeGap : new boolean[]{false, true}) { + for (boolean boundary : new boolean[]{false, true}) { + ContinuationStackChecks checks = covered(carriers, 100); + checks.accept(sample(7, false, nativeGap ? "#[skipped frames]" : "#native_leaf", + WORKLOAD + "#" + layout + "Frames", + boundary ? (carriers ? CARRIER : "#JVM Continuation") + : WORKLOAD + "#" + layout + "Workload")); + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, + () -> checks.assertValid("cpu")); + assertEquals(1, failure.getFailures().size()); + assertTrue(failure.getFailures().get(0).getMessage() + .startsWith("Incomplete walks reported complete")); + } + } + } + } + } + + /** Truncation permits missing Java ancestors while a native gap still consumes the budget. */ + @Test + public void acceptsTruncatedMissingJavaAncestorsWithNativeGapWithinFailureBudget() { + for (boolean carriers : new boolean[]{false, true}) { + for (String layout : new String[]{"register", "stack"}) { + for (boolean boundary : new boolean[]{false, true}) { + ContinuationStackChecks checks = covered(carriers, 100); + checks.accept(sample(7, true, "#[skipped frames]", WORKLOAD + "#" + layout + "Frames", + boundary ? (carriers ? CARRIER : "#JVM Continuation") : "#native_frame")); + checks.assertValid("wall"); + } + } + } + } + + /** Error markers must not bypass carrier uniqueness and ordering checks. */ + @Test + public void rejectsInvalidCarrierAncestryDespiteNativeGap() { + for (JfrEvent event : Arrays.asList( + sample(7, true, "#[skipped frames]", WORKLOAD + "#registerFrames", CARRIER, CARRIER), + sample(7, true, "#[skipped frames]", CARRIER, WORKLOAD + "#registerFrames"))) { + ContinuationStackChecks checks = covered(true, 100); + checks.accept(event); + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, + () -> checks.assertValid("cpu")); + assertEquals(1, failure.getFailures().size()); + assertTrue(failure.getFailures().get(0).getMessage() + .startsWith("Missing, duplicated or misplaced carrier frames")); + } + } + + /** Ten successful samples cannot conceal widespread generic failures without workload frames. */ + @Test + public void rejectsGenericFailuresOnKnownCarriers() { + ContinuationStackChecks checks = covered(true, 10); + for (int i = 0; i < 1000; i++) checks.accept(sample(7, true, "#break_not_walkable")); + AssertionError failure = assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + assertTrue(failure.getMessage().contains("including carrier-only failures")); + assertTrue(failure.getMessage().contains("carrier-7")); + assertTrue(failure.getMessage().contains("break_not_walkable")); + } + + /** Failures retain their layout even when a carrier frame happens to be present. */ + @Test + public void rejectsFailedWalksThatStillReachCarrierAncestry() { + ContinuationStackChecks checks = covered(true, 10); + for (int i = 0; i < 100; i++) { + checks.accept(sample(7, true, WORKLOAD + "#stackFrames", "#break_not_walkable", CARRIER)); + } + AssertionError failure = assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + assertTrue(failure.getMessage().contains("successful boundary walks for layout 1")); + } + + /** One healthy layout must not hide a degraded layout in the combined success rate. */ + @Test + public void boundsEachLayoutIndependently() { + ContinuationStackChecks checks = covered(true, 10); + for (int i = 0; i < 1000; i++) { + checks.accept(sample(7, true, WORKLOAD + "#registerFrames", CARRIER)); + } + checks.accept(sample(7, true, WORKLOAD + "#stackFrames", "#break_not_walkable")); + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, () -> checks.assertValid("cpu")); + assertEquals(1, failure.getFailures().size()); + assertTrue(failure.getMessage().contains("successful boundary walks for layout 1")); + } + + /** Fully thawed samples cannot satisfy the required coverage of omitted outer callers. */ + @Test + public void requiresPartialThawCoverageForEachLayout() { + ContinuationStackChecks checks = new ContinuationStackChecks(true, Collections.singleton(7L)); + for (int i = 0; i < 10; i++) { + checks.accept(sample(7, true, WORKLOAD + "#registerFrames", CARRIER)); + checks.accept(sample(7, false, WORKLOAD + "#stackFrames", WORKLOAD + "#stackWorkload", CARRIER)); + } + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, () -> checks.assertValid("cpu")); + assertEquals(1, failure.getFailures().size()); + assertTrue(failure.getMessage().contains("Insufficient partial-thaw samples for layout 1")); + } + + /** Empty stacks on known carriers must not escape the generic failure budget. */ + @Test + public void rejectsEmptyStacksOnKnownCarriers() { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(sample(7, true)); + AssertionError failure = assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + assertTrue(failure.getMessage().contains("reason=empty stack")); + } + + /** Rare generic stops are bounded by the declared 99% gate, rather than an absolute count. */ + @Test + public void boundsFailuresRelativeToSuccessfulSamples() { + ContinuationStackChecks checks = covered(true, 100); + checks.accept(sample(7, true, WORKLOAD + "#registerFrames", "#break_not_walkable")); + checks.accept(sample(7, true, WORKLOAD + "#stackFrames", "#break_not_walkable")); + checks.assertValid("cpu"); + checks.accept(sample(7, true, "#break_not_walkable")); + assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + } + + /** Unrelated native threads do not count against workload-carrier success. */ + @Test + public void scopesGenericFailuresToIdentifiedCarriers() { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(sample(99, true, "#break_not_walkable")); + checks.assertValid("cpu"); + Set carriers = new HashSet<>(); + ContinuationStackChecks.collectCarriers(carriers, sample(7, true, WORKLOAD + "#registerFrames", CARRIER)); + assertEquals(Collections.singleton(7L), carriers); + } + + /** Error-only continuation samples fail even when no workload frames or known carrier survive. */ + @Test + public void rejectsContinuationErrorsWithoutApplicationFrames() { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(sample(99, true, "#break_cont_entry_fp")); + AssertionError failure = assertThrows(AssertionError.class, () -> checks.assertValid("wall")); + assertTrue(failure.getMessage().contains("Continuation recovery errors")); + assertTrue(failure.getMessage().contains("thread=carrier-99 osId=99 truncated=true")); + assertTrue(failure.getMessage().contains(".break_cont_entry_fp")); + } + + /** Wrong ancestry and omitted callers reported complete must remain failures. */ + @Test + public void rejectsDuplicatedMisplacedOrFalselyCompleteAncestry() { + for (JfrEvent event : Arrays.asList( + sample(7, true, WORKLOAD + "#registerFrames", CARRIER, CARRIER), + sample(7, true, CARRIER, WORKLOAD + "#registerFrames"), + sample(7, false, WORKLOAD + "#registerFrames", CARRIER))) { + ContinuationStackChecks checks = covered(true, 10); + checks.accept(event); + assertThrows(AssertionError.class, () -> checks.assertValid("cpu")); + } + } + + /** A CPU failure must not prevent reporting the independent wall-clock failure. */ + @Test + public void reportsBothSamplingEnginesWhenBothFail() { + ContinuationStackChecks cpu = covered(true, 10), wall = covered(true, 10); + cpu.accept(sample(7, true, "#break_cont_entry_fp")); + wall.accept(sample(7, true, "#break_cont_carrier_sp")); + MultipleFailuresError failure = assertThrows(MultipleFailuresError.class, + () -> assertAll(() -> cpu.assertValid("cpu"), () -> wall.assertValid("wall"))); + assertEquals(2, failure.getFailures().size()); + assertTrue(failure.getMessage().contains("break_cont_entry_fp")); + assertTrue(failure.getMessage().contains("break_cont_carrier_sp")); + } +} diff --git a/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadCarrierChaosTest.java b/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadCarrierChaosTest.java new file mode 100644 index 0000000000..78e42ca562 --- /dev/null +++ b/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadCarrierChaosTest.java @@ -0,0 +1,80 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ +package com.datadoghq.profiler.wallclock; + +import com.datadoghq.profiler.CStackAwareAbstractProfilerTest; +import com.datadoghq.profiler.ContinuationStackChecks; +import com.datadoghq.profiler.Platform; +import com.datadoghq.profiler.chaos.VirtualThreadRemountAntagonist; +import com.datadoghq.profiler.junit.CStack; +import java.time.Duration; +import java.util.HashSet; +import java.util.Set; +import org.junit.jupiter.api.TestTemplate; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Checks CPU and wall-clock stack correctness while a shared chaos workload repeatedly remounts. */ +public class VirtualThreadCarrierChaosTest extends CStackAwareAbstractProfilerTest { + private final boolean carrierFrames; + + /** Selects carrier walking or the ordinary VM-mode control. */ + public VirtualThreadCarrierChaosTest(@CStack String cstack) { + super(cstack); + carrierFrames = "vmx".equals(cstack); + } + + @Override + protected boolean isPlatformSupported() { + return Platform.isJavaVersionAtLeast(21) && !Platform.isJ9() && !Platform.isZing() + && !(isInCI() && Platform.isMusl() && !Platform.isAarch64()); + } + + @Override + protected String getProfilerCommand() { + return "cpu=2ms,wall=2ms,walltpt=64,filter=,jstackdepth=512"; + } + + /** Requires exercised return barriers and checks both layouts in each sampling engine. */ + @TestTemplate + @ValueSource(strings = {"vm", "vmx"}) + public void retainsCarrierAncestryDuringRepeatedRemounts() throws Exception { + String seconds = System.getenv("DDPROF_VTHREAD_STRESS_SECONDS"); + long duration = seconds == null ? 6 : Long.parseLong(seconds); + assertTrue(duration >= 2 && duration <= 3600, "Stress duration must be 2..3600 seconds"); + waitForProfilerReady(2_000); + VirtualThreadRemountAntagonist workload = new VirtualThreadRemountAntagonist(); + try { + workload.start(); + Thread.sleep(duration * 1000); + } finally { + try { + workload.stopGracefully(Duration.ofSeconds(10)); + } finally { + stopProfiler(); + } + } + assertTrue(workload.registerCycles() >= 10, "Register-argument workers made insufficient progress"); + assertTrue(workload.stackCycles() >= 10, "Stack-argument workers made insufficient progress"); + assertTrue(profiler.getDebugCounters().getOrDefault("walkvm_cont_barrier_hit", 0L) >= 20, + "The stress run must sample continuation return barriers"); + verifyCStackSettings(); + Set carriers = reduceEvents("datadog.ExecutionSample", HashSet::new, ContinuationStackChecks::collectCarriers); + carriers.addAll(reduceEvents("datadog.MethodSample", HashSet::new, ContinuationStackChecks::collectCarriers)); + assertAll( + () -> assertStacks("datadog.ExecutionSample", carriers), + () -> assertStacks("datadog.MethodSample", carriers)); + } + + /** Checks both engines against the same carriers without retaining every sampled stack. */ + private void assertStacks(String eventType, Set carriers) { + ContinuationStackChecks counts = reduceEvents(eventType, + () -> new ContinuationStackChecks(carrierFrames, carriers), ContinuationStackChecks::accept); + System.out.println(eventType + " carrier=" + carrierFrames + " " + counts); + counts.assertValid(eventType); + } +} diff --git a/doc/README.md b/doc/README.md index 42a8675e53..ce932c73a1 100644 --- a/doc/README.md +++ b/doc/README.md @@ -36,7 +36,7 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). - [GradleTasks](building/GradleTasks.md) - Available Gradle tasks reference - [JdkUpgrades](building/JdkUpgrades.md) - Two-JDK pattern and JDK upgrade touchpoints - [NativeBuildPlugin](building/NativeBuildPlugin.md) - Native C++ compilation plugin -- [TestingGuide](building/TestingGuide.md) - Test strategy: tiers, sanitizers, CI layout, and local workflows +- [TestingGuide](building/TestingGuide.md) - Test strategy: tiers, sanitizers, CI layout, shared virtual-thread chaos coverage, and local workflows - [Versioning](building/Versioning.md) - Version computation from git tags ### Reference diff --git a/doc/building/TestingGuide.md b/doc/building/TestingGuide.md index 95cd09bb32..4bd09fb497 100644 --- a/doc/building/TestingGuide.md +++ b/doc/building/TestingGuide.md @@ -1,3 +1,5 @@ + + # Testing Guide This document describes the complete test strategy for the java-profiler project: @@ -162,6 +164,22 @@ as artifacts. --- +### Virtual-thread remount correctness + +`ddprof-test` consumes the `chaosElements` outgoing configuration from +`ddprof-stresstest`. Its artifact is the existing `chaosJar`, so the correctness +test and standalone harness run the same `VirtualThreadRemountAntagonist`. +The ordinary debug integration suite runs `VirtualThreadCarrierChaosTest` for +six seconds per `vm`/`vmx` mode on supported HotSpot JDK 21+ runtimes. + +CPU and wall samples must exercise both argument layouts and partial thaw. +Continuation errors, invalid ancestry and falsely complete Java stacks fail +unconditionally. The shared `ContinuationStackChecks` validator also requires +99% successful classified walks; native gaps and ambiguous carrier-only failures +consume that budget. Its unit tests cover the classifier and acceptance checks. +See [the chaos harness guide](../../ddprof-stresstest/src/chaos/README.md#virtual-thread-carrier-recovery) +for the complete classification rules, diagnostics and longer-run controls. + ## Tier 3 — dd-trace Integration Tests (GitLab, Every Push) **Pipeline:** `.gitlab/dd-trace-integration/.gitlab-ci.yml` From 8961d178e0a8063cd72652d086dacd50e86fe238 Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Wed, 28 Oct 2026 09:23:24 +0100 Subject: [PATCH 2/7] fix(profiler): recover arm64 native stacks across sampling signals --- AGENTS.md | 1 + ddprof-lib/src/main/cpp/dwarf.cpp | 94 +++++----- ddprof-lib/src/main/cpp/dwarf.h | 17 +- ddprof-lib/src/main/cpp/dwarfStep.inline.h | 34 +++- .../src/main/cpp/hotspot/hotspotSupport.cpp | 8 +- ddprof-lib/src/main/cpp/stackFrame.h | 7 + ddprof-lib/src/test/cpp/dwarf_ut.cpp | 93 ++++++++++ .../test/cpp/returnAddressAttribution_ut.cpp | 38 ++-- ddprof-lib/src/test/cpp/signalFrame_ut.cpp | 162 ++++++++++++++++++ .../GraalNativeUnwindRegressionTest.java | 162 ++++++++++++++++++ doc/README.md | 2 +- doc/architecture/JvmSupportLayer.md | 21 +++ 12 files changed, 550 insertions(+), 89 deletions(-) create mode 100644 ddprof-lib/src/test/cpp/signalFrame_ut.cpp create mode 100644 ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/GraalNativeUnwindRegressionTest.java diff --git a/AGENTS.md b/AGENTS.md index 05b6dcbc19..13034f91af 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,6 +152,7 @@ before working on a subsystem. | String dictionary (arena chunks, rotation/clear protocols) | [doc/architecture/StringDictionary.md](doc/architecture/StringDictionary.md) | | RefCountGuard (lock-free RAII reader-drain before reclamation) | [doc/architecture/RefCountGuard.md](doc/architecture/RefCountGuard.md) | | TLS context (thread-local tracing context, distributed tracing) | [doc/architecture/TLSContext.md](doc/architecture/TLSContext.md) | +| Native stack unwinding (DWARF CFA rules, arm64 LR/PLT and validated Linux signal-frame recovery) | [doc/architecture/JvmSupportLayer.md](doc/architecture/JvmSupportLayer.md) | | Profiling engines (CPU, wall-clock, allocation, native memory, liveness) | [doc/architecture/ProfilingEngines.md](doc/architecture/ProfilingEngines.md) | | JFR integration (FlightRecorder, metadata, constant pools, buffers) | [doc/architecture/JfrIntegration.md](doc/architecture/JfrIntegration.md) | | Live-heap reference chains (implementation, signal scheduling, design) | [doc/architecture/LiveHeapReferenceChainsImplementation.md](doc/architecture/LiveHeapReferenceChainsImplementation.md), [doc/architecture/ReferenceChainsSignalsExplained.md](doc/architecture/ReferenceChainsSignalsExplained.md), [doc/ReferenceChainsDesign.md](doc/ReferenceChainsDesign.md), [doc/ReferenceChainsCollectionSummary.md](doc/ReferenceChainsCollectionSummary.md) | diff --git a/ddprof-lib/src/main/cpp/dwarf.cpp b/ddprof-lib/src/main/cpp/dwarf.cpp index 9a643306d8..f06a2d88b9 100644 --- a/ddprof-lib/src/main/cpp/dwarf.cpp +++ b/ddprof-lib/src/main/cpp/dwarf.cpp @@ -86,7 +86,7 @@ enum { }; FrameDesc FrameDesc::empty_frame = {0, DW_REG_SP | EMPTY_FRAME_SIZE << 8, - DW_SAME_FP, -EMPTY_FRAME_SIZE}; + DW_SAME_FP, INITIAL_PC_OFFSET}; FrameDesc FrameDesc::default_frame = {0, DW_REG_FP | LINKED_FRAME_SIZE << 8, -LINKED_FRAME_SIZE, -LINKED_FRAME_SIZE + DW_STACK_SLOT}; @@ -219,48 +219,10 @@ void DwarfParser::parseEhFrame(const char *eh_frame, size_t size) { u32 cie_id = get32(); if (cie_id == 0) { - // CIE: update code and data alignment factors. - // Layout after cie_id: [1-byte version][augmentation string \0][code_align LEB][data_align SLEB] - // [return_address_register][augmentation data (if 'z')]... - // return_address_register and everything after data_align are not consumed; _ptr = record_end - // at the bottom of the loop skips them. - // - // `cie` below is overwritten by every CIE encountered. The DWARF spec allows - // multiple CIEs with different augmentation strings in a single .eh_frame section, so - // strictly speaking each FDE should resolve its own CIE via the backward cie_id offset. - // We intentionally skip that: macOS binaries compiled by clang typically emit a single CIE - // per module, and this parser is only called for macOS __eh_frame sections. Multi-CIE - // binaries are not produced by the toolchains we target here. The state is local to this - // function so that this single-CIE policy stays independent of parseFde()'s per-FDE one. - cie = defaultCieInfo(); // every field assigned fresh per CIE, never latched - if (_ptr >= record_end) { - _ptr = record_end; - continue; - } - _ptr++; // skip version - if (_ptr >= record_end) { - _ptr = record_end; - continue; - } - cie.has_z_augmentation = (*_ptr == 'z'); - while (_ptr < record_end) { - char c = *_ptr++; - if (c == 0) break; - if (c == 'S') cie.is_signal_frame = true; - } - if (_ptr >= record_end) { - _ptr = record_end; - continue; - } - cie.code_align = getLeb(record_end); - cie.data_align = getSLeb(record_end); - // Same terms as parseCie(): a zero factor is either a malformed value or - // a record that ended before the field. - cie.valid = cie.code_align != 0 && cie.data_align != 0; - if (!cie.valid) { - cie.code_align = sizeof(instruction_t); - cie.data_align = -(int)sizeof(void *); - } + // The linear macOS path keeps its sequential-CIE policy. Header and + // initial-rule decoding are shared with the indexed ELF path. + _ptr = record_start; + cie = parseCie(); } else { // FDE: parse frame description for the covered PC range. Without a // readable CIE ahead of it -- none seen yet, or the last one rejected -- @@ -358,6 +320,17 @@ DwarfParser::CieInfo DwarfParser::parseCie() { return cie; } + // Consume the return-address column and optional augmentation block to + // locate the CIE's initial rules. Register columns remain architecture-based. + if (_ptr >= cie_end) return cie; + if (version == 1) get8(); else getLeb(cie_end); + if (cie.has_z_augmentation) { + if (_ptr >= cie_end) return cie; + u32 augmentation_size = getLeb(cie_end); + if (augmentation_size > (size_t)(cie_end - _ptr)) return cie; + _ptr += augmentation_size; + } + parseInstructions(0, cie_end, cie, &cie.initial_pc_offset); _ptr = cie_end; cie.valid = true; return cie; @@ -421,13 +394,14 @@ void DwarfParser::parseFde() { recordFlags(cie)); } -void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie) { +void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie, int* initial_pc_offset) { // `end` is derived from an untrusted record length; never let it run past // the section. Reads inside the loop are clamped by the get*/skip* helpers, // but clamping here keeps the loop bound honest and _ptr in range. if (end > _section_end) { end = _section_end; } + ReadWindowScope read_window(_section_end, end); const u32 code_align = cie.code_align; const int data_align = cie.data_align; const u32 record_flags = recordFlags(cie); @@ -435,12 +409,12 @@ void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie u32 cfa_reg = DW_REG_SP; int cfa_off = EMPTY_FRAME_SIZE; int fp_off = DW_SAME_FP; - int pc_off = -EMPTY_FRAME_SIZE; + int pc_off = cie.initial_pc_offset; u32 rem_cfa_reg = DW_REG_SP; int rem_cfa_off = EMPTY_FRAME_SIZE; int rem_fp_off = DW_SAME_FP; - int rem_pc_off = -EMPTY_FRAME_SIZE; + int rem_pc_off = cie.initial_pc_offset; while (_ptr < end) { u8 op = get8(); @@ -448,19 +422,20 @@ void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie case 0: switch (op) { case DW_CFA_nop: + break; case DW_CFA_set_loc: _ptr = end; break; case DW_CFA_advance_loc1: - addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); + if (initial_pc_offset == nullptr) addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); loc += get8() * code_align; break; case DW_CFA_advance_loc2: - addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); + if (initial_pc_offset == nullptr) addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); loc += get16() * code_align; break; case DW_CFA_advance_loc4: - addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); + if (initial_pc_offset == nullptr) addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); loc += get32() * code_align; break; case DW_CFA_offset_extended: @@ -477,11 +452,18 @@ void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie break; case DW_CFA_restore_extended: case DW_CFA_undefined: - case DW_CFA_same_value: - if (getLeb() == DW_REG_FP) { + case DW_CFA_same_value: { + u32 reg = getLeb(); + if (reg == DW_REG_FP) { fp_off = DW_SAME_FP; + } else if (reg == DW_REG_PC && (op == DW_CFA_restore_extended || + (EMPTY_FRAME_SIZE == 0 && op == DW_CFA_same_value))) { + // Restore uses the CIE's rule, which can be a non-default slot. + // same_value leaves the return address in LR on link-register CPUs. + pc_off = op == DW_CFA_restore_extended ? cie.initial_pc_offset : DW_LINK_REGISTER; } break; + } case DW_CFA_register: skipLeb(); skipLeb(); @@ -569,7 +551,7 @@ void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie } break; case DW_CFA_advance_loc: - addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); + if (initial_pc_offset == nullptr) addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); loc += (op & 0x3f) * code_align; break; case DW_CFA_offset: @@ -587,12 +569,18 @@ void DwarfParser::parseInstructions(u32 loc, const char *end, const CieInfo &cie case DW_CFA_restore: if ((op & 0x3f) == DW_REG_FP) { fp_off = DW_SAME_FP; + } else if ((op & 0x3f) == DW_REG_PC) { + pc_off = cie.initial_pc_offset; } break; } } - addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); + if (initial_pc_offset != nullptr) { + *initial_pc_offset = pc_off; + } else { + addRecord(loc, cfa_reg, cfa_off, fp_off, pc_off, record_flags); + } } // Parse a limited subset of DWARF expressions, which is used in diff --git a/ddprof-lib/src/main/cpp/dwarf.h b/ddprof-lib/src/main/cpp/dwarf.h index 29589b3b03..6b5091b632 100644 --- a/ddprof-lib/src/main/cpp/dwarf.h +++ b/ddprof-lib/src/main/cpp/dwarf.h @@ -140,10 +140,12 @@ class DwarfParser { // interrupted PC. Becomes FrameDesc::FLAG_SIGNAL_FRAME on every row. bool is_signal_frame; bool valid; + // Return-address stack offset (or LR) established by the CIE. + int initial_pc_offset; }; static CieInfo defaultCieInfo() { - return CieInfo{(u32)sizeof(instruction_t), -(int)sizeof(void*), false, false, false}; + return CieInfo{(u32)sizeof(instruction_t), -(int)sizeof(void*), false, false, false, INITIAL_PC_OFFSET}; } // FrameDesc::flags value implied by `cie`. @@ -151,6 +153,17 @@ class DwarfParser { return cie.is_signal_frame ? FrameDesc::FLAG_SIGNAL_FRAME : 0; } + // CFI instruction operands must stay within their own CIE/FDE record. + class ReadWindowScope { + const char*& _end; + const char* _previous; + public: + ReadWindowScope(const char*& end, const char* limit) : _end(end), _previous(end) { _end = limit; } + ~ReadWindowScope() { _end = _previous; } + ReadWindowScope(const ReadWindowScope&) = delete; + ReadWindowScope& operator=(const ReadWindowScope&) = delete; + }; + // Real toolchains emit long runs of FDEs sharing one CIE, so the last // resolved CIE is memoized by its start address: every FDE must resolve // its own CIE, but need not re-parse one already seen. @@ -283,7 +296,7 @@ class DwarfParser { CieInfo parseCie(); CieInfo resolveCie(const char* cie_ptr); void parseFde(); - void parseInstructions(u32 loc, const char* end, const CieInfo& cie); + void parseInstructions(u32 loc, const char* end, const CieInfo& cie, int* initial_pc_offset = nullptr); int parseExpression(); void addRecord(u32 loc, u32 cfa_reg, int cfa_off, int fp_off, int pc_off, u32 flags); diff --git a/ddprof-lib/src/main/cpp/dwarfStep.inline.h b/ddprof-lib/src/main/cpp/dwarfStep.inline.h index 638f3c782a..f94d43588e 100644 --- a/ddprof-lib/src/main/cpp/dwarfStep.inline.h +++ b/ddprof-lib/src/main/cpp/dwarfStep.inline.h @@ -14,6 +14,7 @@ #include "safeAccess.h" #include "stackFrame.h" #include "stackWalker.inline.h" +#include // One step of the DWARF unwind, shared by StackWalker::walkDwarf and // HotspotSupport::walkVM. The two had drifted into separate copies; 61959b06f @@ -23,6 +24,9 @@ // Advances sp/fp/walk_pc from the current frame to its caller. Returns false // where the walk must stop -- an unusable CFA rule, an implausible sp or fp, // or no forward progress -- and true when the caller may record another frame. +// A recognized Linux arm64 signal trampoline also replaces the frame view +// with the interrupted ucontext, preserving its LR and exact PC. Neither +// ucontext is modified. // On a false return sp/fp/walk_pc may already have been partly updated; both // callers stop the walk, so no caller observes that. // @@ -44,7 +48,7 @@ __attribute__((always_inline)) inline bool advanceDwarfFrame(WalkPc& walk_pc, uintptr_t& sp, uintptr_t& fp, uintptr_t bottom, int depth, - const StackFrame& frame, Profiler* profiler) { + StackFrame& frame, Profiler* profiler) { using StackWalkValidation::inDeadZone; using StackWalkValidation::aligned; using StackWalkValidation::MAX_FRAME_SIZE; @@ -58,6 +62,32 @@ inline bool advanceDwarfFrame(WalkPc& walk_pc, uintptr_t& sp, uintptr_t& fp, const void* attribution_pc = walk_pc.attribution(); uintptr_t prev_sp = sp; CodeCache* cc = profiler->findLibraryByAddress(attribution_pc); +#if defined(__linux__) && defined(__aarch64__) + // Linux omits CFI for __kernel_rt_sigreturn. Its SP points to rt_sigframe: + // siginfo_t followed by ucontext_t (arch/arm64/kernel/signal.c). + // Match the vDSO's exact mov x8, #139; svc #0 sequence before reading it. + if (cc != nullptr && strcmp(cc->name(), "[vdso]") == 0) { + auto* ip = (uint32_t*)walk_pc.raw(); + uint32_t instruction = SafeAccess::safeFetch32((int*)ip, 0); + bool trampoline = instruction == 0xd2801168U && SafeAccess::safeFetch32((int*)(ip + 1), 0) == 0xd4000001U; + trampoline |= instruction == 0xd4000001U && SafeAccess::safeFetch32((int*)(ip - 1), 0) == 0xd2801168U; + if (trampoline) { + constexpr size_t required = sizeof(siginfo_t) + offsetof(ucontext_t, uc_mcontext.pc) + sizeof(uint64_t); + if ((sp & 15) != 0 || sp >= bottom || bottom - sp < required) return false; + auto* context = (ucontext_t*)(sp + sizeof(siginfo_t)); + uintptr_t interrupted_sp = SafeAccess::safeFetch64((int64_t*)&context->uc_mcontext.sp, 0); + uintptr_t interrupted_fp = SafeAccess::safeFetch64((int64_t*)&context->uc_mcontext.regs[29], 0); + const void* interrupted_pc = stripPointer((const void*)SafeAccess::safeFetch64((int64_t*)&context->uc_mcontext.pc, 0)); + if (interrupted_sp <= sp || interrupted_sp >= bottom || (interrupted_sp & 15) != 0 || + inDeadZone(interrupted_pc) || ((uintptr_t)interrupted_pc & 3) != 0) return false; + sp = interrupted_sp; + fp = interrupted_fp; + walk_pc.setExactAddress(interrupted_pc); + frame = StackFrame(context); + return true; + } + } +#endif FrameDesc f = cc != NULL ? cc->findFrameDesc(attribution_pc) : FrameDesc::fallback_default_frame(); u8 cfa_reg = (u8)f.cfa; @@ -121,7 +151,7 @@ inline bool advanceDwarfFrame(WalkPc& walk_pc, uintptr_t& sp, uintptr_t& fp, walk_pc.setRecoveredPc( stripPointer(SafeAccess::load(INJECT_FAULT_ADDRESS_LIKELY((void**)pc_addr))), f.isSignalFrame()); - } else if (depth == 1) { + } else if (depth == 1 || (!walk_pc.isReturnAddress() && frame.isInterruptedPC(walk_pc.raw()))) { // Matches the memory-slot path above: StackFrame::link() returns // the raw link register, which carries PAC bits on aarch64 and // would otherwise be fed to findFrameDesc as a nonsense address. diff --git a/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp b/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp index 3fab16cad1..5bbdd11376 100644 --- a/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp +++ b/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp @@ -456,7 +456,7 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex uintptr_t carrier_fp, carrier_sp; const void* carrier_pc; bool valid_carrier_sp; - bool sampled_entry = depth == 0 && !walk_pc.isReturnAddress() && continuation_entry_nm != nullptr + bool sampled_entry = !walk_pc.isReturnAddress() && frame.isInterruptedPC(walk_pc.raw()) && continuation_entry_nm != nullptr && CodeHeap::findNMethod(walk_pc.raw()) == continuation_entry_nm; if (sampled_entry) { // Publication/removal of JavaThread::_cont_entry happens inside @@ -701,7 +701,7 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex } } - if (depth == 0) { + if (!walk_pc.isReturnAddress() && frame.isInterruptedPC(walk_pc.raw())) { VMMethod* method = (VMMethod*)frame.method(); jmethodID method_id = getMethodId(method); if (method_id != JMETHODID_NOT_WALKABLE) { @@ -757,7 +757,7 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex if (nm->isFrameCompleteAt(walk_pc.raw())) { const void* epilogue_pc = walk_pc.raw(); - if (depth == 1 && frame.unwindEpilogue(nm, (uintptr_t&)epilogue_pc, sp, fp)) { + if (!walk_pc.isReturnAddress() && frame.isInterruptedPC(walk_pc.raw()) && frame.unwindEpilogue(nm, (uintptr_t&)epilogue_pc, sp, fp)) { walk_pc.setReturnAddress(epilogue_pc); continue; } @@ -831,7 +831,7 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex } continue; } else { - if (features.vtable_target && nm->isVTableStub() && depth == 0) { + if (features.vtable_target && nm->isVTableStub() && !walk_pc.isReturnAddress() && frame.isInterruptedPC(walk_pc.raw())) { uintptr_t receiver = frame.jarg0(); if (receiver != 0) { VMKlass* klass = VMKlass::fromOop(receiver); diff --git a/ddprof-lib/src/main/cpp/stackFrame.h b/ddprof-lib/src/main/cpp/stackFrame.h index bc3834f9ae..071656f1dc 100644 --- a/ddprof-lib/src/main/cpp/stackFrame.h +++ b/ddprof-lib/src/main/cpp/stackFrame.h @@ -1,5 +1,6 @@ /* * Copyright The async-profiler authors + * Copyright 2026, Datadog, Inc. * SPDX-License-Identifier: Apache-2.0 */ @@ -25,6 +26,12 @@ class StackFrame { _ucontext = (ucontext_t*)ucontext; } + // True only for the interrupted instruction in the current register context. + // Signal-frame recovery can replace this view without modifying either context. + bool isInterruptedPC(const void* address) { + return _ucontext != nullptr && pc() == (uintptr_t)address; + } + void restore(uintptr_t saved_pc, uintptr_t saved_sp, uintptr_t saved_fp) { if (_ucontext != nullptr) { pc() = saved_pc; diff --git a/ddprof-lib/src/test/cpp/dwarf_ut.cpp b/ddprof-lib/src/test/cpp/dwarf_ut.cpp index 27faaa4f1d..a3fd52f98b 100644 --- a/ddprof-lib/src/test/cpp/dwarf_ut.cpp +++ b/ddprof-lib/src/test/cpp/dwarf_ut.cpp @@ -10,6 +10,7 @@ #include #include +#include #include // Test name for crash handler @@ -87,6 +88,98 @@ static DwarfParser* parseBuf(const std::vector& buf) { return new DwarfParser("test", base, base, buf.size()); } +static std::vector buildEhFrameHdrImage(const std::vector& body, + const std::vector& fde_offsets); + +// Build a real CIE/FDE instruction stream and own the parser's resulting rows. +static std::unique_ptr parseInstructionRows( + const std::vector& instructions, int& count, + const std::vector& initial_rules = {}, bool indexed = false) { + std::vector buf; + appendCie(buf); + buf[13] = DW_REG_PC; + buf[0] += initial_rules.size(); + buf.insert(buf.end(), initial_rules.begin(), initial_rules.end()); + uint32_t fde_offset = static_cast(buf.size()); + uint32_t cie_offset = fde_offset + 4; + put32(buf, 13 + static_cast(instructions.size())); + put32(buf, cie_offset); + put32(buf, 0); + put32(buf, 256); + put8(buf, 0); + buf.insert(buf.end(), instructions.begin(), instructions.end()); + appendTerminator(buf); + if (indexed) { + auto image = buildEhFrameHdrImage(buf, {fde_offset}); + const char* base = reinterpret_cast(image.data()); + DwarfParser parser("native-clock", base, base, image.size(), + DwarfParser::EhFrameHdrTag{}, base + image.size()); + count = parser.count(); + return {parser.table(), &free}; + } + const char* base = reinterpret_cast(buf.data()); + DwarfParser parser("native-clock", base, base, buf.size()); + count = parser.count(); + return {parser.table(), &free}; +} + +TEST(DwarfEhFrame, RestoreUsesCieDefinedReturnAddressRule) { + // A real CIE can define a return slot different from the ABI default. + // Exercise both ELF-header and linear readers, and both restore encodings. + const uint8_t reg = DW_REG_PC; + for (bool indexed : {false, true}) { + for (const auto& restore : std::vector>{{uint8_t(0xc0 | reg)}, {0x06, reg}}) { + std::vector code{uint8_t(0x80 | reg), 3, 0x41}; + code.insert(code.end(), restore.begin(), restore.end()); + code.push_back(0x41); + int count; + auto rows = parseInstructionRows(code, count, {0, uint8_t(0x80 | reg), 2}, indexed); + ASSERT_GE(count, 3); + EXPECT_EQ(-24, rows.get()[0].pc_off); + EXPECT_EQ(-16, rows.get()[1].pc_off); + } + } +} + +TEST(DwarfEhFrame, LeafReturnAddressUsesArchitectureRegisterOrStackSlot) { + int count; + auto rows = parseInstructionRows({}, count); + ASSERT_EQ(2, count); + EXPECT_EQ(INITIAL_PC_OFFSET, rows.get()[0].pc_off); + EXPECT_EQ(DW_SAME_FP, rows.get()[0].fp_off); +} + +#ifdef __aarch64__ +TEST(DwarfEhFrame, RestoredLinkRegisterReplacesSavedReturnAddressSlot) { + // clock_gettime saves LR at CFA-8, then restores it before returning. + // Compact restore, extended restore and same_value must all stop reading + // the released stack slot once LR holds the return address again. + for (const auto& restore : std::vector>{{0xde}, {0x06, 30}, {0x08, 30}}) { + std::vector code{0x0e, 16, 0x9e, 1, 0x41}; + code.insert(code.end(), restore.begin(), restore.end()); + code.insert(code.end(), {0x0e, 0, 0x41}); + int count; + auto rows = parseInstructionRows(code, count); + ASSERT_GE(count, 3); + EXPECT_EQ(-8, rows.get()[0].pc_off); + EXPECT_EQ(DW_LINK_REGISTER, rows.get()[1].pc_off); + EXPECT_EQ((u32)DW_REG_SP, rows.get()[1].cfa); + } +} + +TEST(DwarfEhFrame, SameValueUsesLrEvenWhenCieDefinesStackSlot) { + int count; + auto rows = parseInstructionRows({0x08, 30}, count, {0x9e, 2}); + ASSERT_EQ(2, count); + EXPECT_EQ(DW_LINK_REGISTER, rows.get()[0].pc_off); +} + +TEST(DwarfEhFrame, PltLeafReturnsThroughLinkRegister) { + EXPECT_EQ(DW_LINK_REGISTER, FrameDesc::empty_frame.pc_off); + EXPECT_EQ(DW_SAME_FP, FrameDesc::empty_frame.fp_off); +} +#endif + TEST(DwarfEhFrame, EmptySection) { std::vector buf; DwarfParser* dwarf = parseBuf(buf); diff --git a/ddprof-lib/src/test/cpp/returnAddressAttribution_ut.cpp b/ddprof-lib/src/test/cpp/returnAddressAttribution_ut.cpp index 72c7dd1a3c..eb8aff9830 100644 --- a/ddprof-lib/src/test/cpp/returnAddressAttribution_ut.cpp +++ b/ddprof-lib/src/test/cpp/returnAddressAttribution_ut.cpp @@ -1206,6 +1206,8 @@ asm( "prof_ra_pcoff_seed:\n" ".cfi_startproc\n" ".cfi_def_cfa sp, 16\n" + // This fabricated frame stores its caller PC in memory, not in LR. + ".cfi_offset 30, -8\n" "nop\nnop\n" ".cfi_endproc\n" ".size prof_ra_pcoff_seed, .-prof_ra_pcoff_seed\n" @@ -1348,24 +1350,9 @@ TEST_F(ReturnAddressAttributionTest, Test5b_DwPcOffsetAppliesOffsetToRawPc) { // =========================================================================== // Test 6 -- Link-register recovery sets the flag (coverage row 7). // -// Reachability, so nobody reads a green run here as coverage: -// - x86_64: structurally unreachable. EMPTY_FRAME_SIZE > 0 makes the -// guard "EMPTY_FRAME_SIZE > 0 || pc_off != DW_LINK_REGISTER" always true, -// so the arm never executes. -// - aarch64: reachable only with a frame table that actually carries -// pc_off == DW_LINK_REGISTER. DwarfParser never emits it -- it starts -// pc_off at -EMPTY_FRAME_SIZE and no DW_CFA_ handler assigns that value -- -// so no .eh_frame fixture, including the one below, can reach the arm. -// The only producer in the tree is SFrameParser (sframe.cpp, gated by -// SFrameParser.PerFRE_RA_Untracked in sframe_ut.cpp), which needs a -// .sframe section in the scanned ELF image; the gtest binary is not built -// with one, and CodeCache::setDwarfTable() cannot be used to inject one -// because it asserts the cache is unpublished. -// -// The fixture is kept so that it starts gating on its own if a .sframe- -// enabled build ever appears, and so the precondition check documents which -// row shape the arm needs. It deliberately does not increment -// g_gating_assertions_reached on the skip path. +// On aarch64 the FDE's initial return-address rule is LR. The parsed row +// must preserve that rule so the real DWARF walker can recover and attribute +// the sampled link register. x86_64 has a stack return-address slot instead. // =========================================================================== #if defined(__aarch64__) @@ -1401,13 +1388,8 @@ TEST_F(ReturnAddressAttributionTest, Test6_LinkRegisterRecoverySetsFlag) { GTEST_SKIP() << "prof_ra_lr_fn not visible to findLibraryByAddress in this build"; } FrameDesc row = lib->findFrameDesc((const void*)&prof_ra_lr_fn); - if (row.pc_off != DW_LINK_REGISTER || (row.fp_off & DW_PC_OFFSET)) { - GTEST_SKIP() << "expected: .eh_frame cannot express pc_off == DW_LINK_REGISTER, so this " - << "fixture's row parsed as (pc_off=" << row.pc_off << ", fp_off=" - << row.fp_off << ") and the link-register arm is not reached. The arm is " - << "covered by inspection only; its producer is gated by " - << "SFrameParser.PerFRE_RA_Untracked. See this test's banner comment."; - } + ASSERT_EQ(DW_LINK_REGISTER, row.pc_off); + ASSERT_EQ(0, row.fp_off & DW_PC_OFFSET); // Adjacency is not a precondition here: unlike Tests 1/3/4/5, the link // register below is set directly rather than derived from a real // call-site return address, so the fixture works wherever the assembler @@ -1472,6 +1454,8 @@ asm( ".cfi_startproc\n" ".cfi_signal_frame\n" ".cfi_def_cfa sp, 16\n" + // This fabricated frame stores its caller PC in memory, not in LR. + ".cfi_offset 30, -8\n" "nop\nnop\n" ".cfi_endproc\n" ".size prof_ra_sig_seed, .-prof_ra_sig_seed\n" @@ -1538,10 +1522,10 @@ TEST_F(ReturnAddressAttributionTest, Test8_SignalFrameRowKeepsExactAddress) { // none of the gating tests reached its assertions, surfacing that // degradation instead of letting it pass silently. Declared last so it runs // after all of them under gtest's default (registration-order) test order. -// Test6 is excluded by construction -- see its banner comment. +// Test6 also gates the link-register path on aarch64. TEST(ReturnAddressAttributionGatingCoverageTest, AtLeastOneGatingAssertionRan) { EXPECT_GT(g_gating_assertions_reached, 0) - << "every gating test in this file (Test1/2c/3/4/5/5b/8) skipped its precondition " + << "every gating test in this file (Test1/2c/3/4/5/5b/6/8) skipped its precondition " << "check on this toolchain -- the attribution fix has zero effective " << "regression coverage in this build; see each test's GTEST_SKIP reason " << "in the test log above for which precondition failed"; diff --git a/ddprof-lib/src/test/cpp/signalFrame_ut.cpp b/ddprof-lib/src/test/cpp/signalFrame_ut.cpp new file mode 100644 index 0000000000..fce2e860b9 --- /dev/null +++ b/ddprof-lib/src/test/cpp/signalFrame_ut.cpp @@ -0,0 +1,162 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ + +#include +#include "codeCache.h" +#include "dwarfStep.inline.h" +#include "libraries.h" +#include +#include +#include + +#if defined(__linux__) && defined(__aarch64__) +namespace { +alignas(16) uint32_t trampoline_code[] = {0xd503201f, 0xd2801168, 0xd4000001, 0xd503201f}; +alignas(16) uint32_t ordinary_code[] = {0xd503201f, 0xd2801168, 0xd4000001, 0xd503201f}; +alignas(16) uint32_t leaf_code[] = {0xd503201f, 0xd65f03c0}; +const void* const caller_pc = (const void*)0x5a5f00000108ULL; + +void registerLibraries() { + static const bool registered = [] { + static CodeCache vdso("[vdso]", -1, (const char*)trampoline_code, + (const char*)(trampoline_code + 4)); + static CodeCache ordinary("ordinary-syscall", -1, (const char*)ordinary_code, + (const char*)(ordinary_code + 4)); + static CodeCache leaf("native-clock-leaf", -1, (const char*)leaf_code, + (const char*)(leaf_code + 2)); + static const FrameDesc leaf_row{0, DW_REG_SP, DW_SAME_FP, DW_LINK_REGISTER}; + leaf.setDwarfTable(nullptr, 0, leaf_row); + bool ok = Libraries::instance()->addLibraryForTest(&vdso); + ok = Libraries::instance()->addLibraryForTest(&ordinary) && ok; + return Libraries::instance()->addLibraryForTest(&leaf) && ok; + }(); + ASSERT_TRUE(registered); +} + +class SignalFrameTest : public ::testing::Test { +protected: + static constexpr size_t context_size = sizeof(siginfo_t) + sizeof(ucontext_t); + using SignalStack = std::array; + + void SetUp() override { registerLibraries(); } + + // A kernel rt_sigframe precedes the interrupted stack on Linux arm64. + static ucontext_t* savedContext(SignalStack& stack) { + auto* context = new (stack.data() + sizeof(siginfo_t)) ucontext_t{}; + context->uc_mcontext.pc = (uintptr_t)leaf_code; + context->uc_mcontext.sp = (uintptr_t)(stack.data() + context_size); + context->uc_mcontext.regs[29] = 1; // compiled Java can use FP as a general register + context->uc_mcontext.regs[30] = (uintptr_t)caller_pc; + return context; + } + + static bool step(WalkPc& pc, uintptr_t& sp, uintptr_t& fp, uintptr_t bottom, + int depth, StackFrame& frame) { + return advanceDwarfFrame(pc, sp, fp, bottom, depth, frame, Profiler::instance()); + } +}; + +TEST_F(SignalFrameTest, RestoresExactPcAndRegistersThenUsesInterruptedLr) { + for (int instruction : {1, 2}) { + alignas(16) SignalStack stack{}; + auto* saved = savedContext(stack); + ucontext_t sampled{}; + sampled.uc_mcontext.pc = (uintptr_t)(trampoline_code + instruction); + sampled.uc_mcontext.regs[30] = 0x5a5f00000908ULL; // handler LR must not escape + StackFrame frame(&sampled); + WalkPc pc; + pc.setReturnAddress(trampoline_code + instruction); + uintptr_t sp = (uintptr_t)stack.data(), fp = 0; + uintptr_t bottom = (uintptr_t)(stack.data() + stack.size()); + ASSERT_TRUE(step(pc, sp, fp, bottom, 2, frame)); + EXPECT_EQ((const void*)leaf_code, pc.raw()); + EXPECT_FALSE(pc.isReturnAddress()); + EXPECT_EQ(saved->uc_mcontext.sp, sp); + EXPECT_EQ(1u, fp); + EXPECT_EQ((uintptr_t)caller_pc, frame.link()); + ASSERT_TRUE(step(pc, sp, fp, bottom, 3, frame)); + EXPECT_EQ(caller_pc, pc.raw()); + EXPECT_TRUE(pc.isReturnAddress()); + EXPECT_EQ((uintptr_t)(trampoline_code + instruction), sampled.uc_mcontext.pc); + EXPECT_EQ(0x5a5f00000908ULL, sampled.uc_mcontext.regs[30]); + } +} + +TEST_F(SignalFrameTest, RejectsIncompleteOrMisalignedSignalFrame) { + alignas(16) SignalStack stack{}; + savedContext(stack); + for (int invalid : {0, 1}) { + ucontext_t sampled{}; + StackFrame frame(&sampled); + WalkPc pc; + pc.setReturnAddress(trampoline_code + 1); + uintptr_t sp = (uintptr_t)stack.data() + invalid, fp = 0; + uintptr_t bottom = invalid == 0 ? sp + sizeof(siginfo_t) : (uintptr_t)(stack.data() + stack.size()); + EXPECT_FALSE(step(pc, sp, fp, bottom, 2, frame)); + EXPECT_EQ((const void*)(trampoline_code + 1), pc.raw()); + } +} + +TEST_F(SignalFrameTest, RejectsBackwardMisalignedOrOutOfRangeInterruptedSp) { + alignas(16) SignalStack stack{}; + auto* saved = savedContext(stack); + uintptr_t base = (uintptr_t)stack.data(), bottom = base + stack.size(); + for (uintptr_t interrupted_sp : {base, base - 16, bottom, base + context_size + 1}) { + saved->uc_mcontext.sp = interrupted_sp; + ucontext_t sampled{}; + StackFrame frame(&sampled); + WalkPc pc; + pc.setReturnAddress(trampoline_code + 1); + uintptr_t sp = base, fp = 0; + EXPECT_FALSE(step(pc, sp, fp, bottom, 2, frame)); + EXPECT_EQ(base, sp); + } +} + +TEST_F(SignalFrameTest, RejectsMissingOrMisalignedInterruptedPc) { + alignas(16) SignalStack stack{}; + auto* saved = savedContext(stack); + for (uintptr_t interrupted_pc : {uintptr_t(0), (uintptr_t)leaf_code + 1}) { + saved->uc_mcontext.pc = interrupted_pc; + ucontext_t sampled{}; + StackFrame frame(&sampled); + WalkPc pc; + pc.setReturnAddress(trampoline_code + 1); + uintptr_t sp = (uintptr_t)stack.data(), fp = 0; + EXPECT_FALSE(step(pc, sp, fp, (uintptr_t)(stack.data() + stack.size()), 2, frame)); + } +} + +TEST_F(SignalFrameTest, DoesNotRestoreAtUnrecognizedVdsoInstruction) { + alignas(16) SignalStack stack{}; + savedContext(stack); + ucontext_t sampled{}; + sampled.uc_mcontext.pc = (uintptr_t)(trampoline_code + 3); + StackFrame frame(&sampled); + WalkPc pc; + pc.setReturnAddress(trampoline_code + 3); + uintptr_t sp = (uintptr_t)stack.data(), fp = 0; + EXPECT_FALSE(step(pc, sp, fp, (uintptr_t)(stack.data() + stack.size()), 2, frame)); + EXPECT_TRUE(frame.isInterruptedPC(trampoline_code + 3)); +} + +TEST_F(SignalFrameTest, DoesNotTreatOrdinarySyscallCodeAsSignalTrampoline) { + alignas(16) SignalStack stack{}; + savedContext(stack); + ucontext_t sampled{}; + sampled.uc_mcontext.pc = (uintptr_t)(ordinary_code + 1); + StackFrame frame(&sampled); + WalkPc pc; + pc.setReturnAddress(ordinary_code + 1); + uintptr_t sp = (uintptr_t)stack.data(), fp = 0; + EXPECT_FALSE(step(pc, sp, fp, (uintptr_t)(stack.data() + stack.size()), 2, frame)); + EXPECT_TRUE(frame.isInterruptedPC(ordinary_code + 1)); +} +} // namespace +#else +TEST(SignalFrameTest, RequiresLinuxAarch64SignalAbi) { + GTEST_SKIP() << "Linux arm64 signal trampoline ABI"; +} +#endif diff --git a/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/GraalNativeUnwindRegressionTest.java b/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/GraalNativeUnwindRegressionTest.java new file mode 100644 index 0000000000..eb94b9b06b --- /dev/null +++ b/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/GraalNativeUnwindRegressionTest.java @@ -0,0 +1,162 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ +package com.datadoghq.profiler.wallclock; + +import com.datadoghq.profiler.CStackAwareAbstractProfilerTest; +import com.datadoghq.profiler.JfrEvent; +import com.datadoghq.profiler.JfrFrame; +import com.datadoghq.profiler.Platform; +import com.datadoghq.profiler.junit.CStack; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.FutureTask; +import java.util.concurrent.TimeUnit; +import org.junit.jupiter.api.TestTemplate; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Isolates native gaps around System.nanoTime from continuation remounts. Run the same test on + * GraalVM and a HotSpot control, with and without preserved frame pointers, against each profiler + * candidate. Platform threads provide a control without continuation entry or return barriers. + */ +public class GraalNativeUnwindRegressionTest extends CStackAwareAbstractProfilerTest { + private volatile boolean running; + private volatile long sink; + + /** Selects the VM walker with or without carrier walking. */ + public GraalNativeUnwindRegressionTest(@CStack String cstack) { + super(cstack); + } + + @Override + protected boolean isPlatformSupported() { + // Other JITs can inline nanoTime without exposing a Java transition frame. + // Controls explicitly disable its intrinsic and opt in through this property. + boolean control = Boolean.getBoolean("ddprof_test.native_unwind_control"); + return Platform.isJavaVersionAtLeast(21) && !Platform.isJ9() && !Platform.isZing() + && (control || Platform.isGraal() && Platform.isLinux() && Platform.isAarch64()) + && !(isInCI() && Platform.isMusl() && !Platform.isAarch64()); + } + + @Override + protected String getProfilerCommand() { + return "cpu=2ms,wall=2ms,walltpt=64,filter=,jstackdepth=128"; + } + + /** Requires native coverage on a platform thread, independently of continuation recovery. */ + @TestTemplate + @ValueSource(strings = {"vm", "vmx"}) + public void retainsNativeCoverageOnPlatformThread() throws Exception { + runNanoTimeWorkload(false); + } + + /** Uses a never-parked virtual thread to separate native transitions from partial thaw. */ + @TestTemplate + @ValueSource(strings = {"vm", "vmx"}) + public void retainsNativeCoverageOnVirtualThread() throws Exception { + runNanoTimeWorkload(true); + } + + /** Runs one bounded worker; propagates its failures and always stops profiling. */ + private void runNanoTimeWorkload(boolean virtual) throws Exception { + waitForProfilerReady(2_000); + CountDownLatch started = new CountDownLatch(1); + running = true; + FutureTask task = new FutureTask<>(() -> { + started.countDown(); + nanoTimeWorkload(); + return null; + }); + Thread worker = null; + try { + if (virtual) { + // Reflection preserves the Java 8 compilation target. + worker = (Thread) Thread.class.getMethod("startVirtualThread", Runnable.class).invoke(null, task); + } else { + worker = new Thread(task, "nano-time-native-control"); + worker.start(); + } + assertTrue(started.await(10, TimeUnit.SECONDS), "The native workload did not start"); + Thread.sleep(6_000); + } finally { + running = false; + try { + if (worker != null) { + worker.join(10_000); + assertFalse(worker.isAlive(), "The native workload did not stop"); + task.get(1, TimeUnit.SECONDS); + } + } finally { + stopProfiler(); + } + } + verifyCStackSettings(); + assertAll( + () -> assertNativeCoverage("datadog.ExecutionSample"), + () -> assertNativeCoverage("datadog.MethodSample")); + } + + /** Keeps the real clock call hot without parking, recursion, or a native test shim. */ + private void nanoTimeWorkload() { + long value = 1; + while (running) { + value = (value * 6364136223846793005L) ^ System.nanoTime(); + sink = value; + } + } + + /** Applies the existing 99% coverage requirement to samples with a decoded workload frame. */ + private void assertNativeCoverage(String eventType) { + NativeCoverage coverage = reduceEvents(eventType, NativeCoverage::new, NativeCoverage::accept); + String message = eventType + " " + coverage; + System.out.println(message); + assertAll(message, + () -> assertTrue(coverage.samples >= 100, "Insufficient workload samples"), + () -> assertTrue(coverage.nanoTimeSamples >= 20, + "No evidence of sampled System.nanoTime transitions; result is inconclusive"), + () -> assertTrue((coverage.samples - coverage.nativeGaps) * 100L >= coverage.samples * 99L, + "Fewer than 99% of workload samples avoided native gaps")); + } + + /** Streams counts and one diagnostic stack without retaining the recording's samples. */ + private static final class NativeCoverage { + private long samples; + private long nanoTimeSamples; + private long nativeGaps; + private String firstGap; + + private void accept(JfrEvent event) { + boolean workload = false, nanoTime = false, skipped = false; + for (JfrFrame frame : event.getStackTrace().frames()) { + workload |= GraalNativeUnwindRegressionTest.class.getName().equals(frame.className()) + && "nanoTimeWorkload".equals(frame.methodName()); + nanoTime |= "java.lang.System".equals(frame.className()) && "nanoTime".equals(frame.methodName()); + skipped |= "[skipped frames]".equals(frame.methodName()); + } + if (!workload) return; + samples++; + if (nanoTime) nanoTimeSamples++; + if (nanoTime && skipped) { + nativeGaps++; + if (firstGap == null) { + StringBuilder text = new StringBuilder(); + for (JfrFrame frame : event.getStackTrace().frames()) { + text.append("\n ").append(frame.className()).append('.').append(frame.methodName()); + } + firstGap = text.toString(); + } + } + } + + @Override + public String toString() { + return "samples=" + samples + " nanoTimeSamples=" + nanoTimeSamples + " nativeGaps=" + nativeGaps + + (firstGap == null ? "" : " firstGap=" + firstGap); + } + } +} diff --git a/doc/README.md b/doc/README.md index ce932c73a1..3c0c853808 100644 --- a/doc/README.md +++ b/doc/README.md @@ -25,7 +25,7 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). - [ProfilingEngines](architecture/ProfilingEngines.md) - CPU, wall-clock, allocation, native-memory and liveness engines - [RefCountGuard](architecture/RefCountGuard.md) - Lock-free RAII reference-counting primitive used to drain readers before resource reclamation - [ReferenceChainsSignalsExplained](architecture/ReferenceChainsSignalsExplained.md) - Reference-chain scheduling and cost control -- [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM-implementation abstraction (HotSpot, J9, Zing), compiled/interpreted continuation entries, arm64 exception phases, and return-barrier entry/interior recovery +- [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM abstraction (HotSpot, J9, Zing), continuation recovery, and native DWARF/link-register rules and Linux signal-frame recovery - [SanityChecks](architecture/SanityChecks.md) - Startup sanity checks - [SigsegvPatching](architecture/SigsegvPatching.md) - SIGSEGV-safe library patching - [StringDictionary](architecture/StringDictionary.md) - Concurrency model: RefCountGuard, clearAll, and rotation protocols diff --git a/doc/architecture/JvmSupportLayer.md b/doc/architecture/JvmSupportLayer.md index 5f0d45b1b1..d0be1d62c9 100644 --- a/doc/architecture/JvmSupportLayer.md +++ b/doc/architecture/JvmSupportLayer.md @@ -163,6 +163,27 @@ walker, with raw `Method*` frames, and with lazy jmethodID resolution. | `klassRegistry.{h,cpp}` | Redefinition marks keyed by `Klass*` for `fjmethodid=lazy` (`HotspotSupport::_klass_registry`, `hotspotSupport.h:161`) | | `klassTraceId.{h,cpp}` | Finds `Klass::_trace_id` by probing, so a stale raw frame can be told apart from a class that reused the same `Klass*` | +Native DWARF steps are shared by `StackWalker::walkDwarf` and the VM walker +through `dwarfStep.inline.h`. On arm64, an FDE's initial return-address source +is LR, and PLT leaf descriptors use the same rule. Compact and extended +`DW_CFA_restore` for the return address restore the CIE-defined source, +including a non-default stack offset; +`DW_CFA_same_value` also keeps the return address in LR. A saved return-address +slot applies only while its CFI offset rule is active. On x64 the initial +return address remains at CFA-8. The sampled LR is used only for a leaf frame; +recovered PCs retain the normal return-address attribution and PAC stripping. + +Linux arm64's vDSO `__kernel_rt_sigreturn` omits DWARF metadata. The shared +native step recognizes its exact syscall instruction pair only in `[vdso]`, +bounds-checks the `siginfo_t`/`ucontext_t` signal-frame layout, and restores the +interrupted PC, SP, FP and register-context view. The interrupted PC is exact; +its LR can seed the next leaf even after native handler frames were recorded. +The VM walker also recognizes interrupted Java entry, interpreter, epilogue +and vtable phases after that prefix. Invalid instructions, stack bounds or +restored register values reject recovery. The sampled and saved ucontexts +remain unchanged. Restoration requires a forward, aligned SP within the walk's existing stack +bound; transitions outside that bound are rejected. + Directly sampled `enterSpecial` frames use `HotspotStackFrame::continuationEntryFrameWords()` to locate the carrier SP. The helper distinguishes the prologue's register save and storage allocation, From 70e21a107c6499ad4d7f352b0f0f06f076be993e Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Wed, 28 Oct 2026 11:11:34 +0100 Subject: [PATCH 3/7] test(profiler): verify truthful blocking continuation truncation --- AGENTS.md | 4 ++++ .../wallclock/VirtualThreadWallClockTest.java | 24 +++++++++++++++++-- doc/README.md | 2 +- doc/architecture/JvmSupportLayer.md | 14 +++++++++++ 4 files changed, 41 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a574e785fa..bbdaa20811 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -301,6 +301,10 @@ pointer-not-yet-stored. Never permanently advance an iteration cursor past NULL at the first NULL or track the last contiguous non-NULL entry so missing entries are retried. +**Stack completeness.** A successful carrier crossing does not establish a complete +virtual-thread stack. Preserve walker-reported truncation through JFR serialization; +see [JvmSupportLayer.md](doc/architecture/JvmSupportLayer.md#continuation-completeness). + **JVM abstraction.** Shared code must not refer to JVM-implementation-specific code directly, only through the abstraction files: `jvmSupport.{h,inline.h,cpp}`, `jvmThread.{h,cpp}`, `vmEntry.{h,cpp}`, `javaApi.cpp`. Implementation-specific code lives under `hotspot/`, diff --git a/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java b/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java index f9c0f0e3b6..b584eb6dd8 100644 --- a/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java +++ b/ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java @@ -12,6 +12,8 @@ import org.junit.jupiter.api.TestTemplate; import com.datadoghq.profiler.JfrEvent; import com.datadoghq.profiler.JfrEvents; +import com.datadoghq.profiler.JfrFrame; +import com.datadoghq.profiler.JfrStackTrace; import org.junit.jupiter.params.provider.ValueSource; @@ -185,11 +187,12 @@ public void samplesCarrierFramesFromCpuBoundVT(@CStack String cstack) throws Exc * The VT parks itself 100 times. The main thread sleeps 10ms between each unpark so the * wall-clock sampler (1ms period) has time to fire while the VT is mounted with frozen * frames still in the StackChunk (cont_returnBarrier as the return PC). + * Samples that do not recover carrier ancestry must report truncation. */ @RetryTest(5) @TestTemplate @ValueSource(strings = {"vm", "vmx", "fp", "dwarf"}) - public void samplesCarrierFramesFromBlockingVT(@CStack String cstack) throws Exception { + public void marksMissingCarrierFramesFromBlockingVTTruncated(@CStack String cstack) throws Exception { // Carrier-frame unwinding is only enabled for vmx (cstack=vm does not set carrier_frames). // fp/dwarf use ASGCT and cannot cross the continuation boundary at all. assumeTrue(cstack.equals("vmx"), @@ -238,6 +241,23 @@ public void samplesCarrierFramesFromBlockingVT(@CStack String cstack) throws Exc verifyCStackSettings(); - assertCarrierFramesVisible(verifyEvents("datadog.MethodSample")); + int workloadSamples = 0; + for (JfrEvent sample : verifyEvents("datadog.MethodSample")) { + JfrStackTrace trace = sample.getStackTrace(); + boolean workload = false; + boolean carrier = false; + for (JfrFrame frame : trace.frames()) { + workload |= getClass().getName().equals(frame.className()) && frame.methodName() != null + && frame.methodName().startsWith("lambda$marksMissingCarrierFramesFromBlockingVTTruncated$"); + carrier |= "java.util.concurrent.ForkJoinWorkerThread".equals(frame.className()) + && "run".equals(frame.methodName()); + } + if (!workload) continue; + workloadSamples++; + assertTrue(carrier || trace.isTruncated(), + "A blocking virtual-thread sample omitted carrier ancestry without reporting truncation: " + + sample.getStackTraceString()); + } + assertTrue(workloadSamples > 0, "No sample captured the blocking virtual-thread workload"); } } diff --git a/doc/README.md b/doc/README.md index 39d51ddf17..395d8f3e7c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -25,7 +25,7 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). - [ProfilingEngines](architecture/ProfilingEngines.md) - CPU, wall-clock, allocation, native-memory and liveness engines - [RefCountGuard](architecture/RefCountGuard.md) - Lock-free RAII reference-counting primitive used to drain readers before resource reclamation - [ReferenceChainsSignalsExplained](architecture/ReferenceChainsSignalsExplained.md) - Reference-chain scheduling and cost control -- [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM-implementation abstraction (HotSpot, J9, Zing) +- [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM-implementation abstraction (HotSpot, J9, Zing), continuation completeness - [SanityChecks](architecture/SanityChecks.md) - Startup sanity checks - [SigsegvPatching](architecture/SigsegvPatching.md) - SIGSEGV-safe library patching - [StringDictionary](architecture/StringDictionary.md) - Concurrency model: RefCountGuard, clearAll, and rotation protocols diff --git a/doc/architecture/JvmSupportLayer.md b/doc/architecture/JvmSupportLayer.md index 8a0ecccac3..87b2a2340b 100644 --- a/doc/architecture/JvmSupportLayer.md +++ b/doc/architecture/JvmSupportLayer.md @@ -185,6 +185,20 @@ Caveats: - **`cstack=vm` needs `VMStructs`.** If `VMStructs::hasStackStructs()` is false, `cstack=vm` is downgraded to DWARF or `no` (`profiler.cpp:1625-1629`). +#### Continuation completeness + +HotSpot can leave older virtual-thread callers frozen in heap `StackChunk` objects +after a partial thaw. `walkVM` classifies continuation boundaries with `ContBoundary`: +a verified thawed boundary proves no callers remain frozen; return barriers and +unverified crossings mark the trace truncated. This walker does not recover heap +callers, so reaching a carrier does not clear that flag. + +`Recording::writeStackTraceHeader` preserves the walker flag even when the final +Java frame is an entry method such as `Thread.run`. A verified thawed crossing also +preserves truncation established earlier in the walk. The blocking virtual-thread +wall-clock regression checks truthful truncation when carrier ancestry is missing; +carrier recovery itself remains a separate correctness requirement. + ### `j9/` | File | Role | From dc206d58ee8f4bf5e93c9b515e0bf54f63d4c88b Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Wed, 28 Oct 2026 11:33:14 +0100 Subject: [PATCH 4/7] docs(profiler): explain preserved stack truncation --- doc/architecture/JfrIntegration.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/architecture/JfrIntegration.md b/doc/architecture/JfrIntegration.md index aec55b2abb..c44844b2c6 100644 --- a/doc/architecture/JfrIntegration.md +++ b/doc/architecture/JfrIntegration.md @@ -390,11 +390,12 @@ per frame: method key varint (MethodInfo::_key) frame type byte ``` -- **Truncated flag:** when the outermost frame (`frames[num_frames - 1]`) is a - Java frame, the trace is reported truncated unless that frame is an entry - point (`Thread.run()` or an override in a `Thread` subclass, or - `main(String[])`); for native outermost frames it is `CallTrace::truncated` - (`flightRecorder.cpp:1946-1955`, `366-388`). +- **Truncated flag:** `Recording::writeStackTraceHeader()` preserves + `CallTrace::truncated` for every root type. A Java outermost frame + (`frames[num_frames - 1]`) also marks the trace truncated unless it is an + entry point (`Thread.run()` or an override in a `Thread` subclass, or + `main(String[])`). An entry root cannot clear truncation established by the + walker; native roots use the walker flag unchanged. - **Java frames:** the frame type and BCI are decoded from the packed `bci` field (`FrameType::decode()` / `FrameType::bci()`), and the line number comes from the method's line-number table (`flightRecorder.cpp:1962-1967`, From 6059050bd41074b54a38a12f8d4ae7266aad848d Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Wed, 28 Oct 2026 11:43:50 +0100 Subject: [PATCH 5/7] fix(profiler): consolidate arm64 runtime caller recovery --- AGENTS.md | 6 +- .../native/gtest/GtestTaskBuilder.kt | 5 +- ddprof-lib/src/main/cpp/dwarfStep.inline.h | 22 ++- .../src/main/cpp/hotspot/hotspotSupport.cpp | 2 +- .../src/main/cpp/stackFrame_aarch64.cpp | 42 ++++- ddprof-lib/src/main/cpp/stackWalker.cpp | 2 +- .../src/test/cpp/continuationWalk_ut.cpp | 4 +- .../test/cpp/returnAddressAttribution_ut.cpp | 95 ++++++++++- ddprof-lib/src/test/cpp/signalFrame_ut.cpp | 105 +++++++++++- .../src/test/cpp/stackFrame_aarch64_ut.cpp | 158 ++++++++++++++++++ doc/README.md | 4 +- doc/architecture/JvmSupportLayer.md | 22 +++ doc/building/NativeBuildPlugin.md | 6 + 13 files changed, 452 insertions(+), 21 deletions(-) create mode 100644 ddprof-lib/src/test/cpp/stackFrame_aarch64_ut.cpp diff --git a/AGENTS.md b/AGENTS.md index 13034f91af..ae9beda844 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,7 @@ + + + # Native Build Plugin Architecture This document describes the architecture of the Kotlin-based native build plugin (`com.datadoghq.native-build`) used for C++ compilation in the Datadog Java Profiler project. @@ -360,6 +362,10 @@ The `build-logic` directory contains all native build plugins: ## GtestPlugin Integration +Static Google Test and Google Mock archives are linked with their entry-point +archives first: `gtest_main`, `gmock_main`, `gmock`, then `gtest`. This order also +links architecture-gated test sources that register no cases on the build host. + The `GtestPlugin` consumes configurations from `NativeBuildPlugin`: ```kotlin From 50d2704d289f2b07317ca72c4860f92001fd4a41 Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Wed, 28 Oct 2026 12:06:26 +0100 Subject: [PATCH 6/7] fix(profiler): validate continuation exception-exit layouts --- .../src/main/cpp/hotspot/hotspotStackFrame.h | 5 +- .../cpp/hotspot/hotspotStackFrame_x64.cpp | 21 ++- .../src/main/cpp/hotspot/hotspotSupport.cpp | 42 +++-- .../src/test/cpp/continuationEntryX64_ut.cpp | 154 ++++++++++++++++++ .../src/test/cpp/continuationWalk_ut.cpp | 124 ++++++++++++-- doc/architecture/JvmSupportLayer.md | 10 ++ 6 files changed, 331 insertions(+), 25 deletions(-) create mode 100644 ddprof-lib/src/test/cpp/continuationEntryX64_ut.cpp diff --git a/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame.h b/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame.h index 684302d89f..3057b6c4fb 100644 --- a/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame.h +++ b/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame.h @@ -123,10 +123,11 @@ class HotspotStackFrame : public StackFrame { // words up to the carrier SP. Two or more words include saved FP/PC; before // saving and after restoring them, FP is still in the sampled register and // PC is in LR (aarch64, zero words) or at SP (x64, one word). Returns -1 for - // unsupported prologues or invalid frame metadata. Interpreter-only entry + // unsupported code sections or invalid frame metadata. Interpreter-only entry // setup is located before the verified compiled entry; arm64 VM-leaf spills - // add two temporary words. Never use a return PC: + // add two temporary words. Never infer a sampled layout from a return PC: // its instruction has not executed with the sampled register values. + // A negative result may conservatively reject an unsupported code section. int continuationEntryFrameWords(VMNMethod* nm, const void* pc); #ifdef __aarch64__ diff --git a/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_x64.cpp b/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_x64.cpp index d270f921bf..2b8302e5b7 100644 --- a/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_x64.cpp +++ b/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_x64.cpp @@ -146,11 +146,12 @@ int HotspotStackFrame::continuationEntryFrameWords(VMNMethod* nm, const void* pc int words = nm->frameSize(); instruction_t* ip = (instruction_t*)pc; instruction_t* entry = (instruction_t*)nm->entry(); + instruction_t* code = (instruction_t*)nm->code(); if (words < 4 || words > 256 || (words & 1) != 0) return -1; + if (!nm->contains(pc) || !nm->contains(entry) || entry < code || ip < code) return -1; if (ip < entry) { // The interpreter-only adapter has its own frame setup before the // verified compiled entry. Keep the search within that earlier code. - instruction_t* code = (instruction_t*)nm->code(); instruction_t* end = entry < code + 256 ? entry : code + 256; instruction_t* save = code; for (; save + 7 < end; save++) { @@ -186,6 +187,24 @@ int HotspotStackFrame::continuationEntryFrameWords(VMNMethod* nm, const void* pc return ip >= entry + allocated ? words : -1; } + // gen_continuation_enter emits its exception section after the normal + // add rsp; pop rbp; ret. That section restores FP, spills the exception + // around a resolver call, then moves the return PC into RDX. Its register + // sources cannot be represented by this saved-FP/stack-PC layout helper. + // Reject it instead of applying the complete frame size to its changing SP. + size_t completed = ip - entry; + if (completed > 1024) return -1; + unsigned int storage = (words - 2) * sizeof(uintptr_t); + for (size_t offset = 7; offset + 6 <= completed; offset++) { + instruction_t* scan = entry + offset; + // Cleanup first restores JavaThread::_cont_entry from RBX. Include + // that store so an add/pop/ret byte pattern inside movabs is ignored. + if (scan[-7] != 0x49 || scan[-6] != 0x89 || scan[-5] != 0x9f || scan[0] != 0x48 || scan[2] != 0xc4) continue; + if (storage < 128 && scan[1] == 0x83 && scan[3] == storage && scan[4] == 0x5d && scan[5] == 0xc3) return -1; + if (offset + 9 <= completed && scan[1] == 0x81 && scan[3] == (storage & 0xff) && + scan[4] == (storage >> 8) && scan[5] == 0 && scan[6] == 0 && scan[7] == 0x5d && scan[8] == 0xc3) return -1; + } + if (*ip == 0x5d) return 2; // pop FP: storage already deallocated if (*ip == 0xc3) return 1; // ret: FP already restored return words; diff --git a/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp b/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp index 9ca4c96377..4b55fba7b6 100644 --- a/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp +++ b/ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp @@ -493,7 +493,34 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex } } else { uintptr_t entry_fp; - if (VMContinuationEntry::type_size() > 0) { + VMContinuationEntry* recovered_entry = nullptr; + VMNMethod* entry_nm = continuation_entry_nm; + uintptr_t spill_bytes = 0; +#ifdef __x86_64__ + if (entry_nm != nullptr && CodeHeap::findNMethod(walk_pc.raw()) == entry_nm && + frame.continuationEntryFrameWords(entry_nm, walk_pc.raw()) < 0) { + // Only classify unsupported code here; a native return PC does + // not describe the sampled registers. Exception cleanup may + // already have unlinked the entry advertised by the thread. + fillFrame(frames[depth++], BCI_ERROR, "break_cont_entry_fp"); + return false; + } +#endif +#ifdef __aarch64__ + if (kind != ContBoundary::Frozen && entry_nm != nullptr && CodeHeap::findNMethod(walk_pc.raw()) == entry_nm) { + spill_bytes = HotspotStackFrame::continuationEntrySpillWords(entry_nm, walk_pc.raw()) * sizeof(uintptr_t); + } +#endif + if (spill_bytes != 0) { + // Exception cleanup unlinks the entry before resolving its + // handler. The recognized spill still lies below its live frame. + entry_fp = sp > UINTPTR_MAX - spill_bytes ? 0 : entry_nm->continuationEntryFP(sp + spill_bytes, 0); + if (VMContinuationEntry::type_size() > 0 && entry_fp != 0) { + VMContinuationEntry* entry = (VMContinuationEntry*)(sp + spill_bytes); + if (entry->entryFP() != entry_fp) entry_fp = 0; + else recovered_entry = entry; + } + } else if (VMContinuationEntry::type_size() > 0) { // Exported entries (JDK 27+) form a chain for nested continuations. cont_entry = (cont_entry != nullptr) ? cont_entry->parent() : vm_thread->contEntry(); if (cont_entry == nullptr) { @@ -503,7 +530,6 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex } entry_fp = cont_entry->entryFP(); } else { - VMNMethod* entry_nm = continuation_entry_nm; bool known_boundary = VMStructs::isContReturnBarrier(walk_pc.raw()) || VMStructs::isContEntryReturnPc(walk_pc.raw()) || (entry_nm != nullptr && CodeHeap::findNMethod(walk_pc.raw()) == entry_nm); @@ -519,14 +545,7 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex } else if (entry_nm != nullptr && known_boundary) { // Fast thaw does not repair FP links. Recover the entry from SP, // stack-argument padding and enterSpecial's generated frame size. - uintptr_t spill_bytes = 0; -#ifdef __aarch64__ - if (kind != ContBoundary::Frozen && CodeHeap::findNMethod(walk_pc.raw()) == entry_nm) { - spill_bytes = HotspotStackFrame::continuationEntrySpillWords(entry_nm, walk_pc.raw()) * sizeof(uintptr_t); - } -#endif - entry_fp = sp > UINTPTR_MAX - spill_bytes ? 0 : - entry_nm->continuationEntryFP(sp + spill_bytes, kind == ContBoundary::Frozen ? continuation_stack_args : 0); + entry_fp = entry_nm->continuationEntryFP(sp, kind == ContBoundary::Frozen ? continuation_stack_args : 0); } else { entry_fp = contBoundaryEntryFpIsCurrentFp(kind) ? fp : 0; } @@ -537,6 +556,9 @@ __attribute__((no_sanitize("address"))) int HotspotSupport::walkVM(void* ucontex fillFrame(frames[depth++], BCI_ERROR, "break_cont_entry_fp"); return false; } + // The next nested crossing must advance from the recovered entry, + // even if an earlier crossing left the cursor at its child. + if (recovered_entry != nullptr) cont_entry = recovered_entry; // Range-check before dereferencing; crash protection handles stale memory. uintptr_t* saved = (uintptr_t*)INJECT_FAULT_ADDRESS_UNLIKELY(entry_fp); carrier_fp = saved[0]; diff --git a/ddprof-lib/src/test/cpp/continuationEntryX64_ut.cpp b/ddprof-lib/src/test/cpp/continuationEntryX64_ut.cpp new file mode 100644 index 0000000000..eb0428af1c --- /dev/null +++ b/ddprof-lib/src/test/cpp/continuationEntryX64_ut.cpp @@ -0,0 +1,154 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ + +#include +#include +#include +#include "hotspot/hotspotStackFrame.h" +#include "hotspot/vmStructs.inline.h" + +#ifdef __x86_64__ +namespace { + +class ScopedOffset { + int& _slot; + int _saved; +public: + ScopedOffset(int& slot, int value) : _slot(slot), _saved(slot) { slot = value; } + ~ScopedOffset() { _slot = _saved; } + ScopedOffset(const ScopedOffset&) = delete; + ScopedOffset& operator=(const ScopedOffset&) = delete; +}; + +class ContinuationEntryX64Test : public ::testing::Test, protected VMStructs { +protected: + struct Method { + int words = 10; + int size = sizeof(Method); + short complete = 8; + const void* code_begin = code; + const void* entry = code; + unsigned char code[1200]; + } method; + ScopedOffset frame_size{_frame_size_offset, offsetof(Method, words)}; + ScopedOffset blob_size{_blob_size_offset, offsetof(Method, size)}; + ScopedOffset complete{_frame_complete_offset, offsetof(Method, complete)}; + ScopedOffset code_offset{_code_offset, -1}; + ScopedOffset code_address{_code_address, offsetof(Method, code_begin)}; + ScopedOffset entry_offset{_nmethod_entry_offset, -1}; + ScopedOffset entry_address{_nmethod_entry_address, offsetof(Method, entry)}; + + void SetUp() override { + memset(method.code, 0x90, sizeof(method.code)); + const unsigned char setup[] = {0x55, 0x48, 0x8b, 0xec, 0x48, 0x83, 0xec, 0x40}; + memcpy(method.code, setup, sizeof(setup)); + } + + int wordsAt(int offset) { + HotspotStackFrame frame(nullptr); + return frame.continuationEntryFrameWords(reinterpret_cast(&method), method.code + offset); + } + + void exitSequence(bool large = false) { + // gen_continuation_enter (OpenJDK 21u, 25u and main): the normal + // add rsp; pop rbp; ret precedes the separate exception exit. + const unsigned char normal[] = {0x48, 0x83, 0xc4, 0x40, 0x5d, 0xc3}; + const unsigned char normal_large[] = {0x48, 0x81, 0xc4, 0x80, 0, 0, 0, 0x5d, 0xc3}; + const unsigned char restore_parent[] = {0x49, 0x89, 0x9f, 0x38, 0x05, 0, 0}; + memcpy(method.code + 25, restore_parent, sizeof(restore_parent)); + memcpy(method.code + 32, large ? normal_large : normal, large ? sizeof(normal_large) : sizeof(normal)); + // The resolver's address setup/alignment is omitted. The exception + // stack phases, including the final return PC in RDX, are retained. + const unsigned char exception[] = { + 0x48, 0x83, 0xc4, 0x40, // cleanup: add rsp, 64 + 0x5d, // pop rbp + 0x49, 0x8b, 0xff, // mov rdi, r15 + 0x48, 0x8b, 0x34, 0x24, // mov rsi, [rsp] + 0x50, // push rax (exception oop) + 0xe8, 0, 0, 0, 0, // call resolver + 0x48, 0x8b, 0xd8, // mov rbx, rax (handler) + 0x58, // pop rax + 0x5a, // pop rdx (return PC) + 0xff, 0xe3, // jmp rbx + }; + memcpy(method.code + 64, exception, sizeof(exception)); + } +}; + +TEST_F(ContinuationEntryX64Test, RejectsExceptionSectionBeforeAndAfterRestoringRegisters) { + exitSequence(); + for (int phase : {0, 4, 5, 8, 12, 13, 18, 21, 22, 23}) { + EXPECT_EQ(-1, wordsAt(64 + phase)) << "exception byte offset=" << phase; + } +} + +TEST_F(ContinuationEntryX64Test, PreservesNormalEntryBodyAndExitPhases) { + exitSequence(); + EXPECT_EQ(1, wordsAt(0)); + EXPECT_EQ(2, wordsAt(1)); + EXPECT_EQ(2, wordsAt(4)); + EXPECT_EQ(10, wordsAt(8)); + EXPECT_EQ(10, wordsAt(32)); + EXPECT_EQ(2, wordsAt(36)); + EXPECT_EQ(1, wordsAt(37)); +} + +TEST_F(ContinuationEntryX64Test, HandlesLargeImmediateSetupAndCleanup) { + method.words = 18; + method.complete = 11; + const unsigned char setup[] = {0x55, 0x48, 0x8b, 0xec, 0x48, 0x81, 0xec, 0x80, 0, 0, 0}; + memcpy(method.code, setup, sizeof(setup)); + exitSequence(true); + EXPECT_EQ(2, wordsAt(4)); + EXPECT_EQ(18, wordsAt(11)); + EXPECT_EQ(2, wordsAt(39)); + EXPECT_EQ(1, wordsAt(40)); + EXPECT_EQ(-1, wordsAt(64)); +} + +TEST_F(ContinuationEntryX64Test, RequiresMatchingStoragePopAndReturn) { + exitSequence(); + method.code[35] = 0x30; // wrong storage size + EXPECT_EQ(10, wordsAt(48)); + method.code[35] = 0x40; + method.code[36] = 0x58; // pop rax instead of rbp + EXPECT_EQ(10, wordsAt(48)); + method.code[36] = 0x5d; + method.code[37] = 0x90; // no ret + EXPECT_EQ(10, wordsAt(48)); +} + +TEST_F(ContinuationEntryX64Test, DoesNotScanBeforeTheVerifiedEntry) { + exitSequence(); + method.entry = method.code + 48; + method.complete = 48; + EXPECT_EQ(10, wordsAt(64)); +} + +TEST_F(ContinuationEntryX64Test, IgnoresEpilogueBytesInsideAnImmediate) { + const unsigned char literal[] = {0x48, 0xb8, 0x48, 0x83, 0xc4, 0x40, 0x5d, 0xc3, 0, 0}; + memcpy(method.code + 16, literal, sizeof(literal)); + EXPECT_EQ(10, wordsAt(30)); +} + +TEST_F(ContinuationEntryX64Test, RejectsOutOfBlobAndUnboundedCodePositions) { + EXPECT_EQ(-1, wordsAt(1100)); + method.size = offsetof(Method, code) + 16; + EXPECT_EQ(-1, wordsAt(32)); +} + +TEST_F(ContinuationEntryX64Test, RejectsInvalidFrameSizes) { + for (int words : {0, 2, 9, 258}) { + method.words = words; + EXPECT_EQ(-1, wordsAt(8)); + } +} + +} // namespace +#else +TEST(ContinuationEntryX64Test, RequiresX64) { + GTEST_SKIP() << "x64 continuation entry decoding"; +} +#endif diff --git a/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp b/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp index 73ee21250a..642d20f232 100644 --- a/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp +++ b/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp @@ -221,8 +221,11 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { ExceptionMove, ExceptionBranch, AuthenticatedExceptionMove, AuthenticatedExceptionBranch, ExceptionMoveWithoutBranch, ExceptionBranchWithoutMove, InterpretedEntry, UnsupportedInterpretedEntry, LeafSpillBeforeSave, LeafSpillMove, LeafSpillCall, LeafSpillRestore, LeafSpillAfterRestore, - LeafSpillOneMove, LeafSpillMovn, LeafSpillLogicalImmediate, LeafSpillFourMoves, LeafSpillNativePrefix, LeafSpillUnknownSequence, - BarrierInterior, BarrierInteriorRoot, BarrierInteriorNativePrefix + LeafSpillOneMove, LeafSpillMovn, LeafSpillLogicalImmediate, LeafSpillFourMoves, LeafSpillNativePrefix, + LeafSpillNativePrefixExportedNull, LeafSpillNativePrefixExportedParent, LeafSpillNativePrefixExportedTrackedChild, + LeafSpillNativePrefixExportedInvalidSize, LeafSpillUnknownSequence, + BarrierInterior, BarrierInteriorRoot, BarrierInteriorNativePrefix, + X64ExceptionNativePrefixExportedNull, X64ExceptionNativePrefixExportedParent }; // Generated prologue/setup and normal cleanup excerpts, captured from @@ -231,7 +234,13 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { // cpu/{aarch64,x86}/sharedRuntime_*.cpp. The body between them is omitted. // Metadata and registers are synthetic; the production walker decodes them. void assertCarrierFromEntry(EntryPhase phase, EntryCase test_case = EntryCase::Native) { - alignas(16) uintptr_t stack_storage[36]{}; + bool x64_exception_prefix = test_case == EntryCase::X64ExceptionNativePrefixExportedNull || + test_case == EntryCase::X64ExceptionNativePrefixExportedParent; + bool tracked_child = test_case == EntryCase::LeafSpillNativePrefixExportedTrackedChild; + bool invalid_exported_size = test_case == EntryCase::LeafSpillNativePrefixExportedInvalidSize; + bool leaf_native_prefix = test_case == EntryCase::LeafSpillNativePrefix || + test_case == EntryCase::LeafSpillNativePrefixExportedNull || test_case == EntryCase::LeafSpillNativePrefixExportedParent || tracked_child || invalid_exported_size; + alignas(16) uintptr_t stack_storage[52]{}; uintptr_t* stack = stack_storage + 4; // room for a native frame and the temporary VM-leaf spill struct ConstMethod { void* constants = nullptr; unsigned short idnum = 0; unsigned short slots = 0; } const_method; struct Method { ConstMethod* const_method; } method{&const_method}; @@ -303,6 +312,17 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { const int offsets[] = {0, 1, 4, 27, 68, 72, 73, 8}; #endif memcpy(blob.nm.code, instructions, sizeof(instructions)); +#ifdef __x86_64__ + if (x64_exception_prefix) { + const unsigned char exception[] = { + 0x48, 0x83, 0xc4, 0x40, 0x5d, // cleanup and pop rbp + 0x49, 0x8b, 0xff, 0x48, 0x8b, 0x34, 0x24, // resolver arguments + 0x50, 0xe8, 0, 0, 0, 0, // push exception and call resolver + 0x48, 0x8b, 0xd8, 0x58, 0x5a, 0xff, 0xe3 // restore exception/PC, jump + }; + memcpy((char*)blob.nm.code + 80, exception, sizeof(exception)); + } +#endif if (barrier_interior) { auto prefix = barrierInstructions(); memcpy(blob.nm.code, prefix.data(), prefix.size()); @@ -398,10 +418,13 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { ScopedValue constants(_constmethod_constants_offset, offsetof(ConstMethod, constants)); ScopedValue idnum(_constmethod_idnum_offset, offsetof(ConstMethod, idnum)); ScopedValue slots(_constmethod_stack_arg_slots_offset, offsetof(ConstMethod, slots)); - bool exported_parent = test_case == EntryCase::ExportedParentEntry; - ScopedValue cont_entry_offset(_cont_entry_offset, exported_parent ? offsetof(ThreadLayout, entry) : -1); - ScopedValue entry_size(TYPE_SIZE_NAME(VMContinuationEntry), exported_parent ? 64 : 0); - if (exported_parent) thread.entry = &stack[16]; // enclosing entry before publication/after removal + bool exported_parent = test_case == EntryCase::ExportedParentEntry || test_case == EntryCase::LeafSpillNativePrefixExportedParent || + test_case == EntryCase::X64ExceptionNativePrefixExportedParent; + bool exported_entry = exported_parent || test_case == EntryCase::LeafSpillNativePrefixExportedNull || tracked_child || invalid_exported_size || x64_exception_prefix; + ScopedValue cont_entry_offset(_cont_entry_offset, exported_entry ? offsetof(ThreadLayout, entry) : -1); + ScopedValue parent_offset(_cont_entry_parent_offset, 0); + ScopedValue entry_size(TYPE_SIZE_NAME(VMContinuationEntry), exported_entry ? (invalid_exported_size ? 80 : 64) : 0); + if (exported_entry) thread.entry = exported_parent ? &stack[16] : nullptr; ScopedValue type_size(TYPE_SIZE_NAME(VMNMethod), sizeof(NMethod)); ScopedValue method_size(TYPE_SIZE_NAME(VMMethod), sizeof(Method)); ScopedValue constmethod_size(TYPE_SIZE_NAME(VMConstMethod), sizeof(ConstMethod)); @@ -418,6 +441,9 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { JitCodeCache::DynamicCodeGenerated(nullptr, "cont return barrier", blob.nm.code, sizeof(blob.nm.code)); } const void* sampled_pc = (const char*)blob.nm.code + offsets[(int)phase] + prefix; +#ifdef __x86_64__ + if (x64_exception_prefix) sampled_pc = (const char*)blob.nm.code + 98; // resolver return PC +#endif #ifdef __aarch64__ if (exception_exit && prefix != 0) sampled_pc = (const char*)sampled_pc + sizeof(uint32_t); // after autiaz if (test_case == EntryCase::ExceptionBranch || test_case == EntryCase::AuthenticatedExceptionBranch || @@ -426,14 +452,14 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { int instruction = test_case == EntryCase::LeafSpillBeforeSave ? 0 : test_case == EntryCase::LeafSpillMove || test_case == EntryCase::LeafSpillUnknownSequence ? 1 : test_case == EntryCase::LeafSpillCall ? 4 : - test_case == EntryCase::LeafSpillRestore || test_case == EntryCase::LeafSpillNativePrefix ? 5 : 6; + test_case == EntryCase::LeafSpillRestore || leaf_native_prefix ? 5 : 6; if (test_case >= EntryCase::LeafSpillOneMove && test_case <= EntryCase::LeafSpillFourMoves) instruction = 1; sampled_pc = blob.nm.code + 7 + instruction; } #endif ScopedValue return_pc(_cont_entry_return_pc, - test_case == EntryCase::ReturnPc ? sampled_pc : _cont_entry_return_pc); - VMTestAccessor::install(true, exported_parent ? 27 : 21); + test_case == EntryCase::ReturnPc ? sampled_pc : tracked_child ? blob.nm.code + 31 : _cont_entry_return_pc); + VMTestAccessor::install(true, exported_entry ? 27 : 21); bool nonpointer_fp = test_case == EntryCase::CompiledNonPointerFp || test_case == EntryCase::NativeNonPointerFp || test_case == EntryCase::BlobNonPointerFp; @@ -447,6 +473,14 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { stack[11] = (uintptr_t)THREAD_ENTRY_PC; stack[12] = (uintptr_t)&stack[26]; stack[13] = (uintptr_t)THREAD_ENTRY_PC; + if (test_case == EntryCase::X64ExceptionNativePrefixExportedParent) { + // The published parent is independently walkable. Following it + // would fabricate ancestry for the already unlinked child. + stack[24] = (uintptr_t)&stack[40]; + stack[25] = (uintptr_t)CARRIER_PC; + stack[26] = (uintptr_t)&stack[42]; + stack[27] = (uintptr_t)THREAD_ENTRY_PC; + } int sp_slot = phase == EntryPhase::Complete || phase == EntryPhase::BeforeSpRestore || phase == EntryPhase::AfterAllocation ? 0 : 8; #ifdef __aarch64__ @@ -484,12 +518,45 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { if (test_case == EntryCase::MisalignedSp) frame.sp() += 8; if (test_case == EntryCase::OutOfBoundsSp) frame.sp() = (UINTPTR_MAX / 2) & ~(uintptr_t)15; frame.fp() = sampled_fp; - if (test_case == EntryCase::LeafSpillNativePrefix || test_case == EntryCase::BarrierInteriorNativePrefix) { +#ifdef __x86_64__ + if (x64_exception_prefix) { + // Cleanup popped the entry's RBP, then pushed the exception at + // stack[8]. The native callee saves that restored caller RBP and + // the resolver return PC below it; unwinding restores SP=stack+8. + stack[6] = saved_fp; + stack[7] = (uintptr_t)sampled_pc; + stack[8] = 1; // exception oop, not a saved frame pointer + frame.pc() = (uintptr_t)LEAF_PC; + frame.sp() = (uintptr_t)&stack[6]; + frame.fp() = (uintptr_t)&stack[6]; + } else +#endif + if (leaf_native_prefix || x64_exception_prefix || test_case == EntryCase::BarrierInteriorNativePrefix) { stack[-4] = sampled_fp; stack[-3] = (uintptr_t)sampled_pc; frame.pc() = (uintptr_t)LEAF_PC; frame.sp() = (uintptr_t)(stack - 4); } + if (tracked_child) { + // Cross a child first, recover its parent's unlinked resolver frame, + // then reach the grandparent boundary. Replaying the parent fails. + thread.entry = stack; + stack[0] = (uintptr_t)&stack[14]; + stack[8] = (uintptr_t)&stack[40]; + stack[9] = (uintptr_t)LEAF_PC; + stack[10] = (uintptr_t)&stack[40]; + stack[11] = (uintptr_t)sampled_pc; + stack[14] = (uintptr_t)&stack[26]; + stack[22] = (uintptr_t)&stack[40]; + stack[23] = (uintptr_t)_cont_entry_return_pc; + stack[34] = (uintptr_t)&stack[40]; + stack[35] = (uintptr_t)CARRIER_PC; + stack[36] = (uintptr_t)&stack[40]; + stack[37] = (uintptr_t)THREAD_ENTRY_PC; + frame.pc() = (uintptr_t)_cont_entry_return_pc; + frame.sp() = (uintptr_t)stack; + frame.fp() = (uintptr_t)&stack[8]; + } StackWalkFeatures features{}; features.carrier_frames = test_case != EntryCase::BarrierInteriorRoot; bool truncated = false; @@ -504,7 +571,14 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { EXPECT_TRUE(truncated); return; } - if (test_case == EntryCase::LeafSpillNativePrefix) { + if (invalid_exported_size || x64_exception_prefix) { + ASSERT_EQ(2, depth); + ASSERT_EQ(BCI_ERROR, frames[1].bci); + EXPECT_STREQ("break_cont_entry_fp", (const char*)frames[1].method_id); + EXPECT_TRUE(truncated); + return; + } + if (leaf_native_prefix) { assertCarrier(depth); EXPECT_FALSE(truncated); return; @@ -982,6 +1056,16 @@ TEST_F(ContinuationWalkTest, InterpretedEntryRejectsUnsupportedPrologue) { assertCarrierFromEntry(EntryPhase::Complete, EntryCase::UnsupportedInterpretedEntry); } +#ifdef __x86_64__ +TEST_F(ContinuationWalkTest, ExceptionResolverReturnPcRejectsUnlinkedExportedEntry) { + assertCarrierFromEntry(EntryPhase::Complete, EntryCase::X64ExceptionNativePrefixExportedNull); +} + +TEST_F(ContinuationWalkTest, ExceptionResolverReturnPcDoesNotReplayExportedParent) { + assertCarrierFromEntry(EntryPhase::Complete, EntryCase::X64ExceptionNativePrefixExportedParent); +} +#endif + TEST_F(ContinuationWalkTest, ReturnBarrierInteriorRetainsContinuationAncestry) { for (EntryCase test_case : {EntryCase::BarrierInterior, EntryCase::BarrierInteriorRoot, EntryCase::BarrierInteriorNativePrefix}) { assertCarrierFromEntry(EntryPhase::Complete, test_case); @@ -1019,6 +1103,22 @@ TEST_F(ContinuationWalkTest, EntryVmLeafSpillWithNativePrefixRetainsCarrier) { assertCarrierFromEntry(EntryPhase::Complete, EntryCase::LeafSpillNativePrefix); } +TEST_F(ContinuationWalkTest, EntryVmLeafSpillWithNativePrefixIgnoresUnlinkedExportedEntry) { + assertCarrierFromEntry(EntryPhase::Complete, EntryCase::LeafSpillNativePrefixExportedNull); +} + +TEST_F(ContinuationWalkTest, EntryVmLeafSpillWithNativePrefixIgnoresExportedParent) { + assertCarrierFromEntry(EntryPhase::Complete, EntryCase::LeafSpillNativePrefixExportedParent); +} + +TEST_F(ContinuationWalkTest, EntryVmLeafSpillWithExportedMetadataAdvancesTrackedChildToParent) { + assertCarrierFromEntry(EntryPhase::Complete, EntryCase::LeafSpillNativePrefixExportedTrackedChild); +} + +TEST_F(ContinuationWalkTest, EntryVmLeafSpillRejectsInconsistentExportedFrameSize) { + assertCarrierFromEntry(EntryPhase::Complete, EntryCase::LeafSpillNativePrefixExportedInvalidSize); +} + TEST_F(ContinuationWalkTest, ExceptionExitAfterRestoringRegistersRetainsCarrier) { for (EntryCase test_case : {EntryCase::ExceptionMove, EntryCase::ExceptionBranch}) { assertCarrierFromEntry(EntryPhase::AfterRestore, test_case); diff --git a/doc/architecture/JvmSupportLayer.md b/doc/architecture/JvmSupportLayer.md index 9779fd14d6..1989e08984 100644 --- a/doc/architecture/JvmSupportLayer.md +++ b/doc/architecture/JvmSupportLayer.md @@ -173,6 +173,11 @@ or return-PC stack slot (x64). Unknown prologues fail rather than applying the complete-frame size. Pointer-authentication instructions are recognized on arm64, and recovered PCs have PAC bits stripped. +On x64, the generated exception section after the validated normal cleanup is +unsupported. Its FP and return-PC sources change around handler resolution. Both +direct samples and native resolver return PCs stop with an error and truncation +instead of applying a complete-frame layout or following an unlinked entry. + Interpreter-only execution uses a separate entry before the verified compiled entry on both architectures. The decoder locates that entry's recognized frame setup with a bounded scan before the compiled entry. Its prologue and established @@ -219,6 +224,11 @@ alignment and carrier admission checks apply before proceeding. Walks with a known argument count retain SP-based recovery. Missing or invalid metadata still rejects the walk, and frozen boundaries remain marked truncated. +Recognized arm64 VM-leaf spills also recover an entry that exception cleanup has +already unlinked from the thread. Exported layouts must agree with the entry FP +derived from the nmethod size. The nested-entry cursor records that physical entry +so the next crossing follows its parent once. + Native thaw calls return inside the barrier stub, where SP has already changed. The runtime-stub lookup identifies these PCs by matching the registered stub's start to the known return-barrier entry. They use the current thread's entry From bbe09e6d897cf0a0d55389cfdc055d68e8043f0b Mon Sep 17 00:00:00 2001 From: Paul Fournillon Date: Wed, 28 Oct 2026 14:39:28 +0100 Subject: [PATCH 7/7] fix(profiler): recover arm64 JNI wrapper frame phases --- AGENTS.md | 2 +- .../cpp/hotspot/hotspotStackFrame_aarch64.cpp | 34 ++++- .../src/test/cpp/continuationWalk_ut.cpp | 67 +++++++++- .../src/test/cpp/unwindHelperContract_ut.cpp | 125 +++++++++++++++++- doc/README.md | 2 +- doc/architecture/JvmSupportLayer.md | 10 ++ 6 files changed, 227 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index bfc92aecac..688ae30f45 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -158,7 +158,7 @@ before working on a subsystem. | String dictionary (arena chunks, rotation/clear protocols) | [doc/architecture/StringDictionary.md](doc/architecture/StringDictionary.md) | | RefCountGuard (lock-free RAII reader-drain before reclamation) | [doc/architecture/RefCountGuard.md](doc/architecture/RefCountGuard.md) | | TLS context (thread-local tracing context, distributed tracing) | [doc/architecture/TLSContext.md](doc/architecture/TLSContext.md) | -| Native stack unwinding (DWARF CFA rules, arm64 temporary argument areas and LR/PLT, validated Linux signal-frame recovery) | [doc/architecture/JvmSupportLayer.md](doc/architecture/JvmSupportLayer.md) | +| Native stack unwinding (DWARF CFA rules, arm64 temporary argument areas, JNI wrapper phases and LR/PLT, validated Linux signal-frame recovery) | [doc/architecture/JvmSupportLayer.md](doc/architecture/JvmSupportLayer.md) | | Profiling engines (CPU, wall-clock, allocation, native memory, liveness) | [doc/architecture/ProfilingEngines.md](doc/architecture/ProfilingEngines.md) | | JFR integration (FlightRecorder, metadata, constant pools, buffers) | [doc/architecture/JfrIntegration.md](doc/architecture/JfrIntegration.md) | | Live-heap reference chains (implementation, signal scheduling, design) | [doc/architecture/LiveHeapReferenceChainsImplementation.md](doc/architecture/LiveHeapReferenceChainsImplementation.md), [doc/architecture/ReferenceChainsSignalsExplained.md](doc/architecture/ReferenceChainsSignalsExplained.md), [doc/ReferenceChainsDesign.md](doc/ReferenceChainsDesign.md), [doc/ReferenceChainsCollectionSummary.md](doc/ReferenceChainsCollectionSummary.md) | diff --git a/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_aarch64.cpp b/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_aarch64.cpp index f54e7225b6..afb5acb222 100644 --- a/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_aarch64.cpp +++ b/ddprof-lib/src/main/cpp/hotspot/hotspotStackFrame_aarch64.cpp @@ -276,11 +276,16 @@ NOSANALIGSANITIZE bool HotspotStackFrame::unwindPrologue(VMNMethod* nm, uintptr_ unsigned int offset = (*ip >> 12) & 0x1f8; sp += offset + 16; pc = link(); - } else if (ip[0] == 0x910003fd && ip[-1] == 0xa9bf7bfd) { + } else if ((ip[0] == 0x910003fd && ip[-1] == 0xa9bf7bfd) || + (ip >= entry + 2 && ip[-2] == 0xa9bf7bfd && ip[-1] == 0x910003fd && + nm->frameSize() >= 2 && nm->frameSize() <= 512 && (nm->frameSize() & 1) == 0 && + ip[0] == (0xd10003ffU | ((nm->frameSize() - 2) * sizeof(uintptr_t) << 10)))) { // stp x29, x30, [sp, #-16]! // mov x29, sp + // sub sp, sp, #locals (not executed yet) + fp = ((uintptr_t*)sp)[0]; + pc = (uintptr_t)stripPointer((const void*)((uintptr_t*)sp)[1]); sp += 16; - pc = ((uintptr_t*)sp)[-1]; } else if (ip <= entry + 16 && isFrameComplete(entry, ip)) { sp += nm->frameSize() * sizeof(void*); fp = ((uintptr_t*)sp)[-2]; @@ -297,6 +302,31 @@ NOSANALIGSANITIZE bool HotspotStackFrame::unwindEpilogue(VMNMethod* nm, uintptr_ // {poll_return} // ret instruction_t* ip = (instruction_t*)pc; + instruction_t* entry = (instruction_t*)nm->entry(); + // JNI wrappers use leave(), then test pending_exception before ret. + // At ldp only the FP/LR save remains; after it SP and LR are restored. + // Require the whole generated tail so a body load/branch cannot select LR. + for (int distance = 1; distance <= 4 && ip - entry >= distance; distance++) { + instruction_t* tail = ip - distance; + if (!nm->contains(tail) || !nm->contains((const char*)(tail + 1) + sizeof(instruction_t) - 1) || + tail[0] != 0x910003bf || tail[1] != 0xa8c17bfd) continue; + instruction_t* check = tail + 2; + if (!nm->contains((const char*)check + sizeof(instruction_t) - 1)) continue; + if (*check == 0xd503239f || *check == 0xd50323bf) check++; // autiaz/autiasp + if (!nm->contains((const char*)(check + 2) + sizeof(instruction_t) - 1) || ip > check + 1 || + (*check & 0xffc003ffU) != 0xf9400388 || // ldr x8, [x28, #offset] + (check[1] & 0xff00001fU) != 0xb5000008 || // cbnz x8, forward_exception + check[2] != 0xd65f03c0) continue; + if (ip == tail + 1) { + fp = ((uintptr_t*)sp)[0]; + pc = (uintptr_t)stripPointer((const void*)((uintptr_t*)sp)[1]); + sp += 16; + } else { + // Linux signal delivery may overwrite the slots below restored SP. + pc = (uintptr_t)stripPointer((const void*)link()); + } + return true; + } if (*ip == 0xd65f03c0 || isPollReturn(ip)) { // ret pc = link(); return true; diff --git a/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp b/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp index 5c8043d66e..2443c754c5 100644 --- a/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp +++ b/ddprof-lib/src/test/cpp/continuationWalk_ut.cpp @@ -227,7 +227,8 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { LeafSpillNativePrefixExportedNull, LeafSpillNativePrefixExportedParent, LeafSpillNativePrefixExportedTrackedChild, LeafSpillNativePrefixExportedInvalidSize, LeafSpillUnknownSequence, BarrierInterior, BarrierInteriorRoot, BarrierInteriorNativePrefix, - X64ExceptionNativePrefixExportedNull, X64ExceptionNativePrefixExportedParent + X64ExceptionNativePrefixExportedNull, X64ExceptionNativePrefixExportedParent, + NativeWrapper, NativeWrapperBranch, NativeWrapperReturn, NativeWrapperReturnPc }; // Generated prologue/setup and normal cleanup excerpts, captured from @@ -236,14 +237,16 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { // cpu/{aarch64,x86}/sharedRuntime_*.cpp. The body between them is omitted. // Metadata and registers are synthetic; the production walker decodes them. void assertCarrierFromEntry(EntryPhase phase, EntryCase test_case = EntryCase::Native) { + bool native_wrapper = test_case == EntryCase::NativeWrapper || test_case == EntryCase::NativeWrapperBranch || + test_case == EntryCase::NativeWrapperReturn || test_case == EntryCase::NativeWrapperReturnPc; bool x64_exception_prefix = test_case == EntryCase::X64ExceptionNativePrefixExportedNull || test_case == EntryCase::X64ExceptionNativePrefixExportedParent; bool tracked_child = test_case == EntryCase::LeafSpillNativePrefixExportedTrackedChild; bool invalid_exported_size = test_case == EntryCase::LeafSpillNativePrefixExportedInvalidSize; bool leaf_native_prefix = test_case == EntryCase::LeafSpillNativePrefix || test_case == EntryCase::LeafSpillNativePrefixExportedNull || test_case == EntryCase::LeafSpillNativePrefixExportedParent || tracked_child || invalid_exported_size; - alignas(16) uintptr_t stack_storage[52]{}; - uintptr_t* stack = stack_storage + 4; // room for a native frame and the temporary VM-leaf spill + alignas(16) uintptr_t stack_storage[56]{}; + uintptr_t* stack = stack_storage + 8; // room for a native frame and the temporary VM-leaf spill struct ConstMethod { void* constants = nullptr; unsigned short idnum = 0; unsigned short slots = 0; } const_method; struct Method { ConstMethod* const_method; } method{&const_method}; struct NMethod { @@ -385,6 +388,18 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { } if (test_case == EntryCase::LeafSpillUnknownSequence) blob.nm.code[11] = 0xd503201f; } +#endif + #ifdef __aarch64__ + if (native_wrapper) { + // Temurin 21 JNI wrapper: 96-byte frame, leave(), exception check, ret. + blob.nm.frame_size = 12; + blob.nm.method = &method; + blob.nm.code[2] = 0xd10143ff; // sub sp, sp, #80 + blob.nm.code[14] = 0xd503201f; // JNI leave() uses the established FP. + blob.nm.code[17] = 0xf9400788; // ldr x8, [x28, #8] + blob.nm.code[18] = 0xb5000048; // cbnz x8, forward_exception + blob.nm.code[19] = 0xd65f03c0; + } #endif if (test_case == EntryCase::UnsupportedPrologue) blob.nm.code[0] = 0; if (test_case == EntryCase::InterpretedEntry || test_case == EntryCase::UnsupportedInterpretedEntry) { @@ -435,7 +450,7 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { ScopedValue heap2(_code_heap[2], nullptr); ScopedValue heap_low(_code_heap_low, heap.low); ScopedValue heap_high(_code_heap_high, heap.high); - ScopedValue entry_nm(_enter_special_nm, reinterpret_cast(barrier_interior ? &carrier.nm : &blob.nm)); + ScopedValue entry_nm(_enter_special_nm, native_wrapper ? nullptr : reinterpret_cast(barrier_interior ? &carrier.nm : &blob.nm)); ScopedValue return_barrier(_cont_return_barrier, barrier_interior ? blob.nm.code : _cont_return_barrier); ScopedValue inferred_entry_offset(_cont_entry_thread_offset, barrier_interior ? 0 : _cont_entry_thread_offset); if (barrier_interior) { @@ -447,6 +462,8 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { if (x64_exception_prefix) sampled_pc = (const char*)blob.nm.code + 98; // resolver return PC #endif #ifdef __aarch64__ + if (test_case == EntryCase::NativeWrapperBranch) sampled_pc = blob.nm.code + 18; + if (test_case == EntryCase::NativeWrapperReturn) sampled_pc = blob.nm.code + 19; if (exception_exit && prefix != 0) sampled_pc = (const char*)sampled_pc + sizeof(uint32_t); // after autiaz if (test_case == EntryCase::ExceptionBranch || test_case == EntryCase::AuthenticatedExceptionBranch || test_case == EntryCase::ExceptionBranchWithoutMove) sampled_pc = (const char*)sampled_pc + sizeof(uint32_t); @@ -488,9 +505,11 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { #ifdef __aarch64__ if (leaf_spill && test_case != EntryCase::LeafSpillBeforeSave && test_case != EntryCase::LeafSpillAfterRestore) sp_slot = -2; #endif + if (native_wrapper && (phase == EntryPhase::Complete || phase == EntryPhase::BeforeSpRestore || + test_case == EntryCase::NativeWrapperReturnPc)) sp_slot = -2; uintptr_t sampled_fp = (uintptr_t)&stack[8]; if (phase == EntryPhase::AfterSave) sampled_fp = saved_fp; - if (phase == EntryPhase::BeforeSave || phase == EntryPhase::AfterRestore) { + if ((phase == EntryPhase::BeforeSave || phase == EntryPhase::AfterRestore) && test_case != EntryCase::NativeWrapperReturnPc) { sampled_fp = saved_fp; stack[8] = 0; // Below sampled SP; may be overwritten by signal delivery. #ifdef __aarch64__ @@ -559,10 +578,36 @@ class ContinuationWalkTest : public ::testing::Test, protected VMStructs { frame.sp() = (uintptr_t)stack; frame.fp() = (uintptr_t)&stack[8]; } + if (test_case == EntryCase::NativeWrapperReturnPc) { + // A recovered return PC is not an interrupted epilogue. Its full + // frame is still present; the sampled LR belongs to the native leaf. + stack[-4] = sampled_fp; + stack[-3] = (uintptr_t)sampled_pc; + frame.pc() = (uintptr_t)LEAF_PC; + frame.sp() = (uintptr_t)(stack - 4); +#ifdef __aarch64__ +#ifdef __APPLE__ + machine_context.__ss.__lr = 1; +#else + context.uc_mcontext.regs[30] = 1; +#endif +#endif + } StackWalkFeatures features{}; features.carrier_frames = test_case != EntryCase::BarrierInteriorRoot; bool truncated = false; int depth = HotspotSupportTestAccessor::walk(&context, frames, features, MAX_DEPTH, &truncated); + if (native_wrapper) { + int prefix_depth = test_case == EntryCase::NativeWrapperReturnPc ? 1 : 0; + ASSERT_EQ(2 + prefix_depth, depth); + if (prefix_depth != 0) EXPECT_STREQ("continuation_sampled_leaf", (const char*)frames[0].method_id); + EXPECT_EQ(FrameType::encode(FRAME_JIT_COMPILED, 0), frames[prefix_depth].bci); + EXPECT_EQ(JMETHODID_NOT_WALKABLE, frames[prefix_depth].method_id); + EXPECT_EQ(BCI_NATIVE_FRAME, frames[1 + prefix_depth].bci); + EXPECT_STREQ("continuation_carrier", (const char*)frames[1 + prefix_depth].method_id); + EXPECT_FALSE(truncated); + return; + } if (barrier_interior) { if (test_case == EntryCase::BarrierInteriorNativePrefix) assertCarrier(depth); else { @@ -1146,6 +1191,18 @@ TEST_F(ContinuationWalkTest, SampledEntryWithPointerAuthenticationRetainsCarrier assertCarrierFromEntry(phase, EntryCase::PointerAuthentication); } } +TEST_F(ContinuationWalkTest, NativeWrapperPhasesRetainCaller) { + for (EntryPhase phase : {EntryPhase::BeforeSave, EntryPhase::AfterSave, EntryPhase::BeforeAllocation, + EntryPhase::Complete, EntryPhase::BeforeSpRestore, EntryPhase::AfterDeallocation, + EntryPhase::AfterRestore}) { + SCOPED_TRACE((int)phase); + assertCarrierFromEntry(phase, EntryCase::NativeWrapper); + } + assertCarrierFromEntry(EntryPhase::AfterRestore, EntryCase::NativeWrapperBranch); + assertCarrierFromEntry(EntryPhase::AfterRestore, EntryCase::NativeWrapperReturn); + assertCarrierFromEntry(EntryPhase::AfterRestore, EntryCase::NativeWrapperReturnPc); +} + #endif } // namespace diff --git a/ddprof-lib/src/test/cpp/unwindHelperContract_ut.cpp b/ddprof-lib/src/test/cpp/unwindHelperContract_ut.cpp index 3209f0acf5..7386ead225 100644 --- a/ddprof-lib/src/test/cpp/unwindHelperContract_ut.cpp +++ b/ddprof-lib/src/test/cpp/unwindHelperContract_ut.cpp @@ -48,6 +48,8 @@ // befriends. class VMStructsTestAccessor { public: + static offset getBlobSizeOffset() { return VMStructs::_blob_size_offset; } + static void setBlobSizeOffset(offset value) { VMStructs::_blob_size_offset = value; } static offset getFrameSizeOffset() { return VMStructs::_frame_size_offset; } static void setFrameSizeOffset(offset value) { VMStructs::_frame_size_offset = value; } static offset getNMethodEntryOffset() { return VMStructs::_nmethod_entry_offset; } @@ -197,8 +199,8 @@ TEST_F(UnwindHelperContractTest, AsgctOverloadLeavesUcontextReadyForAsyncGetCall // unwindPrologue / unwindEpilogue // // Same contract, but these two take a VMNMethod*, so they need a stand-in. -// Only two fields are ever read on the branches under test: entry(), to -// locate the start of the method's code, and frameSize(). The fixture is a +// The branches under test read entry(), frameSize(), and the blob size +// used to bound native-wrapper tail decoding. The fixture is a // byte buffer laid out to satisfy exactly those, with the offsets pointed at // it for the duration of the test and restored afterwards -- the offsets are // process-global, and a real JVM run would have populated them from @@ -217,6 +219,7 @@ class UnwindNMethodHelperContractTest : public UnwindHelperContractTest { void SetUp() override { UnwindHelperContractTest::SetUp(); + _saved_blob_size_offset = VMStructsTestAccessor::getBlobSizeOffset(); _saved_frame_size_offset = VMStructsTestAccessor::getFrameSizeOffset(); _saved_entry_offset = VMStructsTestAccessor::getNMethodEntryOffset(); _saved_entry_address = VMStructsTestAccessor::getNMethodEntryAddress(); @@ -224,6 +227,8 @@ class UnwindNMethodHelperContractTest : public UnwindHelperContractTest { memset(_nmethod, 0, sizeof(_nmethod)); *(const void**)(bytes() + kEntryAddressOffset) = code(); *(int*)(bytes() + kFrameSizeOffset) = 4; + *(int*)(bytes() + 24) = sizeof(_nmethod); + VMStructsTestAccessor::setBlobSizeOffset(24); VMStructsTestAccessor::setFrameSizeOffset(kFrameSizeOffset); // -1 selects entry()'s pre-JDK-23 arm: a single load of the stored @@ -233,6 +238,7 @@ class UnwindNMethodHelperContractTest : public UnwindHelperContractTest { } void TearDown() override { + VMStructsTestAccessor::setBlobSizeOffset(_saved_blob_size_offset); VMStructsTestAccessor::setFrameSizeOffset(_saved_frame_size_offset); VMStructsTestAccessor::setNMethodEntryOffset(_saved_entry_offset); VMStructsTestAccessor::setNMethodEntryAddress(_saved_entry_address); @@ -249,15 +255,17 @@ class UnwindNMethodHelperContractTest : public UnwindHelperContractTest { // int the layout above stores in it. char* bytes() { return (char*)_nmethod; } - static uintptr_t _nmethod[4]; + instruction_t* code() { return reinterpret_cast(_nmethod + 4); } + static uintptr_t _nmethod[32]; private: + offset _saved_blob_size_offset; offset _saved_frame_size_offset; offset _saved_entry_offset; offset _saved_entry_address; }; -uintptr_t UnwindNMethodHelperContractTest::_nmethod[4]; +uintptr_t UnwindNMethodHelperContractTest::_nmethod[32]; // pc sitting on the method's first instruction: nothing of the frame has been // built yet, so the sender pc is still wherever the call left it. @@ -348,4 +356,113 @@ TEST_F(UnwindNMethodHelperContractTest, UnwindEpilogueAtReturnRecoversTheRawSend #endif } +#ifdef __aarch64__ +TEST_F(UnwindNMethodHelperContractTest, NativeWrapperPendingAllocationRestoresSavedFpAndSender) { + code()[0] = 0xa9bf7bfd; + code()[1] = 0x910003fd; + code()[2] = 0xd10143ff; // sub sp, sp, #80: 12-word full frame + *(int*)(bytes() + kFrameSizeOffset) = 12; + uintptr_t* slot = stackSlot(); + slot[0] = kOtherSlot; + slot[1] = kSenderRa; + for (int phase : {1, 2}) { + uintptr_t pc = (uintptr_t)(code() + phase); + uintptr_t sp = (uintptr_t)slot; + uintptr_t fp = sp; + HotspotStackFrame frame(&_uc); + ASSERT_TRUE(frame.unwindPrologue(nmethod(), pc, sp, fp)); + EXPECT_EQ(kSenderRa, pc); + EXPECT_EQ((uintptr_t)(slot + 2), sp); + EXPECT_EQ(kOtherSlot, fp); + } + for (int words : {-2, 1, 3, 11, 13, 514}) { + SCOPED_TRACE(words); + *(int*)(bytes() + kFrameSizeOffset) = words; + uintptr_t pc = (uintptr_t)(code() + 2); + uintptr_t sp = (uintptr_t)slot; + uintptr_t fp = sp; + HotspotStackFrame frame(&_uc); + ASSERT_TRUE(frame.unwindPrologue(nmethod(), pc, sp, fp)); + EXPECT_EQ(kLinkReg, pc); + EXPECT_EQ((uintptr_t)slot, sp); + EXPECT_EQ((uintptr_t)slot, fp); + } +} + +TEST_F(UnwindNMethodHelperContractTest, NativeWrapperEpilogueRestoresOnlyTheLiveRegisterSave) { + const uint32_t tail[] = {0x910003bf, 0xa8c17bfd, 0xf9400788, 0xb5000048, 0xd65f03c0}; + memcpy(code() + 4, tail, sizeof(tail)); + for (int phase = 1; phase < 5; phase++) { + SCOPED_TRACE(phase); + uintptr_t* slot = stackSlot(); + slot[0] = kOtherSlot; + slot[1] = kSenderRa; + uintptr_t pc = (uintptr_t)(code() + 4 + phase); + uintptr_t sp = (uintptr_t)slot; + uintptr_t fp = phase == 1 ? sp : kOtherSlot; + HotspotStackFrame frame(&_uc); + ASSERT_TRUE(frame.unwindEpilogue(nmethod(), pc, sp, fp)); + EXPECT_EQ(phase == 1 ? kSenderRa : kLinkReg, pc); + EXPECT_EQ((uintptr_t)slot + (phase == 1 ? 16 : 0), sp); + EXPECT_EQ(kOtherSlot, fp); + } +} + +TEST_F(UnwindNMethodHelperContractTest, NativeWrapperEpilogueRejectsAdjacentLookalikes) { + const uint32_t tail[] = {0x910003bf, 0xa8c17bfd, 0xf9400788, 0xb5000048, 0xd65f03c0}; + for (int changed = 0; changed < 5; changed++) { + SCOPED_TRACE(changed); + memcpy(code() + 4, tail, sizeof(tail)); + code()[4 + changed] ^= 1; // Wrong register/instruction, not the generated tail. + uintptr_t pc = (uintptr_t)(code() + 6); + uintptr_t sp = (uintptr_t)stackSlot(); + uintptr_t fp = sp; + HotspotStackFrame frame(&_uc); + EXPECT_FALSE(frame.unwindEpilogue(nmethod(), pc, sp, fp)); + } +} +TEST_F(UnwindNMethodHelperContractTest, NativeWrapperEpilogueAcceptsAuthenticationAndBoundsItsCodeReads) { + for (uint32_t auth : {0xd503239fU, 0xd50323bfU}) { + const uint32_t tail[] = {0x910003bf, 0xa8c17bfd, auth, 0xf9400788, 0xb5000048, 0xd65f03c0}; + memcpy(code() + 4, tail, sizeof(tail)); + for (int phase = 1; phase < 5; phase++) { + SCOPED_TRACE(phase); + uintptr_t* slot = stackSlot(); + slot[0] = kOtherSlot; + slot[1] = kSenderRa | (1ULL << 60); // Saved LR can still carry PAC bits. + setLinkRegister(kLinkReg | (1ULL << 60)); + uintptr_t pc = (uintptr_t)(code() + 4 + phase); + uintptr_t sp = (uintptr_t)slot; + uintptr_t fp = phase == 1 ? sp : kOtherSlot; + HotspotStackFrame frame(&_uc); + ASSERT_TRUE(frame.unwindEpilogue(nmethod(), pc, sp, fp)); + EXPECT_EQ(phase == 1 ? kSenderRa : kLinkReg, pc); + EXPECT_EQ((uintptr_t)slot + (phase == 1 ? 16 : 0), sp); + EXPECT_EQ(kOtherSlot, fp); + } + } + uintptr_t pc = (uintptr_t)(code() + 7); + uintptr_t sp = (uintptr_t)stackSlot(); + uintptr_t fp = sp; + HotspotStackFrame frame(&_uc); + // Ret remains readable in the allocation but lies outside this code blob. + for (int bytes_present = 0; bytes_present < 4; bytes_present++) { + SCOPED_TRACE(bytes_present); + pc = (uintptr_t)(code() + 7); + *(int*)(bytes() + 24) = (char*)(code() + 9) - bytes() + bytes_present; + EXPECT_FALSE(frame.unwindEpilogue(nmethod(), pc, sp, fp)); + } + for (int bytes_present = 0; bytes_present < 4; bytes_present++) { + pc = (uintptr_t)(code() + 5); // Pending ldp, before the optional AUTI word. + *(int*)(bytes() + 24) = (char*)(code() + 6) - bytes() + bytes_present; + EXPECT_FALSE(frame.unwindEpilogue(nmethod(), pc, sp, fp)); + } + pc = (uintptr_t)(code() + 7); + *(int*)(bytes() + 24) = sizeof(_nmethod); + *(const void**)(bytes() + kEntryAddressOffset) = code() + 5; + EXPECT_FALSE(frame.unwindEpilogue(nmethod(), pc, sp, fp)); +} + +#endif + } // namespace diff --git a/doc/README.md b/doc/README.md index bc73d4cb90..274623aee3 100644 --- a/doc/README.md +++ b/doc/README.md @@ -25,7 +25,7 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). - [ProfilingEngines](architecture/ProfilingEngines.md) - CPU, wall-clock, allocation, native-memory and liveness engines - [RefCountGuard](architecture/RefCountGuard.md) - Lock-free RAII reference-counting primitive used to drain readers before resource reclamation - [ReferenceChainsSignalsExplained](architecture/ReferenceChainsSignalsExplained.md) - Reference-chain scheduling and cost control -- [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM abstraction, continuation completeness and carrier recovery, native DWARF and arm64 stack/signal-restorer unwinding +- [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM abstraction, continuation completeness and carrier recovery, native DWARF and arm64 JNI wrapper phases and stack/signal-restorer unwinding - [SanityChecks](architecture/SanityChecks.md) - Startup sanity checks - [SigsegvPatching](architecture/SigsegvPatching.md) - SIGSEGV-safe library patching - [StringDictionary](architecture/StringDictionary.md) - Concurrency model: RefCountGuard, clearAll, and rotation protocols diff --git a/doc/architecture/JvmSupportLayer.md b/doc/architecture/JvmSupportLayer.md index 8fc6e2c582..ac6d6eb342 100644 --- a/doc/architecture/JvmSupportLayer.md +++ b/doc/architecture/JvmSupportLayer.md @@ -184,6 +184,16 @@ C1 object-arraycopy has a second live save area around its subtype-check call. Matching pair pops on both paths of the following result branch establish the additional 16 bytes; the branch-target probe is bounded to 1,024 bytes. +Interrupted arm64 JNI wrappers use their generated register-save phases rather +than the complete frame size. Before the local-storage subtraction, only the +saved FP/LR pair is popped. The bounded normal tail decoder requires +`mov sp, fp; ldp fp, lr, [sp], #16`, optional LR authentication, the thread's +pending-exception load, its conditional branch and `ret`. At the pending `ldp` +it restores the two live stack words; later phases use the interrupted LR and +never read the saved pair below the restored SP. The epilogue repair applies to +exact interrupted PCs, while recovered return addresses keep the normal +compiled-frame path. + Linux arm64's vDSO `__kernel_rt_sigreturn` omits DWARF metadata. The shared native step recognizes its exact syscall instruction pair only in `[vdso]`, bounds-checks the `siginfo_t`/`ucontext_t` signal-frame layout, and restores the