Skip to content

test(nativemem): cover allocator probe outcomes and overhead accounting - #911

Open
kaahos wants to merge 8 commits into
mainfrom
paul.fournillon/fix-malloc-footprint-test
Open

kaahos wants to merge 8 commits into
mainfrom
paul.fournillon/fix-malloc-footprint-test

Conversation

@kaahos

@kaahos kaahos commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?:

Correct the small-allocation overhead expectation and add separate regression coverage for successful and inconclusive header discovery. Check footprint and accounting totals against allocator-reported usable sizes after allocations and partial frees.

Motivation:

The ARM64 release job in #877 reports a zero header estimate and 104 usable bytes for a 96-byte request. That is 8.33% observed overhead, which fails the existing assertion that requires more than 10%. Zero is an explicit fallback when the header probe is inconclusive.

Accepting that fallback also needs a separate test that requires a nonzero estimate when contiguous allocations expose a header. Otherwise an estimator that always returns zero could pass unnoticed.

Additional Notes:

  • Production code is unchanged.
  • Retain the 2,000-node map scenario. Observe actual rebound node allocations and check exact logical-byte and overhead totals after inserts, partial erases, and container destruction. Expected overhead uses allocator-reported usable sizes plus the cached header estimate.
  • A standalone executable prepares contiguous and fragmented layouts using real glibc allocations before the first cached header probe. Both layouts execute the same percentage assertions as the main test. Its layout preconditions fail explicitly if they are not met.
  • The isolated probe checks run on glibc. The main accounting checks retain the active platform or sanitizer allocator.
  • Independent agentic review: passed, no remaining findings.

How to test the change?:

Passed on Linux ARM64:

./.claude/commands/build-and-summarize spotlessApply :ddprof-lib:gtestDebug_mallocFootprint_ut :ddprof-lib:gtestRelease_mallocFootprint_ut :ddprof-lib:gtestDebug_countingAllocator_ut
./.claude/commands/build-and-summarize :ddprof-lib:gtestAsan_mallocFootprint_ut -Pnative.forceCompiler=g++
  • Debug and release: all 7 footprint tests pass under both normal startup and reproduced zero-header startup.
  • Both isolated allocator layouts also pass on Linux x64.
  • A map-node-only overhead mutation passes the direct-allocation test and fails the map scenario, confirming distinct coverage.
  • Temporary mutation checks reject the original small_pct > 10 assertion, an always-zero header estimate, omitted usable-size rounding, and omitted CountingAllocator overhead recording.

For Datadog employees:

Unsure? Have a question? Request a review!

@dd-octo-sts

dd-octo-sts Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #41047605373 | Commit: 4611617 | Duration: 20m 9s (longest job)

✅ All 108 test jobs passed

Status Overview

JDK glibc-aarch64/asan glibc-aarch64/debug glibc-aarch64/release glibc-aarch64/tsan glibc-amd64/asan glibc-amd64/debug glibc-amd64/release glibc-amd64/tsan musl-aarch64/debug musl-aarch64/release musl-amd64/debug musl-amd64/release
8 - - - - ✅ ✅ ✅ ✅ - - - -
8-ibm - - - - ✅ ✅ ✅ ✅ - - - -
8-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
8-librca - - - - - - - - ✅ ✅ ✅ ✅
8-orcl - - - - ✅ ✅ ✅ ✅ - - - -
11 - - - - ✅ ✅ ✅ ✅ - - - -
11-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
11-librca - - - - - - - - ✅ ✅ ✅ ✅
17 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-librca - - - - - - - - ✅ ✅ ✅ ✅
21 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
21-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
21-librca - - - - - - - - ✅ ✅ ✅ ✅
25 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
25-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
25-librca - - - - - - - - ✅ ✅ ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 108 | Passed: 108 | Failed: 0


Updated: 2026-10-30 11:44:59 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 00578a23

@kaahos kaahos changed the title test(nativemem): allow zero-header footprint estimates test(nativemem): cover allocator probe outcomes and overhead accounting Oct 27, 2026
@kaahos
kaahos marked this pull request as ready for review October 27, 2026 14:25
@kaahos
kaahos requested a review from a team as a code owner October 27, 2026 14:25
@kaahos

kaahos commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T14:29:41.312215Z c86fe0d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c86fe0d69c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Sphinx Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c86fe0d

@jbachorik

jbachorik commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

🔵 Sphinx review — INFO

Finding — Mutation test-adequacy analysis skipped 334 test-path candidates in: ddprof-lib/src/test/cpp/mallocFootprintChecks.h, ddprof-lib/src/test/cpp/mallocFootprint_ut.cpp, ddprof-lib/src/test/resources/native-libs/malloc-footprint-probe/Makefile, ddprof-lib/src/test/resources/native-libs/malloc-footprint-probe/main.cpp. These test files were not assessed; verify they actually pin the change under review.

Suggestion — Confirm the added or changed tests cover the modified behavior, or note the gap explicitly.

kaahos and others added 4 commits October 28, 2026 16:29
Track external probe headers during incremental builds. Validate executable paths and reject unknown allocator layouts.
Track the compiler as a task input so changing toolchains rebuilds helpers without forwarding sanitizer flags.

Add Gradle/Make regression coverage for compiler selection and incremental rebuilds.
Reject test configurations without usable-size introspection instead of treating requested bytes as an independent measurement. Keep the production fallback unchanged.
@kaahos kaahos added test:release Run CI tests with release configuration test:asan Run CI tests with AddressSanitizer configuration test:tsan Run CI tests with ThreadSanitizer configuration test:fuzz test:reliability labels Oct 30, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/148886730

@kaahos
kaahos requested a review from jbachorik October 30, 2026 12:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:asan Run CI tests with AddressSanitizer configuration test:fuzz test:release Run CI tests with release configuration test:reliability test:tsan Run CI tests with ThreadSanitizer configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants