[ntuple] add support for on-demand loading of page lists by jblomer · Pull Request #23524 · root-project/root

jblomer · 2026-09-28T12:24:32Z

Key feature for huge-file support: don't necessarily load all page list (and thus cluster group details) upfront but allow for them to be loaded on demand.

Follow-up changes to limit the memory footprint of cluster metadata:

Add "pinned cluster details" for pinned clusters
Optimize FindClusterId() such that in common cases we don't need to do a full binary search but
- we search in the current cluster group and in one adjacent one (next or previous) plus in the pinned ones
- For "top-level columns", we use the entry-based FindClusterId(), which points us directly to the right cluster group
Drop the least recently used cluster details when cluster details for a new cluster group are loaded. Aim at keeping at most two cluster groups (plus pinned ones) in memory.

github-actions · 2026-09-28T20:37:03Z

Test Results

24 files 24 suites 4d 2h 14m 23s ⏱️
3 885 tests 3 885 ✅ 0 💤 0 ❌
83 202 runs 83 202 ✅ 0 💤 0 ❌

Results for commit b66cf23.

♻️ This comment has been updated with latest results.

hahnjo

Some comments inline. I'm wondering about the descriptor guards (which get quite a bit more complicated): when do we need them? A page source should be accessed by at most one thread at a time, no?

jblomer · 2026-10-03T11:26:58Z

Some comments inline. I'm wondering about the descriptor guards (which get quite a bit more complicated): when do we need them? A page source should be accessed by at most one thread at a time, no?

The cluster prefetch accesses it in parallel. I think it's worthy seeing if we can collect all the locators to prefetch in the main thread, which indeed would allow for removing the locks altogether. I don't think it would fundamentally change this PR, however. I'd consider the additions to the lock guards mostly mechanical.

hahnjo

Looks good from my side

hahnjo

(didn't click the right button, see #23524 (review))

Factor out loading a page list (and thus populating cluster group details) into a separate method, to be reused later.

... by ensuring cluster details are available when cluster descriptors are requested.

To be called after attach in cases where it is clear upfront that the full information of the cluster details is needed.

With on-demand loading of page lists, the connection to storage (file handle, DAOS container, etc.) can be used concurrently, by the page list loading and by the prefetcher. Hence requires locking.

jblomer requested review from enirolf, hahnjo, pcanal and silverweed September 28, 2026 12:24

jblomer self-assigned this Sep 28, 2026

jblomer requested review from hageboeck and vepadulano as code owners September 28, 2026 12:24

jblomer added the in:RNTuple label Sep 28, 2026

jblomer marked this pull request as draft September 28, 2026 12:24

jblomer force-pushed the ntuple-huge-file-read branch 2 times, most recently from 65532f7 to 63316a1 Compare September 28, 2026 15:15

jblomer force-pushed the ntuple-huge-file-read branch 2 times, most recently from c8d354d to b39589c Compare September 29, 2026 21:24

hahnjo reviewed Oct 1, 2026

View reviewed changes

jblomer force-pushed the ntuple-huge-file-read branch 8 times, most recently from e28913e to 372e990 Compare October 4, 2026 11:38

jblomer marked this pull request as ready for review October 4, 2026 11:42

jblomer requested a review from dpiparo as a code owner October 4, 2026 11:42

jblomer requested a review from hahnjo October 4, 2026 11:42

jblomer force-pushed the ntuple-huge-file-read branch 3 times, most recently from f602672 to 55d62e1 Compare October 7, 2026 12:02

jblomer force-pushed the ntuple-huge-file-read branch 2 times, most recently from 91c5bf3 to ec1718a Compare October 17, 2026 20:31

jblomer requested a review from pcanal October 17, 2026 20:31

hahnjo reviewed Oct 19, 2026

View reviewed changes

Comment thread tree/ntuple/inc/ROOT/RPageStorage.hxx Outdated

Comment thread tree/ntuple/inc/ROOT/RPageStorage.hxx Outdated

Comment thread tree/ntuple/src/RPageStorage.cxx Outdated

hahnjo approved these changes Oct 19, 2026

View reviewed changes

jblomer added 24 commits October 19, 2026 21:41

[ntuple] fix mock page source in cluster unit test

bcc41ab

[ntuple] fix descriptor locking in join table's entry mapping

c8984e1

[ntuple] fix descriptor locking in merger unit test

9740ae2

[ntuple] minor fix in RClusterPool

58cc690

[ntuple] add private RPageSource::LoadPageList()

e7897f5

Factor out loading a page list (and thus populating cluster group details) into a separate method, to be reused later.

[ntuple] allow for manual release of RSharedDescriptorGuard

bb6efb0

[ntuple] make R[Shared|Excl]DescriptorGuard default constructible

14dbd6c

[ntuple] fix moving RSharedDescriptorGuard

7b1a258

[ntuple] make RExclDescriptorGuard move constructible

52d67c4

[ntuple] allow for manual release of RExclDescriptorGuard

42c8cab

[ntuple] add RPageSource::RAnyDescriptorGuard

1a42491

[ntuple] add private RPageSource::EnsureClusterDetails()

fae152c

[ntuple] add RPageStorage::FindClusterGroupId()

be7fa92

[ntuple] prepare for loading cluster details on demand

e07be3a

... by ensuring cluster details are available when cluster descriptors are requested.

[ntuple] properly lock RPageSource::FindNextClusterId()

8343ad2

[ntuple] properly lock entry-based RPageSource::FindClusterId()

958ee36

[ntuple] properly lock element-based RPageSource::FindClusterId()

903a1f2

[ntuple] use FindClusterId() to get first cluster in merger

44409a0

[ntuple] add RPageSource::LoadAllPageLists()

91ae775

To be called after attach in cases where it is clear upfront that the full information of the cluster details is needed.

[ntuple] lock access to storage backend

a7ec7c7

With on-demand loading of page lists, the connection to storage (file handle, DAOS container, etc.) can be used concurrently, by the page list loading and by the prefetcher. Hence requires locking.

[ntuple] add metadata mode read option

1219fc4

[ntuple] remove dead code in FindClusterId()

86decd7

[ntuple] descriptor guard GetRef() --> GetConstRef()

8c58bd9

[ntuple] fix I/O counters of file backend

b66cf23

jblomer force-pushed the ntuple-huge-file-read branch from ec1718a to b66cf23 Compare October 19, 2026 20:15

Navigation Menu

Repository navigation

[ntuple] add support for on-demand loading of page lists - #23524

[ntuple] add support for on-demand loading of page lists#23524
jblomer wants to merge 24 commits into
root-project:masterfrom
jblomer:ntuple-huge-file-read

jblomer commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

hahnjo left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jblomer commented Oct 3, 2026 •

edited

Loading

Uh oh!

hahnjo left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hahnjo left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jblomer commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

hahnjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jblomer commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hahnjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hahnjo left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jblomer commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

jblomer commented Oct 3, 2026 •

edited

Loading