diff --git a/_gsocblogs/2026/blog_Final_Fine_grained_storage_for_the_DUNE_experiment_AhmedIdani.md b/_gsocblogs/2026/blog_Final_Fine_grained_storage_for_the_DUNE_experiment_AhmedIdani.md new file mode 100644 index 000000000..31d0bade1 --- /dev/null +++ b/_gsocblogs/2026/blog_Final_Fine_grained_storage_for_the_DUNE_experiment_AhmedIdani.md @@ -0,0 +1,158 @@ +--- +project: DUNE +title: "Fine grained storage for the DUNE experiment" +author: Ahmed Idani +photo: blog_authors/AhmedIdani.jpeg +date: 24.08.2026 +year: 2026 +layout: blog_post +logo: DUNE-logo.png +intro: | + DUNE is building a new data processing framework, Phlex, whose I/O layer FORM lets each pipeline stage write its own data product into its own container without waiting for anyone else. That makes writing cheap, but it moves the cost onto the reader: someone who needs several products of the same event can follow only one of them in its natural order and has to read the rest out of order. I spent the summer building a benchmark on ROOT's RNTuple to find out what that actually costs. Short version: you pay in decompression, not in disk, and the size of the bill is set by how many events share a page. At one event per page the read order is free. At four thousand, the same bytes cost 48 times as much. +--- + +| | | +| ------------ | ------------------------------------------------------------------------------------------------------------------------ | +| Name | [Ahmed Idani](https://github.com/Ahmed-Idani) | +| Organization | [CERN-HSF](https://hepsoftwarefoundation.org/activities/gsoc.html), [Argonne National Laboratory](https://www.anl.gov/) | +| Mentors | [Wanwei Wu](https://www.anl.gov/profile/wanwei-wu), [Peter Van Gemmeren](https://www.anl.gov/profile/peter-van-gemmeren) | +| Project | [Fine grained storage for the DUNE experiment](https://hepsoftwarefoundation.org/gsoc/2026/proposal_DUNE_FORM.html) | +{: .table} + +## Important Links + +- **Project proposal:** [Fine grained storage for the DUNE experiment](https://hepsoftwarefoundation.org/gsoc/2026/proposal_DUNE_FORM.html) +- **Project repository:** [wwuoneway/fine-grained-storage](https://github.com/wwuoneway/fine-grained-storage/tree/gsoc-2026-final) +- **Final work report** (the long version, with every figure, table and configuration file): [GSoC 2026 Final Work Report](https://gist.github.com/Ahmed-Idani/2a9d3bdc4b03b0e3d0cf4da4e82c258f) + +## The problem + +[DUNE](https://www.dunescience.org/) is a neutrino experiment. Its scale creates a computing problem: PB-scale datasets, GB-scale event objects, and a lot of different workflows that all want different slices of the same detector data. At that scale the traditional way of processing physics data, one whole event at a time, starts to get in the way. + +So the experiment is building [Phlex](https://github.com/Framework-R-D/phlex), a framework that works with fine grained data instead. Its I/O layer, [FORM](https://indico.fnal.gov/event/72820/contributions/341330/attachments/199633/278226/FORM_DUNE_DPF2026.pdf), lets each stage of the pipeline write its own data product into its own container, eagerly, without synchronising with the other writers. + +Writing that way is easy. Reading is where it gets interesting. + +Say two stages are writing `Hit` and `Track` into two containers at the same time, and nobody is keeping them in step. Then the row an event occupies in one container has nothing to do with the row it occupies in the other. Event 5 can be row 5 in `Hit` and row 900 in `Track`. Now a reader comes along and wants both products of the same event. It can follow one container in its cheap, natural order, front to back. The other one it has to jump around in. + +That is the whole question of this project, and it is narrow enough to measure: + +> On [RNTuple](https://root.cern/doc/master/group__NTuple.html), the columnar format DUNE is moving to, how much does it cost to read a data product out of its storage order? Where does that cost go, and what decides how big it is? + +One detail of RNTuple's design ends up deciding the entire answer, so it is worth putting on the table now. RNTuple does not store rows. It stores each field as its own column, cuts each column into **pages**, and compresses every page as a single unit. That makes a page the smallest thing you can read: ask for one row and you decompress the whole page it happens to sit in. Keep an eye on how many events fit into one page, because everything below comes back to that number. + +## What I built + +The temptation at the start was to reproduce FORM. I am glad my mentors talked me out of it. What I built instead is a standalone benchmark suite whose only dependency is ROOT: it links neither Phlex nor FORM, and it does not reimplement them. It just rebuilds the smallest system that still has the same problem, which is each data product living in its own container, in its own order on disk, reachable at read time only through an index. + +FORM's vocabulary maps onto it directly: + +| FORM concept | This benchmark | +| ------------ | -------------------------------------------------------------- | +| Container | one RNTuple per data product, all sharing one `.root` file | +| Technology | RNTuple, the format under study | +| Index | a TTree on disk, mapping `event_id` to each product's Token | +| Token | `{container, entry}`, naming the row a product's data lives at | +| Data product | `position` and `momentum`, one of each per particle | +{: .table .table-bordered} + +It runs in three phases, all driven from one configuration file. + +**1. Generate.** An event here is one simulated collision holding some number of particles, and a particle is described by two things, where it is and how it is moving: + +```cpp +struct Position { float x, y, z; }; // where the particle is +struct Momentum { float px, py, pz; }; // how it is moving +``` + +Those two structs are the benchmark's data products. Every event carries a vector of each, both the same length, so particle `i` of an event is the pair `(positions[i], momenta[i])`. The particle count may vary from event to event, though for every dataset in this post I hold it fixed, so that bytes per event is a knob I set rather than a distribution I have to argue about. Everything is drawn from a single seeded `mt19937_64` and written to plain binary files, so any dataset can be regenerated exactly. Positions are uniform over a detector-shaped box, momentum components are Gaussian, and any triplet below a minimum magnitude is redrawn. + +> These distributions are not a physics simulation and are not meant to reproduce DUNE data. What this study needs from the data is its _shape_: bytes per event, and how many events share a page. That is set by the particle count and the struct layout, not by the values themselves. + +**2. Write.** Each product goes into its own RNTuple, and one row is one whole event. But RNTuple does not keep a row together on disk: every field becomes its own column. So a container declares two fields, the `event_id` and the particle vector, and RNTuple ends up storing five columns for it: `event_id`, an offsets column recording where each event's particles begin, and one `float` column each for `px`, `py` and `pz`. Those last three carry essentially all of the bytes, which is why the rest of this post calls them the **payload columns**. Alongside the containers goes the index TTree that says which row holds which event. + +**3. Read.** Read it all back with the stopwatch running, splitting the wall time into storage I/O and decompression using ROOT's own per-container counters, and recording bytes fetched, bytes decompressed, pages fetched and pages unzipped. + +### Turning disorder into a knob + +Reading a file sequentially, then randomly, and comparing gives you two points and no explanation. So I made disorder adjustable instead: start from the sequential order, walk it once, and swap each position with another no further than `distance` away. It only ever swaps, so the result is still a permutation: every event is read exactly once, and only the order changes. Two endpoints become a curve. + +I also simulate the container mismatch on the read side rather than the write side. Files are written in plain event order and the reader jumps around instead. It is the same experiment seen from the other end, the same pages in the same sequence for the same cost, but far cheaper to run: the visit order is a seeded list I can change between cases, while concurrent writers would leave a different layout behind on every run. + +### Getting the machine out of the way + +My first numbers measured the Linux scheduler more than they measured RNTuple. This is a hybrid CPU, so an unpinned case would land on a performance core in one repetition and an efficiency core in the next, an effect bigger than most of the knobs I was trying to study. + +So everything below runs pinned to isolated cores, one CPU per physical core so the reader and ROOT's I/O thread never end up sharing execution units, with the frequency governor locked to `performance` and ASLR off. The page cache is also dropped before every repetition, with `posix_fadvise(POSIX_FADV_DONTNEED)`, which needs no root and touches only my own files. + +That last one has a trap in it that cost me a while. `DONTNEED` only drops **clean** pages, and a file I had just written in the write phase was still full of dirty ones, so it stayed comfortably cached and my "cold" runs were not cold at all. The fix is a `sync` first, to push those pages out so the kernel is allowed to drop them. + +## What I found + +I ran two studies, 110 measured cases between them. Every dataset holds the same 104,857,600 particles, about 1,200 MiB of raw momentum, just cut into different numbers of events. So the payload never changes. The only thing that changes is how many events end up sharing a page. Write settings are [ROOT's defaults](https://root.cern.ch/doc/master/classROOT_1_1RNTupleWriteOptions.html) throughout: zstd level 5, 1 MiB maximum page. + +### Study one: the disk keeps up, the decompressor does not + +Four datasets, from one event per page up to about four thousand. Ten access patterns each: sequential, then scatter at distance 1, 2, 4, all the way to 256. Cold cache, and the cluster cache deliberately **off**. That last one is ROOT's read-ahead: RNTuple groups pages into clusters, and with the cluster cache on it fetches and unpacks a whole cluster on a background thread before you ask for it. Switching it off means no read-ahead, so every page is fetched only when the reader actually asks for it, and the time it takes stays on the critical path where I can measure it. ROOT's implicit multithreading is off as well, in both studies, so every number below is single-threaded decompression. + +
+
+
+Figure 1. Where the read wall time goes, against scatter distance, all four panels on one scale. Decompression reproduces the shape of the total. I/O and everything else stay flat against the axis.
+
+
+
+Figure 2. Bytes out of the decompressor for every byte the read actually asked for. Every dataset starts on the same line, then bends off it and flattens at a ceiling set by its own events per page.
+