mmap/munmap EL1 fastpath - #307
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
1c0c386 to
62f5a59
Compare
e5622b0 to
ac06472
Compare
Make anonymous mappings lazy: sys_mmap records the region while first touch creates page tables, commits host memory, and zeros reused backing. Serve common private anonymous read-write requests from per-vCPU EL1 arenas, with publication rings that let the host reconcile mappings under mmap_lock. Keep shim.S focused on exception entry, register-frame preservation, and HVC dispatch by compiling the EL1 arena consumer as freestanding C. The mmap fast-path allocation policy then has a C home that munmap can share. Materialize untouched guest memory before host access and preserve PROT_NONE reservations. Let partial guest writes materialize lazy destinations, use tracked mremap protections for dirty state, and keep neighboring PTEs intact when mremap grows across block boundaries. Cover lazy reuse, refill, fork, and first-touch behavior. Close sysprog21#165
Extend the freestanding C EL1 fast path to retire compatible anonymous mappings, invalidate their translations before return, and defer host metadata cleanup until mmap_lock is next acquired. Return drained arena generations to per-vCPU allocators and refill arenas from recent registration history so mmap and munmap remain effective under reuse and mixed mapping sizes. Preserve the producer window around vCPU kicks and fork, and expose counters for both munmap fallback reasons. Keep dirty backing lazy on every anonymous munmap path; Keep unrelated VM exits out of mmap_lock when no EL1 slot has pending work. Lock-taking paths still drain unconditionally, and revocation skips controls when their shim mapping is unavailable. Prove the guest-influenced arena sizing arithmetic, isolate benchmark samples by process, and cover retirement, reuse, refill, fallback, and cross-vCPU publication behavior. Store the per-thread blocked mask with one atomic release in deliver_signal_locked, signal_deliver_fault, and signal_set_state. signal_pending() and thread_signal_deliverable() read the field lock-free from other vCPU threads, so the plain read-modify-write left those reads racing against a torn store under ThreadSanitizer.
ac06472 to
850de13
Compare
|
Can you explain benchmark results? In particular, dirty-every-page items. |
The 3–10x gap between PR95 and PR50 likely because improper measurement setup, where continuous munmap operations for each size were executed within the same process. After isolating each size and attempt into a newly created process, the results show a much smaller difference between PR95 and PR50. |
| if (start < | ||
| atomic_load_explicit(&candidate->arena_base, memory_order_relaxed)) | ||
| return false; | ||
| if (end > atomic_load_explicit(&candidate->cursor, memory_order_acquire)) |
There was a problem hiding this comment.
The ownership test here is [arena_base, cursor), but the host allocator can place a non-arena mapping inside that span. munmap_retire_commit_locked lowers the gap hint to the retired start (mem.c:297) and mmap_fastpath_skip_reserved reserves only [cursor, limit) (mem.c:1079), so a hole freed below the cursor is handed out to any mapping type.
Reproduced on this branch: three 64 KiB anon mappings, free the middle one, mmap(fd) into the hole, munmap it.
p1=0x200000000 p2=0x200010000 p3=0x200020000
freed p2
file q=0x200010000 reused-hole=1
FATAL src/syscall/mem.c:265: munmap retire: non-fast mapping in arena [0x200010000-0x200020000)
EL1 cleared the file mapping's PTEs and broadcast TLBI with the host uninvolved. test-mmap-fastpath (11/11) and test-mmap-lazy (16/16) pass, so nothing in the suite covers a non-arena mapping inside a live arena.
Simplest fix: reserve all of [arena_base, arena_limit) and drop the gap-hint lowering, so this bounds check stays a proof of ownership rather than a heuristic.
| atomic_load_explicit(&c->cursor, memory_order_acquire); | ||
| uint64_t limit = | ||
| atomic_load_explicit(&c->arena_limit, memory_order_relaxed); | ||
| if (cursor < limit && *start < limit && end > cursor) { |
There was a problem hiding this comment.
This reserves only the unallocated tail. Holes below cursor stay visible to find_free_gap_inner, which is what lets a file-backed mapping land inside a live arena and be claimed later by el1_munmap_match_arena. Widening the condition to [arena_base, arena_limit) costs VA the arena already owns and restores the invariant EL1 depends on.
| log_fatal( | ||
| "mmap fast path: region metadata exhausted while " | ||
| "draining vCPU slot %d", | ||
| slot); |
There was a problem hiding this comment.
log_fatal does not terminate. log_impl returns after printing, and the convention in this tree is log_fatal(...); abort(); (guest.c:311). All eight new log_fatal sites in this file lack the abort(), so every fail-closed guard added here is inert and execution continues into exactly the state the guard was written to prevent.
This site is the clearest case, since the comment above it names the outcome it is meant to rule out:
ERROR guest.c:2430: region table full (4096/4096), cannot track [0x202328000-0x202329000)
FATAL mem.c:230: mmap fast path: region metadata exhausted while draining vCPU slot 0
elfuse exit=139
The guest took SIGSEGV on first touch after fragmenting past GUEST_MAX_REGIONS. Beyond adding the abort(), this one is worth making unreachable: the slow path returns ENOMEM for the same condition, so gate the arena on free region slots instead of discovering exhaustion after EL1 has already handed the address to the guest.
There was a problem hiding this comment.
Sharpening the reservation suggestion: counting currently free g->regions[] slots is not enough on its own. The reservation has to cover every entry already sitting unpublished in a ring (up to SHIM_MMAP_RING_SIZE per vCPU), since those addresses are in guest hands but not yet in the tracker. Reserve against that bound before enabling an arena, have EL1 bail to HVC once its reservation is spent, and give slots back as publications drain or retire.
| entry->len = length; | ||
| entry->prot = prot; | ||
| /* publish the bump cursor before the entry */ | ||
| atomic_store_explicit(&control->cursor, cursor, memory_order_relaxed); |
There was a problem hiding this comment.
cursor is stored relaxed before the release store to tail, so the acquire load of cursor in el1_munmap_match_arena (line 392) gets no synchronizes-with edge to that tail store.
A sibling vCPU can therefore observe the bumped cursor, accept the range, and publish a retirement, while the host acquire-loading this ring's tail still sees the old value. The drain then removes nothing, and the publication reinstates the region on a later pass: the inversion the PR description says the tail snapshot rules out. It can also surface as the stale arena generation/range guard further down.
Release-store tail first, then release-store cursor. The acquire load at line 392 then orders the publication ahead of the retirement, and the host inherits that ordering through the retire ring.
2 participants
The primary goal is to reduce VM exits for
mmapandmunmap.Architecture
Anonymous mappings are lazy:
sys_mmaponly records the region; page-table creation, host memory commit, and zeroing of reused backing all happen at first touch.Each vCPU owns a private contiguous virtual address arena. When the guest traps into EL1 for
mmap/munmap, EL1 serves the request directly out of its own arena and never leaves EL1. The host's job is to keep those arenas supplied with VA and to reconcile their effects into host-side region/PTE state, both done lazily undermmap_lockrather than synchronously per call.mmap fast path
EL1 serves exactly one shape:
file-backed mapping is excluded. The reason is that EL1 is a freestanding, no-syscall context that cannot read a file or install a page-cache overlay itself — that work is inherently the host's.
The host prepares a contiguous VA region per vCPU. Each vCPU is the sole producer of its own 32-entry publication ring and bump cursor, and publishes the extent into its ring. The host installs the corresponding PTEs.
Arena refill is dynamic, not fixed-size. Any host path that already holds
mmap_lock— a syscall, a page fault, or a natural VM exit — opportunistically tops up the current thread's arena once its remaining drops under that recent high-water mark. Arena exhaustion, an unsupported shape, a full ring, or a stale arena generation each bail to the ordinary hostmmapslow path.munmap fast path
EL1 validates that the target range belongs to a live arena, clears the covering page-table entries directly, and issues a broadcast TLBI, all without leaving EL1
The address is unusable to the guest the instant EL1 returns. Only after clearing the PTEs and completing the TLBI does EL1 publish a retirement record — address, length, and the arena generation it came from — to a second, separate 32-entry SPSC ring. The host has not been involved yet at that point; its bookkeeping is reconciled lazily by
mmap_fastpath_drain_locked(), which walks every vCPU's retire ring and commits the deferred cleanup. That drain runs on every natural VM exit as well as on every syscall or fault that needs the lock.Cross-thread ordering matters here, and A and B are often touching the same address, not two unrelated ones: EL1's fast
munmapscans every vCPU's arena, not just its own, because handing a fresh allocation to another thread to free is the common case. If the host were to drain B's teardown of address X before it has drained A's publication of that same X, the removal would be a no-op (X isn't in the region table yet) and the publication drain right after would insert X as live — reviving, in host bookkeeping, an address EL1 already invalidated. The drain avoids this by snapshotting every vCPU's retire-ring tail first, draining every mmap publication next, and only then consuming retirements up to that snapshot, so within one drain pass a publication a retirement depends on is always applied before the retirement removes it:EL1 self-throttles rather than pinging the host mid-flight. If a producer's retire ring is within one slot of full,
munmapbails out of the fast path entirely and falls through to the ordinary syscall trap, which drains the ring as a side effect of takingmmap_lock. Separately, once a vCPU's own unconsumed retired bytes cross a 256 MiB soft threshold, it sets an advisorycleanup_requestedflag; this never forces an HVC, it only marks that real cleanup work is waiting for the next natural drain.behavior while fork
Before the fork snapshot is taken, the parent-side handler acquires
mmap_lockthrough the fork variant of the acquire path, drains every vCPU's publication and retire ring, and waits for any in-flight lazy materialization to finish. It then revokes every per-vCPU arena descriptor. Because sibling vCPUs are already quiesced for the fork snapshot, this revocation cannot race a live EL1 producer.The child does not inherit the parent's arena or ring layout. The child's main thread begins with a fresh, empty arena rather than a stale one aliased to the parent's.
Any
clone()call that publishes a stack for the new thread goes through this too. Any anonymous fast-path allocation that has become a live thread stack must remain reachable only through the ordinary stack-lifetime bookkeeping, not through EL1's arena-generation check, so revoking first guarantees no later EL1 fastmunmapcan tear down a live stack range by still recognizing it as belonging to a live arena. Only aclone()with a nullchild_stackskips this.behavior while exec
sys_execve()takesmmap_lockimmediately before the point of no return. That both closes the EL1 producer gate and drains every vCPU's mmap and munmap rings, sog->regions[]and PTE state agree with each other before the address space they describe is destroyed.guest_reset()then zeroes the entire shim-data page, arena, both rings.execdoes not explicitly do arena's control block comes back. The replacement image's first eligible slow-pathmmapby calling the refill path and turns the fast path back.Frama-C proof coverage
The guest-influenced arena sizing and capacity arithmetic is pulled out of
src/syscall/mem.cintosrc/proved/mmap-fastpath.hspecifically so it can be proved.Four functions are proved, every input treated as fully guest-influenced (
request_lenis the guest's own mmap length; the history window is built from a sequence of guest-chosen lengths), because a slip here either wedges the allocator by undersizing an arena forever, or letsmmap_fastpath_request_fitsaccept a request that runs pastarena_limit:mmap_fastpath_request_fits: whether alen-byte request still fits beforelimit, covering the zero-length, sub-block, and block-aligned cases. Proved to never answer "fits" when the aligned start would actually run pastlimit.mmap_fastpath_pow2_clamped: rounds a target size up to the nearest power of two inside[MIN, MAX].mmap_fastpath_window_max: the largest of the last 16 registered mapping sizes. Proved as an upper bound over the whole history window.mmap_fastpath_arena_size: the target arena size given recent history and the request about to be served. Proved to always land in[MIN, MAX], with-wp-rteseparately closing both multiplication-overflow guards and the division-by-zero case.92 of 93 discharged goals (the 93rd,
align_up_ok_ensures_rejects_only_on_wrap, is a pre-existingalign.hgoal that also fails standalone undermake verify-alignin this environment, not something this change introduced).Benchmark results
A. One timed mmap/munmap fast-path pair per fresh process
All values are nanoseconds. Speedup is
OrbStack / elfuse; values above 1mean elfuse is faster.
B. munmap after materializing one 4 KiB page
Fifteen fresh processes are used per size. Values are outer medians of the
per-process medians.
C. munmap after dirtying every 4 KiB page
One fresh process is used per size. Each process performs one warmup, then the
listed number of timed operations. The table reports the in-process
distribution.
munmaplatency has a step right at the 2 MiB boundary. Under 2 MiB, which walks and clears one 4 KiB L3 leaf at a time.Arena size is clamped to
MMAP_FAST_ARENA_MIN = 64 MiBandMMAP_FAST_ARENA_MAX = 32 GiBper vCPU.tests/bench-mmap-isolated allto reproduce mmap/munmap-vs-size numbersCloses #165