+ {%- include excalidraw.html file="00-course-map" alt="Course map: four stages, Foundations, Core systems, Operating context, and Mastery, shown as bands containing the fourteen parts with their chapter ranges, and the six artifacts fwatch, shmkv, chatterd, pmon, bugfarm, and sysagent along the bottom, each with an arrow to the part where it grows." caption="Figure 0.1: the course at a glance." -%}
+ {%- include excalidraw.html file="01-toolchain-landscape" alt="Three toolchain stacks: GCC/CMake/gdb for C++, go1.26.5/go build/dlv for Go, and rustc/cargo/clippy for Rust, each converging through its own demo.sh onto one shared demo.sh runner contract." caption="Figure 1.1: three toolchains, one demo.sh contract shared by every example in the book" -%}
+ {%- include excalidraw.html file="02-lab-topology" alt="Lab topology: a host band with libvirtd/KVM, the ~/.cache/systems-lab image cache holding the pristine Fedora Cloud Base image plus one qcow2 overlay per guest, and the scripts/lab directory; below it, the libvirt NAT network 192.168.124.0/24 containing systems-target (4 GB, .7) and systems-peer (2 GB, .95), both on kernel 6.19.10. An amber deploy arrow runs from the scripts to systems-target; a dashed line links the two guests for later two-host chapters; a note marks the lab-ready snapshot taken on both." caption="Figure 2.1: the systems-* lab, cached base image + per-VM overlays on the host, two guests on libvirt NAT, one amber deploy path" -%}
+ {%- include excalidraw.html file="03-telemetry-pipeline" alt="Telemetry pipeline: a host demo and a VM demo send OTLP HTTP to port 4318 on the lsp-lgtm container, where the OpenTelemetry Collector fans out to Loki for logs, Tempo for traces, and Mimir for metrics; Grafana on port 3000 queries all three and serves the developer's browser. The VM demo reaches the host at the libvirt gateway address 192.168.124.1." caption="Figure 3.1: one container, whole pipeline: OTLP in on :4318, Grafana out on :3000" -%}
+ {%- include excalidraw.html file="04-user-kernel-path" alt="A user-space band with your code, a language wrapper, and the syscall instruction, plus a dashed vdso page; an amber trap arrow crosses into a kernel band through entry_SYSCALL_64 and the syscall table to sys_getrandom, with a dashed return arrow carrying rax back." caption="Figure 4.1: one getrandom(2) crossing the boundary, marshal registers, trap, dispatch by number, return rax, or take the vDSO fast path and never trap at all" -%}
+ {%- include excalidraw.html file="04-runtime-syscall-layers" alt="Three side-by-side stacks: C++ code over glibc wrappers over syscall instructions in libc, Go code over the Go runtime over 511 syscall instructions in a static binary, Rust code over the rustix linux_raw backend over 32 inlined syscall instructions, all with amber arrows labeled 70, 283, and 73 calls into one dark kernel ABI box." caption="Figure 4.2: the per-language layer stacks over the one register ABI, annotated with this chapter's real strace -c totals" -%}
+ {%- include excalidraw.html file="05-error-flow-three-ways" alt="One failure's journey: write(2) on /dev/full returns -ENOSPC in the kernel; in user space that becomes errno 28, which fans out into three columns (C++23's std::expected with std::error_code mapping into a Failure struct, Go's unix.Errno wrapped into *opError with Unwrap for errors.Is/As, and Rust's Result with Errno mapped via map_err into a thiserror CopyError variant), all three converging on one outcome box: exit 3 and the line copyx: write /dev/full: No space left on device." caption="Figure 5.1: one failure, three spellings, one contract: ENOSPC from write(2) to the user, in each type system" -%}
+ {%- include excalidraw.html file="06-three-concurrency-models" alt="Three columns (C++23, Go, Rust) each showing the same producer, queue, four-worker, cancellation shape: main thread into a mutex/deque/condition_variable_any queue into four jthreads with a sigtimedwait watcher; walker goroutine into an unbuffered channel into four goroutines with signal.NotifyContext; main thread into an Arc-Mutex-wrapped mpsc receiver into four scoped threads with a signal-hook AtomicBool." caption="Figure 6.1: one program shape, three vocabularies; same kernel threads underneath." -%}
+ {%- include excalidraw.html file="06-cancellation-drain" alt="SIGINT fans out as one flipped flag to three parties: the walker stops enumerating, idle workers wake and exit, busy workers finish the file in hand: then the queue closes and results drain into a final box: partial sorted lines, parhash: interrupted, exit 130." caption="Figure 6.2: the drain: stop accepting, finish what you hold, report what completed." -%}
+ {%- include excalidraw.html file="07-fd-table-vfs" alt="Three stacked layers: the fwatch process fd table holding fds 0-2 and four dirfds 3-6 with cloexec on, each dirfd pointing down to its own kernel open file description holding flags 02700000 and a readdir position, each description pointing down to a VFS inode numbered 69926 through 69929; dashed openat arrows chain fd 3 to fd 4 to fd 5 to fd 6." caption="Figure 7.1: one paused fwatch walk: four dirfds in the per-process fd table, four open file descriptions carrying flags and offset, four inodes; every number in this figure comes from the /proc inspection later in the chapter" -%}
+ {%- include excalidraw.html file="08-write-path-page-cache" alt="Three columns over user space, kernel, and storage bands. In buffered mode, write(2) copies into the page cache and data reaches the NVMe device either by later asynchronous writeback or by an fdatasync at the end; in fsync-every mode an fdatasync every N blocks forces it down; in direct mode an O_DIRECT write bypasses the page cache and DMAs straight from the user buffer to the device." caption="Figure 8.1: one write path, three flush points: where each iobench mode hands bytes to the device" -%}
+ {%- include excalidraw.html file="09-event-loop-anatomy" alt="Two bands. In the user-space band, four loop steps run left to right (arm timerfd, epoll_wait, dispatch on the data token, drain and act) with a return arrow labeled loop. In the kernel band, an eventpoll interest-list box holds tfd 3, 4, 5, fed by three source boxes: the inotify fd with wd 1 and mask 0x3c6, the CLOCK_MONOTONIC timerfd, and the signalfd carrying blocked SIGINT and SIGTERM; an amber readiness arrow carries EPOLLIN and the token back up into epoll_wait." caption="Figure 9.1: the anatomy of fwatch watch: one loop, three event sources; every fd number, token, and mask in this figure reappears verbatim in the /proc inspection later in the chapter" -%}
+ {%- include excalidraw.html file="10-uring-rings" alt="Three columns: user-space fwatch code filling SQEs and popping CQEs, the mmap'd shared memory holding the SQ ring, SQE array, and CQ ring, and the kernel consuming submissions and posting completions, with io_uring_enter as the single syscall between them." caption="Figure 10.1: the io_uring anatomy: two rings and an SQE array shared via mmap; io_uring_enter is the only syscall in the data path" -%}
+ {%- include excalidraw.html file="10-readiness-vs-completion" alt="Two columns. Left, readiness: epoll_wait asks which fds are ready, the kernel reports readiness events, the application still performs read and write itself, and the operation can still fail with EAGAIN. Right, completion: the application queues SQEs describing the I/O, one io_uring_enter submits the batch, the kernel performs the I/O inline or via io-wq workers, and CQEs carry the finished results." caption="Figure 10.2: the mental-model pair: readiness (epoll) hints per fd; completion (io_uring) reports finished results per operation" -%}
+ {%- include excalidraw.html file="11-process-lifecycle-states" alt="Five process states drawn as boxes: fork/clone3 leads into Running R inside an alive band that also holds Sleeping S and Stopped T with SIGSTOP and SIGCONT arrows between them; an amber exit arrow crosses into an after-exit band holding Zombie Z, annotated defunct in ps f and exit status plus rusage only, with a final amber wait4/waitid arrow down to a dashed reaped box where the pid table slot is free." caption="Figure 11.1: the lifecycle as ps STAT letters; the kernel keeps a zombie until someone reaps it, and every annotation in the zombie box is observed later in this chapter" -%}
+ {%- include excalidraw.html file="11-fork-exec-wait-seq" alt="Two lanes, parent pmon and child. Parent creates pipe2 O_CLOEXEC, calls fork drawn as clone SIGCHLD with a COW note, and an arrow labeled returns 0 in the child crosses to the child lane where execvp runs; a dashed arrow back carries EOF on success or errno then exit 127. The child box CMD runs then exits or is killed leads by an amber arrow labeled zombie until reaped, SIGCHLD to the parent's waitpid/wait4 box, then down to report and mirror the fate, status s exits s and signal n exits 128 plus n." caption="Figure 11.2: pmon's spine: fork → exec → wait, with the CLOEXEC self-pipe carrying the exec verdict back to the parent" -%}
+ {%- include excalidraw.html file="12-signal-delivery-paths" alt="Two bands. Kernel band: kill and child-exit sources feed a per-process disposition table and a per-thread mask, converging on an amber pending set that coalesces one bit per signal, then a dashed delivery box gated on unblocked. User band: three columns for C++ signalfd read, Go runtime handler into a signal.Notify channel, and Rust self-pipe handler setting an AtomicBool and writing a doorbell byte, each ending in the main loop reading plain data." caption="Figure 12.1: every signal passes the same three kernel words (disposition, mask, pending) and pmon hears it three ways, none of them in signal context" -%}
+ {%- include excalidraw.html file="12-supervisor-state-machine" alt="State machine: an amber child-running state with infinite poll timeout, a backoff-wait state with deadline poll timeout, an amber transition between them labeled with the doubling backoff, a return transition when the backoff elapses, exits to exit 0 on clean exit or SIGTERM/SIGINT and to exit 1 when restarts reach the maximum, and a dashed SIGHUP reload box that stops the child, resets the budget, and spawns a replacement." caption="Figure 12.2: pmon supervise as a two-state machine; every transition is triggered by an ordinary value read in the main loop" -%}
+ {%- include excalidraw.html file="13-pid-reuse-race-pidfd" alt="Two horizontal timeline lanes. Top lane, five numbered steps: spawn remembering pid 4242, child crashes leaving a zombie, a wait reaps it freeing the pid, the kernel recycles 4242 for an unrelated process, and kill 4242 delivers SIGTERM to the stranger, with an amber dashed arrow noting same integer different process. Bottom lane, four numbered steps: spawn plus pidfd_open pinning child 4242 to pidfd 4, the child exit turning the fd readable under poll, waitid P_PIDFD reaping exactly the process behind the fd, and pidfd_send_signal returning ESRCH after death instead of hitting a recycled pid." caption="Figure 13.1: the pid-reuse race in five numbered steps, and the pidfd lane that closes it; the fdinfo and strace annotations are from this chapter's real runs" -%}
+ {%- include excalidraw.html file="14-credential-sets-caps" alt="Two bands. The top band shows the three uids: real, effective, saved, each moving 0 to 65534: setresuid moves all three at once and the saved uid is the way back that the drop destroys for good. The bottom band shows pmon's kept-cap path as six boxes: pmon as root with full permitted and effective sets, prctl PR_SET_KEEPCAPS, setresuid to 65534 with permitted kept, capset pinning 0x400 into permitted inheritable and effective, prctl AMBIENT_RAISE putting 0x400 into ambient, and execve into bindprobe bound to port 80 at uid 65534. A dashed negative-control row underneath skips the prctl and capset steps: setresuid scrubs permitted to zero and execve reaches bindprobe with every set zero and bind denied, exit 3. Notes carry the real /proc status values: the kept-cap child reads 0000000000000400 in CapInh CapPrm CapEff CapAmb with CapBnd 000001ffffffffff." caption="Figure 14.1: the three-uid state machine and the five capability sets across pmon's drop; every hex value comes from /proc/<pid>/status on systems-target, read during the cross-check below" -%}
+ {%- include excalidraw.html file="15-address-space-map" alt="A vertical stack of address-space regions from high to low: vsyscall, the stack growing down, an unmapped gap, the mmap region containing vdso and vvar pages, ld-linux and libc segments, and an accented 64 MiB anonymous rw-p mapping, another gap, the brk-grown heap, data and relro segments, and the executable text at the bottom, each labeled with a real address from this chapter's runs." caption="Figure 15.1: one x86-64 process address space, high addresses at the top; every address label comes from the /proc maps output walked below" -%}
+ {%- include excalidraw.html file="15-fault-flow" alt="A left-to-right flow: a user write to buf, then an MMU miss with a CPU trap, then the kernel page-fault handler looking up the VMA, fanning out to four outcomes: no VMA or bad permissions raising SIGSEGV, an anonymous first write getting a new zero-filled frame as a minor fault, a file page found in the page cache mapped with fault-around as a minor fault, and an uncached file page requiring a disk read as a major fault. Notes list the measured counts from this chapter's runs." caption="Figure 15.2: one touched page's journey through the fault handler; the counts in the notes are this chapter's getrusage measurements" -%}
+ {%- include excalidraw.html file="16-mapping-page-cache" alt="Two user-space process bands above a kernel band and a disk band. Process A holds an mmap MAP_SHARED node marked rw-s and a dashed MAP_PRIVATE node; process B holds its own MAP_SHARED node. Amber and grey arrows from both mappings converge on one page-cache node for store.kv in the kernel band; a dashed arrow back to the private node is labeled first write copies the page; an amber msync MS_SYNC arrow and a dashed writeback-eventually arrow lead down to the store.kv-on-disk node." caption="Figure 16.1: one file, one set of page-cache pages, many mappings: A's store and B's load meet in the same physical memory, msync forces those pages to disk, and a MAP_PRIVATE view forks its own copy on first write" -%}
+ {%- include excalidraw.html file="16-shmkv-layout" alt="A byte-layout band shows store.kv as 1034 bytes: a dark magic node with bytes 53 48 4b 56 31 00, an amber u32 little-endian slot_count node with bytes 04 00 00 00, and four 256-byte slot boxes. A zoom band below splits one slot into a key field covering bytes 0 to 64 and a value field covering 64 to 256, both NUL-padded. Three language nodes (C++ writes it, Go reads it, Rust reads it) point up into the layout." caption="Figure 16.2: the shmkv v0 format, byte for byte; the header bytes shown are the real od output from this chapter's run, and cmp proves a Rust-built and a C++-built store file identical" -%}
+ {%- include excalidraw.html file="17-arena-vs-gc" alt="Three columns compare reclamation strategies for the same 200,000 short-lived strings. Left, free one by one: objects flow into glibc's main arena and tcache with 64 cached chunks per size class, pages stay with the process (mallinfo2 reports arena 6340KB after everything is freed), peak RSS 3908KB for C++ and 2548KB for Rust with one free per object. Middle, arena and pool: a reusable 256KiB slab with a bump pointer, survivors copied out, the whole batch reclaimed in O(1) by reset, peak RSS 4240KB C++ and 2544KB Rust with zero per-object frees. Right, tracing GC: the churn becomes garbage the collector must find, the pacer triggers on GOGC growth or the GOMEMLIMIT soft limit, concurrent mark and sweep runs 9 cycles by default for 8064KB peak, or 47 cycles at 5676KB with GOMEMLIMIT=8MiB." caption="Figure 17.1: three ways to take memory back: per-object free through glibc malloc, wholesale arena release, and traced collection under a pacer; every number is from an allocbench run on the reference host" -%}
+ {%- include excalidraw.html file="18-pipe-internals" alt="Three panels: a writer process box holding fd 1 the write end, a kernel band labelled struct pipe_inode_info inode 16462630 containing a row of eight page slots from the sixteen-slot ring with head and tail pointers marked, and a reader process box holding fd 5 the read end with fdinfo flags 02000000; notes record the 65536-byte capacity read via F_GETPIPE_SZ, the F_SETPIPE_SZ rounding of 1 up to 4096, the 1048576 pipe-max-size ceiling, and the warning that PIPE_BUF atomicity is not message framing." caption="Figure 18.1: one pipe from this chapter's live supervise run: the child's fd 1 and the supervisor's fd 5 share the ring behind pipe inode 16462630; every number is from /proc or a live fcntl" -%}
+ {%- include excalidraw.html file="18-log-topology" alt="Three horizontal bands. Top, pmon supervise pid 2633754: the sh child pid 2633756 writes fd 1 and fd 2 into two kernel pipe nodes with inodes 16462630 and 16462631, which feed a poll loop node listing signalfd 4, pidfd 6, and pipe read ends 5 and 7, appending framed out and err lines into a live.log node opened append-only as fd 3. Middle, pmon tail pid 2634759: a log fd 3 node with fdinfo pos 10 connects by an amber splice arrow labelled with the real splice call returning 22 to a FIFO writer fd 4 node whose flags 02100001 show O_NONBLOCK cleared. Bottom, kernel FIFO plus reader: the live.fifo inode of type S_IFIFO flows into a cat reader holding fd 3; a detached reader causes EPIPE, and the relay re-splices undelivered bytes to the next reader." caption="Figure 18.2: pmon v4's log pipeline end to end; every pid, fd number, pipe inode, flag value, and splice return comes from the live runs traced in this chapter" -%}
+ {%- include excalidraw.html file="19-uds-control-plane" alt="Two process bands over a kernel band: the pmon supervise process holds listener fd 4 bound to /tmp/…/ctl.sock, an accepted fd 5 annotated with the SO_PEERCRED result uid=1000 pid=2624432, and log fd 3 opened O_RDWR|O_APPEND feeding a dashed child box for sleep 300 pid 2622905; the pmctl process holds client fd 3 which connects down to socket inode 16422777 in the kernel band, paired by a bidirectional amber arrow with socket inode 16411996 owned by the supervisor, exactly as ss -xp reported." caption="Figure 19.1: pmon v5's control plane during one status call: every pid, fd number, and socket inode in this figure comes from the live run traced later in the chapter" -%}
+ {%- include excalidraw.html file="19-scm-rights-transfer" alt="Two fd-table bands over a shared kernel band: sender slot 3 with close-on-exec set at open, and receiver slot 4 labeled new lowest free with close-on-exec set by MSG_CMSG_CLOEXEC, both pointing at one dark open-file-description box holding pos 47 and flags 02102002, which points down to inode 95934 for pmon.log; sendmsg with cmsg_data 3 names the sender slot, an amber dashed arrow shows the kernel installing the new receiver slot, and recvmsg reports cmsg_data 4." caption="Figure 19.2: SCM_RIGHTS across the three fd layers: two per-process slots, one open file description, one inode; pos, flags, inode, and both fd numbers are from the held-open run below" -%}
+ {%- include excalidraw.html file="20-futex-wait-wake" alt="Three lanes: watcher process, kernel, writer process. The watcher's seqlock read finds nothing new and calls futex FUTEX_WAIT on word value 3; the kernel re-checks the word equals 3 and sleeps the task on a wait queue keyed by inode and offset 12. The writer publishes k4 under the seqlock bumping seq 6 to 7 to 8, stores futex word 4 with release ordering, and calls FUTEX_WAKE INT_MAX which returns 1; an amber arrow marks the watcher runnable again, and it back-fills ids from the 8-slot ring. A dashed note explains the EAGAIN lost-wakeup guard." caption="Figure 20.1: one shmkv wake end to end: the watcher parks on the mapped word, the writer publishes and wakes, the kernel's wait queue keyed by (inode, offset) connects them; every syscall in this figure appears verbatim in the strace cross-check below" -%}
+ {%- include excalidraw.html file="20-ipc-matrix" alt="A ladder of six boxes ordered left to right by latency: futex wake p50 under 1 microsecond p99 4, eventfd 2 and 6, pipe 2 and 5, POSIX mq 2 and 14, UNIX socket 3 and 12, then a dark box for the 1 millisecond poll loop at p50 1056 and p99 1083 microseconds with an amber arrow labeled roughly 350x. Below, two summary boxes: the scheduler evidence of 8 versus 3774 voluntary context switches in a 4 second window, and rusage counts of 120 versus 2002 voluntary switches over 2000 rounds." caption="Figure 20.2: the coordination ladder as measured on this host: every blocking channel lands within one order of magnitude, and the poll loop pays its nap" -%}
+ {%- include excalidraw.html file="21-server-lifecycle" alt="A top band labeled serve: one listener thread holds five boxes left to right: socket() AF_INET SOCK_STREAM|CLOEXEC, setsockopt SO_REUSEADDR = 1 rebind past TIME_WAIT, bind() sockaddr_in 127.0.0.1:9000, listen(128) passive socket fd 3, and an amber accept4() loop SOCK_CLOEXEC producing new fd 4, 5. Two amber spawn-thread arrows fall from the accept loop into a lower band, one std::jthread or goroutine or std::thread per accepted connection, holding serve_client fd 4 (bob) and serve_client fd 5 (alice). An amber note reads SIGINT arrow g_stop = 1; poll(2) wakes arrow stop accepting; listener fd closes; threads join; exit 0." caption="Figure 21.1: chatterd serve's lifecycle: the four setup calls, the accept4 loop, and one thread per accepted fd; every fd number here reappears in the ss and strace captures later" -%}
+ {%- include excalidraw.html file="21-frame-format" alt="A row of 21 byte cells for the server's DELIVER broadcast of alice's message to bob. The first six cells, amber, hold 43 48 01 03 00 0f labeled magic CH, version 0x01, type 0x03 DELIVER, length (u16BE) = 15. The next five cells hold 61 6c 69 63 65 labeled name = alice. One dark cell holds 00 labeled NUL. Nine cells hold 68 65 6c 6c 6f 20 62 6f 62 labeled text = hello bob. Below: 00 0f = 0x000f = 15 = len(alice) + 1 (NUL) + len(hello bob). Then the frames that got here: alice's own MSG frame (type 0x02) carried just hello bob as its payload, no name; JOIN (type 0x01) payload is name alone, sent once at connect. A footnote notes length is u16BE, 0..65535, counts payload bytes only, and the 6 header bytes are never counted; WELCOME (0x04) and PING (0x05) are declared and reserved but neither sent nor expected in this chapter." caption="Figure 21.2: the wire frame byte for byte: the DELIVER broadcast of alice's message to bob; 43 48 01 03 00 0f is the exact magic+version+type+length header captured off the socket in the cross-check below" -%}
+ {%- include excalidraw.html file="22-connection-state-machine" alt="Two bands. The top READ PATH band runs left to right: a recv() loop draining nonblocking until EAGAIN feeds an inbuf that reassembles partial frames across reads, into take_frame which asks whether one whole 6+LEN frame is ready, into MSG to broadcast which appends the frame to every connection's outbuf. The bottom WRITE PATH and BACKPRESSURE band: broadcast drops for each conn into queue() plus flush() which send()s what fits now; on send() EAGAIN the remainder stays in outbuf (the backpressure queue) because the kernel socket buffer is full; an amber arrow labelled full: arm EPOLLOUT runs down-left to an epoll_wait node that later reports EPOLLOUT for this conn, and an amber arrow labelled EPOLLOUT wakeup runs back up-right to flush(), closing the loop. A note reads: level-triggered, EPOLLIN always armed, EPOLLOUT armed only while outbuf is non-empty." caption="Figure 22.1: chatterd's epoll engine as one per-connection read/write state machine; the EPOLLOUT backpressure loop is the whole reason a slow client cannot stall the server" -%}
+ {%- include excalidraw.html file="22-netpoller-vs-epoll" alt="Two side-by-side bands. Left, titled C++/Rust, you write the epoll loop: one OS thread (Threads: 1) calling epoll_wait (4 calls for a 50-conn flood) connected by a bidirectional amber arrow labelled EPOLLIN/EPOLLOUT + token to a kernel eventpoll interest list holding tok 0 listener, tok 1 signalfd, tok 2..N every client socket, one process; a note reads backpressure = outbuf + EPOLLOUT (explicit). Right, titled Go, you write goroutines; the runtime hides the epoll: a goroutine per conn (read + write) plus hub with channels fanning out over ~7 OS threads total, a bidirectional amber arrow park/resume goroutine down to a dark runtime netpoller (hidden) box running epoll_pwait (165 calls, edge-triggered EPOLLET), then down to the same kernel sockets, one process, N conns (ss -tp); a note reads backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads." caption="Figure 22.2: the chapter's central comparison: the epoll loop you write by hand in C++/Rust is the epoll loop Go's runtime writes for you, driven by epoll_pwait one layer down" -%}
+ {%- include excalidraw.html file="23-two-vm-topology" alt="A libvirt virbr0 bridge band on 192.168.124.0/24 holds a central amber multicast-group box for 239.23.7.1 port 51824 TTL 1, with systems-target at 192.168.124.7 on the left and systems-peer at 192.168.124.95 on the right, each joined to the group on enp1s0 and holding udp fd 4 and tcp 9241; two dashed amber bidirectional beacon arrows (35 bytes from target, 34 bytes from peer) reach the group, and a solid bidirectional TCP arrow labelled one chat frame each way joins the two guests directly." caption="Figure 23.1: the two-guest lab topology: both VMs join 239.23.7.1, beacon to the group, and bridge each discovery to a direct TCP :9241 chat frame; every address, port, and byte-length here is from the two-host run later in the chapter" -%}
+ {%- include excalidraw.html file="23-discovery-sequence" alt="Two horizontal bands. The upper blue band, UDP multicast discovery, holds three boxes left to right: announce loop sendto to group emitting CHATTERD1 target 9241; receive loop recv one datagram splitting four fields and checking magic; and an amber parse-and-dedup box that skips its own name and calls seen.first before dialing. An amber arrow labelled new peer bridge to TCP drops into the lower grey band, TCP bridge, whose boxes are connect with retry, write_frame slash read_frame with a 6-byte header (magic, version, type, 2-byte big-endian length) plus payload, and a dark stdout box printing discovered peer and peer says. A note reads loss and reorder tolerated." caption="Figure 23.2: one peer's path from a connectionless beacon to a connection-oriented frame: recv, dedup by name, then dial and speak the canonical chat frame already introduced (JOIN out, DELIVER back)" -%}
+ {%- include excalidraw.html file="24-clock-taxonomy" alt="Top band: the four clocks clock_gettime can read as boxes: CLOCK_REALTIME (dashed, wall time, settable, can step backward), CLOCK_MONOTONIC (amber, since a boot epoch, never steps, steady_clock/Instant), CLOCK_BOOTTIME (monotonic plus suspend time), and PROCESS_CPUTIME_ID (this process's on-CPU time). Bottom band: while a peer is silent and ntpd corrects the wall clock, a dashed REALTIME box shows a wall-clock deadline rewinding so a dead peer looks alive, beside an amber MONOTONIC box where elapsed is always non-negative and the timeout fires on real silence; an amber arrow from CLOCK_MONOTONIC points to it labeled chatterd reads this." caption="Figure 24.1: the clock taxonomy and the monotonic-vs-wall-clock hazard: a wall-clock deadline slips when the clock is stepped; a CLOCK_MONOTONIC deadline does not" -%}
+ {%- include excalidraw.html file="24-heartbeat-timeout-seq" alt="Two lifelines: connector B (amber) on the left, listener A (grey, ending early) on the right. Top: JOIN 'B' from B to A and JOIN 'A' back, then a note that both print linked and exchange messages. Middle: amber PING arrows every 300 ms in both directions, each resetting B's deadline, with a note that any frame sets deadline = mono_now() + timeout. Then A killed, socket EOF, and a dashed PING from B that is dropped on the broken pipe. Bottom on B's side: an amber box 'deadline fires on silence: chatterd: peer A timed out', then three stacked reconnect boxes 107 ms, 245 ms, 712 ms chained by amber arrows, beside a note explaining equal-jitter backoff delay = B/2 + rand(0..B/2) with B doubling per attempt and capped at max-backoff." caption="Figure 24.2: chatterd v3 from link to loss to reconnect: the monotonic deadline fires on silence, then jittered exponential backoff (real 107/245/712 ms from this run) paces the redials" -%}
+ {%- include excalidraw.html file="28-stack-frame-anatomy" alt="A column of five stacked frames from a real gdb bt full on app's core file, growing toward lower addresses as they go down: frame 0 std::println<int,int> at print colon 164, the actual crash PC; frame 1 deref at main.cpp:52 showing child equals pid 4242 exit_slot 0x0; frame 2 handle_child at main.cpp:56 with return address 0x4016b2; frame 3 run_supervisor at main.cpp:65 with return address 0x40177c and locals child, guard equals optimized out; frame 4 main at main.cpp:104 with return address 0x400964. Beside it, a DWARF cross-check column shows readelf/objdump decodedline results: main.cpp:52 maps to 0x401680, an exact match to gdb's own breakpoint address; main.cpp:56 maps to 0x4016b2, an exact match to frame 2's return address; main.cpp:65 maps to 0x401772, close to but not exactly frame 3's return address of 0x40177c since that is a post-call return address, not a statement start. A footer note says every frame function is noinline so -O2 cannot fold the call chain away, and ShutdownGuard's destructor never runs because SIGSEGV does not unwind the stack." caption="Figure 28.1: the five real frames gdb names on app's core, cross-checked address for address against the DWARF line table readelf and objdump report independently" -%}
+ {%- include excalidraw.html file="28-remote-debug-topology" alt="Two stacked bands. The upper host band, marked verified this session, shows the local core-dump path: app crash producing a SIGSEGV, an arrow into systemd-coredump, coredumpctl dump extracting core.app, and gdb app core.app doing a full bt. Beside it, a second host-side node reads gdb (client), target remote systems-target:2345, with an amber bidirectional arrow labeled gdb remote serial protocol, TCP:2345 descending into the lower systems-target VM band, marked documented, not run this session. That band holds a gdbserver :2345 --once app crash node feeding an app (crash target) child process node below it, plus a dashed ghost box off to the side reading CLion, Remote GDB Server run config, unverified. A footer note says gdbserver is preinstalled on the lab VM image via cloud-init but not on this host, and no systems-target VM was up this session." caption="Figure 28.2: the local core-dump path this chapter's evidence is built on, beside the gdbserver/target-remote/CLion path documented for the systems-target VM" -%}
+ {%- include excalidraw.html file="29-tool-coverage-matrix" alt="A grid with defect down the rows (leak, uninit, uaf, overflow, race) and C++, Go, Rust across the columns. C++/leak: valgrind memcheck caught it, definitely lost. C++/uninit: valgrind memcheck caught it, uninitialised value(s); ASan alone missed it, needs MemorySanitizer. C++/uaf: valgrind caught it (invalid write, freed) and ASan caught it (heap-use-after-free). C++/overflow: valgrind caught it (0 bytes after a block) and ASan caught it (heap-buffer-overflow). C++/race: memcheck is not built for this and was not the tool tried; helgrind caught it (possible data race) and ThreadSanitizer caught it. Go/leak: self-detected via goroutines before and after; valgrind was confused, 630 errors and 0 real signal. Go/uninit, uaf, overflow: prevented by the runtime, exit 64, no tool needed. Go/race: go -race caught it, WARNING DATA RACE, exit 66. Rust/leak: valgrind caught it, 65536 bytes lost, and miri on nightly additionally reported memory leaked. Rust/uninit, uaf, overflow, race: prevented at compile time by the borrow checker, exit 64; a note says unsafe Rust plus miri catches the use-after-free class separately, shown elsewhere in the chapter. A footer legend says amber cells mean a tool run this session caught the defect, dark cells mean the language prevents it outright, and plain cells mean a real blind spot this session measured; every cell traces to a command in Build run observe or Cross-check." caption="Figure 29.1: the five bugfarm defects against three languages, every cell backed by a command this chapter ran" -%}
+ {%- include excalidraw.html file="29-shadow-memory" alt="Two stacked bands. The upper application-memory band shows three boxes: a left redzone at addresses ending ff80 through ffff, a 4-byte int* p at address ending 0010 marked as new'd then delete'd (writing through it is undefined behavior), and a right redzone at addresses ending 0018 through 007f. The lower shadow-memory band shows the corresponding three shadow bytes: fa labeled heap left redzone, fd labeled freed heap region and marked as the exact byte the write trips, and fa labeled heap left redzone again for the next block's guard. An arrow labeled shift right by 3 then add offset connects the freed block down to its fd shadow byte. A footer formula reads shadow address equals app address shifted right 3 plus offset, one comparison before every load or store, not a full CPU emulation, and a footer gives the real legend from this session's ASan run (00 addressable, 01 through 07 partially addressable, fa heap redzone, fd freed heap region, f1 through f8 stack states, f9 global redzone), plus the measured gap: 0.002s native, 0.046s under ThreadSanitizer, 1.0s under valgrind memcheck, for the same tiny program." caption="Figure 29.2: ASan's shadow-memory model, with the real freed-block shadow bytes this chapter's `uaf` run produced" -%}
+ {%- include excalidraw.html file="30-ebpf-tooling-stack" alt="Three stacked bands. The top kernel band holds five small boxes: kprobe/kretprobe on kernel functions, uprobe/uretprobe on userspace functions, tracepoint as the stable kernel ABI, perf_event as timer/PMU sampling, and BTF/CO-RE for struct layout without headers. Amber arrows descend from three of these into the middle eBPF-tooling band, which holds bcc-tools (execsnoop/opensnoop/biolatency/offcputime/funccount/profile), bpftrace (one-liners and scripts, probe:target /filter/ action), bpftool (prog show/map show), and c++filt (demangle the exact symbol you attach to); libbpf loads a small verified BPF program per probe that the kernel runs in-kernel on each hit. Below that, a third band holds SystemTap as the sibling technology: its classic kernel-module backend needing matching kernel-devel, beside stap --runtime=dyninst, which patches the target binary directly in userspace with no kernel module. A fourth band at the bottom shows the three target binaries: C++'s busy_hash under the SysV ABI with a correct arg0, Rust's cmd_work also correct under the SysV ABI once c++filt demangles its v0-mangled name, and Go's cmdWork in a dark box reading arg0/RDI wrong at 1, RAX right at 9, with a dashed amber arrow from it up to the eBPF-tooling band labeled arg0 mismatch. A footer note says every amber box ran live against systems-target this session; SystemTap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, and the dyninst backend attached without erroring." caption="Figure 30.1: the eBPF observation stack this chapter exercises live: kernel probe primitives at the bottom, bcc-tools/bpftrace/bpftool/c++filt above them, SystemTap as the sibling technology beside them, and the one real register-ABI mismatch that only shows up on the Go binary" -%}
+ {%- include excalidraw.html file="30-uprobe-attachment" alt="Three parallel columns, one per language, each four boxes tall. C++ column: nm reports the mangled symbol t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0; c++filt (auto Itanium ABI) demangles it to (anonymous namespace)::cmd_work(int); a uprobe attaches using the MANGLED string as the attach target; the probe fires with seconds_arg=5 for --seconds 5, and arg0==RDI is correct because C++ follows the plain SysV x86-64 ABI. Rust column: nm reports t _RNvCs6FzKXswKZCc_3app8cmd_work; c++filt auto-detects Rust v0 mangling and demangles it to app[hash]::cmd_work; a uprobe attaches using that same mangled string; the probe fires with seconds_arg=7 for --seconds 7, and arg0==RDI is again correct under the SysV ABI. Go column: nm reports T main.cmdWork, never mangled; no demangling step is needed since the package-qualified name is already the symbol; a uprobe attaches to the plain unmangled name main.cmdWork; the probe fires for --seconds 9 but RDI(arg0) reads 1, which is wrong, while RAX correctly reads 9; a dashed amber arrow connects this box back up to the tooling band above, labeled arg0 mismatch. A footer band spans all three columns explaining that Go's ABIInternal (1.17+) passes integer arguments in the order RAX, RBX, RCX, RDI, RSI, R8, R9, ... rather than the SysV order RDI, RSI, RDX, ... that bpftrace's arg0/arg1 assume, and that goroutine stacks also grow by copying, so a captured stack address can move out from under a probe that read it from the stack instead of a register." caption="Figure 30.2: the same uprobe-attach recipe (find the symbol, demangle it to read it, attach to the mangled string) across three binaries, ending in the one real, measured argument-register mismatch that shows up only on Go" -%}
+ {%- include excalidraw.html file="31-flamegraph-reading" alt="A schematic flamegraph with two labeled axes: y is stack depth, bottom the root frame runtime.main and top the leaf running at the instant a sample was taken; x is share of samples, explicitly not time order or call order. A stack of boxes narrows from a full-width root frame up through main/main.run, into a wide accent box for spin_hot/hotSpin covering about 99% of the row's width, into an equally wide inlined divisor-loop frame directly above it, topped by a leaf sliver. A callout box explains that color here just marks which language tab a frame came from, since real profilers usually color by module or at random. A footer note says the way to find the hot path is to scan for the widest unbroken column reaching toward the top." caption="Figure 31.1: reading a flamegraph: width is the share of samples, height is stack depth, and the widest column reaching toward the top is the function spending the cycles" -%}
+ {%- include excalidraw.html file="31-toolbelt-map" alt="One shared target box, app hot or alloc with an identical result across languages, feeding three columns. The C++ column lists perf record with call-graph fp as its accent entry naming spin_hot at 99.9%, then perf report/perf stat, clang-tidy, GCC -fanalyzer, and a dashed gdb box marked as the debugging digression. The Go column lists runtime/pprof as its accent entry, cpuprofile into go tool pprof top naming main.hotSpin, then memprofile with alloc_space naming main.allocChurn, runtime/trace with go tool trace, delve, and a dashed go vet box. The Rust column lists perf record with call-graph fp as its accent entry naming spin_hot inlined near 100%, then cargo flamegraph producing a real SVG, cargo clippy and cargo fmt check, a dashed tokio-console box marked as a forward reference to async-runtime debugging, not applicable here since there is no async runtime, and a dashed gdb/rust-gdb box marked as the debugging digression. A note explains the amber boxes are the CPU-profiling step this chapter cross-checks against each other, and the dashed boxes are noted but out of scope." caption="Figure 31.2: one shared target, three native toolbelts: the amber boxes are what this chapter's cross-check compares; the dashed boxes point to where gdb and tokio-console belong instead" -%}
+ {%- include excalidraw.html file="32-namespace-overview" alt="Two columns, host and pmon containerize's PID-1 child, with eight namespace-type rows: PID, mount, UTS, network, time, IPC, user, cgroup. The first four rows are amber boxes with an arrow marked 'differs' between real /proc/<pid>/ns/* inode numbers: pid 4026531836 vs 4026532530, mount 4026531832 vs 4026532528, UTS 4026531838 vs 4026532529, network 4026531833 vs 4026532531, read on systems-target. The last four rows are grey dashed boxes marked 'shared with host' with identical inode numbers on both sides: time 4026531834, IPC 4026531839, user 4026531837, cgroup 4026531835, because pmon containerize only requests CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID and leaves the other four alone by design." caption="Figure 32.1: the eight Linux namespaces, host vs pmon's PID-1 child, every inode number read live from /proc/<pid>/ns/* on systems-target" -%}
+ {%- include excalidraw.html file="32-cgroup-hierarchy" alt="The cgroup v2 unified hierarchy from root down to a leaf. The root band shows cgroup.controllers listing cpuset, cpu, io, memory, hugetlb, pids, rdma, misc, dmem, and cgroup.subtree_control enabling plus memory plus cpu, delegated to children. An arrow labeled enables leads down into the pmon-oomdemo leaf band, showing memory.max as 67108864 (64 MiB), memory.swap.max as 0 (no swap headroom), cpu.max as max 100000 (unthrottled), cgroup.procs holding pmon's own pid written before fork, and memory.pressure PSI reading some avg10=0.00. A dark box below states the hog process (sh, real anon-rss 64384kB) writes past the 64 MiB ceiling. An arrow labeled 'OOM killer fires' leads into a kernel band showing the real dmesg lines from this session: oom-kill constraint=CONSTRAINT_MEMCG oom_memcg=/pmon-oomdemo task=sh pid=27013, and 'Memory cgroup out of memory: Killed process 27013 (sh) anon-rss:64384kB', with pmon's own report alongside it: signal=9 (KILL), exit 137." caption="Figure 32.2: the cgroup v2 unified hierarchy, root to leaf, ending in the real dmesg OOM-kill line this session produced" -%}
+ {%- include excalidraw.html file="32-clone3-flow" alt="Three lanes. The left lane, C++/Rust, shows unshare taking effect on the caller, then clone3 with flags=0 (fork semantics) creating a child that prints its pid/hostname proof lines and execs CMD, labeled PASS, a real run this session. The middle lane, Go's first attempt, shows runtime.LockOSThread pinning the goroutine then a raw unshare call, followed by os/exec.Start issuing an internal vfork/pidfd probe fork, then the real fork for CMD failing with clone returning -1 ENOMEM and the message fork/exec: cannot allocate memory, real output from this session. The bottom lane, Go's shipped fix, shows setup_cgroup writing its own pid first same as the other two languages, then exec.Command with SysProcAttr.Cloneflags carrying CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID, one clone(2) call creating the child already inside all four namespaces with no prior unshare and no LockOSThread needed, the child re-execing itself as __ns_init__ to mount / MS_PRIVATE, sethostname, print the proof lines, and exec CMD as PID 1, labeled PASS 13/13 across cpp, go, rust on systems-target, this session. A note explains the ENOMEM is a real, kernel-documented restriction: unshare(CLONE_NEWPID) allows the caller to create exactly one child in the fresh namespace before further forks there return ENOMEM, and os/exec's own vfork/pidfd fast path performs an internal probe fork that silently spends that one shot." caption="Figure 32.3: C++/Rust's unshare-then-clone3 shape beside Go's two attempts: a real ENOMEM, then the SysProcAttr.Cloneflags fix that passes" -%}
+ {%- include excalidraw.html file="33-seccomp-filter-flow" alt="Two stacked bands. The top userspace band shows fwatch issuing a syscall such as epoll_wait, read, or socket, beside a note that this is classic BPF: you write the filter, Part 8's bpftrace only observes. The bottom kernel band shows the filter running on every syscall entry: a BPF_LD_ABS load of the syscall number and architecture, a compiled jump table of 26 entries for cpp and rust or 39 for go, a match arrow to SECCOMP_RET_ALLOW that returns control to the caller, and a no-match arrow down to the filter's default action. Two default-action boxes sit side by side: SCMP_ACT_ERRNO(EPERM), the one fwatch actually installs, where the process keeps running and sees EPERM on socket(2); and a dashed, unused SCMP_ACT_KILL_PROCESS box, where a separate probe program's getpid() call ends the whole process with SIGSYS. Footer notes report the real guest transcript: socket(AF_INET,...) = -1 EPERM under fwatch's filter, and the ACT_KILL_PROCESS probe's getpid() ending in killed by SIGSYS (core dumped)." caption="Figure 33.1: a syscall's path through fwatch's seccomp-bpf filter: one cBPF program, run again on every entry, deciding ALLOW versus its default action" -%}
+ {%- include excalidraw.html file="33-landlock-ruleset" alt="Two stacked bands. The top band builds the ruleset: landlock_create_ruleset called first with NULL/0/VERSION to probe the ABI, returning 9 on the host versus 7 on the systems-target guest despite both being Fedora 44 kernels; then landlock_create_ruleset(attr) with handled_access_fs set to READ_FILE and READ_DIR, feeding into landlock_add_rule binding a PATH_BENEATH rule to one directory's O_PATH file descriptor; then prctl(PR_SET_NO_NEW_PRIVS,1) feeding into landlock_restrict_self, marked irrevocable, with a note that Go's build also passes the TSYNC flag here to cover every OS thread already running. The bottom band shows the filesystem view after restrict_self: the sandboxed directory /tmp/lsp33-watch-sandbox with READ_FILE and READ_DIR still allowed and the watch loop working normally, beside /tmp/lsp33-outside-target/secret.txt where open(O_RDONLY) returns EACCES, with an arrow from the ruleset's restrict_self box down to each outcome, solid to the grant and dashed to the denial. A footer note across the bottom states that handled_access_net was left at zero in this ruleset, and that Landlock's TCP bind/connect rights have existed since ABI 4 and UDP rights only since ABI 10, so even at ABI 7 or 9 a Landlock-only sandbox does not cover the network by itself." caption="Figure 33.2: the Landlock ruleset/rule model, with the ABI probed live rather than assumed and the real grant/deny split from this chapter's probes" -%}
+ {%- include excalidraw.html file="34-image-layering" alt="Two side-by-side bands. The left, discarded builder stage starts from registry.access.redhat.com/ubi10/ubi:10.2 at 220 MB measured, runs dnf install of gcc-c++, golang, or rustup, and statically links the binary (cpp -static-libstdc++ -static-libgcc, go CGO_ENABLED=0, rust rustup 1.97.1). An amber COPY --from=builder arrow crosses into the right, shipped runtime stage, which starts from registry.access.redhat.com/ubi10-micro:10.2 at 24.3 MB measured, copies only the one binary to /usr/local/bin/app, and sets ENTRYPOINT/CMD. A footer note reports the real measured final image sizes (cpp and go both 27.2 MB, rust 24.9 MB), barely above the bare micro base, and explains that static linking is what makes copying one binary sufficient." caption="Figure 34.1: app's multi-stage Containerfile; the 220 MB builder never ships, only the binary crosses into the 24.3 MB runtime base" -%}
+ {%- include excalidraw.html file="34-pid1-signal-flow" alt="Two side-by-side bands contrasting serve and naive receiving SIGTERM as a container's PID 1. The serve band: podman stop sends SIGTERM, the signalfd/Notify/Signals wake fires, job_spawner.request_stop() runs, kill(worker, SIGTERM) plus waitpid/Wait() reaps it, a shutting-down line prints, and the process exits 0, measured at cpp 192ms, go 60 to 87ms, rust 101ms, all well inside a 3000ms grace budget. The naive band: podman stop sends SIGTERM into a kernel that finds no explicit handler on PID 1, so the signal is dropped for cpp and rust; podman waits out the full grace period and falls back to SIGKILL, measured at cpp 2108ms and rust 2074ms. A muted box notes the Go exception: the Go runtime installs its own fatal-signal handler at startup regardless of user code, so naive still dies in about 60ms, but with no shutting-down line and no cleanup, in every language." caption="Figure 34.2: serve's graceful pid-1 shutdown sequence beside naive's kernel-masked signal, with this session's real podman stop timings" -%}
+ {%- include excalidraw.html file="34-ns-debugging-map" alt="A host band above a podman-user-namespace band. podman exec reaches the container PID 1 directly from inside the container runtime. podman unshare nsenter first joins podman's own user namespace, then nsenter --target --pid --mount reaches the same PID 1 from there, both confirming Name: app and /proc/1/exe resolving to /usr/local/bin/app. A dashed, failing arrow shows plain nsenter without podman unshare returning EPERM: reassociate to namespaces failed. Below, real evidence boxes show the subuid mapping user:524288:65536 and a live zombie scan of /proc/[0-9]*/stat reporting zero." caption="Figure 34.3: podman exec and podman unshare nsenter reach the same container PID 1 from two different starting namespaces; plain nsenter cannot" -%}
+ {%- include excalidraw.html file="35-kvm-virtio-stack" alt="Three horizontal bands. Top: host, VMX root mode, showing kvm.ko and kvm_intel.ko managing EPT tables, a QEMU/VMM userspace box running the vCPU thread and virtio-blk backend, and host NVMe/btrfs where a real fsync lands. Middle: guest, VMX non-root mode, showing the guest kernel serving getpid in-guest with no host trap, overheadbench in guest ring 3 issuing the syscall bench's raw getpid loop and the io bench's open/write/fsync calls, a virtio-blk front-end driver, and a virtqueue descriptor ring. Bottom: the trapped path for fsync: fsync in the guest causes a VM exit, the KVM exit handler hands off to the QEMU vCPU thread, the virtio-blk backend issues the real host fsync and triggers VM enter back into the guest. A fourth band gives the measured overheadbench deltas: syscall about 88ns/call on both host and guest because getpid never triggers a vmexit, mem about 14 to 15 GB/s on both because the EPT walk cost is amortized after the TLB warms up, and io about 1.3k ops/s on the host against about 0.5k ops/s on the guest, the real vmexit tax." caption="Figure 35.1: the VMX root/non-root boundary, the virtio-blk round trip that fsync pays, and where overheadbench's own measured host-vs-guest deltas land" -%}
+ {%- include excalidraw.html file="35-isolation-spectrum" alt="Four boxes left to right along a spectrum from shared host kernel to separate guest kernel. Bare process: no isolation boundary, syscall equals host syscall, escape equals root on host. Podman container: namespaces plus cgroup v2 plus seccomp, one host kernel and one syscall table, syscall and mem about equal host and io about equal host on a bind mount, escape equals a kernel bug in namespace or cgroup code. MicroVM, Firecracker or cloud-hypervisor built on rust-vmm: own guest kernel with a minimal virtio device set, about 125 milliseconds boot and under 5 MiB overhead for Firecracker, escape equals a bug in the roughly five devices the VMM exposes or in KVM itself. Full KVM/QEMU VM, the systems-target lab guest: own guest kernel with a full device model, syscall about equal host and io about three times slower due to virtio-blk, escape equals a bug in the full emulated device model or in KVM. A note below cites that rust-vmm crates such as kvm-ioctls, virtio-queue, and vm-memory underpin both Firecracker v1.16.1 and Cloud Hypervisor v53.0." caption="Figure 35.2: process, container, microVM, and full VM, each priced by escape surface and by this chapter's measured overheadbench numbers" -%}
+ {%- include excalidraw.html file="36-proc-datasource-map" alt="A pipeline diagram of sysagent's take_snapshot. Left, a single /proc/stat read at t0 landing in a small box labeled 'cumulative counters since boot, not a percentage.' Center, a band labeled 'gathered concurrently with the --interval-ms sleep' containing five parallel boxes (/proc/loadavg, /proc/meminfo, /proc/diskstats, /proc/net/dev, and cgroup PSI (cpu.pressure/memory.pressure/io.pressure, falling back to /proc/pressure)), each writing into its own field of one shared Snapshot, all joined before the sleep's deadline is allowed to pass. A second /proc/stat read at t1 sits to the right of the band; arrows from both t0 and t1 feed a delta box computing cpu_util_pct/cpu_user_pct/cpu_system_pct as 100 times (busy1-busy0)/(total1-total0). A note reads 'two cumulative reads, --interval-ms apart, become one percentage.' Below, the completed Snapshot feeds two consumers drawn side by side: sample --json printing the object once to stdout, and serve --port P's GET /metrics handler calling take_snapshot fresh on every request. A small inset box shows the PSI fallback path: this process's own cgroup v2 delegate (resolved via the '0::' line in /proc/self/cgroup) tried first, the system-wide /proc/pressure/* files tried second, and psi_available:false (not an error) if neither is readable." caption="Figure 36.1: sysagent's take_snapshot: a t0/t1 /proc/stat delta bracketing five concurrently-gathered /proc + cgroup PSI sources, all landing in one Snapshot that sample --json and serve /metrics both emit" -%}
+ {%- include excalidraw.html file="37-use-method-checklist" alt="A pipeline diagram of loadmix analyze. Left column, four sequential pre-samples run once at start: uptime, dmesg tail, top -b -n1, and ss -s, each landing in a single printed 'tool' line. Center, a band labeled 'five interval samplers, spawned concurrently' containing five parallel boxes (vmstat 1 N, mpstat -P ALL 1 N, iostat -xz 1 N, sar -n DEV 1 N, and pidstat 1 N) each writing to its own temp file, all five joined by a single wait so the band's total wall time is N seconds, not 5N; a note under the band reads 'N-second checklist costs ~N seconds'. Right, each sampler's output feeds a column-name parser (Row::get against the tool's own header row) into a 4x3 signal matrix: rows cpu/mem/io/net, columns Utilization/Saturation/Errors, each cell a fired=true-or-false verdict against a fixed threshold, cpu's Saturation threshold shown as cpus+2 to make the guest-relative scaling visible. An arrow from the matrix's Saturation column into a final verdict box reads 'resource with the highest Saturation-value/threshold ratio wins'. A small inset box below the diagram shows the saturate side: app --resource cpu|mem|io|net --seconds N spinning a worker pool that the analyze pipeline, run concurrently against it, is able to see and attribute." caption="Figure 37.1: the analyze pipeline: four one-shot pre-samples, five interval samplers run concurrently, column-name parsing into a 4×3 USE signal matrix, and the highest-Saturation-ratio verdict" -%}
+ {%- include excalidraw.html file="38-otlp-pipeline" alt="A left-to-right pipeline. Left, a band labeled 'serve: one traced request per input line' containing: a parent span box 'request (attr request.seq)' with two child span boxes below it, 'work' (sleep 1+seq%4 ms) and 'respond' (write ok seq), plus two metric boxes, 'requests_total (+1 per request)' and 'request_duration (ms)'. An arrow labeled OTLP leaves the band into an 'OTLP/HTTP exporter' box annotated 'batch ~2s, flush on SIGINT', which POSTs, over an amber arrow labeled POST /v1/{traces,metrics}, into an 'OTel collector :4318' box. The collector fans out along two arrows: 'spans' up to a 'Tempo: traces' box and 'metrics' across to a 'Prometheus: metrics' box, and both of those feed a 'Grafana query :3000' box on the right. A caption along the bottom reads: cpp, go, rust: each tags service.name = lsp-otel-
, so the three never collide in the shared stack." caption="Figure 38.1: lsp-otel's telemetry path: a per-request span tree plus two metrics, batched through the OTLP/HTTP exporter to the collector on :4318, fanned to Tempo and Prometheus, and read back through Grafana, which is exactly how the export is verified" -%}
+ {%- include excalidraw.html file="39-benchmark-pipeline" alt="Two bands. Top, benchlab real path: a warmup loop of W untimed calls to run_once feeding a timed loop of N iterations each timed with the monotonic clock into a raw samples array with a running checksum, splitting into a sort-and-percentile path producing min/median/p99/max and a parallel co_correct path producing co_p99_ns, co_n, and checksum, both landing in a printed table. Bottom, benchlab --lie: one untimed call straight to a wall-clock delta straight to a printed elapsed_ns, no percentiles because there is nothing to compute one from. A note reads same codec, same fixed workload: the harness, not the code under test, is what changes the story." caption="Figure 39.1: the same codec measured two ways, warmup, per-iteration timing, and coordinated-omission correction versus one unwarmed wall-clock sample" -%}
+ {%- include excalidraw.html file="39-coordinated-omission" alt="Two stacked bands. Top, benchlab's raw closed-loop timer: iterations 1 through 3 each a small box around 40 nanoseconds, iteration 4 an accent box labeled STALL page fault slash sched, 1057 nanoseconds, one sample. Bottom, what a fixed-rate caller would have seen: the same three small requests, the same stalling request 4, then two dashed ghost boxes labeled request 5 queued and request 6 queued at roughly 1018 and 979 nanoseconds, backfilled synthetic samples continuing down toward the expected interval floor. A dashed vertical line connects the two request 4 boxes labeled same stall. Captions read the raw p99 over a 1000-sample window buries the single stall sample, while the corrected co_p99 backfills the queued requests so co_p99_ns is always at least p99_ns." caption="Figure 39.2: one stall, recorded once by the closed loop (top) versus what a fixed-rate caller experiences (bottom); co_correct backfills the difference" -%}
+ {%- include excalidraw.html file="40-fastpath-datapath" alt="Two bands showing one echo round trip each. Top, the naive server: a client frame arrives at a server that may run on any of the online CPUs; per message it calls make_unique to allocate a fresh 64-byte heap buffer, then a blocking read(2) that puts the thread to sleep (an arrow labeled voluntary context switch loops out through a box labeled kernel scheduler and back, waking the thread when data arrives) then write(2) echoes and the heap buffer is freed. A counter reads voluntary_ctxt_switches roughly one per message. Bottom, the fastpath server: pinned by sched_setaffinity to a single CPU shown boxed and shaded, one reused 64-byte stack buffer for the whole connection, and a busy-poll loop that spins on non-blocking recv(2) over EAGAIN, never entering the scheduler, no context-switch arrow, the thread stays on its core. A counter reads voluntary_ctxt_switches roughly zero. A note reads same 64-byte frame, same round trip: the difference is who the thread has to ask for its CPU back." caption="Figure 40.1: one echo round trip, naive versus fastpath, a per-message heap buffer and a blocking read that sleeps through the scheduler, against one reused buffer on a pinned core that never gives the CPU up" -%}
+ {%- include excalidraw.html file="41-fleet-topology" alt="Two side-by-side VM boxes labeled systems-target (192.168.124.7) and systems-peer (192.168.124.95), each drawn as a large rounded rectangle representing one fleet. Inside each box, a top band labeled 'pmon: capability bounding set dropped, no_new_privs=1' encloses three child boxes side by side: chatterd, sysagent, and fwatch, each connected upward to a small pmon box with a supervise/restart arrow labeled 'fork+exec, restart on unexpected exit, forward SIGTERM'. The fwatch box additionally sits inside its own smaller nested box labeled 'Landlock ruleset: reads restricted to sandbox dir' with its enforced ABI number shown as a variable (7 on this guest's 6.19.10 kernel). A bidirectional arrow runs directly between the two chatterd boxes, labeled 'bridge: bridge@target / bridge@peer, DELIVER re-broadcast with from=NICK@node'. Below both VM boxes, a horizontal band represents the libvirt virbr0 bridge and its gateway address 192.168.124.1; two upward arrows from each guest's sysagent box pass through the gateway band labeled 'OTLP/HTTP :4318' and converge on a host-side box labeled 'LGTM stack (Grafana + Prometheus + Tempo)' sitting outside and above both VM boxes. A small caption near the gateway notes 'same route deploy-to-vm.sh forwards OTEL_EXPORTER_OTLP_ENDPOINT over.'" caption="Figure 41.1: the two-VM fleet: pmon supervises chatterd/sysagent/fwatch on each of systems-target and systems-peer inside a capability-bounding-set-drop boundary, fwatch additionally sandboxed under Landlock, chatterd bridged directly between the two guests, and both guests' sysagent exporting OTLP over the libvirt gateway to the host's LGTM stack" -%}
+ {%- include excalidraw.html file="41-supervise-lifecycle" alt="A state-machine diagram for one supervised child. A box labeled starting has an arrow to a box labeled up, annotated 'fork+exec succeeded, pid recorded, pmon: started service=NAME pid=PID printed'. From up, a dashed arrow labeled 'blocking wait (waitpid/cmd.Wait/nix::waitpid)' loops back into a decision diamond: 'shutting_down flag set?'. The no-branch leads to a box labeled restarting, annotated 'pmon: restart service=NAME attempt=N reason=... printed, restarts counter incremented, 300ms backoff sleep', with an arrow looping back up to starting. The yes-branch leads to a box labeled down. Off to the side, a separate small loop shows a clock icon labeled 'health tick every --health-interval-ms' with an arrow into a box printing 'pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0'. A third flow at the bottom shows SIGTERM/SIGINT arriving at a box labeled 'signal thread (sigwait / signal.Notify channel / SigSet::wait)', with an arrow to 'shutting_down.store(true)', then a fan-out arrow to 'kill(child_pid, SIGTERM) for every recorded pid', ending in a box labeled 'pmon: shutdown, exit 0' once every per-child supervisor has returned." caption="Figure 41.2: pmon's per-child supervise/restart lifecycle (starting → up → wait → restarting → up again on an unexpected exit), the health tick alongside it, and the SIGTERM → forward-to-children → shutdown path" -%}
+ {%- include excalidraw.html file="42-host-script-boundary" alt="Three stacked host-runtime boxes on the left: C++23/sol2 against PUC-Rio Lua 5.4, Go/gopher-lua against Lua 5.1, Rust/mlua against a vendored PUC-Rio Lua 5.4, each with a bidirectional arrow into a center box labeled policy.lua, one file, marked on_exit(info) going in and decision{action,delay_ms,reason} coming back. The policy.lua box sits inside a shaded sandbox-boundary band with a ghost box reading blocked: os, io, package, debug (never opened); nilled: load/loadfile/dofile/require/collectgarbage/loadstring. A host table box on the right connects to policy.lua with a bidirectional arrow labeled host.log / host.now_ms. A caption note reads Lua 5.1 (gopher-lua) vs Lua 5.4 (sol2, mlua) forces the shared subset: one policy.lua drives all three." caption="Figure 42.1: the host↔script boundary: three differently-built sandboxed Lua states, one shared policy.lua, and the host table it calls back into" -%}
+ {%- include excalidraw.html file="42-sighup-reload-lifecycle" alt="A horizontal sequence of six boxes inside a band labeled one pmon process: PID never changes: load v1 (base=500ms max_restarts=8), phase-1 decisions (web: stop; worker: restart 500ms to 1000ms), awaiting SIGHUP (a dashed ghost box), reload (fresh Lua state rebuilt, on_load logs v1.1.0), phase-2 decisions (api: restart 100ms; worker: escalate after 2 restarts), and exit 0. Above the awaiting-SIGHUP box, a ghost box shows the operator action: edit the policy file, then kill -HUP $(pgrep app), with an arrow into the flow. Below it, three sub-boxes show the three per-language consumption mechanisms: C++ signalfd + poll, Go signal.Notify(ch), Rust signal_hook flag (AtomicBool + 20ms poll). A bottom note reads: the process is not restarted; only a fresh, sandboxed Lua state is built in place and the decision logic changes underneath it." caption="Figure 42.2: the SIGHUP reload lifecycle: v1's decisions, a real SIGHUP consumed three different ways, a fresh Lua state built in place, and v1.1.0's tighter decisions, same process throughout" -%}
+ {%- include excalidraw.html file="43-proc-macro-pipeline" alt="Two horizontal bands. Top band: the app crate, main.rs's annotated source (#[derive(SysError)] enum PmonError, #[instrument] fn hash_chunk) feeding rustc, which packages a TokenStream and hands it off; on the right, rustc splices tokens back into the AST, typechecks, and compiles the result into the app binary (generated Display + Error impls, the instrumented hash_chunk). Bottom band: sysmacros, a separate proc-macro crate compiled first, containing two nodes: syn::parse_macro_input! turning the TokenStream into a real DeriveInput/ItemFn AST, and quote! emitting a new TokenStream2 (the Display impl or the wrapped function). Arrows cross the band boundary labeled TokenStream (macro input) going down and TokenStream (macro output) coming back up. A note at the bottom reads: sysmacros never runs at app runtime; it is pure tokens-in/tokens-out, executed only while rustc is compiling app." caption="Figure 43.1: the compile-time-only pipeline: annotated source becomes a TokenStream, the separately-compiled sysmacros crate parses it with syn and re-emits new tokens with quote, and rustc splices the result into app" -%}
+ {%- include excalidraw.html file="43-miri-bounds-check" alt="An 8-byte buffer BUF: [u8; 8] drawn as eight numbered byte boxes 0 through 7, labeled above as magic: u32 (bytes 0..4) and slots: u32 (bytes 4..8), followed by four dashed ghost boxes numbered 8 through 11 marked past the end (never allocated). Below the real bytes, a blue SOUND window box spans bytes 0 through 7, labeled: read_record(buf, oob=false), read_unaligned::(ptr.add(0)), window [0..8) fully inside the 8-byte alloc, cargo +nightly miri run: exit 0, no UB. Below and offset right, an amber window box spans bytes 4 through 11 (crossing from the real buffer into the ghost bytes), labeled: --oob-read: read_record(buf, oob=true), read_unaligned::(ptr.add(4)), window [4..12), 4 bytes past the 8-byte alloc, miri: only 4 bytes from the end of the allocation (main.rs:57). Dashed amber boundary markers point from the OOB window up into the byte row at the offset-4 and offset-12 boundaries." caption="Figure 43.2: the same 8-byte Record read from two offsets: the SOUND window [0..8) miri accepts, and the --oob-read window [4..12) that runs 4 bytes past the allocation, exactly what miri's diagnostic names" -%}
+ {%- include excalidraw.html file="44-gmp-scheduler" alt="A global run queue band at top holds overflow G's. Below, two P columns (P0.M0 running a spinner G, P1.M1 running a ticker G) each show a local run queue (256 slots + runnext), plus a box for P2 through P15 on this 16-CPU host. A dashed amber steal half arrow runs P1 into P0. To the right, a dashed sysmon box (no P, its own M) lists retake, forcegc, netpoll backstop, and the Go 1.25+ GOMAXPROCS recheck, with a solid amber SIGURG to asyncPreempt arrow into P0's running M. Caption: M:N: many G onto few M via P; GOMAXPROCS=1 still preempts since sysmon's signal needs no spare P." caption="Figure 44.1: the GMP model: G's queued locally (256 slots + runnext) and globally, GOMAXPROCS P's bound to M's, work-stealing between them, and sysmon's SIGURG reaching in from outside the P pool" -%}
+ {%- include excalidraw.html file="44-goroutines-not-threads" alt="Top band: 2000 goroutines blocked on os.Pipe reads feed one shared netpoller (epoll_pwait, per-fd pollDesc) labeled parked, no M; the OS-thread box reads threads=12, bound=80. Bottom band, the contrast: N goroutines calling syscall.Read on a raw, never-epoll-registered fd each block one M in read(2); the thread box shows the count climbing toward N via sysmon retake, handoffp, startm/newm. Caption: goroutines != threads: 2000 parked, 12 threads, bound 80." caption="Figure 44.2: the netpoller contrast: 2,000 goroutines parked in one shared epoll with no M held, against a raw blocking read that pins one M per blocked goroutine" -%}
+ {%- include excalidraw.html file="45-analysis-suite-landscape" alt="Three side-by-side boxes under a banner reading three philosophies for fleet-wide observation. Left: a browser icon into a Cockpit box, labeled zero-scripting web console, cockpit-ws multiplexing sessions over a socket answering on port 9090. Middle: a stap CLI box compiling into a kernel-module icon, labeled SystemTap: bcc's harder-to-build kernel-module-compiling sibling, needing matching kernel-devel. Right: a pmcd hub box with two labeled PMDA arms: pmda-openmetrics reading a file:// document and pmda-podman reading a podman.sock, both arrows curving back to two smaller ghost boxes captioned containers and OpenMetrics, showing PCP pulls metrics out of things this book already built. Caption: three ways to observe a fleet: click, script, or scrape." caption="Figure 45.1: three philosophies for fleet-wide observation: Cockpit's zero-scripting console, SystemTap's kernel-module scripting (bcc's harder-to-build sibling), and PCP pulling metrics out of two things this book already built" -%}
+ {%- include excalidraw.html file="45-pcp-build-gate" alt="A vertical three-stage pipeline. Stage one, builder, installs pcp packages, copies lsp45.prom, starts pmcd, and runs two ./Install calls against that live pmcd. An arrow labeled FROM builder AS smoke descends into stage two, smoke, which restarts pmcd and asserts both PMDAs are listed plus a real value 42 and a real kernel.all.load sample, gated by grep -q calls that can fail the whole podman build. A ghost, disconnected box beside stage two reads FROM builder AS runtime: the skipped path, dashed and crossed out, with a note that podman silently never builds a stage nothing chains onto. The real arrow, solid and amber, is labeled FROM smoke AS runtime and descends into stage three, runtime, which reuses the smoke stage's already-activated PMNS/pmcd.conf and carries only EXPOSE 44321 and the pmcd entrypoint. Caption: chaining onto smoke, not builder, is what forces the gate stage to build." caption="Figure 45.2: the PCP image's build gate: chaining runtime onto smoke forces the gate stage to build, and a PMDA that fails to register fails the build, not just the demo" -%}
+ {%- include excalidraw.html file="46-cmake-preset-graph" alt="A tree diagram. At the top, a hidden _base configure preset (generator Ninja, binaryDir sourceDir/build/presetName) fans out to six visible presets: release (RelWithDebInfo, GCC), debug (Debug, GCC), asan (Debug + -fsanitize=address,undefined), and ubsan (Debug + -fsanitize=undefined -fno-sanitize-recover=undefined) sit in one column; release-clang and debug-clang sit beside them, each drawn inheriting from release/debug respectively with an arrow labeled inherits plus a CMAKE_CXX_COMPILER=clang++ override. Each of the six feeds its own build/ directory box below it. A seventh preset, conan, also inherits _base but is drawn held apart in a dashed box labeled isolated: only reachable when TOOLBOX_ENABLE_CONAN=ON, needed by no other preset, so the default build never touches Conan. A caption note names which preset the parity gate uses (release vs release-clang), which the UBSan gate uses (ubsan), and which gdb uses (debug)." caption="Figure 46.1: the CMakePresets inheritance graph, one _base config fanning out to release/debug/asan/ubsan and their clang twins, each into build/${presetName}, with the Conan sub-target preset held apart so the default build needs no Conan" -%}
+ {%- include excalidraw.html file="46-toolbox-pipeline" alt="One source box, toolbox.cpp plus digest and smell translation units, splits into two build arrows (one through GCC, one through clang) each landing in its own build/ binary. Both binaries feed into a report command box, and their outputs converge on an amber diff box marked empty, the byte-identical parity gate, digest=0x481984990deee5ff. Below the source box, a static-analysis arm branches off into clang-tidy (flagging performance-unnecessary-value-param in smell.cpp) and clang-format --dry-run --Werror (clean on tracked sources, nonzero on a misformatted fixture). A separate runtime arm branches into the ubsan build, which traps the seeded signed-integer overflow with an abort icon, and the debug build, which feeds gdb plus a Python pretty-printer producing Digest(0x481984990deee5ff). Two dashed accelerator boxes, ccache and mold, sit off to the side labeled shown, not verified on this host." caption="Figure 46.2: the toolbox pipeline, one source built by two compilers into byte-identical output (the parity gate), with static tools (clang-tidy, clang-format) and runtime tools (UBSan, gdb pretty-printers) around it, and ccache/mold as optional accelerators" -%}
+ {%- include excalidraw.html file="47-go-toolchain-pipeline" alt="A go.mod box showing the go 1.26 and toolchain go1.26.5 directives feeds through a GOTOOLCHAIN resolution step into the go binary. From the go binary, two arrows fan out: one into an amber toolbox report box that computes an FNV-1a digest over a fixed embedded payload and prints digest=0x481984990deee5ff, the other into an amber go generate box that runs internal/gen against a hand-maintained tool table and diffs its output against the committed tools_generated.go, landing in a diff-empty gate. Both boxes are labeled go-native, verified offline on this host." caption="Figure 47.1: the Go toolchain and version-management flow: go.mod's go/toolchain directives resolve through GOTOOLCHAIN into the go binary, which drives the deterministic build (report digest) and the go-generate diff-clean gate" -%}
+ {%- include excalidraw.html file="47-go-tool-gates" alt="One Go module box splits into two gate lanes. The amber hard lane lists gofmt -l, go generate diff-clean, go vet -tags vetdemo naming a printf finding, go tool pprof -top naming hotLoop, and the FNV-1a digest: all shipping inside the go binary and verifying offline. A dashed ghost lane beside it lists gofumpt, golangci-lint, staticcheck, and delve as gated-if-present, govulncheck marked network required, and benchstat, all shown as reference and not verified on this host." caption="Figure 47.2: one module, two gate lanes: the go-native hard gates (gofmt, go generate, go vet, pprof, digest) that ship with the go binary and verify offline, beside the gated-if-present tools (gofumpt, golangci-lint, staticcheck, delve, govulncheck, benchstat) shown as reference" -%}
+ {%- include excalidraw.html file="48-rust-toolchain-pipeline" alt="A rust-toolchain.toml box listing channel 1.97.1, edition 2024, and the rustfmt and clippy components feeds into a rustup box that resolves the pinned toolchain and installs it if missing, which in turn drives a cargo/rustc 1.97.1 box. From cargo/rustc two arms fan out: a deterministic build arm running cargo run -- report, producing the zero-dependency toolbox binary and an FNV-1a digest over a fixed payload, landing in an amber digest gate asserting 0x481984990deee5ff and noting the same value appears in the C++ and Go chapters; and a cargo test arm running the #[cfg(test)] suite in digest.rs, landing in an amber cargo-test gate that asserts the same literal so any drift fails. Both gates are labeled rust-native and verified offline on this host." caption="Figure 48.1: the Rust toolchain and pinning flow: rust-toolchain.toml's channel and components resolve through rustup into cargo/rustc, which drives the deterministic build (report digest) and the cargo test gate" -%}
+ {%- include excalidraw.html file="48-rust-tool-gates" alt="One Rust crate box splits into two gate lanes. The amber hard lane lists the rustup pin at channel 1.97.1 and edition 2024, the FNV-1a digest gate at 0x481984990deee5ff, cargo fmt --check clean on tracked sources and nonzero on the fixture, cargo clippy -D warnings with --features lintbait naming needless_return, and cargo test asserting the digest literal: all running offline from the pinned toolchain alone because rustfmt and clippy ship as toolchain components. A dashed ghost lane beside it lists cargo-nextest, cargo-deny with its deny.toml, cargo-llvm-cov asserting a filename, and cargo-audit marked advisory-db slash network, plus cargo-watch, cargo-flamegraph, and sccache marked not gated, all shown as reference and not verified on this host." caption="Figure 48.2: one crate, two gate lanes: the pinned-toolchain hard gates (rustup pin, digest, cargo fmt, cargo clippy, cargo test) that need no cargo install, beside the gated-if-present tools (cargo-nextest, cargo-deny, cargo-llvm-cov, cargo-audit) and the shown-not-gated accelerators (cargo-watch, cargo-flamegraph, sccache)" -%}
+