diff --git a/_docs/00-outline.md b/_docs/00-outline.md index ef11ab5..a819fb3 100644 --- a/_docs/00-outline.md +++ b/_docs/00-outline.md @@ -2,11 +2,11 @@ title: "Outline" order: 0 part: "Setting Up" -description: "The full arc of this book — fourteen parts, forty-nine chapters, six programs that grow with you, three languages the whole way." +description: "The full arc of this book: fourteen parts, forty-nine chapters, six programs that grow with you, three languages the whole way." --- -This book teaches Linux systems programming three times at once — in modern -**C++23**, **Go**, and **Rust** — for developers who already program and want to +This book teaches Linux systems programming three times at once, in modern +**C++23**, **Go**, and **Rust**, for developers who already program and want to understand what their code asks of the kernel. Every hands-on chapter ships a runnable example implemented idiomatically in all three languages, presented in switchable code tabs, and every example doubles as a standalone demo. @@ -14,14 +14,14 @@ switchable code tabs, and every example doubles as a standalone demo. {% include excalidraw.html file="00-course-map" alt="Course map: four journey stages - Foundations, Core systems, Operating context, and Mastery - shown as bands containing the fourteen parts with their chapter ranges, and the six artifacts fwatch, shmkv, chatterd, pmon, bugfarm, and sysagent along the bottom, each with an arrow to the part where it grows." - caption="Figure 0.1 — the course at a glance." %} + caption="Figure 0.1: the course at a glance." %} Two threads run through the whole book and never stop: -- **Errors, three ways** — how `errno` becomes `std::expected`, a wrapped Go +- **Errors, three ways**: how `errno` becomes `std::expected`, a wrapped Go `error`, or a Rust `Result`, and what a *policy* for `EINTR`, `EAGAIN`, and partial failure looks like in each language. -- **The concurrency lens** — which of the three concurrency models each +- **The concurrency lens**: which of the three concurrency models each chapter's example leans on (`std::jthread` and atomics, goroutines and channels, ownership and `Send`/`Sync`), and why. @@ -34,7 +34,7 @@ artifacts that grow chapter by chapter: |---|---|---| | `pmon` | a process supervisor: fork/exec, signals, pidfds, privilege dropping, namespaces, pid 1 | 11 → 14, 18 → 19, 32, 34, 41 | | `fwatch` | a file watcher: polling → inotify/epoll → io_uring → sandboxed with Landlock | 07 → 10, 33, 41 | -| `shmkv` | an mmap-backed key-value store shared across processes — and across languages | 16, 20 | +| `shmkv` | an mmap-backed key-value store shared across processes, and across languages | 16, 20 | | `chatterd` | a peer-to-peer chat daemon: TCP → epoll at scale → UDP discovery → async → low-latency | 21 → 27, 34, 38, 40, 41 | | `sysagent` | a metrics agent reading /proc, /sys, and cgroups, exporting OpenTelemetry | 36 → 38, 41 | | `bugfarm` | a corpus of seeded bugs for learning gdb, valgrind, sanitizers, and eBPF tools | 28 → 31 | @@ -54,7 +54,7 @@ artifacts that grow chapter by chapter: | **Debugging** | gdb and remote debugging, valgrind vs sanitizers vs miri, eBPF observation tools, per-language toolbelts | 28–31 | | **Containers and Virtualization** | namespaces and cgroups, seccomp and Landlock, our programs in containers, KVM and the isolation spectrum | 32–35 | | **Observability** | /proc and /sys by hand, the USE method, OpenTelemetry into Grafana/Loki/Tempo/Mimir | 36–38 | -| **Performance and Low Latency** | benchmarking without self-deception, pinned-core fast paths, the capstone fleet | 39–41 | +| **Performance and Low Latency** | benchmarking without self-deception, pinned-core fast paths, the final project fleet | 39–41 | | **Deep Dives** | embedding Lua, Rust macros for systems code, the Go runtime | 42–44 | | **Appendices: Tooling** | Cockpit, SystemTap, and PCP; the C++, Go, and Rust toolboxes | 45–48 | | **Compendium: C++ Concurrency** | concurrency vs parallelism, pthreads, std threading, Boost.Thread, C++20 coroutines, Boost.Fiber, Boost.Asio, and a measured comparison of all six | 49–56 | @@ -62,17 +62,17 @@ artifacts that grow chapter by chapter: ## Where things run Examples run **locally** on your Fedora 44 machine unless they need root, kernel -observation, or a second host — those run in the disposable KVM lab +observation, or a second host; those run in the disposable KVM lab (`systems-target`, and `systems-peer` for two-host networking) that Chapter 2 provisions. Observability chapters use a local Podman LGTM stack (Grafana, Loki, Tempo, Mimir) plus Performance Co-Pilot from its container image. Nothing experimental ever runs un-snapshotted on your workstation. -> **Note** — every example section carries a verification status. A demo is +> **Note**: every example section carries a verification status. A demo is > marked *verified* only when it has produced its claimed observable effect on -> the stated environment — never merely because it compiled. That claim is +> the stated environment, never merely because it compiled. That claim is > enforced by machinery, not habit: the runner, `scripts/test-all-examples.py`, > builds each example in all three languages and then executes the example's own > Lua `verify.lua` harness, which must observe the behavior to report a PASS. If > the VM lab or the LGTM stack an example depends on is down, the runner reports -> a SKIP — never a false PASS. +> a SKIP, never a false PASS. diff --git a/_docs/01-prerequisites.md b/_docs/01-prerequisites.md index 7992212..e89897a 100644 --- a/_docs/01-prerequisites.md +++ b/_docs/01-prerequisites.md @@ -2,12 +2,12 @@ title: "Prerequisites and toolchains" order: 1 part: "Setting Up" -description: "Prepare a Fedora 44 host for systems programming in C++23, Go, and Rust — pinned toolchains, one check script that verifies everything, and a first tri-language example that asks the kernel who you are." +description: "Prepare a Fedora 44 host for systems programming in C++23, Go, and Rust, with pinned toolchains, one check script that verifies everything, and a first tri-language example that asks the kernel who you are." duration: 60 minutes --- Every chapter in this book ends with something you can run, and almost every -one of them shows the same program three times — in C++23, Go, and Rust. That +one of them shows the same program three times: in C++23, Go, and Rust. That only works if all three toolchains on your machine agree with the ones the book was written against, which is why this chapter is less "install these packages" and more "here is *why* each pin exists, and here is one script that @@ -22,53 +22,53 @@ every later example reuses. {% include excalidraw.html file="01-toolchain-landscape" - alt="Three toolchain stacks — GCC/CMake/gdb for C++, go1.26.5/go build/dlv for Go, rustc/cargo/clippy for Rust — each converging through its own demo.sh onto one shared demo.sh runner contract." - caption="Figure 1.1 — three toolchains, one demo.sh contract shared by every example in the book" %} + alt="Three toolchain stacks: GCC/CMake/gdb for C++, go1.26.5/go build/dlv for Go, and rustc/cargo/clippy for Rust, each converging through its own demo.sh onto one shared demo.sh runner contract." + caption="Figure 1.1: three toolchains, one demo.sh contract shared by every example in the book" %} -> **Tools used** — `gcc`, `cmake`, `ninja`, `go`, `cargo`, `rustup`, `strace` -> (all host — the toolchains are this chapter's subject). Everything here is +> **Tools used**: `gcc`, `cmake`, `ninja`, `go`, `cargo`, `rustup`, `strace` +> (all host; the toolchains are this chapter's subject). Everything here is > checked by `scripts/check-host.sh`. ## One host, three toolchains -The book targets **Fedora 44** deliberately. Systems programming is the one +The book targets **Fedora 44** because systems programming is the one domain where "any recent Linux" is not good enough: chapters later in the book read kernel tracepoints, load eBPF programs, and poke at cgroup files whose -layout shifts between kernel versions. Fedora ships close to upstream — the -reference host here runs kernel `7.1` — so what you observe matches +layout shifts between kernel versions. Fedora ships close to upstream: the +reference host here runs kernel `7.1`, so what you observe matches what current kernel documentation describes. It also ships **GCC 16.1.1** as the system compiler, which matters immediately: the C++ examples use C++23's `std::println` and `std::expected`, and those need GCC 14 or newer. On Fedora -44 you get that for free; on an LTS distribution you would be building a +44 that already works; on an LTS distribution you would be building a compiler first. Go and Rust are pinned differently, and the difference is worth understanding because it decides *where* the pin lives: - **Go** pins inside the module. `examples/01-hello-syscalls/go/go.mod` - declares `toolchain go1.26.5`, so any reasonably modern `go` binary — - including Fedora's packaged one — will download and use exactly go1.26.5 on + declares `toolchain go1.26.5`, so any reasonably modern `go` binary, + including Fedora's packaged one, will download and use exactly go1.26.5 on first build. The pin travels with the code; your system Go only bootstraps it. - **Rust** pins via `rust-toolchain.toml` (`channel = "1.97.1"`), but that file is honored by **rustup**, not by Fedora's packaged `rustc`. Install - Rust through rustup or the pin silently does nothing — which is exactly the + Rust through rustup or the pin silently does nothing, which is exactly the kind of drift that later makes an example "work on my machine" only. C++ has no equivalent in-tree pin mechanism, so it pins by *floor* instead: the check script requires GCC ≥ 14 and CMake ≥ 3.25 (the first version with the `CMakePresets.json` v6 schema the examples use). Clang is installed as a -first-class alternate — every C++ example carries `release-clang` / +first-class alternate: every C++ example carries `release-clang` / `debug-clang` presets, because two compilers disagreeing about your code is the cheapest static analysis you will ever get. -## What `scripts/check-host.sh` verifies — and why +## What `scripts/check-host.sh` verifies, and why Rather than a page of `dnf install` lines to transcribe, the repo ships -`scripts/check-host.sh`: run it, read the table, fix the `[fail]` rows. Two -design choices in the script are worth noticing before the package list. +`scripts/check-host.sh`: run it, read the table, fix the `[fail]` rows. The +script makes two design choices that matter before the package list. -First, it deliberately does **not** use `set -e`: +First, it does **not** use `set -e`: ```bash # Deliberately no -e: failing checks are recorded in the table, not @@ -79,42 +79,42 @@ set -uo pipefail A host-check script that dies on the first missing tool tells you one problem per run; this one accumulates every result into a table and exits non-zero only at the end, so one pass gives you the full shopping list. Second, it -distinguishes **hard requirements** (`fail` — gates the exit code) from -**soft** ones (`warn` — informational). That split encodes what the book +distinguishes **hard requirements** (`fail`, which gates the exit code) from +**soft** ones (`warn`, informational). That split encodes what the book actually needs versus what is merely convenient. The hard requirements, and the reason each is hard: -- **`gcc`/`g++` ≥ 14, `cmake` ≥ 3.25, `clang`, `ninja`** — the C++ floor +- **`gcc`/`g++` ≥ 14, `cmake` ≥ 3.25, `clang`, `ninja`**: the C++ floor described above. Ninja is required because every preset in `CMakePresets.json` sets `"generator": "Ninja"`. -- **`go`** — any version, because of the toolchain-directive trick; the +- **`go`**: any version, because of the toolchain-directive trick; the script only *warns* if it is not already 1.26.x, noting the directive "auto-downloads go1.26.5 on first build." -- **`rustup` + `rustc` + `cargo`** — a hard fail if rustup is absent, with +- **`rustup` + `rustc` + `cargo`**: a hard fail if rustup is absent, with the hint "do not use Fedora's system rust". If rustup is present but on a different version, it is only a warning: `rust-toolchain.toml` installs 1.97.1 on first build anyway. -- **`lua` ≥ 5.4 (or LuaJIT) and `python3`** — Lua drives the per-example +- **`lua` ≥ 5.4 (or LuaJIT) and `python3`**: Lua drives the per-example `verify.lua` checks that CI runs; the harness accepts LuaJIT locally (the check script records a warning recommending lua 5.4 for parity with CI). Python drives the site's tooling (including the diagram generator that produced Figure 1.1). -- **`podman` ≥ 5.0** — the observability chapters run the LGTM stack +- **`podman` ≥ 5.0**: the observability chapters run the LGTM stack (Grafana, Loki, Tempo, Mimir) as containers; the reference host has 5.8.4. -- **`virsh`, `virt-install`, `qemu-img`, `cloud-localds`, and `/dev/kvm`** — +- **`virsh`, `virt-install`, `qemu-img`, `cloud-localds`, and `/dev/kvm`**: the KVM lab that the next chapter provisions. `/dev/kvm` missing means virtualization is disabled in firmware, which no package can fix, so the script checks the device node itself. -- **`git` and `gh`** — the repo workflow. +- **`git` and `gh`**: the repo workflow. Hard rows also cover the debug and observation tooling the cross-checks lean on from here forward: **gdb, valgrind, perf, strace, ltrace**, and the -**sanitizer runtimes** (`libasan`/`libtsan` — without them the `asan` and +**sanitizer runtimes** (`libasan`/`libtsan`; without them the `asan` and `tsan` presets compile but cannot link). The soft rows: **Conan 2.x** (only some C++ demos pull dependencies through it) and the Go lint/debug trio **golangci-lint, staticcheck, dlv** (CI runs them; locally they are -nice-to-have). Membership in the `libvirt` group is also warn-only — +nice-to-have). Membership in the `libvirt` group is also warn-only: everything works with `sudo`, it is just tedious without the group. There is no Ruby row: the site builds and deploys in CI, so nothing local depends on Jekyll. @@ -165,8 +165,8 @@ run() { The `TARGET` branch is the part that pays off later: set `TARGET` to a lab VM name and `run` deploys the host-built binary to the guest via `scripts/lab/deploy-to-vm.sh` instead of running it locally. Privileged -examples in later chapters need exactly this — build on the host, execute on -a disposable kernel — and because the contract is uniform, `TARGET=… ./demo.sh +examples in later chapters need exactly this: build on the host, execute on +a disposable kernel. Because the contract is uniform, `TARGET=… ./demo.sh go run` means the same thing in every chapter. Alongside the demo sits `verify.lua`, which CI runs once per language: it executes `./demo.sh $LSP_LANG run` and asserts the output matches the Lua pattern `pid %d+ on Linux`. That @@ -176,7 +176,7 @@ compile. ## How the code works The program itself is three files, one per language, each printing exactly one -line — `pid on ()` — using two of the +line: `pid on ()`, using two of the oldest syscalls in the catalog: `getpid(2)` and `uname(2)`. {% include codetabs.html langs="C++|Go|Rust" %} @@ -265,11 +265,11 @@ fn main() { ``` All three call `uname(2)`, which fills a caller-supplied `utsname` struct with -fixed-size, NUL-terminated byte arrays — the kernel writes into *your* memory; +fixed-size, NUL-terminated byte arrays. The kernel writes into *your* memory; there is no allocation and no pointer handed back. Each language then has to answer the same two questions: *how do I get at those bytes?* and *what do I -do if the call fails?* — and the three answers are a preview of the error- -handling policy the whole book runs on. +do if the call fails?* The three answers preview the error-handling policy +the whole book runs on. **C++** wraps the raw call in `system_info()`, which returns `std::expected`. The kernel reports failure as `-1` @@ -277,23 +277,23 @@ plus `errno`; the wrapper immediately converts that into `std::error_code{errno, std::system_category()}` so the error is a value, not ambient global state that the next libc call could overwrite. `[[nodiscard]]` makes ignoring the result a warning, and `main` branches on the `expected` -rather than on `errno`. The `::uname` scope qualifier is not decoration — the +rather than on `errno`. The `::uname` scope qualifier is not decoration: the struct and the function are *both* named `uname`, and `::` disambiguates the global C function from the type. The struct's `char[]` fields print directly because `std::println` treats them as C strings. -**Go** goes through `golang.org/x/sys/unix` (v0.47.0) — the maintained home +**Go** goes through `golang.org/x/sys/unix` (v0.47.0), the maintained home for raw syscall bindings since the standard `syscall` package was frozen. `unix.Uname` returns an ordinary `error`, which `helloLine` wraps with `%w` so callers can still `errors.Is`/`errors.As` through the added context. The `Utsname` fields arrive as `[65]byte` arrays, and `unix.ByteSliceToString` -does the careful conversion: scan to the first NUL, slice, convert — without it +does the careful conversion: scan to the first NUL, slice, convert. Without it you would print 65 bytes of name plus garbage. **Rust** uses `rustix` (1.1.4), a safe binding layer that makes syscalls directly rather than through libc. Note what is *missing*: no `Result`, no `unwrap`. `rustix::system::uname()` is typed as infallible because on Linux -`uname` cannot fail when handed a valid struct the caller owns — the binding +`uname` cannot fail when handed a valid struct the caller owns; the binding encodes the manpage's failure contract in the signature. The returned fields are `CStr`-like views, and `to_string_lossy()` handles the (theoretical) non-UTF-8 case instead of panicking. Three languages, three defaults: C++ @@ -312,9 +312,9 @@ Check the host first, then run the whole example: The check script prints its `[ ok ]`/`[warn]`/`[fail]` table and ends with a summary line; on the reference host every required check passes. The demo then -builds and runs each language in turn — CMake configures the `release` preset +builds and runs each language in turn: CMake configures the `release` preset into `cpp/build/release/`, `go build` resolves go1.26.5 and `x/sys` on first -run, `cargo` installs the pinned 1.97.1 toolchain if needed — and each binary +run, and `cargo` installs the pinned 1.97.1 toolchain if needed. Each binary prints one line of the same shape: ``` @@ -322,14 +322,14 @@ pid 2520094 on Linux 7.1 (x86_64) ``` Three lines, three different PIDs, identical formatting: that is the whole -point. If any language prints something else — or the pattern -`pid %d+ on Linux` fails to match — your toolchain differs from the pin, and +point. If any language prints something else, or the pattern +`pid %d+ on Linux` fails to match, your toolchain differs from the pin, and `check-host.sh` will usually tell you where. ## Cross-check: watch the syscalls happen -The source *claims* it calls `uname(2)` and `getpid(2)`. `strace` — which -intercepts every syscall a process makes — lets you verify that independently +The source *claims* it calls `uname(2)` and `getpid(2)`. `strace`, which +intercepts every syscall a process makes, lets you verify that independently of any language runtime (`sudo dnf install -y strace` if you lack it). On the C++ binary, trimmed to the `uname` call: @@ -353,13 +353,13 @@ pid 2520267 on Linux 7.1 (x86_64) +++ exited with 0 +++ ``` -Same single `uname`, but four extra threads attached before `main` even ran — +Same single `uname`, but four extra threads attached before `main` even ran: the Go scheduler's worker threads, created with `clone(2)`. Run `strace -c` on the C++ binary and `strace -cf` on the Go one (the `-f` follows those threads) and the difference gets numeric: on this host the C++ binary made 64 syscalls total, while the Go binary made 217 across all threads, including 114 `rt_sigaction` calls (the runtime installing its signal handlers) and 4 -`clone`s. Neither number is a problem — but it is the first concrete sighting +`clone`s. Neither number is a problem, but it is the first concrete sighting of a theme that recurs all book long: a language runtime is itself a systems program, and `strace` does not let it hide. @@ -368,20 +368,20 @@ program, and `strace` does not let it hide. If you use CLion, opening `examples/01-hello-syscalls/cpp` should auto-import the `CMakePresets.json` presets (release, debug, asan, and the clang pair) as build configurations, and the Rust and Go plugins cover the other two -directories. Everything in the book, though, is driven from the terminal — no +directories. Everything in the book, though, is driven from the terminal; no step ever requires the IDE. -

unverified — the CLion preset auto-import and plugin flow has not yet been exercised for this repo; confirm the five presets appear and that the Rust plugin picks up rust-toolchain.toml.

+

unverified: the CLion preset auto-import and plugin flow has not yet been exercised for this repo; confirm the five presets appear and that the Rust plugin picks up rust-toolchain.toml.

## What you learned - Fedora 44 gives current-kernel observability and a C++23-capable system GCC; Go pins itself via the `go.mod` toolchain directive, Rust via - `rust-toolchain.toml` — but only under rustup. + `rust-toolchain.toml`, but only under rustup. - `scripts/check-host.sh` separates hard requirements from soft ones and reports all failures in one pass instead of dying on the first. -- Every example follows one `demo.sh` contract — `build`, `run`, and - `TARGET=` for deploy-to-guest — and is *verified* only by its observable +- Every example follows one `demo.sh` contract: `build`, `run`, and + `TARGET=` for deploy-to-guest. It is *verified* only by its observable output, which `verify.lua` pattern-matches. - `strace` confirms what a binary actually asks the kernel for, and already reveals runtime differences: 64 syscalls for the C++ hello, 217 for Go's. @@ -392,4 +392,4 @@ snapshotted kernel instead of yours. --- -

verified — runner table PASS for cpp, go, and rust on the Fedora 44 host (kernel 7.1, GCC 16.1.1, go1.26.5, rustc 1.97.1): each binary printed pid <PID> on Linux 7.1 (x86_64), and the strace excerpts above are real trimmed output from those binaries. The CLion section remains unverified as noted.

+

verified: runner table PASS for cpp, go, and rust on the Fedora 44 host (kernel 7.1, GCC 16.1.1, go1.26.5, rustc 1.97.1): each binary printed pid <PID> on Linux 7.1 (x86_64), and the strace excerpts above show trimmed output captured from those binaries. The CLion section remains unverified as noted.

diff --git a/_docs/02-lab-setup.md b/_docs/02-lab-setup.md index 1800b55..ccf60e0 100644 --- a/_docs/02-lab-setup.md +++ b/_docs/02-lab-setup.md @@ -2,16 +2,16 @@ title: "The KVM lab" order: 2 part: "Setting Up" -description: "Provision two disposable Fedora KVM guests with cloud-init — an observation-first toolset, snapshot/revert discipline, and a deploy script whose output proves your code ran on the guest kernel, not yours." +description: "Provision two disposable Fedora KVM guests with cloud-init: an observation-first toolset, snapshot/revert discipline, and a deploy script whose output proves your code ran on the guest kernel, not yours." duration: 45 minutes --- Chapter 1 proved that three toolchains build and run the same program on your workstation. This chapter builds the other half of the book's infrastructure: -two disposable Fedora VMs — `systems-target` and `systems-peer` — that absorb +two disposable Fedora VMs (`systems-target` and `systems-peer`) that absorb everything you should never do to the machine you work on. A systems-programming book keeps asking you to run code as root, watch the kernel from the inside, -fill filesystems, kill process trees, and misconfigure networks on purpose. +fill filesystems, kill process trees, and misconfigure networks. The lab is where all of that happens, and one snapshot command makes any mistake reversible in seconds. @@ -22,36 +22,36 @@ destroys the guests by name. {% include excalidraw.html file="02-lab-topology" alt="Lab topology: a host band with libvirtd/KVM, the ~/.cache/systems-lab image cache holding the pristine Fedora Cloud Base image plus one qcow2 overlay per guest, and the scripts/lab directory; below it, the libvirt NAT network 192.168.124.0/24 containing systems-target (4 GB, .7) and systems-peer (2 GB, .95), both on kernel 6.19.10. An amber deploy arrow runs from the scripts to systems-target; a dashed line links the two guests for later two-host chapters; a note marks the lab-ready snapshot taken on both." - caption="Figure 2.1 — the systems-* lab: cached base image + per-VM overlays on the host, two guests on libvirt NAT, one amber deploy path" %} + caption="Figure 2.1: the systems-* lab, cached base image + per-VM overlays on the host, two guests on libvirt NAT, one amber deploy path" %} -> **Tools used** — `virt-install`, `qemu-img`, `cloud-localds`, `virsh`, `ssh` +> **Tools used**: `virt-install`, `qemu-img`, `cloud-localds`, `virsh`, `ssh` > (host). Everything here is checked by `scripts/check-host.sh`; the guest > toolset itself is installed by cloud-init. ## Why a disposable lab -Three kinds of chapters force the issue. **Privileged demos** — capabilities, -namespaces, cgroup writes, mount tricks — need root, and root that fat-fingers -a path on your workstation is a bad afternoon. **Kernel observation** — bpftrace -one-liners, perf profiles, the BCC tools — attaches instrumentation to a live +Three kinds of chapters force the issue. **Privileged demos** (capabilities, +namespaces, cgroup writes, mount tricks) need root, and root that fat-fingers +a path on your workstation is a bad afternoon. **Kernel observation** (bpftrace +one-liners, perf profiles, the BCC tools) attaches instrumentation to a live kernel, and you want that kernel to be one you can reboot without losing your -editor. **Destructive demos** — OOM kills, disk-full behavior, deliberately -broken network configs — only teach if you actually break something, which +editor. **Destructive demos** (OOM kills, disk-full behavior, broken network +configs) only teach if you actually break something, which means you need something safe to break. A VM answers all three at once, and adds the property no container gives you: **its own kernel**. Namespaces share the host kernel, so a container can't show you kernel-version drift, can't be probed as an isolated kernel, and can't be snapshot-reverted with its kernel state. The lab guests run their own -`6.19.10-300.fc44` kernel while the host runs `7.1` — a visible +`6.19.10-300.fc44` kernel while the host runs `7.1`, a visible difference this chapter turns into its cross-check. That said, the VM is not the default. The book's decision rule, applied at the top of every example: > **Run locally unless the demo needs root, needs to observe the kernel, or -> needs two hosts.** Everything else — ordinary syscalls, file I/O, unprivileged -> process work — runs faster and debugs easier on your workstation. +> needs two hosts.** Everything else (ordinary syscalls, file I/O, unprivileged +> process work) runs faster and debugs easier on your workstation. ## Host prerequisites @@ -67,7 +67,7 @@ before doing anything: `virt-install` defines and boots the domain, `qemu-img` builds overlay disks, `cloud-localds` (from `cloud-utils`) packs the cloud-init seed, and membership -in the `libvirt` group lets you talk to the system libvirt instance — every +in the `libvirt` group lets you talk to the system libvirt instance: every script exports `LIBVIRT_DEFAULT_URI=qemu:///system` so the guests live in the system instance, not a per-user session that vanishes with your login. @@ -75,12 +75,12 @@ system instance, not a per-user session that vanishes with your login. `scripts/lab/provision-vm.sh` is a straight pipeline: cached base image → per-VM overlay → cloud-init seed → `virt-install`. Each stage exists for a -reason worth spelling out. +specific reason, detailed below. **The cached base image.** The script downloads `Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2` once into `~/.cache/systems-lab/` and never modifies it. The version is pinned in the -script ("Pin the exact image you tested; bump deliberately") — an unpinned +script ("Pin the exact image you tested; bump deliberately"): an unpinned "latest" would silently change the guest kernel under the whole book. **The overlay disk.** Instead of copying the 400 MB base per guest, the script @@ -91,12 +91,12 @@ qemu-img create -f qcow2 -F qcow2 -b "$CACHE_DIR/$BASE_IMG" "$OVERLAY" "${DISK_G ``` The overlay records only the blocks the guest changes; the base stays pristine -and shared. This is what makes the guests *disposable* — destroying and +and shared. This is what makes the guests *disposable*: destroying and re-provisioning a guest deletes one thin overlay and rebuilds it in seconds, with no re-download. **The cloud-init seed.** Fedora Cloud Base has no users, no packages, no SSH -access — cloud-init is how it becomes yours on first boot. The script fills +access; cloud-init is how it becomes yours on first boot. The script fills two placeholders (`__HOSTNAME__`, `__SSH_PUBKEY__`) in `scripts/lab/cloud-init/user-data.tmpl` with `sed`, then `cloud-localds` packs the result into a tiny seed image that is attached as a CD-ROM. On first boot @@ -109,43 +109,43 @@ with `--cpu host-passthrough` so the guest sees your real CPU features, virtio disk and network for near-native I/O, and `--graphics none --noautoconsole` because everything from here on is SSH. The script also refuses to clobber: an existing domain of the same name is an error, and the -fix is an explicit `./destroy-vm.sh ` — rebuilds are deliberate, never +fix is an explicit `./destroy-vm.sh `: rebuilds are deliberate, never accidental. -## What goes in the guest — and what deliberately doesn't +## What goes in the guest, and what doesn't The package list in `scripts/lab/cloud-init/user-data.tmpl` *is* the book's tooling philosophy, so read it as an argument, not a manifest. The first block is **observation tooling**: `bcc-tools`, `bpftrace`, `bpftool`, `libbpf-tools` -(precompiled CO-RE tools — `opensnoop`, `execsnoop`, `biolatency`), `perf`, +(precompiled CO-RE tools: `opensnoop`, `execsnoop`, `biolatency`), `perf`, `strace`, `ltrace`, `gdb` plus `gdb-gdbserver`, `valgrind`, `sysstat`. That block is the point of the lab: this book uses eBPF **as tooling, not as a build target**, which is why there is *no* clang/llvm/libbpf-devel eBPF build -chain in the guest — the template says so in a comment right above the list. +chain in the guest; the template says so in a comment right above the list. You will trace with bpftrace constantly; you will not compile BPF programs. The second block is **per-language toolchains** (`gcc-c++`, `clang`, `cmake`, -`ninja-build`, `golang`, `rust`/`cargo`) so demos that must compile in-VM — -privileged code, namespace experiments, glibc-parity fallbacks — can. The rest +`ninja-build`, `golang`, `rust`/`cargo`) so demos that must compile in-VM +(privileged code, namespace experiments, glibc-parity fallbacks) can. The rest is `podman`/`crun` for the container chapters, networking tools (`tcpdump`, `socat`, `nmap-ncat`) for the two-host chapters, and `cockpit`, whose socket cloud-init enables so the web console is reachable at -`https://:9090` — a useful visual second opinion on guest load. +`https://:9090`, a useful visual second opinion on guest load. The `runcmd` section ends with the **readiness stamp**: it writes `bpftrace --version` and `uname -r` into `/var/log/lab-ready`. That file is -the lab's contract — if it exists and names a bpftrace, cloud-init finished +the lab's contract: if it exists and names a bpftrace, cloud-init finished and the observation tooling works. Every later chapter can check one file instead of guessing whether provisioning completed. Packages a single demo needs (fio, rt-tests, numactl) stay *out* of the base -on purpose: install them on demand, then re-snapshot to bake them in. A lean +by default: install them on demand, then re-snapshot to bake them in. A lean base keeps provisioning fast and the snapshot meaningful. ## Snapshots: the discipline that makes "destructive" safe -`scripts/lab/snapshot-vm.sh ` captures a snapshot named `lab-ready` — -RAM *and* disk, because the guest is running, so a revert restores it +`scripts/lab/snapshot-vm.sh ` captures a snapshot named `lab-ready`: RAM +*and* disk, because the guest is running, so a revert restores it *running*, tools warm, in seconds. The discipline has two halves: - **Snapshot immediately after provisioning** (and after any deliberate @@ -155,7 +155,7 @@ RAM *and* disk, because the guest is running, so a revert restores it "did that experiment leave state behind?" doubt. Re-running `snapshot-vm.sh` replaces the same-named snapshot, so refreshing -the baseline is idempotent. `destroy-vm.sh` is the nuclear option — it +the baseline is idempotent. `destroy-vm.sh` is the nuclear option: it undefines the domain and deletes its overlay and seed, leaving only the cached base image, from which a fresh guest is minutes away. @@ -165,13 +165,13 @@ Day to day you touch two scripts. `scripts/lab/lab-up.sh` starts whichever of `systems-target`/`systems-peer` are defined but stopped, waits for DHCP leases, and prints IPs; `eval "$(./lab-up.sh --export)"` drops `TARGET_IP` and `PEER_IP` straight into your shell for scripted tests. `scripts/lab/lab-down.sh` -sends ACPI shutdowns, waits up to 60 seconds, and — deliberately — deletes +sends ACPI shutdowns, waits up to 60 seconds, and by design deletes nothing; disks and snapshots survive, and `FORCE=1` hard-stops a guest that ignores ACPI. `vm-ip.sh` answers "what address did libvirt lease this guest?" by reading the DHCP lease (guest-agent fallback). One host-specific note: on this machine the libvirt `default` network is `192.168.124.0/24` with gateway -`192.168.124.1` — not the stock `192.168.122.0/24` you'll see in most libvirt -docs — so never hardcode an address; always ask `vm-ip.sh`. +`192.168.124.1` (not the stock `192.168.122.0/24` you'll see in most libvirt +docs), so never hardcode an address; always ask `vm-ip.sh`. ## Deploying host builds @@ -179,10 +179,10 @@ docs — so never hardcode an address; always ask `vm-ip.sh`. reaches the guest: it resolves the IP, `scp`s the binary to `/home/fedora/`, marks it executable, and `exec`s it over `ssh -t` so Ctrl-C reaches the remote process. Two toggles matter. `SUDO=1` runs the binary under -sudo for the demos that genuinely need root — the default is unprivileged, +sudo for demos that need root; the default is unprivileged, matching the decision rule. And if `OTEL_ENDPOINT` is set, the script forwards it as `OTEL_EXPORTER_OTLP_ENDPOINT` through `env` explicitly, because `sudo` -strips the environment — without that, a privileged guest binary would fall +strips the environment; without that, a privileged guest binary would fall back to its own localhost and its telemetry would never reach the host's observability stack. This works because host builds and the guest are both Fedora 44. That parity holds when the guests are provisioned and lapses as the @@ -259,7 +259,7 @@ links against the guest's own runtime. ## Build, run, observe -Provision both guests — target at the 4 GB default, peer smaller: +Provision both guests, target at the 4 GB default, peer smaller: ```bash [host]$ cd scripts/lab @@ -291,7 +291,7 @@ lab: built once, reverted forever. ## Cross-check: prove the code runs *there* -Chapter 1's example prints the kernel it runs on — which makes it a perfect +Chapter 1's example prints the kernel it runs on, which makes it a perfect independent check that `deploy-to-vm.sh` really executes on the guest rather than, say, quietly running the local binary. Run it both ways: @@ -307,24 +307,24 @@ pid on Linux 6.19.10-300.fc44.x86_64 (x86_64) Same binary, two different kernel strings: `7.1` is your host, `6.19.10-300.fc44` is the guest. The example's per-language `demo.sh` wires -this up — with `TARGET` set, its `run` step calls +this up: with `TARGET` set, its `run` step calls `scripts/lab/deploy-to-vm.sh "$TARGET" ` instead of executing locally, -so every example in the book gets remote execution for free. The Go build -behaves identically (`TARGET=systems-target ./demo.sh go`); both were run on -this lab and printed the guest kernel string. The kernel version in the output -is the observable proof — a claim like "it deployed" is worthless next to the -guest's own `uname` in your terminal. +so every example in the book gets remote execution with no extra wiring. The +Go build behaves identically (`TARGET=systems-target ./demo.sh go`); both +were run on this lab and printed the guest kernel string. The kernel version +in the output is the observable proof; a claim like "it deployed" is +worthless next to the guest's own `uname` in your terminal. ## What you learned -- The lab exists for exactly three reasons — root, kernel observation, and - two-host demos — and the decision rule says everything else runs locally. +- The lab exists for exactly three reasons: root, kernel observation, and + two-host demos; the decision rule says everything else runs locally. - Provisioning is a cheap pipeline: pinned cached base image → qcow2 overlay per guest → cloud-init seed → `virt-install`; guests are disposable because only the overlay is theirs. - The guest toolset is observation-first (bcc/bpftrace/perf/gdbserver), with - in-VM toolchains as a fallback and deliberately no eBPF build chain. -- Snapshot `lab-ready` after provisioning, revert after destructive demos — + in-VM toolchains as a fallback and no eBPF build chain. +- Snapshot `lab-ready` after provisioning, revert after destructive demos, and trust `/var/log/lab-ready` plus a guest kernel string in your output, not assumptions. - **Host/guest parity holds at provisioning and drifts afterwards.** A snapshot @@ -339,4 +339,4 @@ stack**, so every later demo has somewhere to send its telemetry. --- -

verified — both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); /var/log/lab-ready read back bpftrace v0.24.2 and kernel 6.19.10-300.fc44 on both; lab-ready snapshots exist on both; and deploy-to-vm.sh ran the host-built C++ and Go binaries on systems-target, printing the guest kernel string 6.19.10-300.fc44.x86_64 versus 7.1 locally. The “When the guest falls behind the host” section was added later and is verified separately, on the same lab: the host’s libstdc++ reached 16.2.1-2.fc44 (libstdc++.so.6.0.36, max GLIBCXX_3.4.36) while both guests sat at 16.1.1-2.fc44 (libstdc++.so.6.0.35, max GLIBCXX_3.4.35) behind snapshots taken before the update, and every host-built C++ binary then failed on the guest with the quoted version `GLIBCXX_3.4.36' not found. A full-matrix run reproduced the asymmetry the section describes — six VM examples failing in C++ while Go and Rust passed on the same guest in the same run — and the quoted diagnostic commands, the dnf fix, and the re-snapshot were all run here: after revert-vm.sh systems-target lab-ready the guest reported GLIBCXX_3.4.36 and test-all-examples.py --mode vm --lang cpp reported the quoted 8 passed, 0 failed, 0 skipped, including the two-guest 41-capstone-fleet. systems-peer was confirmed to carry the same drift while showing no symptom. Not exercised: unverified — the GLIBC_ form of the symbol check is stated by analogy and was not the failure encountered here, which was GLIBCXX_ only; and building inside the guest with the in-VM toolchains is named as the standing fallback but is not exercised by any example in this book.

+

verified: both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); /var/log/lab-ready read back bpftrace v0.24.2 and kernel 6.19.10-300.fc44 on both; lab-ready snapshots exist on both; and deploy-to-vm.sh ran the host-built C++ and Go binaries on systems-target, printing the guest kernel string 6.19.10-300.fc44.x86_64 versus 7.1 locally. The “When the guest falls behind the host” section was added later and is verified separately, on the same lab: the host’s libstdc++ reached 16.2.1-2.fc44 (libstdc++.so.6.0.36, max GLIBCXX_3.4.36) while both guests sat at 16.1.1-2.fc44 (libstdc++.so.6.0.35, max GLIBCXX_3.4.35) behind snapshots taken before the update, and every host-built C++ binary then failed on the guest with the quoted version `GLIBCXX_3.4.36' not found. A full-matrix run reproduced the asymmetry the section describes: six VM examples failing in C++ while Go and Rust passed on the same guest in the same run, and the quoted diagnostic commands, the dnf fix, and the re-snapshot were all run here: after revert-vm.sh systems-target lab-ready the guest reported GLIBCXX_3.4.36 and test-all-examples.py --mode vm --lang cpp reported the quoted 8 passed, 0 failed, 0 skipped, including the two-guest 41-capstone-fleet. systems-peer was confirmed to carry the same drift while showing no symptom. Not exercised: unverified: the GLIBC_ form of the symbol check is stated by analogy and was not the failure encountered here, which was GLIBCXX_ only; and building inside the guest with the in-VM toolchains is named as the standing fallback but is not exercised by any example in this book.

diff --git a/_docs/03-observability-stack.md b/_docs/03-observability-stack.md index 712df65..f3ea642 100644 --- a/_docs/03-observability-stack.md +++ b/_docs/03-observability-stack.md @@ -8,24 +8,24 @@ duration: "30 minutes" Chapter 1 gave you three toolchains; Chapter 2 gave you two disposable VMs. This chapter adds the last piece of the workbench: somewhere for telemetry to -*go*. Many chapters in this book emit metrics, traces, or logs — a syscall -counter here, a request span there — and printing them to a terminal only gets +*go*. Many chapters in this book emit metrics, traces, or logs (a syscall +counter here, a request span there), and printing them to a terminal only gets you so far. What you want is a real backend you can query, graph, and cross-check against, without turning your workstation into a monitoring deployment. The answer is one container: the Grafana **LGTM** all-in-one image, run rootless under Podman, listening on exactly two ports you will -memorize — **4318** for sending and **3000** for looking. +memorize: **4318** for sending and **3000** for looking. -The infrastructure is in `examples/_infra/` — one compose file and a README. +The infrastructure is in `examples/_infra/`: one compose file and a README. There is no per-chapter example here; every telemetry-emitting example from Chapter 38 onward depends on what you stand up now. {% include excalidraw.html file="03-telemetry-pipeline" alt="Telemetry pipeline: a host demo and a VM demo send OTLP HTTP to port 4318 on the lsp-lgtm container, where the OpenTelemetry Collector fans out to Loki for logs, Tempo for traces, and Mimir for metrics; Grafana on port 3000 queries all three and serves the developer's browser. The VM demo reaches the host at the libvirt gateway address 192.168.124.1." - caption="Figure 3.1 — one container, whole pipeline: OTLP in on :4318, Grafana out on :3000" %} + caption="Figure 3.1: one container, whole pipeline: OTLP in on :4318, Grafana out on :3000" %} -> **Tools used** — `podman`, `systemctl`, `curl`, `openssl`, `python3` (host); +> **Tools used**: `podman`, `systemctl`, `curl`, `openssl`, `python3` (host); > `ip` (systems-target VM). Everything here is checked by > `scripts/check-host.sh`, ships with Fedora, or is preinstalled in the lab VMs. @@ -35,14 +35,14 @@ LGTM is Grafana Labs' acronym for its open-source backend quartet: **L**oki (logs), **G**rafana (the UI), **T**empo (traces), **M**imir (metrics, Prometheus-compatible). In production these are four separately scaled, separately configured services. For local development that topology is pure -overhead — what you need is the *interfaces*, not the scale. So Grafana ships +overhead; what you need is the *interfaces*, not the scale. So Grafana ships `grafana/otel-lgtm`, a single image containing all four **plus an embedded OpenTelemetry Collector** wired to receive OTLP and fan each signal out to the right store, with Grafana's datasources pre-provisioned to read them back. One `podman compose up` and the entire pipeline in Figure 3.1 exists. -The collector is the part worth pausing on. Your programs will never speak -"Loki" or "Mimir" directly — they speak **OTLP** (the OpenTelemetry protocol) +The collector does the real work here. Your programs will never speak +"Loki" or "Mimir" directly; they speak **OTLP** (the OpenTelemetry protocol) to the collector, and the collector routes logs, traces, and metrics to their stores. That indirection is the whole point of OpenTelemetry: instrument once, point `OTEL_EXPORTER_OTLP_ENDPOINT` somewhere, and the backend becomes @@ -59,7 +59,7 @@ concern; debuggability wins on a workbench. ## The compose file, line by line -Read `examples/_infra/compose-lgtm.yaml` — it is short enough to understand +Read `examples/_infra/compose-lgtm.yaml`; it is short enough to understand completely: ```yaml @@ -90,17 +90,17 @@ networks: name: lsp-observability ``` -Each choice has a why. The image is **pinned to `0.8.1`** — an unpinned +Each choice has a why. The image is **pinned to `0.8.1`**: an unpinned `latest` would let the stack silently change under a book whose chapters promise reproducible output. `container_name: lsp-lgtm` gives scripts (and you) a stable handle instead of a generated name. Only three ports are published; Loki, Tempo, and Mimir listen on internal ports *inside* the -container, deliberately unpublished — everything goes in through the collector +container, left unpublished; everything goes in through the collector and comes out through Grafana, which is exactly the discipline production imposes. `mem_limit: 2g` exists because this one container runs five services; the cap keeps a runaway ingest from competing with your builds for RAM. The three `GF_AUTH_*` variables make Grafana skip its login entirely and treat -every visitor as an admin — indefensible anywhere reachable, perfect for a +every visitor as an admin: indefensible anywhere reachable, perfect for a loopback-only lab, and the comment in the file says so. The `healthcheck` polls Grafana's `/api/health` from inside the container with a generous `start_period`, because first boot unpacks and starts all five services and @@ -119,8 +119,8 @@ everyone forgets: ``` Why: `podman compose` is not a compose implementation. It is a thin -dispatcher that delegates to an external provider — on Fedora, the -`docker-compose` plugin — which speaks the Docker Engine API to a daemon +dispatcher that delegates to an external provider: on Fedora, the +`docker-compose` plugin, which speaks the Docker Engine API to a daemon socket. Podman famously has no daemon, but it emulates one: the systemd user unit `podman.socket` listens on `$XDG_RUNTIME_DIR/podman/podman.sock` and socket-activates a Podman service @@ -138,7 +138,7 @@ requires: [lgtm] ``` The verification harness (`scripts/test-all-examples.py`) probes -`http://localhost:4318` before running such an example — and counts *any* +`http://localhost:4318` before running such an example, and counts *any* HTTP response as alive, including a 4xx, because a bare GET to an OTLP ingest path legitimately returns an error status while proving something is listening. If nothing answers, the example is marked **SKIP (lgtm down)** @@ -148,11 +148,11 @@ working example red. ## Reaching the stack from inside a VM -A guest cannot use `localhost:4318` — its loopback is its own. But on a +A guest cannot use `localhost:4318`: its loopback is its own. But on a libvirt NAT network, the **network gateway is the host**, so the host's published ports are reachable at the gateway IP. On this lab that is `192.168.124.1` (libvirt's stock default network is `192.168.122.0/24`, so -most references say `.122.1` — always check yours): +most references say `.122.1`; always check yours): ```bash [vm]$ ip route show default @@ -167,7 +167,7 @@ You pass the endpoint through `deploy-to-vm.sh` via `OTEL_ENDPOINT`: The script forwards it explicitly because of a subtlety worth knowing: `sudo` strips the environment, so `scripts/lab/deploy-to-vm.sh` re-injects it with -`env` on the remote side — +`env` on the remote side: ```bash RENV="" @@ -176,7 +176,7 @@ RENV="" exec ssh -t $SSH_OPTS "fedora@$IP" "sudo env $RENV '$REMOTE' $*" ``` -— otherwise the guest binary would fall back to *its* localhost and telemetry +Otherwise the guest binary would fall back to *its* localhost and telemetry would vanish silently. One host-side requirement: ports 4317/4318 must be open toward the libvirt bridge (on Fedora, the `libvirt` firewalld zone). @@ -198,7 +198,7 @@ healthy ``` {% endraw %} -Open `http://localhost:3000` — no login form, straight to Grafana as an +Open `http://localhost:3000`: no login form, straight to Grafana as an anonymous admin. Take the two-minute tour: **Explore** in the left navigation is where you will live; its datasource picker already lists **Loki**, **Tempo**, and **Mimir** (as a Prometheus datasource), provisioned by the @@ -219,7 +219,7 @@ gate used: hand-post one trace with `curl`, then query it back. That `200` is the collector accepting a minimal but complete OTLP/JSON trace: one resource (carrying `service.name`, the identity every backend groups by), one scope, one 50 ms span. Now query it back **through Grafana's datasource -proxy** — Tempo's own HTTP port is intentionally unpublished, so the proxy +proxy**: Tempo's own HTTP port is unpublished, so the proxy route both reaches Tempo *and* proves the Grafana→Tempo wiring in a single request: @@ -244,12 +244,12 @@ request: ``` (Response trimmed.) The span went curl → collector → Tempo → Grafana proxy → -back to your terminal — every arrow in Figure 3.1's trace path, exercised +back to your terminal: every arrow in Figure 3.1's trace path, exercised once. In the UI, Explore → Tempo → TraceQL query `{ resource.service.name = "lsp-smoke" }` finds the same span with a waterfall view of one bar. -Stop the stack the same way you started it (state is not preserved — fine for +Stop the stack the same way you started it (state is not preserved, fine for a lab): ```bash @@ -270,22 +270,22 @@ lsp-lgtm docker.io/grafana/otel-lgtm:0.8.1 Up ... (healthy) ``` {% endraw %} -`podman ps` is independent of the compose provider — it asks the container +`podman ps` is independent of the compose provider: it asks the container runtime directly, so agreement here means the docker-compose plugin, the user socket, and the runtime all see the same world. The `(healthy)` suffix is the compose file's `curl` healthcheck reporting from *inside* the container (the -image ships curl, not wget — a detail that cost this chapter one debugging +image ships curl, not wget, a detail that cost this chapter one debugging round), while your `curl` hits the *published* port from outside: same endpoint, two network paths, both answering. ## What you learned - The LGTM all-in-one image packs Loki, Grafana, Tempo, Mimir, and an OTel - Collector into one pinned container — production interfaces, workbench + Collector into one pinned container: production interfaces, workbench footprint; senders only ever speak OTLP, and this book standardizes on the curl-able HTTP transport at `:4318`. - `podman compose` delegates to the docker-compose plugin over the Docker - API socket that `systemctl --user enable --now podman.socket` provides — + API socket that `systemctl --user enable --now podman.socket` provides, the one-time prerequisite behind every `up -d`. - Examples declare `requires: [lgtm]`; the harness probes `:4318` and skips, never fails, when the stack is down. @@ -296,9 +296,9 @@ endpoint, two network paths, both answering. The stack sits idle until Chapter 38 instruments real programs in all three languages against it, and Chapter 45 adds the PCP, Cockpit, and SystemTap suites alongside. With the toolchains, the lab, and the telemetry pipeline in -place, the workbench is complete — next, Foundations begins where everything +place, the workbench is complete; next, Foundations begins where everything in this book bottoms out: the syscall boundary and the ABI beneath it. --- -

verified — on the Fedora 44 host: podman.socket enabled, podman compose -f examples/_infra/compose-lgtm.yaml up -d brought lsp-lgtm (otel-lgtm:0.8.1) to healthy, /api/health returned ok, the OTLP HTTP POST to :4318/v1/traces returned 200, and the Tempo query by trace ID through Grafana's datasource proxy returned the span r02-gate-smoke (service lsp-smoke). Not yet exercised: exporting telemetry from inside a lab VM via the 192.168.124.1 gateway — the endpoint-forwarding path is verified only as far as deploy-to-vm.sh running host-built binaries on the guest; a VM-side OTLP export lands with Chapter 38.

+

verified: on the Fedora 44 host: podman.socket enabled, podman compose -f examples/_infra/compose-lgtm.yaml up -d brought lsp-lgtm (otel-lgtm:0.8.1) to healthy, /api/health returned ok, the OTLP HTTP POST to :4318/v1/traces returned 200, and the Tempo query by trace ID through Grafana's datasource proxy returned the span r02-gate-smoke (service lsp-smoke). Not yet exercised: exporting telemetry from inside a lab VM via the 192.168.124.1 gateway; the endpoint-forwarding path is verified only as far as deploy-to-vm.sh running host-built binaries on the guest; a VM-side OTLP export lands with Chapter 38.

diff --git a/_docs/04-syscalls-and-the-abi.md b/_docs/04-syscalls-and-the-abi.md index 84afae3..850177e 100644 --- a/_docs/04-syscalls-and-the-abi.md +++ b/_docs/04-syscalls-and-the-abi.md @@ -2,15 +2,15 @@ title: "Syscalls and the ABI" order: 4 part: "Foundations" -description: "What a system call really is — the syscall instruction, the register-level kernel ABI, and the vDSO fast path — traced through sysprobe, one specimen program that shows how glibc, the Go runtime, and rustix each reach the same kernel." +description: "What a system call really is: the syscall instruction, the register-level kernel ABI, and the vDSO fast path, traced through sysprobe, one specimen program that shows how glibc, the Go runtime, and rustix each reach the same kernel." duration: 40 minutes --- Chapter 1 ended on a number it did not explain: the C++ hello made 64 syscalls, the Go hello made 217, and we waved at "the runtime" and moved on. This chapter is where that debt gets paid. With the lab and observability stack built, Part -1 starts at the only interface that actually matters in this book — the -user/kernel boundary — and answers three questions precisely: what *is* a +1 starts at the only interface that actually matters in this book, the +user/kernel boundary, and answers three questions precisely: what *is* a system call at the instruction level, why do some libc calls never reach the kernel at all (the vDSO), and what path does each of our three languages take from your function call down to the `syscall` instruction. The vehicle is @@ -25,9 +25,9 @@ CLI contract and the exact five-line output that `verify.lua` asserts. {% include excalidraw.html file="04-user-kernel-path" alt="A user-space band with your code, a language wrapper, and the syscall instruction, plus a dashed vdso page; an amber trap arrow crosses into a kernel band through entry_SYSCALL_64 and the syscall table to sys_getrandom, with a dashed return arrow carrying rax back." - caption="Figure 4.1 — one getrandom(2) crossing the boundary: marshal registers, trap, dispatch by number, return rax — or take the vDSO fast path and never trap at all" %} + caption="Figure 4.1: one getrandom(2) crossing the boundary, marshal registers, trap, dispatch by number, return rax, or take the vDSO fast path and never trap at all" %} -> **Tools used** — `strace`, `objdump`, `ldd`, `python3` (host); `bpftrace` +> **Tools used**: `strace`, `objdump`, `ldd`, `python3` (host); `bpftrace` > (lab VM, exercised in Part 8). Everything here is checked by > `scripts/check-host.sh` or preinstalled in the lab VMs. @@ -38,7 +38,7 @@ process runs in ring 3, where the CPU refuses to touch page tables, device registers, or other processes' memory; the kernel runs in ring 0, where it can. The *only* legitimate way across on x86-64 is a handful of trap instructions, and on modern Linux that means `syscall`. The contract around -that instruction — the **syscall ABI** — is register-level and brutally +that instruction (the **syscall ABI**) is register-level and brutally simple: the syscall *number* goes in `rax` (318 for `getrandom` on x86-64), up to six arguments go in `rdi`, `rsi`, `rdx`, `r10`, `r8`, `r9`, and the result comes back in `rax`, where a small negative value means "error @@ -46,15 +46,15 @@ result comes back in `rax`, where a small negative value means "error but the `syscall` instruction itself clobbers `rcx` to save the return address, so the kernel ABI substitutes `r10`. That one register swap is why every language needs at least a sliver of assembly or a wrapper library to -make a syscall — you cannot express "arg 4 in r10" in portable C. +make a syscall: you cannot express "arg 4 in r10" in portable C. When the instruction executes, the CPU switches to ring 0 and jumps to the kernel's `entry_SYSCALL_64`, which moves onto a kernel stack, saves your registers, and indexes `sys_call_table` by `rax` to dispatch the handler. -The handler runs *in your process*, just in kernel mode — `sys_getrandom` +The handler runs *in your process*, just in kernel mode: `sys_getrandom` writes directly into the buffer your `rdi` pointed at. Then the path unwinds and `sysretq` drops back to ring 3 with the result in `rax`. Every trap is a -full round trip — two mode switches plus the kernel work — which is why +full round trip (two mode switches plus the kernel work), which is why syscall *count* is a performance currency, and why `strace`, which uses `ptrace` to stop the process at exactly this boundary, sees every trap a process makes and nothing else. @@ -62,7 +62,7 @@ process makes and nothing else. ## The vDSO: syscalls that never happen That "nothing else" has teeth. Some things the kernel exports are read so -often — the clock, above all — that trapping for each read would be absurd. +often (the clock, above all) that trapping for each read would be absurd. So the kernel maps a small shared library into every process, the **vDSO**, visible in any process's memory map: @@ -76,7 +76,7 @@ visible in any process's memory map: `[vdso]` is executable kernel-provided code; the `[vvar]` pages are read-only data the kernel keeps current (timekeeping state in `vvar_vclock`). A `clock_gettime` through glibc calls into that page, reads the shared data, -and returns — ring 3 the whole way. To see the difference, I compiled a +and returns, ring 3 the whole way. To see the difference, I compiled a 16-line probe that loops `clock_gettime(CLOCK_MONOTONIC, …)` a thousand times, then `getrandom(buf, 16, 0)` a thousand times, and ran it under the book's strace 7.1 build: @@ -91,13 +91,13 @@ book's strace 7.1 build: ``` One thousand `clock_gettime` calls: **zero** syscalls. One thousand -`getrandom` calls: **two** — and neither is the one you would predict. +`getrandom` calls: **two**, and neither is the one you would predict. The full trace shows `getrandom("…", 8, GRND_NONBLOCK) = 8` (glibc seeding internal state at startup) and `getrandom("…", 32, 0) = 32`. Where did our sixteen-byte request go? Kernel 7.1 plus glibc 2.43 route `getrandom` through the vDSO's `vgetrandom`: on first use it traps *once* to pull a 32-byte ChaCha20 key from the kernel, then generates every byte you asked -for — all thousand requests — in user space. The lesson generalizes: `strace` +for (all thousand requests) in user space. The lesson generalizes: `strace` shows you the boundary, not your API calls, and on a current kernel even "obvious" syscalls like `getrandom` may have quietly stopped being syscalls. Keep that 32-byte signature in mind; it reappears inside `sysprobe`. @@ -106,36 +106,36 @@ Keep that 32-byte signature in mind; it reappears inside `sysprobe`. {% include excalidraw.html file="04-runtime-syscall-layers" - alt="Three side-by-side stacks — C++ code over glibc wrappers over syscall instructions in libc, Go code over the Go runtime over 511 syscall instructions in a static binary, Rust code over the rustix linux_raw backend over 32 inlined syscall instructions — all with amber arrows labeled 70, 283, and 73 calls into one dark kernel ABI box." - caption="Figure 4.2 — the per-language layer stacks over the one register ABI, annotated with this chapter's real strace -c totals" %} + alt="Three side-by-side stacks: C++ code over glibc wrappers over syscall instructions in libc, Go code over the Go runtime over 511 syscall instructions in a static binary, Rust code over the rustix linux_raw backend over 32 inlined syscall instructions, all with amber arrows labeled 70, 283, and 73 calls into one dark kernel ABI box." + caption="Figure 4.2: the per-language layer stacks over the one register ABI, annotated with this chapter's real strace -c totals" %} The kernel ABI is one; the roads to it differ, and `file`, `ldd`, and `objdump` on the three freshly built `sysprobe` binaries map them exactly. Counting literal `syscall` instructions with `objdump -d | grep -cw syscall`: -- **C++ — 0.** The binary is dynamically linked (`ldd` lists `libstdc++`, +- **C++: 0.** The binary is dynamically linked (`ldd` lists `libstdc++`, `libm`, `libgcc_s`, `libc`, and `linux-vdso.so.1` itself). Your code calls glibc *wrappers*; every `syscall` instruction lives in `libc.so.6`. The wrapper layer is not thin: it flips `-errno` into the `errno` global, - implements thread-cancellation points, chooses vDSO fast paths — and + implements thread-cancellation points, chooses vDSO fast paths, and sometimes substitutes a different syscall entirely, as we will see with `nanosleep`. -- **Go — 511.** The binary is statically linked; `ldd` says +- **Go: 511.** The binary is statically linked; `ldd` says `not a dynamic executable`. Go carries its own syscall stubs, its own calling convention, and its own stack discipline (goroutine stacks are small and growable, so the runtime must ensure a syscall never runs on one that might move). `golang.org/x/sys/unix` is a veneer over - `runtime`-provided entry points — including, since Go 1.24, the vDSO: + `runtime`-provided entry points, including, since Go 1.24, the vDSO: `x/sys` v0.47.0 links `unix.Getrandom` straight to `runtime.vgetrandom` via `go:linkname` and only falls back to the raw syscall where the vDSO is unsupported. -- **Rust — 32.** The binary still links `libc.so.6` — std uses it for - process startup — but `rustix` compiles with its `linux_raw` backend on +- **Rust: 32.** The binary still links `libc.so.6` (std uses it for + process startup), but `rustix` compiles with its `linux_raw` backend on x86-64, inlining raw `syscall` instructions at each call site and returning the `rax` error directly as `Result<_, Errno>`. No `errno` global is ever involved, and no glibc policy sits between your call and - the kernel — which is exactly why rustix's `getrandom` shows up in + the kernel, which is exactly why rustix's `getrandom` shows up in `strace` and glibc's does not. ## How the code works @@ -145,7 +145,7 @@ of syscall: **open** creates a resource (an fd), **write** moves data through one, **sleep** blocks in the kernel, and **random** asks the kernel for a service with partial-result semantics. Each step prints `step= ok`, the program ends with `sysprobe: 4 steps ok`, and any argument gets `usage: app` -and exit 2 — identical in all three languages, so one `verify.lua` asserts +and exit 2, identical in all three languages, so one `verify.lua` asserts all five lines appear in order for each. Step 1 is the richest, so it is the excerpt; all three are the real functions from `examples/04-syscalls-and-the-abi/`, verbatim. @@ -237,12 +237,12 @@ fn open_scratch(dir: &str) -> Result { ``` The `openat` call opens the *directory itself* with `O_TMPFILE`: the kernel -allocates an inode with a link count of zero, so no name ever appears — no +allocates an inode with a link count of zero, so no name ever appears: no cleanup, no symlink races, gone on close. Not every filesystem supports it, so each version distinguishes "this filesystem can't" (`EOPNOTSUPP`, or the -`EISDIR`/`EINVAL` older kernels and filesystems report) — recoverable, take -the create-then-`unlink` fallback, which reaches the same anonymous-inode -state through a brief named window — from any other errno, which is a real +`EISDIR`/`EINVAL` older kernels and filesystems report), which is recoverable +by taking the create-then-`unlink` fallback (reaching the same anonymous-inode +state through a brief named window), from any other errno, which is a real failure to propagate. Getting that triage *wrong* is invisible on your machine and a crash on someone else's; getting it right is most of what "handling" an errno means. @@ -250,11 +250,11 @@ machine and a crash on someone else's; getting it right is most of what Step 2, `write_all`, loops because `write(2)` may legally write *fewer* bytes than asked, so a single unchecked call is a data-loss bug even when it "works"; each iteration re-slices the buffer by the returned count. The fd -is then closed *deliberately early* — RAII scope exit in C++, a checked -`unix.Close` in Go, `drop(fd)` in Rust — so the trace shows a clean +is then closed early by design (RAII scope exit in C++, a checked +`unix.Close` in Go, `drop(fd)` in Rust) so the trace shows a clean `openat → write → close` group before the sleep starts. Step 3 sleeps 10 ms via `nanosleep(2)`, and on `EINTR` resumes with the *remainder* the kernel -wrote back — resuming with the original request would restart the full 10 ms +wrote back; resuming with the original request would restart the full 10 ms every time a signal landed. Step 4 loops `getrandom(2)` over partial reads for the same reason as `write_all`. `main` in each language is only sequencing: check `argc`, resolve `$TMPDIR` (default `/tmp`), run the four @@ -262,7 +262,7 @@ steps, print per-step lines as they succeed. ### Errors, three ways -All three languages receive the identical fact — `rax` came back negative — +All three languages receive the identical fact (`rax` came back negative) and immediately disagree about what to do with it. glibc stores it into the thread-local `errno` and returns `-1`, so the C++ code's first act is `last_error()`: capture `errno` into a `std::error_code` *now*, before any @@ -271,7 +271,7 @@ value. Go's `unix` package returns a `syscall.Errno` as an ordinary `error`; the code wraps with `%w` (`"openat %s (O_TMPFILE): %w"`) so `errors.Is(err, unix.EOPNOTSUPP)` still matches through the added context. rustix never materializes a global at all: the negative `rax` *is* the `Err(Errno)`, and -the fallback triage becomes a match arm — +the fallback triage becomes a match arm: `Err(Errno::OPNOTSUPP | Errno::ISDIR | Errno::INVAL)`. Same kernel fact, three idioms; Chapter 5 turns this three-way comparison into the book's error-handling policy. @@ -282,14 +282,14 @@ error-handling policy. 7. Before `main` ran, the Go runtime made 4 `clone` calls (its worker threads) and 114 `rt_sigaction` calls (installing handlers so it can preempt goroutines); while our program slept its one 10 ms sleep, a runtime -thread issued 61 `nanosleep` calls of its own — visible in the trace as +thread issued 61 `nanosleep` calls of its own, visible in the trace as `{tv_nsec=20000}` repeated, then doubling `40000`, `80000`, up to `2560000` as the runtime's monitor backs off polling an idle program. C++ and Rust stay a single thread from `execve` to exit, which is exactly why their traces need no `-f`. The portable lesson is the `EINTR` loops: once any process has signal handlers (as Go's runtime just demonstrated), *every* blocking syscall can return early, so retry-on-`EINTR` is not -defensive decoration but the baseline discipline — Chapter 12 (signals) +defensive decoration but the baseline discipline; Chapter 12 (signals) makes it precise. ## Build, run, observe @@ -309,7 +309,7 @@ step=random ok sysprobe: 4 steps ok ``` -The book's harness agrees — `python3 scripts/test-all-examples.py --only +The book's harness agrees: `python3 scripts/test-all-examples.py --only 04-syscalls-and-the-abi` reports `PASS` for cpp, go, and rust. Now run it the way it was designed to be run. This book's strace lives at `scratchpad/usr/bin/strace` (v7.1) on the reference host; the C++ binary, @@ -328,22 +328,22 @@ write(1, "step=open ok\nstep=write ok\nstep="..., 77) = 77 Read that against the source and two "discrepancies" jump out, both wrapper-layer fingerprints. The source says `nanosleep`; the trace says -`clock_nanosleep(CLOCK_REALTIME, …)` — glibc's `nanosleep()` is implemented +`clock_nanosleep(CLOCK_REALTIME, …)`: glibc's `nanosleep()` is implemented on the more general syscall. The source asks for 16 random bytes; the trace -shows one 32-byte `getrandom` — the vDSO key-seeding trap from earlier, after +shows one 32-byte `getrandom`: the vDSO key-seeding trap from earlier, after which our request was served in user space. The Rust trace inverts both: a raw `nanosleep({tv_sec=0, tv_nsec=10000000}, …)` and a literal `getrandom("…", 16, 0) = 16` (plus `O_LARGEFILE` spelled out in its -`openat` flags — rustix passes explicitly what glibc implies). And Go, whose +`openat` flags; rustix passes explicitly what glibc implies). And Go, whose `unix.Getrandom` now rides `runtime.vgetrandom`, shows a single 32-byte `getrandom` for the whole process. One source-level contract, three -distinct boundary signatures — none of them wrong, all of them visible. +distinct boundary signatures: none of them wrong, all of them visible. ## Cross-check: counting every trap Chapter 1 counted 64 syscalls for the C++ hello and 217 for Go's. `strace -c` on the three `sysprobe` binaries (`-f` for Go) deepens that finding with -a full accounting; trimmed to the load-bearing rows from this session's +a full accounting; trimmed to the key rows from this session's runs: ``` @@ -358,34 +358,34 @@ overhead, and the ratio matches Chapter 1's almost exactly (roughly 4× for Go, near-parity for C++ and Rust). C++'s 21 `mmap`s are the dynamic loader mapping four libraries; Rust's 73 is C++-shaped because its *startup* still goes through glibc even though its syscalls do not. Go's 283 is dominated by -runtime self-management — signals, threads, and its sleeping monitor. As an +runtime self-management: signals, threads, and its sleeping monitor. As an independent surface, the `objdump` census (0, 511, and 32 embedded `syscall` instructions) confirms *where* each binary keeps its boundary crossings: nowhere (glibc's), everywhere (a static runtime's), and inline at 32 rustix call sites. Two tools, one story. -

unverified — On the lab VM: the kernel-side view of this chapter — watching sys_enter_* tracepoints fire with bpftrace as sysprobe runs, counting traps from inside the kernel instead of via ptrace — needs root and a disposable kernel, so it is not run here; chapter 30 (Debugging part) exercises exactly this on systems-target.

+

unverified: On the lab VM: the kernel-side view of this chapter (watching sys_enter_* tracepoints fire with bpftrace as sysprobe runs, counting traps from inside the kernel instead of via ptrace) needs root and a disposable kernel, so it is not run here; chapter 30 (Debugging part) exercises exactly this on systems-target.

## What you learned - A syscall is a register protocol around one trapping instruction: number in `rax`, args in `rdi rsi rdx r10 r8 r9` (`r10` because `syscall` clobbers `rcx`), result or `-errno` back in `rax`. -- The vDSO answers hot calls without a trap — 1000 `clock_gettime` calls +- The vDSO answers hot calls without a trap: 1000 `clock_gettime` calls produced zero syscalls here, and on kernel 7.1/glibc 2.43 even `getrandom` is served in user space after one 32-byte key-seeding trap. -- The three languages take three roads to the same ABI — glibc wrappers +- The three languages take three roads to the same ABI: glibc wrappers (0 `syscall` instructions in the binary), the Go runtime's own stubs and - ABI (511, static), rustix's inlined raw backend (32) — and each road + ABI (511, static), rustix's inlined raw backend (32), and each road leaves a distinct fingerprint in `strace`. - `strace -c` puts numbers on runtime overhead: 70 (C++) vs 283 (Go, all - threads) vs 73 (Rust) traps for four steps of real work — the same shape + threads) vs 73 (Rust) traps for four steps of real work, the same shape as Chapter 1's 64-vs-217, now fully attributed. Next, **errors, three ways**: the `-errno` fact each wrapper just handed us -becomes a policy — `std::expected`, wrapped Go errors, and `thiserror`-style -Rust — that every later example in the book obeys. +becomes a policy (`std::expected`, wrapped Go errors, and `thiserror`-style +Rust) that every later example in the book obeys. --- -

verified — all numbers in this chapter are from this session's runs on the Fedora 44 reference host (kernel 7.1, glibc 2.43, strace 7.1): the harness reported PASS for cpp/go/rust and each binary printed the five-line block ending sysprobe: 4 steps ok; strace -c totaled 70 (cpp), 283 (go, -cf), and 73 (rust); the vdso-probe run showed 0 clock_gettime and 2 getrandom syscalls for 1000 calls each, including getrandom("…", 32, 0) = 32; the quoted sequence traces (O_TMPFILE openat, 25-byte write, clock_nanosleep vs raw nanosleep, Go's 20 µs-doubling monitor sleeps) are trimmed real output; objdump counted 0/511/32 syscall instructions; and /proc/self/maps showed the quoted [vdso]/[vvar_vclock] mappings. Not run here: the bpftrace kernel-side count (lab-VM callout above).

+

verified: all numbers in this chapter are from this session's runs on the Fedora 44 reference host (kernel 7.1, glibc 2.43, strace 7.1): the harness reported PASS for cpp/go/rust and each binary printed the five-line block ending sysprobe: 4 steps ok; strace -c totaled 70 (cpp), 283 (go, -cf), and 73 (rust); the vdso-probe run showed 0 clock_gettime and 2 getrandom syscalls for 1000 calls each, including getrandom("…", 32, 0) = 32; the quoted sequence traces (O_TMPFILE openat, 25-byte write, clock_nanosleep vs raw nanosleep, Go's 20 µs-doubling monitor sleeps) are trimmed real output; objdump counted 0/511/32 syscall instructions; and /proc/self/maps showed the quoted [vdso]/[vvar_vclock] mappings. Not run here: the bpftrace kernel-side count (lab-VM callout above).

diff --git a/_docs/05-errors-three-ways.md b/_docs/05-errors-three-ways.md index 6ce9f74..ed6eb5a 100644 --- a/_docs/05-errors-three-ways.md +++ b/_docs/05-errors-three-ways.md @@ -2,20 +2,20 @@ title: "Errors, three ways" order: 5 part: "Foundations" -description: "One error's journey from a failing write(2) into std::expected, a wrapped Go error, and a Rust Result — copyx establishes the book's error taxonomy, the strerror(3) message contract, and the EINTR retry policy every later chapter reuses." +description: "One error's journey from a failing write(2) into std::expected, a wrapped Go error, and a Rust Result: copyx establishes the book's error taxonomy, the strerror(3) message contract, and the EINTR retry policy every later chapter reuses." duration: "45 minutes" --- Chapter 4 walked a syscall down to the boundary and back, and ended on the detail this chapter is built from: when a syscall fails, the kernel does not -throw, log, or explain — it puts a small negative integer in a register and +throw, log, or explain: it puts a small negative integer in a register and returns. Everything your language calls an "error" is constructed, in user -space, from that one number. This chapter takes a single program — `copyx`, a -file copier — and writes it three times so you can watch the *same* kernel +space, from that one number. This chapter takes a single program, `copyx`, a +file copier, and writes it three times so you can watch the *same* kernel failure become a `std::expected`, a wrapped Go `error`, and a Rust `Result`, and still come out the other side as byte-identical output and the same exit -code. Along the way it fixes the policies the rest of the book never -renegotiates: what EINTR means, what a short write demands, and which side of +code. Along the way it fixes the policies every later chapter in this book +relies on: what EINTR means, what a short write demands, and which side of a failure owns the exit code. The code is in `examples/05-errors-three-ways/`. `./demo.sh` there builds all @@ -24,29 +24,29 @@ states the behavior contract each one must satisfy. {% include excalidraw.html file="05-error-flow-three-ways" - alt="One failure's journey: write(2) on /dev/full returns -ENOSPC in the kernel; in user space that becomes errno 28, which fans out into three columns — C++23's std::expected with std::error_code mapping into a Failure struct, Go's unix.Errno wrapped into *opError with Unwrap for errors.Is/As, and Rust's Result with Errno mapped via map_err into a thiserror CopyError variant — all three converging on one outcome box: exit 3 and the line copyx: write /dev/full: No space left on device." - caption="Figure 5.1 — one failure, three spellings, one contract: ENOSPC from write(2) to the user, in each type system" %} + alt="One failure's journey: write(2) on /dev/full returns -ENOSPC in the kernel; in user space that becomes errno 28, which fans out into three columns (C++23's std::expected with std::error_code mapping into a Failure struct, Go's unix.Errno wrapped into *opError with Unwrap for errors.Is/As, and Rust's Result with Errno mapped via map_err into a thiserror CopyError variant), all three converging on one outcome box: exit 3 and the line copyx: write /dev/full: No space left on device." + caption="Figure 5.1: one failure, three spellings, one contract: ENOSPC from write(2) to the user, in each type system" %} -> **Tools used** — `strace`, `head`, `cmp`, `ls`, `python3` (host); `bpftrace` +> **Tools used**: `strace`, `head`, `cmp`, `ls`, `python3` (host); `bpftrace` > (lab VM, exercised in Part 8). Everything here is checked by > `scripts/check-host.sh`, ships with Fedora, or is preinstalled in the lab VMs. ## errno is an ABI, not a variable On x86-64 Linux, a failing syscall returns a value between -1 and -4095 in -`rax`; the number is the negated error code. `ENOSPC` is 28 — not "usually -28", but 28 by kernel ABI, stable for decades, the same for every language +`rax`; the number is the negated error code. `ENOSPC` is 28: not "usually +28" but 28 by kernel ABI, stable for decades, the same for every language and libc on the platform. What differs is the unwrapping. glibc's syscall stubs check the return, store the positive code into the thread-local `errno`, and hand C a `-1`. Go's runtime makes the syscall itself and returns -the code as an ordinary value — `unix.Errno` is just `uintptr` with an +the code as an ordinary value: `unix.Errno` is just `uintptr` with an `Error()` method; there is no global to race on. Rust's `rustix` does the same trick: `Errno` is a wrapper around the raw code, delivered in the `Err` arm of a `Result`, never through ambient state. That shared integer is why a cross-language error *policy* is even possible. Whatever type each language dresses the failure in, the bottom of the stack -is the same number with the same `strerror(3)` text — and `copyx` treats that +is the same number with the same `strerror(3)` text, and `copyx` treats that text as part of its interface. The behavior contract, identical in all three implementations: @@ -66,11 +66,11 @@ Two syscall-level policies are also part of the contract, because they will be true of every program in this book. First, **EINTR is retried**. A `read(2)` or `write(2)` interrupted by a signal before transferring anything fails with `EINTR`; nothing happened, so the only correct response is to -reissue the call — treating it as an error turns every signal into data +reissue the call; treating it as an error turns every signal into data corruption at a distance. Second, **short writes are resumed**. `write(2)` is allowed to accept fewer bytes than you offered; the loop must advance past what the kernel took and continue. (The third sibling, `EAGAIN`, means "would -block" on a non-blocking descriptor — not an error either, but a scheduling +block" on a non-blocking descriptor, not an error either, but a scheduling fact; it becomes central when epoll arrives in the Files and I/O part.) One more subtlety the contract encodes: the write side must *observe* `close(2)`, because deferred I/O errors can surface there; the read side's @@ -82,7 +82,7 @@ Each implementation has the same skeleton: a phase-tagged error type, a retrying `read_some`, a resuming `write_all`, and a `copy_file` that maps each failure onto the taxonomy. The error types come first, because they are where the three type systems diverge most visibly. C++ -(`examples/05-errors-three-ways/cpp/src/main.cpp`) uses a plain struct — +(`examples/05-errors-three-ways/cpp/src/main.cpp`) uses a plain struct: ```cpp struct Failure { @@ -91,7 +91,7 @@ struct Failure { }; ``` -— because by the time `copy_file` fails, classification is *done*: the +This works because by the time `copy_file` fails, classification is *done*: the message is rendered and the exit code chosen. Go (`examples/05-errors-three-ways/go/main.go`) cannot stop there, since Go errors are expected to stay inspectable: @@ -128,7 +128,7 @@ enum CopyError { ``` Now the core loops. The three excerpts below are the real `write_all` and -`copy_file` from each implementation — read one, then diff it mentally +`copy_file` from each implementation, read one, then diff it mentally against the next: {% include codetabs.html langs="C++|Go|Rust" %} @@ -311,17 +311,17 @@ fn copy_file(src_path: &str, dst_path: &str) -> Result { } ``` -The loops are structurally identical — a 64 KiB buffer, read until a zero -return means EOF, `write_all` per chunk, then a checked close — but each +The loops are structurally identical (a 64 KiB buffer, read until a zero +return means EOF, `write_all` per chunk, then a checked close) but each language spends its effort differently. **C++** builds the discipline out of `std::expected`: every fallible helper returns `std::expected`, `last_error()` snapshots `errno` into `std::generic_category()` (the category for POSIX errno values; Chapter 1's -`uname` wrapper used `system_category` — for Linux errno they render the +`uname` wrapper used `system_category`; for Linux errno they render the same, and the book standardizes on `generic_category` from here on), and the move-only RAII `Fd` guarantees no descriptor outlives its owner. The -asymmetry between `dst->close()` — explicit, `[[nodiscard]]`, result mapped -to exit 3 — and `src` being closed silently by the destructor *is* the close +asymmetry between `dst->close()` (explicit, `[[nodiscard]]`, result mapped +to exit 3) and `src` being closed silently by the destructor *is* the close policy from the contract, written in types. **Go** threads one `error` value through everything: `readSome` detects EINTR with `errors.Is(err, unix.EINTR)` rather than `err == unix.EINTR`, so the check @@ -330,7 +330,7 @@ from whatever it is handed using `errors.As`, and `main` gets the exit code back the same way. Note also what Go must do that the others get free: its errno strings are lowercase ("no such file or directory"), so `strerror` capitalizes the first rune to match glibc. **Rust** lets `?` do the -propagation and `map_err` do the classification — each call site names its +propagation and `map_err` do the classification: each call site names its variant, so forgetting to classify a failure is a compile error, and `exit_code()` is an exhaustive `match` the compiler re-checks whenever a variant is added. Its `strerror` has the inverse problem to Go's: `Errno` @@ -340,18 +340,18 @@ suffix is stripped to keep the three outputs byte-identical. ### Errors, three ways `copyx` is the policy statement for the thread that runs through every -chapter, so state the doctrine plainly. In **C++**, errors that are part of -normal operation — a missing file, a full disk — travel as values +chapter, so the doctrine deserves a direct statement here. In **C++**, errors that are part of +normal operation (a missing file, a full disk) travel as values (`std::expected`), and exceptions are reserved for *subsystem boundaries*: the places where a whole component (a parser, a plugin, a request handler) fails as a unit and someone several frames up owns the recovery. Inside syscall-adjacent code, exceptions obscure control flow exactly where you need to see it. In **Go**, the equivalent line is drawn between `error` and `panic`: an `error` is a result, a `panic` is a bug (index out of range, nil -map write) — `copyx` returns errors for everything the kernel can say and +map write), `copyx` returns errors for everything the kernel can say and would panic only on programmer mistakes. In **Rust**, the same line separates `Result` from `panic!`; the book's examples are written to be correct under -`panic = "abort"`, which means a panic is never part of the error path — it +`panic = "abort"`, which means a panic is never part of the error path; it is a crash you fix. On the library-versus-application split: `thiserror` (used here) is for code whose *callers* match on error variants; `anyhow` is for binaries that only report. `copyx` is a binary, but its exit codes make @@ -360,14 +360,14 @@ typed enum earns its boilerplate. ### Concurrency lens -`copyx` is deliberately single-threaded — one loop, one buffer, no shared -state — but the error machinery is already concurrency-load-bearing. C's +`copyx` is deliberately single-threaded (one loop, one buffer, no shared +state), but the error machinery already matters for concurrency. C's `errno` is thread-local precisely so two threads failing simultaneously do not corrupt each other's error state, and `last_error()` copies it into a value *immediately* because the very next libc call on the same thread may overwrite it. Go and Rust sidestep the global entirely: the errno arrives as a return value, so there is nothing thread-local to protect. And the EINTR -policy is really a concurrency policy in disguise — signals are asynchronous +policy is really a concurrency policy in disguise: signals are asynchronous interruptions, and a program whose I/O loops retry EINTR is one whose correctness does not depend on when a signal lands. That property gets load-tested in the Processes, Signals, Privilege part, when `pmon` starts @@ -380,7 +380,7 @@ handling real signals around these same loops. ``` The walkthrough runs the taxonomy per language. Reproduce it by hand against -any of the three binaries — shown here with a 200 000-byte source (sized so +any of the three binaries, shown here with a 200 000-byte source (sized so the final read comes up short of the 64 KiB buffer): ```bash @@ -399,20 +399,20 @@ exit=3 Then run the same three cases with `./go/bin/app` and `./rust/target/release/app`: on this host all three produced the identical three lines and the identical exit codes 0, 2, 3, and `cmp src.bin dst.bin` -exited 0 for each copy — byte-identical output from byte-identical input, +exited 0 for each copy, byte-identical output from byte-identical input, which is the whole claim of the chapter. `/dev/full` is the tool that makes the ENOSPC case repeatable: `ls -li /dev/full` shows a character device with major:minor `1, 7`, a kernel-provided device whose write handler always -returns `-ENOSPC` — a full disk on tap, no root required. The book's runner +returns `-ENOSPC`, a full disk on tap, no root required. The book's runner (`python3 scripts/test-all-examples.py --only 05-errors-three-ways`) drives -eleven behavioral checks per language through `verify.lua` — including the -empty-source edge case (`copied 0 bytes`, exit 0) — and reported +eleven behavioral checks per language through `verify.lua`, including the +empty-source edge case (`copied 0 bytes`, exit 0), and reported `PASS PASS PASS` on this host. ## Cross-check: watch the error being born Everything above shows the error *after* each runtime dressed it up. -`strace -e trace=write` shows the moment it exists — the failing `write(2)` +`strace -e trace=write` shows the moment it exists: the failing `write(2)` itself. On the C++ binary: ```bash @@ -426,23 +426,23 @@ write(2, "\n", 1 Read it bottom-up from the failure: one 65 536-byte `write` to descriptor 4 (the destination; 3 is the source) fails with `ENOSPC`, and the *next* writes -are to descriptor 2 — the program printing its own error message. The entire +are to descriptor 2: the program printing its own error message. The entire journey in Figure 5.1 happened between those two lines, in user space, invisible to the kernel. The Rust binary (`strace -e trace=write ./rust/target/release/app src.bin /dev/full`) fails identically but emits the message as six separate little writes (`"copyx: "`, `"write "`, -`"/dev/full"`, …) — `eprintln!` writing each format fragment straight to +`"/dev/full"`, …), `eprintln!` writing each format fragment straight to unbuffered stderr. The Go binary (add `-f`; the runtime has threads) shows -the failing 65 536-byte write on descriptor **5**, not 4 — this trace shows +the failing 65 536-byte write on descriptor **5**, not 4; this trace shows why: the runtime is already holding `/sys/fs/cgroup/.../cpu.max` open on descriptor 3 for its container-aware scheduling, so the source and destination land one higher. Three runtimes, -three descriptor layouts and stderr habits — and one identical failing +three descriptor layouts and stderr habits, and one identical failing syscall, one identical message, one identical `+++ exited with 3 +++`. -> **On the lab VM** — unverified: -> the aggregate view of this chapter — *which* errnos your whole system's -> syscalls are failing with, live — belongs to eBPF tooling in the class of +> **On the lab VM**: unverified: +> the aggregate view of this chapter (*which* errnos your whole system's +> syscalls are failing with, live) belongs to eBPF tooling in the class of > bpftrace one-liners counting negative syscall returns by errno. That is not > runnable on this host without privileges; Chapter 30 (Debugging part) > exercises it in the `systems-target` VM against these same binaries. @@ -453,8 +453,8 @@ syscall, one identical message, one identical `+++ exited with 3 +++`. register, `ENOSPC` is 28 everywhere on the platform, and glibc's thread-local `errno`, Go's `unix.Errno` value, and rustix's `Errno` are three deliveries of the same integer. -- `copyx`'s taxonomy — exit 2 for the source side, 3 for the destination - side, reasons in `strerror(3)` form — is expressible in all three type +- `copyx`'s taxonomy (exit 2 for the source side, 3 for the destination + side, reasons in `strerror(3)` form) is expressible in all three type systems: `std::expected` mapped into a `Failure`, an `*opError` with `Unwrap` keeping `errors.Is`/`As` truthful, and a `thiserror` enum whose `exit_code()` the compiler keeps exhaustive. @@ -466,10 +466,10 @@ syscall, one identical message, one identical `+++ exited with 3 +++`. catch the exact syscall where an error is born, before any runtime touches it. -Next, the second thread that never stops: **concurrency, three ways** — the +Next, the second thread that never stops: **concurrency, three ways**: the same worker pool as `std::jthread`s and atomics, goroutines and channels, and scoped threads under `Send`/`Sync`. --- -

verified — all evidence in this chapter was produced on the Fedora 44 host this session: each of the three binaries printed copied 200000 bytes (exit 0) with cmp exiting 0, copyx: nope.bin: No such file or directory (exit 2), and copyx: write /dev/full: No space left on device (exit 3); the strace excerpt is real output showing write(4, …, 65536) = -1 ENOSPC before the exit-3; the Go trace showed the failing write on descriptor 5 with cpu.max held on 3, and the Rust trace showed the fragmented stderr writes; ls -li /dev/full showed the 1, 7 character device; and scripts/test-all-examples.py --only 05-errors-three-ways reported PASS for cpp, go, and rust (3 passed, 0 failed, 0 skipped). The lab-VM eBPF callout above is the one unverified item, deferred to Chapter 30.

+

verified: all evidence in this chapter was produced on the Fedora 44 host this session: each of the three binaries printed copied 200000 bytes (exit 0) with cmp exiting 0, copyx: nope.bin: No such file or directory (exit 2), and copyx: write /dev/full: No space left on device (exit 3); the strace excerpt is real output showing write(4, …, 65536) = -1 ENOSPC before the exit-3; the Go trace showed the failing write on descriptor 5 with cpu.max held on 3, and the Rust trace showed the fragmented stderr writes; ls -li /dev/full showed the 1, 7 character device; and scripts/test-all-examples.py --only 05-errors-three-ways reported PASS for cpp, go, and rust (3 passed, 0 failed, 0 skipped). The lab-VM eBPF callout above is the one unverified item, deferred to Chapter 30.

diff --git a/_docs/06-concurrency-three-ways.md b/_docs/06-concurrency-three-ways.md index 8797c93..123bfa8 100644 --- a/_docs/06-concurrency-three-ways.md +++ b/_docs/06-concurrency-three-ways.md @@ -2,22 +2,22 @@ title: "Concurrency, three ways" order: 6 part: "Foundations" -description: "The second thread that runs the whole book — how each language shares work over the same kernel threads. C++23 jthreads and stop tokens, Go's runtime and channels, Rust's compiler-refereed sharing — one parallel checksummer, byte-identical output, and a SIGINT drain you can watch happen." +description: "The second thread that runs the whole book: how each language shares work over the same kernel threads. C++23 jthreads and stop tokens, Go's runtime and channels, Rust's compiler-refereed sharing; one parallel checksummer, byte-identical output, and a SIGINT drain you can watch happen." duration: 50 minutes --- Chapter 5 pinned down the first of this book's two permanent threads: how each language turns an `errno` into something a caller can act on. This chapter -pins down the second — how each language *shares work* — and it does it the +pins down the second, how each language *shares work*, and it does it the only way that is fair: one program, `parhash`, written three times with each language's native concurrency toolkit, required to produce **byte-identical -stdout**. Underneath, all three are the same thing — kernel threads created -with `clone(2)`, parked and woken with `futex(2)` — so what you are really +stdout**. Underneath, all three are the same thing: kernel threads created +with `clone(2)`, parked and woken with `futex(2)`, so what you are really comparing is vocabulary: what C++ hands you as parts, what Go's runtime takes away from you, and what Rust's compiler refuses to let you get wrong. One -scope note up front: everything here is *thread* concurrency. Async — C++ -coroutines, tokio, and what the Go scheduler looks like from below — is -deliberately deferred to Chapter 27, after you have seen the event loops that +scope note up front: everything here is *thread* concurrency. Async (C++ +coroutines, tokio, and what the Go scheduler looks like from below) is +deferred to Chapter 27, after you have seen the event loops that motivate it. The code is in `examples/06-concurrency-three-ways/`. `./demo.sh` builds and @@ -26,10 +26,10 @@ how `verify.lua` checks it. {% include excalidraw.html file="06-three-concurrency-models" - alt="Three columns — C++23, Go, Rust — each showing the same producer, queue, four-worker, cancellation shape: main thread into a mutex/deque/condition_variable_any queue into four jthreads with a sigtimedwait watcher; walker goroutine into an unbuffered channel into four goroutines with signal.NotifyContext; main thread into an Arc-Mutex-wrapped mpsc receiver into four scoped threads with a signal-hook AtomicBool." - caption="Figure 6.1 — one program shape, three vocabularies; same kernel threads underneath." %} + alt="Three columns (C++23, Go, Rust) each showing the same producer, queue, four-worker, cancellation shape: main thread into a mutex/deque/condition_variable_any queue into four jthreads with a sigtimedwait watcher; walker goroutine into an unbuffered channel into four goroutines with signal.NotifyContext; main thread into an Arc-Mutex-wrapped mpsc receiver into four scoped threads with a signal-hook AtomicBool." + caption="Figure 6.1: one program shape, three vocabularies; same kernel threads underneath." %} -> **Tools used** — `cmp`, `truncate`, `python3` (host); bcc-tools +> **Tools used**: `cmp`, `truncate`, `python3` (host); bcc-tools > (`offcputime`, `exitsnoop`) (lab VM, exercised in Part 8). Everything here is > checked by `scripts/check-host.sh`, ships with Fedora, or is preinstalled in > the lab VMs. @@ -39,8 +39,8 @@ how `verify.lua` checks it. `parhash DIR` walks every regular file under `DIR` (symlinks are never followed), streams each through an inline FNV-1a 64 hash with a pool of exactly **4 workers**, then prints one `<16-hex> ` line per file, -sorted, plus a summary. The fixed worker count is part of the contract — the -summary line says `4 workers` in all three languages — and the sort at the end +sorted, plus a summary. The fixed worker count is part of the contract (the +summary line says `4 workers` in all three languages), and the sort at the end is what makes byte-identical output possible at all, because the three languages do not even *walk* the tree in the same order, let alone finish hashing in one. @@ -48,17 +48,17 @@ hashing in one. **C++23** gives you parts, not a machine. `std::jthread` improves on `std::thread` in two ways that this example leans on: it joins in its destructor (a `std::thread` you forget to join calls `std::terminate`), and it -carries a `std::stop_token` — a cooperative cancellation flag that +carries a `std::stop_token`: a cooperative cancellation flag that `condition_variable_any::wait` understands natively. What the standard does *not* give you is everything else: there is no work queue, no channel, no -select, and — critically for this chapter — no signal story. Signals are POSIX +select, and (critically for this chapter) no signal story. Signals are POSIX territory, and mixing them with threads requires a discipline the standard never mentions: block the signal in every thread, then let exactly one thread consume it synchronously. The C++ source is real work: about 230 lines, of which maybe 60 are concurrency plumbing you had to design yourself. **Go** took the opposite bet: the runtime owns the threads and you never see -them. In the GMP scheduler, *goroutines* (G) — cheap, growable-stack tasks — +them. In the GMP scheduler, *goroutines* (G), cheap, growable-stack tasks, are multiplexed by the runtime onto a small set of kernel threads (M), each of which needs a *processor* slot (P, capped by `GOMAXPROCS`, 16 on this host) to run Go code; when a goroutine blocks in a syscall its M can hand the P to @@ -66,19 +66,19 @@ another thread, and idle Ps steal runnable Gs from busy ones. That is the entire mental model this book needs until Chapter 27. What you program against is not the scheduler but the idiom on top: goroutines communicate over channels, `select` races multiple channel operations, and cancellation is a -`context.Context` whose `Done()` channel closes — one convention so universal +`context.Context` whose `Done()` channel closes: one convention so universal that the standard library will hand you a context pre-wired to SIGINT. -**Rust** keeps the threads visible — `std::thread::scope` spawns real kernel -threads, one each — and moves the refereeing into the compiler. `Send` and +**Rust** keeps the threads visible: `std::thread::scope` spawns real kernel +threads, one each, and moves the refereeing into the compiler. `Send` and `Sync` are marker traits computed from a type's contents: `Send` means a value may move to another thread, `Sync` means `&T` may be shared across threads. -The race you would have written in C++ — four workers pushing into one shared -`Vec` because you forgot the mutex — simply does not compile in Rust: `&mut +The race you would have written in C++ (four workers pushing into one shared +`Vec` because you forgot the mutex) does not compile in Rust: `&mut Vec` cannot be aliased across threads, and the compiler says so at the exact line you tried. This example hits a subtler case of the same rule: -`mpsc::Receiver` is not `Sync` — sharing one receiver among four workers is -only legal behind a lock — so the type `Arc>>` in +`mpsc::Receiver` is not `Sync`: sharing one receiver among four workers is +only legal behind a lock, so the type `Arc>>` in the source is not a style choice; it is the shortest thing the compiler would accept. Scoped threads add the ergonomic half: because `scope` guarantees every spawned thread joins before the scope returns, workers may borrow plain @@ -87,8 +87,8 @@ reference counting. ## How the code works -All three sources — `examples/06-concurrency-three-ways/cpp/src/main.cpp`, -`go/main.go`, and `rust/src/main.rs` — share one shape: a producer walks the +All three sources (`examples/06-concurrency-three-ways/cpp/src/main.cpp`, +`go/main.go`, and `rust/src/main.rs`) share one shape: a producer walks the tree and feeds relative paths into a queue; four workers pull paths, hash files, and emit `(relpath, hash)` results; main collects, sorts by path, and prints. The interesting 30 lines of each are the worker/queue mechanics: @@ -200,32 +200,32 @@ func worker(ctx context.Context, root string, paths <-chan string, results chan< }); ``` -**C++ — the queue you had to build.** `TaskQueue` is the channel C++ does not +**C++: the queue you had to build.** `TaskQueue` is the channel C++ does not ship: a `std::deque` guarded by a mutex, with a `condition_variable_any` for -blocking pops. The load-bearing call is the three-argument +blocking pops. The call that has to be right is the three-argument `cv_.wait(lock, st, pred)`: it sleeps until the predicate holds *or the stop token fires*, which is exactly why the type is `condition_variable_any` and -not plain `condition_variable` — only the `_any` variant accepts a +not plain `condition_variable`: only the `_any` variant accepts a `stop_token`. Without it, a `request_stop()` could not wake an idle worker, -and Ctrl-C would hang until the next `push`. On waking, `pop` deliberately -checks `st.stop_requested()` *before* looking at the queue: a cancelled worker +and Ctrl-C would hang until the next `push`. On waking, `pop` checks +`st.stop_requested()` *before* looking at the queue: a cancelled worker refuses work that is queued but unstarted, while a file it is already mid-hash on always finishes, because cancellation is only ever observed between pops. -`close()` is the producer's end-of-input signal — `notify_all`, because every +`close()` is the producer's end-of-input signal: `notify_all`, because every idle worker must wake to see `closed_` and exit. The cancellation source is a watcher `jthread`: `main` blocks SIGINT process-wide with `pthread_sigmask` *before creating any thread* (the mask is inherited, so no thread can be interrupted asynchronously), and the watcher loops on `sigtimedwait` with a 20 ms timeout, consuming the signal synchronously and calling -`cancel.request_stop()` — no async-signal-safety puzzle, no handler, ever. +`cancel.request_stop()`: no async-signal-safety puzzle, no handler, ever. -**Go — the idiom in eleven lines of select.** The worker's first `select` has +**Go: the idiom in eleven lines of select.** The worker's first `select` has a `default` arm, making it a non-blocking poll of `ctx.Done()`; the second races `Done()` against the next path. Both are needed: Go's `select` picks *pseudorandomly* among ready cases, so if paths keep arriving after cancellation, the second select alone could keep choosing work forever. The poll first guarantees a cancelled worker takes no new task. `rel, ok := -<-paths` with `ok == false` is Go's equivalent of `close()` above — the walker +<-paths` with `ok == false` is Go's equivalent of `close()` above: the walker `close(paths)`s in a `defer`, and every worker unblocks. The wiring in `main` is the part worth memorizing: `signal.NotifyContext(context.Background(), os.Interrupt)` gives a context that SIGINT cancels; a `sync.WaitGroup` @@ -233,8 +233,8 @@ counts the four workers; and a fifth goroutine does `wg.Wait(); close(results)` so that main's `for r := range results` terminates exactly when every in-flight hash has landed. That closer goroutine *is* the drain. -**Rust — sharing only what the types allow.** The four `scope.spawn` closures -borrow `root` directly (`let root = &root;`) — legal only because scoped +**Rust: sharing only what the types allow.** The four `scope.spawn` closures +borrow `root` directly (`let root = &root;`), legal only because scoped threads provably join before `main`'s stack frame dies. Each worker locks the shared receiver *only around* `recv()`, so dequeues serialize but hashing runs four-wide; hold the lock across `hash_file` and you would have built a @@ -245,7 +245,7 @@ there is no more work; dropping main's `result_tx` clone means `result_rx`'s iterator ends precisely when the last worker (holding the last sender) exits. Cancellation is the bluntest of the three: `signal_hook::flag::register` points SIGINT at a shared `AtomicBool`, and walker and workers poll it with -`Ordering::Relaxed` — a plain flag needs no ordering guarantees, a point +`Ordering::Relaxed`: a plain flag needs no ordering guarantees, a point Chapter 26 returns to properly. ### Errors, three ways @@ -253,26 +253,26 @@ Chapter 26 returns to properly. The per-file hash function carries this book's Chapter 5 policy into concurrent code. C++'s `hash_file` returns `std::expected` and is the only one of the -three that must handle `EINTR` by hand — its raw `::read` loop retries on +three that must handle `EINTR` by hand: its raw `::read` loop retries on `errno == EINTR`, while Go's `os.File.Read` and Rust's `File::read` retry inside the runtime. Go wraps with `%w` (`fmt.Errorf("open %s: %w", path, err)`) so callers can still `errors.Is` through the context; Rust does the same with `anyhow::Context`. The *policy* is identical everywhere: an -unreadable file is not fatal — `parhash: skipping ` on stderr, keep +unreadable file is not fatal: `parhash: skipping ` on stderr, keep hashing; a missing argument is `usage: parhash DIR` and exit 2; and an interrupt is exit **130**, the shell convention for 128 + SIGINT's number 2. Exit codes are part of the error surface, and `verify.lua` checks all three. ### Concurrency lens -This chapter *is* the lens the rest of the book looks through, so here is the -first reading. The C++ model buys maximum control — you chose the queue, the -wake policy, the signal discipline — at the price of designing all of it. The +This chapter *is* the lens every later chapter looks through, so here is the +first reading. The C++ model buys maximum control (you chose the queue, the +wake policy, the signal discipline) at the price of designing all of it. The Go model is the least code by far and the only one where a runtime stands between you and the kernel. The Rust model costs you a negotiation with the compiler up front and repays it by making the forgotten-mutex bug unrepresentable. From here on, each example states which model it leans on and -why; when the answer becomes "none of these — the problem wants an event +why; when the answer becomes "none of these: the problem wants an event loop," that is Chapter 27's cue. ## Build, run, observe @@ -297,16 +297,16 @@ cbf29ce484222325 zz-empty.dat parhash: 5 files, 4 workers ``` -The empty file's hash is `cbf29ce484222325` — FNV-1a's offset basis, i.e. the -hash of zero bytes — which makes it a nice canary: if that line differs, the +The empty file's hash is `cbf29ce484222325` (FNV-1a's offset basis, i.e. the +hash of zero bytes), which makes it a nice canary: if that line differs, the hash loop itself is wrong, not the concurrency. On this host the full runner pass (`python3 scripts/test-all-examples.py --only 06-concurrency-three-ways`) reports PASS for all three languages, 8 checks each, Go under `-race`. ## Cross-check: cmp, /proc, and a live Ctrl-C -The contract says the three binaries are interchangeable. `cmp` — which -compares byte streams and says nothing only when they are identical — is the +The contract says the three binaries are interchangeable. `cmp` (which +compares byte streams and says nothing only when they are identical) is the right referee. On the fixture above: ```bash @@ -320,13 +320,13 @@ identical ``` Next, the claim that these are all just kernel threads. Give each binary a -tree big enough to keep it busy — `verify.lua`'s trick is 16 small files plus +tree big enough to keep it busy (`verify.lua`'s trick is 16 small files plus 96 files of 64 MiB created with `truncate`, ~6.1 GiB logical, 64 KiB on disk, -so the workers hash streams of zeros for seconds while your SSD does nothing — +so the workers hash streams of zeros for seconds while your SSD does nothing), and count entries in `/proc//task`, one per thread, while it runs. On this host: **C++ 6** (main, four workers, the sigtimedwait watcher), **Rust -5** (main plus four scoped threads — no watcher, since a flag needs none), and -**Go 10** — the runtime decides its own thread count, and under `-race` on a +5** (main plus four scoped threads, no watcher, since a flag needs none), and +**Go 10**: the runtime decides its own thread count, and under `-race` on a 16-CPU host it settled on ten kernel threads for a program whose source names only six concurrent activities. Ask *what the threads are doing* and `/proc` answers that too: @@ -343,15 +343,15 @@ answers that too: That is the C++ binary mid-run: the main thread parked in `futex_do_wait` (the mutex/CV machinery is futexes underneath), the watcher visibly sitting in -`do_sigtimedwait`, and four workers with `wchan` 0 — running, hashing. +`do_sigtimedwait`, and four workers with `wchan` 0: running, hashing. -Last, the drain — the behavior Figure 6.2 sequences and the one a validator +Last, the drain: the behavior Figure 6.2 sequences and the one a validator can only trust if it was watched happening. Start each binary on the sparse tree, wait 400 ms, send SIGINT, and collect streams and exit codes. All three exited **130** with `parhash: interrupted` on stderr, and each printed **20** completed, well-formed hash lines with no summary line. Twenty is not a coincidence to wave past: roughly 16 files were fully hashed when the signal -landed, and the **4 in-flight 64 MiB files — one per worker — all completed +landed, and the **4 in-flight 64 MiB files, one per worker, all completed after SIGINT**, exactly the finish-what-you-hold guarantee all three implementations promise. The partial *sets* differ, and that is instructive: Go's partial output starts `56750b99cb963a67 aa-01.txt` because @@ -363,50 +363,50 @@ program sorts at the end instead of trusting the walk. {% include excalidraw.html file="06-cancellation-drain" - alt="SIGINT fans out as one flipped flag to three parties — the walker stops enumerating, idle workers wake and exit, busy workers finish the file in hand — then the queue closes and results drain into a final box: partial sorted lines, parhash: interrupted, exit 130." - caption="Figure 6.2 — the drain: stop accepting, finish what you hold, report what completed." %} + alt="SIGINT fans out as one flipped flag to three parties: the walker stops enumerating, idle workers wake and exit, busy workers finish the file in hand: then the queue closes and results drain into a final box: partial sorted lines, parhash: interrupted, exit 130." + caption="Figure 6.2: the drain: stop accepting, finish what you hold, report what completed." %} -> **On the lab VM** — unverified: -> bcc-tools would let you watch the drain from the kernel's side — +> **On the lab VM**: unverified. +> bcc-tools would let you watch the drain from the kernel's side: > `offcputime` showing the idle workers parked in futex waits, `exitsnoop` > catching the thread exits as they cascade. Those tools are not runnable on > this host without root; Chapter 30 (Debugging part) builds that toolkit on > the `systems-target` VM and points it at these exact binaries. -One asymmetry to name plainly: Go ran every check under its race detector, -but the C++ pool ran with no equivalent referee — this example's CMake presets -stop at `asan`, and no ThreadSanitizer build was exercised here. -unverified — the host carries +One asymmetry here: Go ran every check under its race detector, +but the C++ pool ran with no equivalent referee (this example's CMake presets +stop at `asan`, and no ThreadSanitizer build was exercised here). +unverified: the host carries the `libtsan` runtime (`scripts/check-host.sh` checks for it) and a TSan probe links and runs, but running *this* worker pool under TSan, and what its -reports look like when the mutex is deliberately removed, is Chapter 29's +reports look like with the mutex removed, is Chapter 29's job. ## What you learned -- All three models are `clone(2)` + `futex(2)` underneath — `/proc//task` +- All three models are `clone(2)` + `futex(2)` underneath: `/proc//task` showed 6, 10, and 5 kernel threads for the same four-worker program, and `wchan` showed who was parked where. - C++23 gives parts (`jthread`, `stop_token`, `condition_variable_any`), and - you assemble the queue and the signal discipline — block SIGINT everywhere, + you assemble the queue and the signal discipline: block SIGINT everywhere, consume it in one thread via `sigtimedwait`, `request_stop()` the rest. - Go's cancellation idiom is structural: `signal.NotifyContext`, workers that `select` on `Done()` before work, and a `WaitGroup`-driven channel close that *is* the drain. -- Rust turns sharing mistakes into type errors — `mpsc::Receiver` is not +- Rust turns sharing mistakes into type errors: `mpsc::Receiver` is not `Sync`, so the compiler forces `Arc>`; scoped threads let workers borrow stack data because the join is guaranteed. - Cancellation-as-drain is a contract you can verify: stop accepting, finish - what you hold, report what completed, exit 130 — and all three languages + what you hold, report what completed, exit 130, and all three languages honored it live. -Next, Part 2 begins with **file descriptors and the VFS** — the first chapter +Next, Part 2 begins with **file descriptors and the VFS**: the first chapter of `fwatch`, where the question stops being "who runs the code" and becomes "what exactly is that small integer the kernel keeps handing us." --- -

verified — every number above +

verified: every number above comes from runs on the Fedora 44 reference host (kernel 7.1) this session: the runner reported 06-concurrency-three-ways PASS PASS PASS (3 passed, 0 failed, Go built with -race); the diff --git a/_docs/07-fds-raii-and-the-vfs.md b/_docs/07-fds-raii-and-the-vfs.md index 2c039f7..b6ce37e 100644 --- a/_docs/07-fds-raii-and-the-vfs.md +++ b/_docs/07-fds-raii-and-the-vfs.md @@ -2,21 +2,21 @@ title: "fds, RAII, and the VFS" order: 7 part: "Files and I/O" -description: "The three-layer model behind every file descriptor — fd table, open file description, inode — proven with /proc and strace, plus openat/fstatat directory-relative walking and an owning fd wrapper in each language, as fwatch v0 takes its first snapshot." +description: "The three-layer model behind every file descriptor (fd table, open file description, inode) proven with /proc and strace, plus openat/fstatat directory-relative walking and an owning fd wrapper in each language, as fwatch v0 takes its first snapshot." duration: 45 minutes --- This chapter starts the second of the book's long-running artifacts: `fwatch`, the file watcher that will grow inotify, epoll, io_uring, and finally a Landlock sandbox over the next several chapters. Version 0 has no events at -all — it is a deterministic snapshot/diff tool — because before you can watch -files change you need to be precise about what a file descriptor actually +all (it is a deterministic snapshot/diff tool) because before you can watch +files change you need a precise model of what a file descriptor actually *is*. The one new idea here is that an fd is not a file: it is the top of a -three-layer chain — a slot in your process's fd table, pointing at a -system-wide *open file description*, pointing at an *inode* — and every flag, +three-layer chain (a slot in your process's fd table, pointing at a +system-wide *open file description*, pointing at an *inode*), and every flag, offset, and ownership question in this part of the book lands on exactly one of those three layers. Along the way we build the owning fd wrapper each -language uses for the rest of the book, and we walk directory trees the safe +language uses throughout the book, and we walk directory trees the safe way: relative to a directory fd, never by re-resolving paths. The code is in `examples/07-fds-raii-and-the-vfs/`. `./demo.sh` there builds @@ -27,9 +27,9 @@ codes all three languages share. {% include excalidraw.html file="07-fd-table-vfs" alt="Three stacked layers: the fwatch process fd table holding fds 0-2 and four dirfds 3-6 with cloexec on, each dirfd pointing down to its own kernel open file description holding flags 02700000 and a readdir position, each description pointing down to a VFS inode numbered 69926 through 69929; dashed openat arrows chain fd 3 to fd 4 to fd 5 to fd 6." - caption="Figure 7.1 — one paused fwatch walk: four dirfds in the per-process fd table, four open file descriptions carrying flags and offset, four inodes; every number in this figure comes from the /proc inspection later in the chapter" %} + caption="Figure 7.1: one paused fwatch walk: four dirfds in the per-process fd table, four open file descriptions carrying flags and offset, four inodes; every number in this figure comes from the /proc inspection later in the chapter" %} -> **Tools used** — `strace`, `stat`, `ls`, `lsof`, `touch`, `cmp`, `python3` +> **Tools used**: `strace`, `stat`, `ls`, `lsof`, `touch`, `cmp`, `python3` > (host); `opensnoop`/`bpftrace` (lab VM, exercised in Part 8). Everything here > is checked by `scripts/check-host.sh`, ships with Fedora, or is preinstalled > in the lab VMs. @@ -41,30 +41,30 @@ and conflating them is the root of most fd bugs: 1. **The fd table** is per-process: a plain array indexed by small integers, which is why fds are always the lowest free number. Each slot holds two - things only — a pointer to an open file description, and the + things only: a pointer to an open file description, and the **close-on-exec bit**. That placement matters: `O_CLOEXEC` is a property of *your process's slot*, not of the underlying open file. 2. **The open file description** is system-wide: created by each `open`, shared by `dup(2)` and inherited across `fork(2)`. It owns the status flags (`O_RDONLY`, `O_NOFOLLOW`, …) and the file offset. Two processes - that inherited the same description share one offset — the classic + that inherited the same description share one offset, the classic surprise where a parent and child interleave writes; two independent `open`s of the same path get two descriptions and two offsets. -3. **The inode** is the VFS object for the file itself — type, size, mtime, +3. **The inode** is the VFS object for the file itself: type, size, mtime, and the number `ls -li` prints. Many descriptions can point at one inode; an inode with zero directory entries *and* zero descriptions is when the data actually goes away (which is why you can unlink a running binary). `fwatch snapshot` exercises the whole chain: it opens directories (fd table entries), enumerates them through their descriptions' readdir offsets, and -records size and mtime — inode fields — for every regular file. The +records size and mtime (inode fields) for every regular file. The ` ` lines it prints, sorted bytewise by path, are inode metadata reached through fd-relative syscalls without ever re-resolving a full path. ## Open flags, and `O_CLOEXEC` everywhere -Every directory open in all three implementations uses the same four flags — +Every directory open in all three implementations uses the same four flags; this is the C++ spelling, from `examples/07-fds-raii-and-the-vfs/cpp/src/main.cpp`: @@ -73,7 +73,7 @@ constexpr int kDirFlags = O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW; ``` `O_DIRECTORY` makes the open fail with `ENOTDIR` if the name stopped being a -directory between `readdir` and `openat` — classification races are real, and +directory between `readdir` and `openat`. Classification races are real, and we would rather get an error than enumerate a file. `O_NOFOLLOW` refuses to traverse a symlink in the final component, so a walk of a tree you do not fully control cannot be steered out of that tree by swapping a subdirectory @@ -81,13 +81,13 @@ for a symlink to `/etc`. `O_CLOEXEC` deserves its own paragraph because the book will put it on *every* fd it ever opens. Without it, every fd you hold is inherited by every -child process you — or any library you link — ever `fork`/`exec`s. That is +child process you (or any library you link) ever `fork`/`exec`s. That is how a watcher ends up keeping a deleted directory alive because some spawned helper still holds fd 5, or how a privileged fd leaks into an unprivileged child. The tempting fix, "open, then set `FD_CLOEXEC` with `fcntl`", has a hole: in a threaded program another thread can `fork`+`exec` in the window between the two calls, and the fd escapes anyway. `O_CLOEXEC` closes that -window by setting the bit *atomically at open time* — which is why it is +window by setting the bit *atomically at open time*, which is why it is baked into `kDirFlags` rather than applied afterwards. Note which layer it lives on: the per-process fd table slot. A `dup` of the fd starts with the bit clear; the open file description neither knows nor cares. @@ -97,22 +97,22 @@ bit clear; the open file description neither knows nor cares. The naive way to walk a tree is string arithmetic: `readdir`, concatenate `dir + "/" + name`, `stat` the result, recurse. Every one of those `stat`s re-resolves the whole path from `/`, so every step is a fresh race: if any -ancestor component is renamed — or replaced by a symlink — between two calls, +ancestor component is renamed (or replaced by a symlink) between two calls, you are suddenly operating on a different file than the one `readdir` told you about. The `*at` syscall family fixes this by making the *directory fd* the anchor: `openat(dirfd, name, …)` and `fstatat(dirfd, name, …, AT_SYMLINK_NOFOLLOW)` resolve exactly one component, relative to a directory you already hold open. Once `fwatch` holds fd 3 on the root of the tree, -renaming that tree out from under it changes nothing — the fd pins the inode, +renaming that tree out from under it changes nothing: the fd pins the inode, not the name. `AT_SYMLINK_NOFOLLOW` completes the discipline for the stat side: classify the entry itself, never what a symlink points at. (`AT_FDCWD`, which `scan` passes for the very first open, is the pseudo-dirfd meaning -"resolve relative to the current working directory" — the only path-based +"resolve relative to the current working directory", the only path-based resolution in the whole program.) This is not only about races. Anchored walks also survive path-length limits, cross into the capability style of `openat2(2)` and `RESOLVE_BENEATH` that -the Landlock chapter will want, and are *faster* — one component resolved per +the Landlock chapter will want, and are *faster*: one component resolved per call instead of the whole prefix every time. ## How the code works @@ -120,8 +120,8 @@ call instead of the whole prefix every time. Two data structures carry the whole program. A `Tree` maps relative path → `(size, mtime)`: C++ uses `std::map` and Rust `BTreeMap` because both iterate in sorted key order, which -makes snapshot output deterministic for free; Go uses a plain -`map[string]info` and sorts the keys explicitly at print time — same +makes snapshot output deterministic without extra work; Go uses a plain +`map[string]info` and sorts the keys explicitly at print time: same observable behavior, sorting moved to the edge. `Info` is two `int64`s; two snapshots differ exactly when a key set or an `Info` differs, which is all `diff` needs. @@ -164,25 +164,25 @@ private: }; ``` -Copying is deleted because two owners means two `close`s — a double close is -worse than a leak, since between the two calls the number may already name +Copying is deleted because two owners means two `close`s, and a double close +is worse than a leak, since between the two calls the number may already name someone else's freshly opened file. Moving transfers ownership by `std::exchange`-ing the source to `-1`, so the moved-from destructor is a no-op. `release()` is the escape hatch for APIs that *take* ownership, and the C++ code needs it immediately: `fdopendir(3)` adopts the fd into a `DIR*`, after which `closedir(3)` is the one and only cleanup. The -`DirStream` class documents that transfer in code — `DirStream::adopt(Fd)` +`DirStream` class documents that transfer in code: `DirStream::adopt(Fd)` calls `fd.release()` only after `fdopendir` succeeds, so on the failure path the `Fd` destructor still closes it. Exactly one owner on every path, including errors. Rust gets the same semantics from the standard library: `rustix::fs::open` and `openat` return `std::os::fd::OwnedFd`, whose `Drop` is the close and -which is move-only by construction — the borrow checker rejects what C++ +which is move-only by construction; the borrow checker rejects what C++ merely deletes. Functions that need the fd without owning it take it as `BorrowedFd` (via `AsFd`, which is why `walk` can pass `&fd` to `statat` and `openat` and still drop `fd` at the end). Go has no destructors, so the walk -wraps each raw dirfd in `os.NewFile` and `defer f.Close()` owns it — one +wraps each raw dirfd in `os.NewFile` and `defer f.Close()` owns it: one close per directory, on every return path. The walk itself is the same function three times: @@ -324,49 +324,49 @@ Structure first: each `walk` takes *ownership* of the directory fd (by-value `Fd`, `*os.File` with `defer Close`, by-value `OwnedFd`), enumerates entries, `fstatat`s each one relative to that fd, records regular files under `join(rel, name)`, and recurses into subdirectories by `openat`-ing a child -fd and handing ownership down. `display` and `rel` are deliberately separate -strings — errors should say `cannot stat 'tree/sub/b.txt'` with the +fd and handing ownership down. `display` and `rel` stay separate +strings: errors should say `cannot stat 'tree/sub/b.txt'` with the user-supplied prefix, while snapshot keys must be root-relative so a snapshot taken as `fwatch snapshot ./tree` compares cleanly against a rescan of the same tree by any other spelling of its path. -Three per-language details are worth dwelling on. In C++, enumeration goes +Three per-language details stand out. In C++, enumeration goes through `DirStream` because `readdir` needs a `DIR*`, but classification and -descent use `stream->fd()` — `dirfd(3)` recovers the underlying fd from the +descent use `stream->fd()`: `dirfd(3)` recovers the underlying fd from the stream, so the anchor for the `*at` calls is the same open file description the stream reads. In Go, `dirfd := int(f.Fd())` extracts the raw integer, which is safe here *only because* `defer f.Close()` keeps `f` reachable for -the whole function — more on that in the concurrency lens. In Rust, +the whole function, covered in the concurrency lens below. In Rust, `Dir::read_from(&fd)` borrows the fd and takes its own handle for the enumeration cursor (you can see it in the strace below as an extra `openat(fd, ".")`), leaving `fd` free to serve as the `statat`/`openat` anchor; entry order comes from the kernel unsorted in all three languages, and the sorted map makes the output deterministic anyway. -The wiring is thin by design. `scan` opens the root (`AT_FDCWD`-relative — +The wiring is thin by design. `scan` opens the root (`AT_FDCWD`-relative, the only full-path resolution), seeds the recursion, and returns the `Tree`. `cmd_snapshot` prints it. `cmd_diff` re-reads a saved snapshot with an -ordinary buffered file read — parsing lines *from the right*, so the last two +ordinary buffered file read, parsing lines *from the right* so the last two space-separated fields are size and mtime and paths containing spaces -round-trip — then walks the sorted union of old and new key sets emitting +round-trip, then walks the sorted union of old and new key sets emitting `+`/`-`/`~` lines and the final `fwatch: A added, R removed, M modified` -summary, which prints even when all counts are zero. Fragile bits, stated -plainly: mtime has one-second granularity here (`st_mtim.tv_sec`), so a -same-second same-size rewrite is invisible to `diff` — the durability chapter -sharpens this; symlinks, sockets, pipes, and devices are skipped entirely in +summary, which prints even when all counts are zero. A few fragile points: +mtime has one-second granularity here (`st_mtim.tv_sec`), so a +same-second same-size rewrite is invisible to `diff` (the durability chapter +sharpens this); symlinks, sockets, pipes, and devices are skipped entirely in v0; and the whole tree lives in memory, which is fine for the trees `fwatch` will watch and wrong for a backup tool. ## Errors, three ways -All three implementations must produce byte-identical diagnostics — the +All three implementations must produce byte-identical diagnostics: the contract is `fwatch: error: cannot open directory 'X' (errno N)` on stderr -and exit 1 — so each language has to recover the *raw errno* from its own +and exit 1, so each language has to recover the *raw errno* from its own error machinery. C++ converts `errno` to `std::error_code` at the syscall wrapper and formats `Failure` messages with `ec.value()`; fallible functions return `std::expected`, and `main` never touches `errno` directly. Go wraps with `%w` (`failf` → `fmt.Errorf`), keeps the chain queryable, and digs the -number back out with `errors.As(err, &errno)` — while `errors.Is(err, +number back out with `errors.As(err, &errno)`, while `errors.Is(err, unix.ENOENT)` handles the one *recoverable* error: an entry that `readdir` reported but that vanished before `fstatat`, which every implementation silently skips rather than failing the walk. Rust matches @@ -377,19 +377,19 @@ carries the path and the errno up to one printer. ## Concurrency lens -`fwatch` v0 is deliberately single-threaded in all three languages — the walk -is a depth-first recursion with one live fd chain — but two concurrency -points are already load-bearing. First, `O_CLOEXEC`: the fd table is shared +`fwatch` v0 stays single-threaded in all three languages (the walk +is a depth-first recursion with one live fd chain), but two concurrency +points already matter. First, `O_CLOEXEC`: the fd table is shared by every thread in the process, and the fork+exec window it closes is a -*threading* hazard — even "single-threaded" Go code runs on a multithreaded +*threading* hazard: even "single-threaded" Go code runs on a multithreaded runtime that may spawn processes on your behalf. Second, Go's `os.File` has a cleanup registered with the runtime: if a `File` becomes unreachable, the -garbage collector closes its fd for you. That sounds like RAII but is not — +garbage collector closes its fd for you. That resembles RAII but isn't: it is nondeterministic, and it means `int(f.Fd())` is a loan against `f`'s liveness: hold the raw integer past the last reference to `f` and the collector can close the fd while you are still using the number, or worse, after the kernel has reassigned it. The `defer f.Close()` in `walk` is -therefore doing double duty — deterministic cleanup *and* a liveness anchor +therefore doing double duty: deterministic cleanup *and* a liveness anchor for `dirfd`. Rust makes the same mistake unrepresentable (`BorrowedFd`'s lifetime is tied to the `OwnedFd`), and C++ trusts you to keep the `Fd` alive, which is why the book's wrapper never hands out an owning raw int @@ -402,7 +402,7 @@ except through the explicitly named `release()`. ``` Each language builds, snapshots a fixture tree into a temp file, mutates the -tree, and diffs. To drive it by hand the way the rest of this chapter does, +tree, and diffs. To drive it by hand the way the walkthrough below does, make a small tree with pinned mtimes and snapshot it: ```bash @@ -419,8 +419,8 @@ sub/deep/c.bin 3 1000000200 ``` On this host the Go and Rust binaries print those same four lines, and -`cmp` between the three captured outputs confirms they are byte-identical — -that identical observable behavior is what `verify.lua` asserts, and the +`cmp` between the three captured outputs confirms they are byte-identical, +and that identical observable behavior is what `verify.lua` asserts, and the runner (`python3 scripts/test-all-examples.py --only 07-fds-raii-and-the-vfs`) reports `PASS PASS PASS` for cpp, go, and rust. @@ -440,7 +440,7 @@ inodes agree, field for field: 68628 9 1000000100 tree/sub/b.txt ``` -`ls` renders `mtime` 1000000000 as `Sep 8 2001` — the same inode field, +`ls` renders `mtime` 1000000000 as `Sep 8 2001`, the same inode field formatted; the sizes match the snapshot's second column exactly. **The syscalls, under `strace`.** Filtering the C++ binary to the calls the @@ -464,11 +464,11 @@ newfstatat(5, "c.bin", {st_mode=S_IFREG|0644, st_size=3, ...}, AT_SYMLINK_NOFOLL (Loader lines for `ld.so.cache` and the shared libraries trimmed.) Read the first arguments: only the root open uses `AT_FDCWD`; every later `openat` and -`newfstatat` anchors on fd 3, 4, or 5 — the dirfd chain, exactly as Figure +`newfstatat` anchors on fd 3, 4, or 5: the dirfd chain, exactly as Figure 7.1 draws it, with every open carrying `O_CLOEXEC|O_NOFOLLOW|O_DIRECTORY`. The kernel also reports unsorted entry order (`gone.txt` before `a.txt`); the sorted output really does come from the map. The Rust binary shows one -extra `openat(3, ".", …) = 4` per level — `Dir::read_from` acquiring its own +extra `openat(3, ".", …) = 4` per level: `Dir::read_from` acquiring its own cursor handle, just as its comment says. **The fd table of a live walk, in `/proc`.** `snapshot` finishes its walk @@ -491,12 +491,12 @@ mnt_id: 59 ino: 69929 ``` -(fds 0–2 trimmed.) Four dirfds, one per recursion level — the walk holds +(fds 0–2 trimmed.) Four dirfds, one per recursion level: the walk holds `O(depth)` fds, not `O(files)`. The `fdinfo` file reads out the *description* layer: `pos: 1980` is the readdir cursor partway through l3's 3000 entries, and `flags: 02700000` decodes as `O_CLOEXEC` (02000000) + `O_NOFOLLOW` (0400000) + `O_DIRECTORY` (0200000) + `O_LARGEFILE` (0100000, added by the -64-bit ABI) — the close-on-exec bit, verified on a live process. The `ino:` +64-bit ABI): the close-on-exec bit, verified on a live process. The `ino:` lines completed the loop on this host: fds 3–6 reported inodes 69926–69929, and `ls -lid big big/l1 big/l1/l2 big/l1/l2/l3` printed 69926, 69927, 69928, and 69929 for the same directories, while @@ -504,7 +504,7 @@ and 69929 for the same directories, while `NODE` numbers. fd table → description → inode: each layer observed with a different tool, all agreeing. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the eBPF view of this chapter is watching `openat` flags fleet-wide with > `opensnoop`/`bpftrace` instead of per-process `strace`; bcc-tools are not > runnable on this host without privileges, and chapter 30 (Debugging part) @@ -513,7 +513,7 @@ different tool, all agreeing. ## What you learned - A file descriptor is three layers: a per-process fd-table slot (home of the - close-on-exec bit), a shared open file description (flags and offset — + close-on-exec bit), a shared open file description (flags and offset; `/proc//fdinfo` reads it out), and the inode (`ls -li`'s numbers, `fwatch`'s size and mtime). - `O_CLOEXEC` at open time is the only race-free spelling, and @@ -534,7 +534,7 @@ promises. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host this session: the runner printed 07-fds-raii-and-the-vfs PASS PASS PASS (3 passed, 0 failed, 0 skipped); all three binaries emitted the diff --git a/_docs/08-page-cache-and-durability.md b/_docs/08-page-cache-and-durability.md index 3086907..67e9918 100644 --- a/_docs/08-page-cache-and-durability.md +++ b/_docs/08-page-cache-and-durability.md @@ -2,20 +2,20 @@ title: "The page cache and durability" order: 8 part: "Files and I/O" -description: "What write(2) actually promises — and what durability costs: dirty pages and writeback, fsync vs fdatasync vs sync_file_range, O_DIRECT, and the atomic-rename pattern, all measured with one tri-language benchmark on real hardware." +description: "What write(2) actually promises, and what durability costs: dirty pages and writeback, fsync vs fdatasync vs sync_file_range, O_DIRECT, and the atomic-rename pattern, all measured with one tri-language benchmark on real hardware." duration: 45 minutes --- Chapter 7 ended with a file descriptor safely wrapped in RAII and a byte successfully written through the VFS. This chapter is about the uncomfortable question that follows: written *where*? A successful `write(2)` puts your -bytes in kernel memory — the **page cache** — and nothing more; if the power +bytes in kernel memory (the **page cache**) and nothing more; if the power fails a second later, they are gone, and no error will ever tell you so. The gap between "the syscall returned 0 errors" and "the bytes are on the device" is where databases earn their reputations, and this chapter measures that gap with `iobench`, a small write benchmark implemented identically in C++23, Go, and Rust. On this host the same 64 MiB costs 21 ms to "write" and 138 ms to -write *durably* — and by the end you will have watched the difference happen +write *durably*, and by the end you will have watched the difference happen in `/proc/meminfo`, `/proc//io`, and an `strace` syscall table. The code is in `examples/08-page-cache-and-durability/`. `./demo.sh` there @@ -25,9 +25,9 @@ output format, and exit codes shared by all three. {% include excalidraw.html file="08-write-path-page-cache" alt="Three columns over user space, kernel, and storage bands. In buffered mode, write(2) copies into the page cache and data reaches the NVMe device either by later asynchronous writeback or by an fdatasync at the end; in fsync-every mode an fdatasync every N blocks forces it down; in direct mode an O_DIRECT write bypasses the page cache and DMAs straight from the user buffer to the device." - caption="Figure 8.1 — one write path, three flush points: where each iobench mode hands bytes to the device" %} + caption="Figure 8.1: one write path, three flush points: where each iobench mode hands bytes to the device" %} -> **Tools used** — `strace`, `stat`, `grep`, `python3` (host); +> **Tools used**: `strace`, `stat`, `grep`, `python3` (host); > `cachestat`/`biolatency` (lab VM, exercised in Part 8). Everything here is > checked by `scripts/check-host.sh`, ships with Fedora, or is preinstalled in > the lab VMs. @@ -38,23 +38,23 @@ The contract of a successful buffered `write(2)` is precise and smaller than intuition suggests: the kernel has copied your bytes into page-cache pages belonging to the file, marked those pages **dirty**, and made the data visible to every subsequent `read(2)` from any process. That is all. -Durability is explicitly not included — the manpage says so — and the return +Durability is explicitly not included (the manpage says so), and the return value is really just confirming a memcpy into kernel memory. That is why `iobench --mode buffered` reports 3,013 MiB/s on this host's NVMe drive: the timed loop never touches the device. You are benchmarking `memcpy` plus bookkeeping. -The cache earns its keep in both directions. On reads, the kernel notices -sequential access and issues **readahead** — it fetches pages you have not +The cache helps in both directions. On reads, the kernel notices +sequential access and issues **readahead**: it fetches pages you have not asked for yet, so a sequential reader almost never blocks on the device (`posix_fadvise(2)` lets you tune or defeat this; `iobench` is write-only, so readahead sits out this chapter's measurements). On writes, dirty pages -decouple your process from device latency entirely — until something forces +decouple your process from device latency entirely, until something forces the issue. Dirty pages do not accumulate forever. Kernel **writeback** (flusher) threads write them to the device in the background, driven by two kinds of pressure: -age (`vm.dirty_expire_centisecs`, 30 s by default — no page stays dirty +age (`vm.dirty_expire_centisecs`, 30 s by default: no page stays dirty longer) and volume (`vm.dirty_background_ratio` starts background writeback; at `vm.dirty_ratio` the kernel begins throttling the writers themselves, and your "memcpy-speed" `write(2)` suddenly runs at device speed). The systemwide @@ -66,20 +66,20 @@ section watches balloon by a gigabyte and collapse again. When you need the bytes on the device *now*, you pick one of three calls, in decreasing order of guarantee: -- **`fsync(2)`** — writes the file's dirty pages *and* its metadata (size, +- **`fsync(2)`**: writes the file's dirty pages *and* its metadata (size, timestamps, block allocations) to the device, and asks the device to flush its own volatile write cache. When it returns, the data survives power loss. -- **`fdatasync(2)`** — the same, minus metadata that is not needed to read +- **`fdatasync(2)`**: the same, minus metadata that is not needed to read the data back. An mtime update does not gate your bytes, so skipping that metadata write can save a device round trip. A changed file *size* is data-critical metadata and is still flushed. `iobench` uses `fdatasync` throughout, and the append-heavy write pattern of a database WAL is exactly the case it was designed for. -- **`sync_file_range(2)`** — the expert-mode footgun: it can start or wait on +- **`sync_file_range(2)`**: the expert-mode footgun: it can start or wait on writeback for a byte range, but it never writes metadata and never issues a device cache flush, so it is **not** a durability primitive. Its real use - is pacing — nudging writeback along so a later `fdatasync` has less to do — + is pacing: nudging writeback along so a later `fdatasync` has less to do, and its manpage warning ("extremely dangerous" as a durability mechanism) is one of the better reads in section 2. @@ -87,18 +87,18 @@ decreasing order of guarantee: `write(2)` loop, then pays for a single closing `fdatasync` on a separate `fsync_ms=` line, so the price of durability is printed as its own number. `--mode fsync-every` moves `fdatasync` *inside* the timed loop, every N -blocks — durable-as-you-go, the way a database commits. +blocks, durable-as-you-go, the way a database commits. ## O_DIRECT, and getting renames to keep promises The third mode opts out of the cache entirely. Opening with **`O_DIRECT`** -makes each `write(2)` DMA straight from your user buffer to the device — +makes each `write(2)` DMA straight from your user buffer to the device: no copy into the page cache, no dirty pages, no writeback. The price is a contract: buffer address, file offset, and transfer length must all be aligned to the device's logical block size (4,096 bytes covers current hardware), and a filesystem may simply refuse the flag at `open(2)` with -`EINVAL` — which `iobench` treats as a documented outcome (exit 4), because -it genuinely varies by filesystem. +`EINVAL`, which `iobench` treats as a documented outcome (exit 4), because +it varies by filesystem. Why do databases insist on it? Because they already maintain their own buffer pool, sized and evicted with knowledge of the workload that the kernel cannot @@ -106,19 +106,19 @@ have; caching every page twice wastes half the RAM. And because a database needs to *know* when bytes are on disk to order its writes (WAL before data page), a cache that says "done" when nothing happened is a liability. Note what `O_DIRECT` does **not** give you: the device's own write cache is still -in play, so even direct I/O needs `fdatasync` (or `O_DSYNC`) for real +in play, so even direct I/O needs `fdatasync` (or `O_DSYNC`) to guarantee durability. The last durability tool is not a syscall but a pattern. To replace a config -file so that readers see either the old version or the new one — never a -torn half — you use **atomic rename**: write the new content to a temporary +file so that readers see either the old version or the new one, never a +torn half, you use **atomic rename**: write the new content to a temporary file in the *same directory*, `fsync` the temp file, `rename(2)` it over the target (rename within a filesystem is atomic), then `fsync` the *directory* -fd so the rename itself — a directory entry, i.e. metadata living in the -directory, as chapter 7's inode walk showed — is on disk too. Skip step 2 and +fd so the rename itself (a directory entry, i.e. metadata living in the +directory, as chapter 7's inode walk showed) is on disk too. Skip step 2 and a crash can leave you a correctly named empty file; skip step 4 and the old -name can come back after reboot. `iobench` deliberately does not do this — a -benchmark should overwrite in place — but every "save settings" feature you +name can come back after reboot. `iobench` does not do this: a +benchmark should overwrite in place, but every "save settings" feature you ship should. ## How the code works @@ -291,29 +291,29 @@ fn run(opt: &Options) -> Result { Walk the shared skeleton. **The open** adds `O_DIRECT` to `O_WRONLY | O_CREAT | O_TRUNC` only in direct mode, and only the direct-mode -`EINVAL` is treated as the "unsupported filesystem" outcome — an `EINVAL` +`EINVAL` is treated as the "unsupported filesystem" outcome; an `EINVAL` from a buffered open would be a real bug and falls through to the generic error path. **The block** is built once and rewritten every iteration: 64 KiB of `i & 0xFF` bytes at a 4,096-aligned address. C++ gets alignment from `std::aligned_alloc` (freed via a `unique_ptr` with a `FreeDeleter`); Go and Rust over-allocate by one alignment unit and slice to the first aligned -offset inside — Go's comment notes the heap does not move allocations, and +offset inside; Go's comment notes the heap does not move allocations, and Rust keeps the backing `Vec` alive in a struct so the aligned range stays valid. The alignment is mandatory for `O_DIRECT` and harmless otherwise, which is why all three modes share one buffer. **The write** is never a bare `write(2)`. Each language has a `write_all` that loops: a short write advances the slice/span by the returned count, and -`EINTR` — the call interrupted by a signal before writing anything — retries. +`EINTR` (the call interrupted by a signal before writing anything) retries. A 64 KiB write nearly always completes in one call, but "nearly always" is exactly the assumption this book keeps telling you not to compile in. **The sync placement** is the whole experiment: fsync-every calls `fdatasync` inside the timed region every `--every` blocks (plus a trailing sync when the -block count is not a multiple of N — drop that and up to N−1 blocks quietly +block count is not a multiple of N; drop that and up to N−1 blocks quietly escape the durability guarantee being measured), while buffered mode stops the clock, prints the report line, and only then times its single `fdatasync` as `fsync_ms=`. Elapsed time is clamped to at least 1 ns before the -throughput division — on tmpfs a tiny run genuinely can round to 0 ms. +throughput division: on tmpfs a tiny run can round to 0 ms. ### Errors, three ways @@ -326,26 +326,26 @@ wrapped into the general error path (exit 1). C++ compares the comparison portable rather than testing raw `errno`. Go wraps with `%w` and tests with `errors.Is(err, unix.EINVAL)`, so classification still works through layers of context. Rust matches `Err(Errno::INVAL)` *before* -converting into `anyhow::Error` — the guard sits in the `match` because after +converting into `anyhow::Error`: the guard sits in the `match` because after `.context()` the typed errno is boxed away and would need a downcast. Usage errors are a separate exit code (2) diagnosed entirely in `parse_args`, -before any file is created — `verify.lua` checks that a bad `--mode` leaves +before any file is created; `verify.lua` checks that a bad `--mode` leaves nothing on disk. ### Concurrency lens -`iobench` is single-threaded on purpose, but the system it measures is not. +`iobench` is single-threaded by design, but the system it measures is not. The moment the write loop starts dirtying pages, kernel flusher threads wake -and begin writing them back *concurrently with the loop* — in the cross-check +and begin writing them back *concurrently with the loop*; in the cross-check below, a 2 GiB write peaks at only ~1.1 GiB dirty because writeback was draining the pool while the program filled it. The final `fdatasync` is therefore a *rendezvous*, not the whole transfer: it waits for whatever writeback has not already covered. Language runtimes add one more wrinkle from chapter 6: a blocking `fdatasync` parks the calling OS thread, so in Go -the runtime hands the P to another thread and other goroutines keep running — +the runtime hands the P to another thread and other goroutines keep running, one reason "just fsync in a goroutine" feels free until you count threads. And timing uses `steady_clock` / `time.Now` monotonic readings / -`Instant` — wall-clock time can step backwards; monotonic clocks are the only +`Instant`: wall-clock time can step backwards; monotonic clocks are the only thing you should ever subtract. ## Build, run, observe @@ -368,13 +368,13 @@ mode=direct bytes=67108864 ms=81 MiB/s=785.2 ``` Read the buffered pair together: 21 ms to "write" 64 MiB, then 17 ms of -`fdatasync` — durability nearly doubles the end-to-end time, and the 3,013 +`fdatasync`: durability nearly doubles the end-to-end time, and the 3,013 MiB/s headline was never a device number. Syncing every 8 blocks (512 KiB) drops throughput 6.5× to 462.8 MiB/s because the device's flush latency now sits inside the loop 128 times. `O_DIRECT` lands between them at 785.2 MiB/s: every byte really moved to the device inside the timed region, with no copy and no cache. Go and Rust agree with the C++ numbers (buffered 3,283.9 -and 3,483.6 MiB/s with `fsync_ms=17` and `14` on the same runs) — the page +and 3,483.6 MiB/s with `fsync_ms=17` and `14` on the same runs); the page cache does not care which language dirtied it. After any run, `stat -c %s bench.bin` reports exactly 67108864: same bytes, very different guarantees. @@ -383,7 +383,7 @@ guarantees. The claims above are checkable from the outside. First, the syscall pattern: `strace -c` on a 4 MiB fsync-every run with `--every 4` should show 64 data -writes (4 MiB / 64 KiB) and 16 `fdatasync`s (64 / 4). It does — the 65th +writes (4 MiB / 64 KiB) and 16 `fdatasync`s (64 / 4). It does: the 65th write is the report line to stdout: ``` @@ -405,9 +405,9 @@ Dirty: 22520 kB # after exit — fdatasync drained it ``` A gigabyte of "written" data sat in RAM mid-run (and only ~1.1 GiB of the 2 -GiB was dirty at the sample — concurrent writeback was already draining), and -the closing `fdatasync` — 424 ms on that run, against 661 ms for the whole 2 -GiB write loop — put the rest on the device. Third, the per-process view in +GiB was dirty at the sample, concurrent writeback was already draining), and +the closing `fdatasync` (424 ms on that run, against 661 ms for the whole 2 +GiB write loop) put the rest on the device. Third, the per-process view in `/proc//io` during that same 2 GiB run: ``` @@ -426,12 +426,12 @@ sample is the chapter's thesis in two lines: all 2 GiB of `write(2)` calls had returned while the process was still alive *waiting inside `fdatasync`*. Finally, the same binary on tmpfs (this book's scratchpad) shows why none of the magnitudes are portable: buffered 3,547.2 MiB/s with `fsync_ms=0`, -fsync-every 5,113.2, direct 5,304.7 — with no device behind the file, +fsync-every 5,113.2, direct 5,304.7, with no device behind the file, `fdatasync` is nearly free, `O_DIRECT` is accepted but bypasses nothing, and the three modes converge. That is precisely why `verify.lua` asserts output shape and exact file sizes, never throughput. -

unverified — On the +

unverified: On the lab VM: the bcc-tools pair cachestat (page-cache hit/miss counts while iobench runs) and biolatency (the block I/O latency histogram that separates fsync-every's many small flushes from @@ -443,17 +443,17 @@ host; chapter 30 (Debugging part) exercises them on the lab VM.

- A successful `write(2)` means "copied into the page cache", nothing more; dirty pages reach the device via writeback (age and pressure driven) or - when you say so — and on this host that "so" cost 17 ms atop a 21 ms + when you say so, and on this host that "so" cost 17 ms atop a 21 ms 64 MiB write. - `fsync` flushes data + metadata, `fdatasync` skips metadata that does not gate reading the data back, and `sync_file_range` is writeback pacing, not durability. - `O_DIRECT` trades the cache for alignment rules and an `EINVAL`-at-open - probe — and databases take that trade to own their caching and write + probe, and databases take that trade to own their caching and write ordering; crash-safe file replacement is tmpfile → `fsync` → `rename` → `fsync(dir)`. - `strace -c`, `/proc/meminfo`'s `Dirty:`, and `/proc//io` let you - audit any program's durability behavior from outside — 2 GiB of `wchar` + audit any program's durability behavior from outside: 2 GiB of `wchar` with the process still parked in `fdatasync` is the promise gap made visible. @@ -462,7 +462,7 @@ Next, **event loops**: `epoll` and `inotify`, where the question stops being --- -

verified — all numbers above +

verified: all numbers above are from runs performed on the reference host (btrfs on NVMe, Fedora 44, kernel 7.1) while writing this chapter: the mode reports (ms=21 / fsync_ms=17, ms=138, ms=81 at @@ -472,5 +472,5 @@ sequence 8684 → 1118476 → 22520 kB around a 2 GiB buffered run (ms=661, fsync_ms=424), the /proc/<pid>/io samples, the tmpfs convergence numbers, and the runner gate (test-all-examples.py: cpp PASS, go PASS, rust -PASS — 15 behavioral checks per language). The bcc-tools callout above is the +PASS, 15 behavioral checks per language). The bcc-tools callout above is the one unverified item, deferred to the lab VM.

diff --git a/_docs/09-event-loops-epoll-inotify.md b/_docs/09-event-loops-epoll-inotify.md index 9365b2a..dd8ed8d 100644 --- a/_docs/09-event-loops-epoll-inotify.md +++ b/_docs/09-event-loops-epoll-inotify.md @@ -2,20 +2,20 @@ title: "Event loops: epoll and inotify" order: 9 part: "Files and I/O" -description: "The readiness model — nonblocking fds multiplexed by one epoll loop, level versus edge triggering, inotify's per-watch queues and overflow, and timerfd/signalfd as ordinary readable fds — as fwatch v1 grows a real-time watch subcommand and Go's select turns out to be the same kernel machinery wearing channels." +description: "The readiness model: nonblocking fds multiplexed by one epoll loop, level versus edge triggering, inotify's per-watch queues and overflow, and timerfd/signalfd as ordinary readable fds, as fwatch v1 grows a real-time watch subcommand and Go's select turns out to be the same kernel machinery wearing channels." duration: 50 minutes --- Chapter 8 closed on a question durability cannot answer: "which of ten -thousand fds is ready?" This chapter answers it with the **readiness model** -— the kernel will tell you *when an fd has something for you*, and a program +thousand fds is ready?" This chapter answers it with the **readiness model**: +the kernel will tell you *when an fd has something for you*, and a program built on that promise never blocks anywhere except the one call that asks. `fwatch` v1 keeps the polling-era `snapshot` and `diff` subcommands from Chapters 7 and 8 and adds `watch`: a real-time watcher that reports `created`/`modified`/`deleted` per file, debounced 100 ms per path. The C++ -and Rust versions build it the explicit way — one `epoll` loop over three -kernel fds (inotify, timerfd, signalfd), a single thread, nothing hidden — -while the Go version inverts the picture: the runtime's netpoller owns the +and Rust versions build it the explicit way: one `epoll` loop over three +kernel fds (inotify, timerfd, signalfd), a single thread, every step visible +in source, while the Go version inverts the picture: the runtime's netpoller owns the epoll loop, and the same three event sources surface as channels in a `select`. Same kernel machinery, two programming surfaces, and by the end you will have watched both under `strace` and read epoll's own interest list @@ -28,10 +28,10 @@ codes all three languages share. {% include excalidraw.html file="09-event-loop-anatomy" - alt="Two bands. In the user-space band, four loop steps run left to right — arm timerfd, epoll_wait, dispatch on the data token, drain and act — with a return arrow labeled loop. In the kernel band, an eventpoll interest-list box holds tfd 3, 4, 5, fed by three source boxes: the inotify fd with wd 1 and mask 0x3c6, the CLOCK_MONOTONIC timerfd, and the signalfd carrying blocked SIGINT and SIGTERM; an amber readiness arrow carries EPOLLIN and the token back up into epoll_wait." - caption="Figure 9.1 — the anatomy of fwatch watch: one loop, three event sources; every fd number, token, and mask in this figure reappears verbatim in the /proc inspection later in the chapter" %} + alt="Two bands. In the user-space band, four loop steps run left to right (arm timerfd, epoll_wait, dispatch on the data token, drain and act) with a return arrow labeled loop. In the kernel band, an eventpoll interest-list box holds tfd 3, 4, 5, fed by three source boxes: the inotify fd with wd 1 and mask 0x3c6, the CLOCK_MONOTONIC timerfd, and the signalfd carrying blocked SIGINT and SIGTERM; an amber readiness arrow carries EPOLLIN and the token back up into epoll_wait." + caption="Figure 9.1: the anatomy of fwatch watch: one loop, three event sources; every fd number, token, and mask in this figure reappears verbatim in the /proc inspection later in the chapter" %} -> **Tools used** — `strace`, `mktemp`, `stat`, `ls`, `python3` (host); +> **Tools used**: `strace`, `mktemp`, `stat`, `ls`, `python3` (host); > `bpftrace` (lab VM, exercised in Part 8). Everything here is checked by > `scripts/check-host.sh`, ships with Fedora, or is preinstalled in the lab VMs. @@ -39,25 +39,25 @@ codes all three languages share. A blocking `read(2)` couples two things that have no business being coupled: *waiting* for data and *moving* it. With one input source that is fine; with -three, a blocking read on the wrong fd deadlocks you — while you wait for a +three, a blocking read on the wrong fd deadlocks you: while you wait for a file event, the timeout you promised never fires. The classic escape was one thread per source; the scalable escape is to decouple: put every fd in -**nonblocking** mode (`IN_NONBLOCK`, `TFD_NONBLOCK`, `SFD_NONBLOCK` — all +**nonblocking** mode (`IN_NONBLOCK`, `TFD_NONBLOCK`, `SFD_NONBLOCK`, all `O_NONBLOCK` under different names), so `read` either returns data immediately or fails with `EAGAIN`, and delegate all the *waiting* to one syscall that can wait on everything at once. That syscall is `epoll_wait(2)`, and its shape explains its scalability. You -create a kernel object with `epoll_create1(2)` — it is itself an fd, of type -`anon_inode:[eventpoll]` — then *register* interest once per fd with +create a kernel object with `epoll_create1(2)` (it is itself an fd, of type +`anon_inode:[eventpoll]`) then *register* interest once per fd with `epoll_ctl(EPOLL_CTL_ADD, …)`. The registration carries two things: an event -mask (`EPOLLIN` — readable) and a caller-chosen 64-bit cookie +mask (`EPOLLIN`, readable) and a caller-chosen 64-bit cookie (`epoll_event.data`) the kernel returns verbatim when that fd fires. The interest list lives **in the kernel**, which is the whole advantage over the older `poll(2)`/`select(2)`: those pass the entire fd set on every call, so each wait is O(fds); epoll pays the setup cost once and each wakeup returns only the fds that are actually ready. `fwatch` has three fds so any of them -would work — the point of learning epoll on three is that the loop is +would work; the point of learning epoll on three is that the loop is *identical* at thirty thousand, which is where Part 6's network servers take it. @@ -67,10 +67,10 @@ Registration has one decision that changes program correctness, not just performance: **when** does a ready fd count as ready again? - **Level-triggered** (the default): `epoll_wait` reports the fd as long as - data *remains* — a fact about the fd's state. Forget to read everything and + data *remains*, a fact about the fd's state. Forget to read everything and the next wait simply reports it again. - **Edge-triggered** (`EPOLLET`): the fd is reported only on the - *transition* to readable — a fact about a state change. Miss data and no + *transition* to readable, a fact about a state change. Miss data and no further wakeup comes until *new* data arrives; the only correct consumer is one that drains the fd to `EAGAIN` on every wakeup, without exception. @@ -78,26 +78,26 @@ This book uses level triggering everywhere it writes an epoll loop, because level converts a whole class of lost-wakeup bugs into mere extra wakeups. The discipline it demands instead is smaller but real: **consume what made the fd readable**. That is why the loop below reads the timerfd's 8-byte -expiration counter even though it only wants "the timer fired" — leave those +expiration counter even though it only wants "the timer fired"; leave those 8 bytes and the timerfd stays readable, `epoll_wait` returns instantly forever, and your loop spins at 100% CPU. Note that `fwatch` drains its inotify fd to `EAGAIN` anyway: with level triggering that is an optimization (fewer wakeups per burst, and each burst is timestamped coherently); under edge it would be the difference between working and silently stalling. -> **EPOLLONESHOT caveat** — there is a third mode you should know exists +> **EPOLLONESHOT caveat**: there is a third mode you should know exists > before you need it. When *multiple threads* call `epoll_wait` on one epoll > fd, a level-triggered fd that still has data can be reported to two > threads at once, and even an edge can race a second event while thread one > is mid-read. `EPOLLONESHOT` disarms the registration after each report; > the owning thread must re-arm with `EPOLL_CTL_MOD` when done. It is the -> standard fix for multi-threaded epoll — and the reason this book's +> standard fix for multi-threaded epoll, and the reason this book's > single-threaded loops do not need it. One loop, one thread, no re-arm > protocol: that simplicity is a feature, not a limitation. Interestingly, the readiness style is observable per implementation: the strace later in this chapter shows Go's netpoller registering our inotify fd -with `EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET` — the runtime is an +with `EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET`: the runtime is an edge-triggered consumer (it always drains to `EAGAIN` and parks goroutines on the transition), while our C++ and Rust loops register plain level `EPOLLIN`. @@ -105,28 +105,28 @@ on the transition), while our C++ and Rust loops register plain level ## inotify: per-watch queues, and where they overflow `inotify_init1(2)` returns an fd; `inotify_add_watch(2)` attaches a **watch** -to an inode — directory or file — with a mask of event types, and returns a +to an inode (directory or file) with a mask of event types, and returns a small integer *watch descriptor* (`wd`) that names the watch in every event. `fwatch` watches one directory with six mask bits: `IN_CREATE`, `IN_MODIFY`, -`IN_ATTRIB`, `IN_DELETE`, `IN_MOVED_FROM`, `IN_MOVED_TO` — together `0x3c6`, +`IN_ATTRIB`, `IN_DELETE`, `IN_MOVED_FROM`, `IN_MOVED_TO`, together `0x3c6`, a number you will meet again in `/proc`. Reading the fd yields packed -variable-length records — `wd`, `mask`, `cookie`, `len`, then `len` bytes of -NUL-padded name — so all three implementations walk a byte buffer, stepping +variable-length records (`wd`, `mask`, `cookie`, `len`, then `len` bytes of +NUL-padded name), so all three implementations walk a byte buffer, stepping by `sizeof(header) + len`. A record with `len == 0` is an event on the watched object itself (we skip those), and a rename arrives as a `IN_MOVED_FROM`/`IN_MOVED_TO` pair joined by `cookie`. Semantics worth being precise about: events queue **per inotify instance**, in order, and the kernel already coalesces *identical adjacent* events. The -queue is finite — on this host `/proc/sys/fs/inotify/max_queued_events` is -16384 (with `max_user_watches` 524288 and `max_user_instances` 512) — and +queue is finite: on this host `/proc/sys/fs/inotify/max_queued_events` is +16384 (with `max_user_watches` 524288 and `max_user_instances` 512), and when it fills, the kernel does not block the writer and does not drop silently: it appends one `IN_Q_OVERFLOW` event and discards until you catch up. A correct watcher treats `IN_Q_OVERFLOW` as "my view is stale, -re-scan" — which is exactly why `fwatch` kept its `snapshot`/`diff` +re-scan," which is exactly why `fwatch` kept its `snapshot`/`diff` subcommands: the polling machinery from Chapters 7 and 8 is the recovery path for the event machinery of Chapter 9. Also note what a directory watch does -*not* give you: recursion. Subdirectories need their own watches — the cost +*not* give you: recursion. Subdirectories need their own watches, the cost model behind editors and sync clients burning thousands of `wd`s. ## Everything is an fd: timerfd and signalfd @@ -136,14 +136,14 @@ can join the loop, and Linux has spent two decades converting non-fd events into fds. `fwatch` uses two of them. `timerfd_create(2)` gives a timer whose expiration makes an fd readable; reading it returns an 8-byte count of expirations since the last read. The C++ and Rust loops run exactly **one** -timer for two jobs — each iteration re-arms it (one-shot) to +timer for two jobs: each iteration re-arms it (one-shot) to `min(overall deadline, earliest debounce due)`, so whichever deadline is nearest is the one wakeup that gets scheduled. One subtlety hides in `arm_timer`: an `it_value` of all zeros means *disarm*, so a deadline that -is already past is clamped to one nanosecond — "fire immediately" and "never +is already past is clamped to one nanosecond: "fire immediately" and "never fire" are one bit pattern apart. -`signalfd(2)` does the same for signals — with a prerequisite that trips +`signalfd(2)` does the same for signals, with a prerequisite that trips everyone once: you must **block** the signals first (`sigprocmask(2)`), otherwise they are still delivered asynchronously to a handler and the fd reads nothing. Blocked-then-read, SIGINT stops being a control-flow @@ -155,22 +155,22 @@ the shutdown path itself is just another `case` in the dispatch. ## How the code works -Two data structures carry the whole design. `pending` — `std::map` in C++, `BTreeMap` in Rust, `map[string]*pendingEvent` -in Go — maps each path to its coalesced event kind and its debounce deadline +in Go) maps each path to its coalesced event kind and its debounce deadline (`due = now + 100ms`); ordered maps are chosen deliberately so a batch flush prints in name order and `verify.lua` can assert output deterministically (Go sorts the keys at flush time instead). The second structure is the token: C++ registers each fd with `data.u32` set to `tok_inotify`/`tok_timer`/ `tok_signal` (Rust: `TOK_INOTIFY = 0` … `TOK_SIGNAL = 2`), so dispatch is a -`switch` on a number the kernel echoes back — no fd comparisons, no lookup +`switch` on a number the kernel echoes back, no fd comparisons, no lookup table. Coalescing is one shared `merge` rule: within a window, delete wins, a fresh create stays `created` through later writes, and delete-then-recreate -reads as `modified` — the same visible truth `diff` would have reported. +reads as `modified`, the same visible truth `diff` would have reported. -Setup is four fallible constructions — `inotify_init1` + `inotify_add_watch`, +Setup is four fallible constructions (`inotify_init1` + `inotify_add_watch`, `timerfd_create`, `sigprocmask` + `signalfd`, `epoll_create1` + three -`epoll_ctl` ADDs — every fd owned by Chapter 7's RAII wrapper (`Fd` in C++, +`epoll_ctl` ADDs); every fd owned by Chapter 7's RAII wrapper (`Fd` in C++, `OwnedFd`-backed nix types in Rust, `os.File` in Go). Then the loop; here it is in all three languages, verbatim from `examples/09-event-loops-epoll-inotify/`: @@ -347,18 +347,18 @@ is in all three languages, verbatim from } ``` -C++ and Rust are the same program in two spellings — arm, wait forever +C++ and Rust are the same program in two spellings: arm, wait forever (`-1` / `EpollTimeout::NONE`; the timerfd *is* the timeout), dispatch by token, drain or consume, `EINTR` retries the wait. The one visible library difference: nix's `inotify.read_events()` does the record-walking that C++ does by hand over `buf`, returning parsed events with `Option` -names — same bytes, one abstraction up. +names, same bytes, one abstraction up. Go's excerpt has no epoll in it, and that is the entire point. Its `watch` is built from a goroutine and four channel operations: `readEvents` (in `examples/09-event-loops-epoll-inotify/go/main.go`) wraps the `O_NONBLOCK` -inotify fd in an `os.File` — the act that registers it with the runtime's -**netpoller** — and calls `f.Read` in a loop. That read *looks* blocking, +inotify fd in an `os.File` (the act that registers it with the runtime's +**netpoller**) and calls `f.Read` in a loop. That read *looks* blocking, but because the fd is nonblocking and registered, the runtime parks the *goroutine* (not an OS thread) until its epoll loop reports readiness, then resumes it to walk the same packed records (`unix.SizeofInotifyEvent`, @@ -366,8 +366,8 @@ resumes it to walk the same packed records (`unix.SizeofInotifyEvent`, sources never touch an fd in user code: `time.After` is the timeout, a `time.Timer` re-armed to the earliest `due` is the debounce (`rearm` mirrors the C++ `arm_timer` computation exactly), and `signal.Notify` -delivers SIGINT/SIGTERM on a channel. `select` is `epoll_wait` one level up -— it blocks until any source is ready and dispatches — except the "fds" are +delivers SIGINT/SIGTERM on a channel. `select` is `epoll_wait` one level up: +it blocks until any source is ready and dispatches, except the "fds" are channels and the kernel machinery is rented from the runtime. ### Errors, three ways @@ -380,32 +380,32 @@ with `.and_then(…)` so the first failure short-circuits with its `fwatch: watch: /no/such/dir: No such file or directory`, exit 1. Go gets `unix.ENOENT` back from `inotify_add_watch`, matches it with `errors.Is(err, unix.ENOENT)` to emit the friendlier -`no such directory`, and — because the raw fd is not yet wrapped in an -`os.File` at that point — must `unix.Close(ifd)` manually on that one path: +`no such directory`, and, because the raw fd is not yet wrapped in an +`os.File` at that point, must `unix.Close(ifd)` manually on that one path: the seam between raw-fd code and RAII-style ownership is exactly one line wide, and it is the line you forget. Rust's `anyhow::Context` builds the same chain declaratively (`.with_context(|| dir.to_string())`), printed with `{err:#}` to get `context: cause` in one line. Runtime errors inside the -loop split by kind: `EINTR` is not an error (retry the wait — explicitly in +loop split by kind: `EINTR` is not an error (retry the wait, explicitly in C++/Rust, invisibly in Go), `EAGAIN` on the inotify fd is the *success* case "queue drained", and anything else aborts with context. Usage errors exit 2, -runtime failures exit 1, clean shutdown — timeout or signal — exits 0; all +runtime failures exit 1, clean shutdown (timeout or signal) exits 0; all three verified below. ### Concurrency lens -Count the threads doing *your* work: one in C++, one in Rust — and one +Count the threads doing *your* work: one in C++, one in Rust, and one goroutine plus `main` in Go, multiplexed by the runtime onto its thread pool. The single-threaded epoll design is not an accident, it is the concurrency *strategy*: since only one thread touches `pending`, there are no locks, no atomics, and no `EPOLLONESHOT` re-arm protocol; the event loop serializes everything by construction. Go reaches the same freedom-from-locks -differently — `pending` is confined to the `select` goroutine, and other -concerns communicate by channel — which is Chapter 6's "share memory by +differently: `pending` is confined to the `select` goroutine, and other +concerns communicate by channel, which is Chapter 6's "share memory by communicating" applied to I/O. The costs surface in `strace`: idle for 400 ms, the C++ watcher makes exactly **1** `epoll_wait` call (armed once, sleeps, fires at the deadline), while the Go binary makes **10** `epoll_pwait` calls -across its threads — the netpoller is consulted by the scheduler, not just +across its threads: the netpoller is consulted by the scheduler, not just by your `select`, and no `timerfd_create` or `signalfd4` ever appears because Go implements timers in the runtime and signals via handlers feeding channels. Neither number is wrong; one design spends syscalls to buy an @@ -438,7 +438,7 @@ Two lines, not four: the create and the immediate append landed in one 100 ms debounce window and `merge` kept `created`; the `rm` 400 ms later was its own window. Send SIGINT instead of waiting and the last line becomes `fwatch: exiting (signal)` (observed, exit 0). The book's harness agrees on -the full behavior matrix — `python3 scripts/test-all-examples.py --only +the full behavior matrix: `python3 scripts/test-all-examples.py --only 09-event-loops-epoll-inotify` reports `PASS` for cpp, go, and rust (3 passed, 0 failed, 0 skipped). @@ -468,14 +468,14 @@ read(4, "\1\0\0\0\0\0\0\0", 8) = 8 Every mechanism from this chapter is on one screen. The `data=0` wakeups are the inotify token, `data=0x1` the timer token. The first inotify drain -returns **64 bytes — two packed records in one read** (the create, mask +returns **64 bytes: two packed records in one read** (the create, mask `\0\1\0\0` = `IN_CREATE`, and the append, coalesced by our merge into one output line), followed by the mandatory `EAGAIN`. The delete arrives later as one 32-byte record with mask `\0\2\0\0` = `IN_DELETE`. Three timer wakeups, three 8-byte expiration reads: two debounce flushes and the final deadline. Total `epoll_wait` calls for the whole session: five. -Second — and this is the part few people ever look at — epoll's interest +Second, and this is a part few people look at, epoll's interest list is inspectable. With a watcher running (PID via `$!`): ```bash @@ -499,10 +499,10 @@ it_value: (2, 694650159) The eventpoll fdinfo *is* Figure 9.1: three `tfd` (target fd) rows, our tokens 0/1/2 in the `data` column, `events: 19` = `EPOLLIN|EPOLLERR|EPOLLHUP` (the error bits are always reported; you never register them). The inotify -fdinfo shows `wd:1`, our exact mask `3c6`, and the watched inode `142a5` — +fdinfo shows `wd:1`, our exact mask `3c6`, and the watched inode `142a5`, hex for 82597, which `stat -c '%i'` on the watched directory confirms. The timerfd fdinfo shows `clockid: 1` (`CLOCK_MONOTONIC`) and, caught mid-run, -`it_value: (2, 694650159)` — about 2.69 s remaining on a 3000 ms run, i.e. +`it_value: (2, 694650159)`, about 2.69 s remaining on a 3000 ms run, i.e. armed to the overall deadline because no debounce was pending. Third, the Go binary on the same session, tracing what *our code never @@ -519,21 +519,21 @@ epoll_pwait(5, [], 128, 799, NULL, 0) = 0 The runtime built its own epoll instance, registered its wakeup fd, then added our inotify fd (4) **edge-triggered** with an opaque runtime cookie -where we used token integers — 9 `epoll_pwait` calls in this run, issued +where we used token integers: 9 `epoll_pwait` calls in this run, issued from several threads, none from `main.go`. The idle comparison quantifies the two architectures: over a 400 ms idle watch, `strace -c` shows the C++ binary making exactly 1 `epoll_wait`, 1 `epoll_create1`, 1 `timerfd_create`, 1 `timerfd_settime`, 1 `signalfd4`, and 1 `inotify_add_watch`; the Rust binary is syscall-for-syscall identical (the same six calls, once each); the -Go binary makes 10 `epoll_pwait` and zero timerfd/signalfd calls. Two tools -— the trace and `/proc` — one consistent story: readiness is a kernel fact, +Go binary makes 10 `epoll_pwait` and zero timerfd/signalfd calls. Two tools, +the trace and `/proc`, tell one consistent story: readiness is a kernel fact, and both surfaces are views of it. -

unverified — On the -lab VM: the kernel-side view of this loop — watching wakeup sources +

unverified: On the +lab VM: the kernel-side view of this loop, watching wakeup sources fire with bpftrace on tracepoint:syscalls:sys_enter_epoll_wait and kprobes on the eventpoll internals, and forcing a real -IN_Q_OVERFLOW by shrinking max_queued_events — +IN_Q_OVERFLOW by shrinking max_queued_events, needs root and a disposable kernel, so it is not run here; chapter 30 (Debugging part) exercises exactly this on systems-target.

@@ -541,28 +541,28 @@ needs root and a disposable kernel, so it is not run here; chapter 30 - Readiness decouples waiting from reading: nonblocking fds, one `epoll_wait`, and a kernel-side interest list you can literally read back - from `/proc//fdinfo/` — tokens, masks, and all. + from `/proc//fdinfo/`: tokens, masks, and all. - Level triggering plus "consume what woke you" is the book's default; edge demands drain-to-`EAGAIN` always (Go's netpoller does exactly that, registering with `EPOLLET`), and `EPOLLONESHOT` exists for the multi-threaded case this design avoids. - inotify queues events per instance with per-watch `wd`s and a finite - queue (16384 here) that overflows into `IN_Q_OVERFLOW` — the signal to + queue (16384 here) that overflows into `IN_Q_OVERFLOW`, the signal to fall back to `snapshot`/`diff`; watches do not recurse. -- timerfd and signalfd turn timeouts and SIGINT into readable fds — one +- timerfd and signalfd turn timeouts and SIGINT into readable fds: one re-armed one-shot timer served both the debounce and the deadline, and a blocked-then-read signal became an ordinary dispatch case. - Go's goroutine + channels + `select` is the same machinery inverted: 6 explicit setup syscalls and 1 wait for C++/Rust idle, versus 10 runtime `epoll_pwait`s and no timerfd/signalfd for Go. -Next, **io_uring**: the completion model — instead of the kernel telling you -an fd is *ready* so you can do the work, you submit the work and the kernel -tells you it is *done*. +Next, **io_uring**: the completion model, where instead of the kernel +telling you an fd is *ready* so you can do the work, you submit the work and +the kernel tells you it is *done*. --- -

verified — all evidence in +

verified: all evidence in this chapter was produced on the Fedora 44 reference host (kernel 7.1, strace 7.1) this session: the runner reported PASS for cpp, go, and rust (3 passed, 0 failed, 0 skipped); the @@ -572,7 +572,7 @@ a.txt, event: deleted a.txt, fwatch: exiting (signal) with exit 0 (missing dir exited 1, bad usage 2); the C++ strace excerpt is real trimmed output (5 epoll_waits, the 64-byte two-record inotify read, the EAGAINs, three 8-byte -timerfd reads); the /proc quotes are real (tfd +timerfd reads); the /proc output matches the live run (tfd rows 3/4/5 with data 0/1/2, mask:3c6, ino:142a5 matching stat -c '%i' = 82597, clockid: 1, it_value: (2, 694650159)); the Go trace showed diff --git a/_docs/10-io-uring.md b/_docs/10-io-uring.md index 1811e04..75d34f3 100644 --- a/_docs/10-io-uring.md +++ b/_docs/10-io-uring.md @@ -2,13 +2,13 @@ title: "io_uring: the completion model" order: 10 part: "Files and I/O" -description: "fwatch grows a batched sync engine and meets the second of the two I/O mental models — completion. SQ and CQ rings shared with the kernel, one io_uring_enter carrying sixteen operations, real strace counts showing the syscall collapse, and real timings showing why fewer syscalls is not automatically faster." +description: "fwatch grows a batched sync engine and meets the second of the two I/O mental models, completion. SQ and CQ rings shared with the kernel, one io_uring_enter carrying sixteen operations, real strace counts showing the syscall collapse, and real timings showing why fewer syscalls is not automatically faster." duration: 60 minutes --- Chapter 9 ended with `fwatch` blocked in an event loop, and with a promise: -epoll would not be the last word on Linux I/O. Epoll answers one question — -*which of these fds may I touch without blocking?* — and then leaves every +epoll would not be the last word on Linux I/O. Epoll answers one question: +*which of these fds may I touch without blocking?* Then it leaves every actual `read(2)` and `write(2)` for you to make, one syscall per operation. This chapter adds `fwatch sync`, a small directory-copy subcommand with two engines: `--engine rw`, the plain read/write loop you already know, and @@ -18,8 +18,8 @@ the process, and submits sixteen of them per syscall. That is the **completion model**, and readiness-versus-completion is the mental-model pair this whole Part has been building toward. The chapter also delivers two deliberate surprises: our uring engine is measurably *slower* on this -workload, and one of our three languages refuses to play at all — both on -purpose, both worth understanding. +workload, and one of our three languages refuses to play at all. Both are +intended, and both are worth understanding. The code is in `examples/10-io-uring/`. `./demo.sh` there builds and runs all three implementations; its `README.md` covers the `scan`/`watch`/`sync` @@ -28,9 +28,9 @@ subcommands and the two-engine contract. {% include excalidraw.html file="10-uring-rings" alt="Three columns: user-space fwatch code filling SQEs and popping CQEs, the mmap'd shared memory holding the SQ ring, SQE array, and CQ ring, and the kernel consuming submissions and posting completions, with io_uring_enter as the single syscall between them." - caption="Figure 10.1 — the io_uring anatomy: two rings and an SQE array shared via mmap; io_uring_enter is the only syscall in the data path" %} + caption="Figure 10.1: the io_uring anatomy: two rings and an SQE array shared via mmap; io_uring_enter is the only syscall in the data path" %} -> **Tools used** — `strace`, `diff`, `cmp`, `grep`, `python3` (host); +> **Tools used**: `strace`, `diff`, `cmp`, `grep`, `python3` (host); > `bpftrace` (lab VM, exercised in Part 8). Everything here is checked by > `scripts/check-host.sh`, ships with Fedora, or is preinstalled in the lab VMs. @@ -38,25 +38,25 @@ subcommands and the two-engine contract. Every I/O interface you have met so far is synchronous at the moment of data movement: `read(2)` in chapter 7, `write(2)` through the page cache in -chapter 8, and epoll in chapter 9 — which, for all its scalability, only +chapter 8, and epoll in chapter 9, which, for all its scalability, only *schedules* those same calls. Readiness is a hint about the future ("fd 7 will not block right now"), and it can even be wrong: a readable socket can still return `EAGAIN` by the time you get there. -io_uring inverts the contract. You write a **submission queue entry** (SQE) — +io_uring inverts the contract. You write a **submission queue entry** (SQE), a 64-byte struct naming an opcode (`IORING_OP_READ`), an fd, a buffer -address, a length, and a file offset — into an array the kernel mapped into +address, a length, and a file offset, into an array the kernel mapped into your address space. You publish it by advancing the tail index of the **submission queue** (SQ) ring. Some time later the kernel leaves a **completion queue entry** (CQE) in the CQ ring: `res` is the operation's result (bytes transferred, or a negative errno), and `user_data` is an opaque value you attached to the SQE so you can match the completion back to the request. The syscall `io_uring_enter(2)` exists only to say "I have queued N -things; wake me when M of them are done" — and with `IORING_ENTER_GETEVENTS` +things; wake me when M of them are done." With `IORING_ENTER_GETEVENTS` it does both halves in one crossing. One syscall can therefore carry any number of operations, which is the entire economic argument: the fixed cost of entering the kernel is amortized across a batch. There is even a mode that -eliminates the syscall entirely — `IORING_SETUP_SQPOLL` starts a kernel +eliminates the syscall entirely: `IORING_SETUP_SQPOLL` starts a kernel thread that polls the SQ ring so a busy application never traps at all. We mention it and move on: our engine does not use it, and the manpage itself warns that it "may sound immediately appealing as an automatic go-faster @@ -66,27 +66,27 @@ below make the same point without SQPOLL. {% include excalidraw.html file="10-readiness-vs-completion" alt="Two columns. Left, readiness: epoll_wait asks which fds are ready, the kernel reports readiness events, the application still performs read and write itself, and the operation can still fail with EAGAIN. Right, completion: the application queues SQEs describing the I/O, one io_uring_enter submits the batch, the kernel performs the I/O inline or via io-wq workers, and CQEs carry the finished results." - caption="Figure 10.2 — the mental-model pair: readiness (epoll) hints per fd; completion (io_uring) reports finished results per operation" %} + caption="Figure 10.2: the mental-model pair: readiness (epoll) hints per fd; completion (io_uring) reports finished results per operation" %} -## Where uring wins — and why containers say no +## Where uring wins, and why containers say no The batch economics pay off where per-operation syscall overhead is a real fraction of the work: thousands of small operations in flight against storage that benefits from deep queues (`O_DIRECT` on NVMe), or servers juggling enormous socket counts, where features like registered buffers and multishot operations remove per-op setup cost too. They pay off *least* when -each operation is already almost free — and a buffered read of a page-cache-hot +each operation is already almost free, and a buffered read of a page-cache-hot file is a memcpy. Small trees of warm files are exactly that, which is why the timings below look the way they do. The same power has a security price. io_uring is a large, fast-moving kernel attack surface, and it lets a process express nearly every syscall family -through one opcode-dispatched interface — which quietly bypasses syscall +through one opcode-dispatched interface, which quietly bypasses syscall filters that were written in terms of `read`/`write`/`connect`. Container -runtimes responded plainly: on this host, podman's default seccomp profile +runtimes responded directly: on this host, podman's default seccomp profile (`/usr/share/containers/seccomp.json`) sets `"defaultAction": "SCMP_ACT_ERRNO"` and its allowlist contains **zero** mentions of `io_uring_setup`, -`io_uring_enter`, or `io_uring_register` — deny by default. Run the uring +`io_uring_enter`, or `io_uring_register`: deny by default. Run the uring engine in an unmodified container and `io_uring_setup` fails with `EPERM`. Chapter 34, when our programs move into containers, deals with that head-on. @@ -94,7 +94,7 @@ Chapter 34, when our programs move into containers, deals with that head-on. `fwatch sync --engine uring` in Go prints `engine=uring: unsupported in Go (see chapter 10)` and exits 64. That is not -laziness — it is the chapter's design lesson. Go already solved "many I/O +laziness: it is the chapter's design lesson. Go already solved "many I/O operations, few OS threads" once, inside the runtime: every goroutine writes ordinary blocking code, and the **netpoller** multiplexes those goroutines over epoll, parking them until readiness arrives. The concurrency model *is* @@ -102,27 +102,27 @@ the I/O model. Grafting a completion engine underneath would mean teaching the scheduler that some I/O finishes on rings instead of unparking on readiness, deciding who owns the buffers while the kernel holds them, and doing it all without breaking the promise that `f.Read(buf)` just works. The -tracking issue the source comment cites — golang/go#31908, open since 2019 — +tracking issue the source comment cites, golang/go#31908, open since 2019, remains on hold for exactly these reasons. Rust, at the other extreme, ships io_uring as a library (`io-uring`, the crate our Rust engine uses) and lets runtimes compete; C++ gives you the raw syscalls and a free hand. Three languages, three answers to the same question: *who owns the event loop?* -> **Research note — io_uring on this kernel, from primary sources.** Claims +> **Research note: io_uring on this kernel, from primary sources.** Claims > about io_uring age fast, so this chapter cites only what this host shows. > Kernel `7.1` reported 18 `IORING_FEAT_*` flags in the -> `io_uring_setup` params during our traced run — from `SINGLE_MMAP` and +> `io_uring_setup` params during our traced run, from `SINGLE_MMAP` and > `NODROP` through `RECVSEND_BUNDLE`, `MIN_TIMEOUT`, `RW_ATTR`, and -> `NO_IOWAIT` — and the shipped `io_uring(7)`, `io_uring_setup(2)`, and +> `NO_IOWAIT`, and the shipped `io_uring(7)`, `io_uring_setup(2)`, and > `io_uring_enter(2)` manpages document the setup/enter contract used here. > Feature lists in older articles (and newer marketing) routinely disagree > with both; when in doubt, trust the flags your kernel returns. ## How the code works -`fwatch` v2 carries the earlier chapters' machinery forward — the v0 +`fwatch` v2 carries the earlier chapters' machinery forward: the v0 polling walk survives as `scan` (now a one-line summary; the per-file -`snapshot`/`diff` pair stays with v1), v1's `watch` keeps its name — and +`snapshot`/`diff` pair stays with v1), v1's `watch` keeps its name, and adds `cmd_sync`: walk the source tree, recreate directories, copy every regular file, print `synced files bytes engine= ms=`. The engines differ only in how one file's bytes move. @@ -131,9 +131,9 @@ The C++ engine is the pedagogical core, because the host has no `liburing-devel` and we turned that into a feature: a self-contained `Ring` class built on raw `syscall(2)` wrappers, so every moving part is visible. `Ring::create` calls `io_uring_setup(2)` with 64 entries, then performs the -three canonical mmaps of the returned fd — the SQ ring (head, tail, mask, +three canonical mmaps of the returned fd: the SQ ring (head, tail, mask, and an index array), the CQ ring (head, tail, and the CQE array), and the -SQE array itself — honoring `IORING_FEAT_SINGLE_MMAP`, which on modern +SQE array itself, honoring `IORING_FEAT_SINGLE_MMAP`, which on modern kernels folds the two rings into one mapping (our trace below shows exactly that: one 2368-byte mapping at offset 0, one SQE mapping at `IORING_OFF_SQES`). The interesting discipline is in the hot-path methods, @@ -162,17 +162,17 @@ from `examples/10-io-uring/cpp/src/main.cpp`: } ``` -`try_get_sqe` load-acquires the SQ *head* — the kernel's consumption cursor — +`try_get_sqe` load-acquires the SQ *head* (the kernel's consumption cursor) to learn how much room remains; the ring is full when our private `local_tail_` has run a whole ring-size ahead of it. Filled entries stay private until `submit_and_wait` store-releases the shared *tail*, which is the publish: everything written to the SQEs before that store is guaranteed visible to the kernel after it. The mask trick (`local_tail_ & sq_mask_`) works because ring sizes are powers of two, so a free-running 32-bit counter -maps onto a slot without ever being reset — the same idiom you will meet +maps onto a slot without ever being reset, the same idiom you will meet again in the lock-free chapter. -On top of the ring, each engine copies one file the same way — here is the +On top of the ring, each engine copies one file the same way: here is the same function three times: {% include codetabs.html langs="C++|Go|Rust" %} @@ -357,24 +357,24 @@ fn copy_file_uring(ring: &mut IoUring, src: &Path, dst: &Path) -> Result { The design is identical in C++ and Rust: for each 128 KiB chunk, queue a READ into a per-slot buffer and a WRITE of that same buffer at the same -destination offset, and chain them with `IOSQE_IO_LINK` — the kernel starts +destination offset, and chain them with `IOSQE_IO_LINK`: the kernel starts the WRITE only after its READ fully succeeded, so ordering *within a pair* needs no round-trip to userspace. Up to eight pairs (sixteen SQEs, which is why the ring is sized 64) go into one `submit_and_wait(2 * pairs)`. Every -operation carries an explicit `off` — completions can arrive in any order, +operation carries an explicit `off`: completions can arrive in any order, so nothing may depend on a shared file position; `user_data` encodes the slot (and read/write bit) so each CQE can be checked against the length that was requested. `drain` then insists on *exactly* `2 * pairs` CQEs, -re-entering the kernel if a wait returned early — a wait can be satisfied +re-entering the kernel if a wait returned early, since a wait can be satisfied before every linked pair has retired. Rust's version is the same shape on the `io-uring` crate, which owns the setup and barriers (dropping the `submission()` view is the tail-publish); the `unsafe` blocks mark the one -contract the compiler cannot check — the kernel holds raw pointers into +contract the compiler cannot check: the kernel holds raw pointers into `bufs` until we have drained the completions. The fragile bits are deliberate and visible: `kChunk`/`kPairs` are hardcoded tuning knobs, the batch waits for *all* its completions before starting the next (no cross-file pipelining), and the buffers are plain allocations, not -registered buffers — all three are exactly where a production engine would +registered buffers. All three are exactly where a production engine would diverge, and the numbers below show the cost. ### Errors, three ways @@ -382,15 +382,15 @@ diverge, and the numbers below show the cost. The completion model moves error handling: failures are no longer a `-1` and an ambient `errno` at the call site, but a negative errno *value* inside each CQE, delivered whenever the kernel got around to that operation. The -C++ engine folds every CQE through one check — `cqe.res < 0 || -cqe.res != lens[slot]` — treating short transfers as errors, and returns +C++ engine folds every CQE through one check, `cqe.res < 0 || +cqe.res != lens[slot]`, treating short transfers as errors, and returns `std::errc::io_error` through the same `expected<…, error_code>` channel as `open` failures; a full SQ ring is `std::errc::no_buffer_space`, and only `io_uring_enter` itself can still fail the old way (its `EINTR` is retried in `Ring::enter`). Rust does the same per-CQE audit but `bail!`s with the got/want lengths, so the error message carries the evidence. And a linked pair fails as a unit: if a READ comes up short, the kernel cancels its -linked WRITE with `-ECANCELED` — one more CQE, one more `res < 0`, caught by +linked WRITE with `-ECANCELED`: one more CQE, one more `res < 0`, caught by the same check. Go's error story here is the bluntest and the most interesting: `exit 64`, refusing at the front door, because the runtime's contract cannot express the engine at all. @@ -398,18 +398,18 @@ contract cannot express the engine at all. ### Concurrency lens Strip away the I/O and Figure 10.1 is a pair of single-producer, -single-consumer lock-free ring buffers — the kernel is simply *the other +single-consumer lock-free ring buffers: the kernel is simply *the other thread*. That is why `try_get_sqe`/`submit_and_wait` are written in acquire/release vocabulary: the store-release of the SQ tail publishes the filled SQEs, the kernel's load-acquire observes them, and the mirror-image pair guards the CQ ring in the other direction (`pop` load-acquires the CQ -tail before reading a CQE). `std::atomic_ref` is doing real work here — the +tail before reading a CQE). `std::atomic_ref` is doing real work here: the ring indices are plain `unsigned` fields in shared memory laid out by the kernel, and `atomic_ref` lets us perform atomic operations on memory we do not get to declare `std::atomic`. Meanwhile the kernel side is concurrent too: `IORING_FEAT_NATIVE_WORKERS` in our trace is the io-wq worker-thread pool, which executes any submitted operation that cannot complete -immediately — our buffered file writes among them, a fact that returns in +immediately, our buffered file writes among them, a fact that returns in the timings. Chapter 26 rebuilds this exact structure in user space, barrier by barrier. @@ -420,7 +420,7 @@ by barrier. [host]$ ./demo.sh cpp run sync --engine uring ``` -For real numbers, this session generated a scratch tree — 403 files, +To measure this directly, this session generated a scratch tree, 403 files, 19,507,360 bytes: 400 small files of 4–64 KiB across eight directories, plus three larger files (about 1 MiB, 3 MiB, and 1.1 MiB) sized to straddle the 128 KiB chunk and the 8-pair batch. Three runs per engine, warm page @@ -431,18 +431,18 @@ cache, `ms=` as printed by the tool: | `rw` | 11, 12, 11 ms | 13, 12, 12 ms | 21 ms | | `uring` | 129, 137, 130 ms | 105, 107, 107 ms | exit 64 (by design) | -Read that plainly: on this workload our uring engine is roughly **ten times -slower**. Nothing is broken — the copies are byte-identical (checked below) -— the model just costs more than it saves here. Every file is page-cache +On this workload, our uring engine is roughly **ten times +slower**. Nothing is broken: the copies are byte-identical (checked below). +The model just costs more than it saves here. Every file is page-cache hot, so each `read(2)` in the rw loop is a short memcpy; the syscalls we so cleverly eliminated were never the bottleneck. The uring path, meanwhile, -pays for what buffered file I/O cannot do asynchronously inline — the +pays for what buffered file I/O cannot do asynchronously inline: the kernel punts those operations to io-wq worker threads, buying context -switches and wakeups per batch — and our engine waits for each batch to +switches and wakeups per batch, and our engine waits for each batch to fully retire before building the next, with unregistered buffers reallocated per file. This is the batch-size lesson the chapter owes you: completion -wins when queues are deep, operations are genuinely asynchronous, and fixed -costs get amortized across real waiting — not when the "I/O" is a memcpy +wins when queues are deep, operations are actually asynchronous, and fixed +costs get amortized across real waiting, not when the "I/O" is a memcpy that was already cheaper than the bookkeeping. On the small verify fixture (5 files, 1,348,634 bytes) the gap shrinks to noise: `rw` printed `ms=0`, uring `ms=2`–`3`. @@ -450,22 +450,22 @@ uring `ms=2`–`3`. ## Cross-check: the syscall collapse, counted Timing said uring lost; `strace -c` proves it did so while *keeping its -promise* — the promise was fewer syscalls, not faster memcpys. Same tree, +promise*: the promise was fewer syscalls, not faster memcpys. Same tree, C++ binary, both engines: ```bash [host]$ strace -c ./cpp/build/release/app sync /src /dst --engine rw ``` -The rw run: **850 `read`**, **444 `write`**, 820 `openat` — 3,467 calls +The rw run: **850 `read`**, **444 `write`**, 820 `openat`: 3,467 calls total. The uring run: **407 `io_uring_enter`**, 1 `io_uring_setup`, and the only remaining `read`/`write` calls (4/1) belong to the dynamic loader. The arithmetic closes exactly: 400 single-batch small files, plus 2 + 3 + 2 batches for the three large files (9, 24, and 10 chunks at 8 pairs per batch) = 407 enters carrying all 19.5 MB. On the 5-file fixture the same -count is 5 enters — one per small file, two for the 9-chunk `d.bin`, zero -for the empty file. The uring table also shows 3,216 `brk` calls — the -allocator churn of building eight 128 KiB buffers per file — a bonus lesson +count is 5 enters: one per small file, two for the 9-chunk `d.bin`, zero +for the empty file. The uring table also shows 3,216 `brk` calls, the +allocator churn of building eight 128 KiB buffers per file, a lesson in how `strace -c` catches *your* overhead, not just the kernel's. The setup trace shows the anatomy of Figure 10.1 verbatim @@ -488,8 +488,8 @@ claim above is a one-liner you can run yourself: `grep -c io_uring /usr/share/containers/seccomp.json` prints `0` on this host, against a `defaultAction` of `SCMP_ACT_ERRNO`. -> **On the lab VM** — unverified: -> the natural next probe is watching io-wq from the inside — +> **On the lab VM**: unverified: +> the natural next probe is watching io-wq from the inside, > `bpftrace -e 'tracepoint:io_uring:io_uring_submit_req { @[comm] = count(); }'` > while the two engines run. bcc/bpftrace are not part of this host's > toolchain by design; chapter 30 (Debugging part) builds that eBPF @@ -499,25 +499,25 @@ claim above is a one-liner you can run yourself: `grep -c io_uring - Readiness and completion are the two I/O mental models: epoll hints that an fd *may* be usable; io_uring reports that an operation *finished*, with - its result in the CQE — and the SQ/SQE/CQ structures live in memory shared + its result in the CQE, and the SQ/SQE/CQ structures live in memory shared via three well-known mmaps of the ring fd. -- One `io_uring_enter` can submit and reap a whole batch — 407 enters - replaced 1,294 data syscalls on our 403-file tree — and `IOSQE_IO_LINK` +- One `io_uring_enter` can submit and reap a whole batch: 407 enters + replaced 1,294 data syscalls on our 403-file tree, and `IOSQE_IO_LINK` expresses intra-pair ordering without a userspace round-trip. - Fewer syscalls is not faster by itself: against a warm page cache our uring engine ran ~10× slower than the rw loop (129 vs 11 ms), because buffered-file completions ride io-wq worker threads and our batches are - small and fully synchronous — completion earns its keep at depth, on real + small and fully synchronous. Completion pays off at depth, on real waiting. - io_uring's power is why container seccomp profiles deny it by default - (podman's allowlist has no io_uring entries on this host — chapter 34), + (podman's allowlist has no io_uring entries on this host; chapter 34), and why Go declines to expose it at all: the netpoller already owns the event loop (golang/go#31908 stays on hold). -Next, Part 3 opens with the **process lifecycle** — `fork`, `exec`, and -`wait` — and the first cut of `pmon`, the process supervisor that stays with -us to the capstone. +Next, Part 3 opens with the **process lifecycle**: `fork`, `exec`, and +`wait`, and the first cut of `pmon`, the process supervisor that stays with +us to the final project. --- -

verified — all numbers above are from this session's runs on the Fedora 44 host (kernel 7.1): the runner table passed 3/3 (cpp, go, rust); the 403-file/19,507,360-byte tree synced with engine=rw in 11–12 ms (C++) and 12–13 ms (Rust) versus 129–137 ms and 105–107 ms with engine=uring; strace -c counted 850 read + 444 write (rw) against 407 io_uring_enter + 1 io_uring_setup (uring, loader-only read/write 4/1); the fixture uring sync used 5 enters and printed synced 5 files 1348634 bytes engine=uring ms=3; the io_uring_setup/mmap trace, the diff -r/cmp byte-equality, the Go refusal (engine=uring: unsupported in Go (see chapter 10), exit 64), and the empty grep -c io_uring against podman's seccomp profile were all reproduced as quoted. Unverified: the bpftrace probe in the lab-VM callout, as marked.

+

verified: all numbers above are from this session's runs on the Fedora 44 host (kernel 7.1): the runner table passed 3/3 (cpp, go, rust); the 403-file/19,507,360-byte tree synced with engine=rw in 11–12 ms (C++) and 12–13 ms (Rust) versus 129–137 ms and 105–107 ms with engine=uring; strace -c counted 850 read + 444 write (rw) against 407 io_uring_enter + 1 io_uring_setup (uring, loader-only read/write 4/1); the fixture uring sync used 5 enters and printed synced 5 files 1348634 bytes engine=uring ms=3; the io_uring_setup/mmap trace, the diff -r/cmp byte-equality, the Go refusal (engine=uring: unsupported in Go (see chapter 10), exit 64), and the empty grep -c io_uring against podman's seccomp profile were all reproduced as quoted. Unverified: the bpftrace probe in the lab-VM callout, as marked.

diff --git a/_docs/11-process-lifecycle.md b/_docs/11-process-lifecycle.md index 66df347..2bc6b7e 100644 --- a/_docs/11-process-lifecycle.md +++ b/_docs/11-process-lifecycle.md @@ -2,36 +2,36 @@ title: "The process lifecycle: fork, exec, wait" order: 11 part: "Processes, Signals, Privilege" -description: "fork, exec, and wait walked end to end as pmon v0 spawns and reaps its first children: copy-on-write, a real zombie caught live in ps, orphans reparented to a subreaper, exit codes vs termination signals, rusage accounting — and the clone flavors os/exec and Command actually issue, read from strace." +description: "fork, exec, and wait walked end to end as pmon v0 spawns and reaps its first children: copy-on-write, a real zombie caught live in ps, orphans reparented to a subreaper, exit codes vs termination signals, rusage accounting; and the clone flavors os/exec and Command actually issue, read from strace." duration: "45 minutes" --- This chapter opens the book's third part with a new long-running artifact: `pmon`, a process monitor that will grow signal forwarding, pidfd-based supervision, and privilege dropping over the next three chapters. Version 0 -does the irreducible job of every supervisor — spawn a command, wait for it, -report how it died and what it cost — because everything else in this part +does the irreducible job of every supervisor: spawn a command, wait for it, +report how it died and what it cost, because everything else in this part builds on one idea: **a process's death is data, and someone must collect it**. Between a child's last instruction and the parent's `wait`, the kernel -holds the corpse — exit status, resource usage, a pid that cannot be reused — -as a *zombie*, and in this chapter you will create one deliberately, watch it +holds the corpse (exit status, resource usage, a pid that cannot be reused) +as a *zombie*, and in this chapter you will create one, watch it in `ps`, read its `/proc` entry field by field, and then reap it. Along the way we take apart what `fork(2)` really copies (almost nothing, thanks to copy-on-write), why exec failure needs a back channel, and what Go's -`os/exec` and Rust's `Command` are actually doing under their tidy APIs — the +`os/exec` and Rust's `Command` are actually doing under their tidy APIs: the answer, straight from `strace`, involves three different flavors of `clone`. The code is in `examples/11-process-lifecycle/`. `./demo.sh` there builds all -three implementations and walks the four child fates — clean exit, `exit 42`, -death by SIGTERM, exec failure — and its `README.md` pins the output contract +three implementations and walks the four child fates (clean exit, `exit 42`, +death by SIGTERM, exec failure) and its `README.md` pins the output contract and exit-code mirroring all three languages share. {% include excalidraw.html file="11-process-lifecycle-states" alt="Five process states drawn as boxes: fork/clone3 leads into Running R inside an alive band that also holds Sleeping S and Stopped T with SIGSTOP and SIGCONT arrows between them; an amber exit arrow crosses into an after-exit band holding Zombie Z, annotated defunct in ps f and exit status plus rusage only, with a final amber wait4/waitid arrow down to a dashed reaped box where the pid table slot is free." - caption="Figure 11.1 — the lifecycle as ps STAT letters; the kernel keeps a zombie until someone reaps it, and every annotation in the zombie box is observed later in this chapter" %} + caption="Figure 11.1: the lifecycle as ps STAT letters; the kernel keeps a zombie until someone reaps it, and every annotation in the zombie box is observed later in this chapter" %} -> **Tools used** — `strace`, `ps`, `kill`, `sh`, `sleep`, `cat` (host); +> **Tools used**: `strace`, `ps`, `kill`, `sh`, `sleep`, `cat` (host); > `execsnoop`/`exitsnoop` (lab VM, exercised in Part 8). Everything here is > checked by `scripts/check-host.sh`, ships with Fedora, or is preinstalled > in the lab VMs. @@ -39,8 +39,8 @@ and exit-code mirroring all three languages share. ## Born twice: fork, then exec Unix splits "run a program" into two syscalls with nothing in between but -your own code. `fork(2)` duplicates the calling process — same program, same -open fds, a new pid — and *returns twice*: the child sees `0`, the parent +your own code. `fork(2)` duplicates the calling process (same program, same +open fds, a new pid) and *returns twice*: the child sees `0`, the parent sees the child's pid. `execve(2)` then replaces the calling process's program image with a new one, keeping the pid and the fd table. The gap between them is the whole point: the child can rearrange fds, drop privileges, or change @@ -48,27 +48,27 @@ directory *as ordinary code running in the child*, before the new program exists. Every shell redirection you have ever typed is three lines of code in that gap. -The apparent extravagance of fork — duplicate an entire address space just to -throw it away at exec — is paid for by **copy-on-write**. The kernel copies +The apparent extravagance of fork (duplicate an entire address space just to +throw it away at exec) is paid for by **copy-on-write**. The kernel copies page *tables*, not pages: parent and child share every physical page, read-only, and only a write by either side triggers a fault that copies the one page written. A fork of a 100 MB process touches kilobytes. You can see the accounting in this chapter's own evidence: the zombie's -`/proc//stat` below records `minflt` of 103 — one hundred and three +`/proc//stat` below records `minflt` of 103: one hundred and three minor faults is the entire page-level cost of forking pmon and execing `sleep`. The COW story has a sharper corollary for the Go and Rust runtimes, visible in the cross-check: when the child is going to exec *immediately*, even copying page tables is waste, so both runtimes ask for -`CLONE_VM|CLONE_VFORK` — share the address space outright and suspend the -parent until the exec — which is fork with the copy amortized to zero. +`CLONE_VM|CLONE_VFORK`, sharing the address space outright and suspending the +parent until the exec, which is fork with the copy amortized to zero. {% include excalidraw.html file="11-fork-exec-wait-seq" alt="Two lanes, parent pmon and child. Parent creates pipe2 O_CLOEXEC, calls fork drawn as clone SIGCHLD with a COW note, and an arrow labeled returns 0 in the child crosses to the child lane where execvp runs; a dashed arrow back carries EOF on success or errno then exit 127. The child box CMD runs then exits or is killed leads by an amber arrow labeled zombie until reaped, SIGCHLD to the parent's waitpid/wait4 box, then down to report and mirror the fate, status s exits s and signal n exits 128 plus n." - caption="Figure 11.2 — pmon's spine: fork → exec → wait, with the CLOEXEC self-pipe carrying the exec verdict back to the parent" %} + caption="Figure 11.2: pmon's spine: fork → exec → wait, with the CLOEXEC self-pipe carrying the exec verdict back to the parent" %} -One hole remains in the two-step model: if `execvp` itself fails — the -command does not exist — the failure happens *in the child*, after fork, and +One hole remains in the two-step model: if `execvp` itself fails (the +command does not exist), the failure happens *in the child*, after fork, and the parent has no direct way to distinguish "exec failed" from "the program ran and exited". pmon closes the hole with a **self-pipe**, and chapter 7's `O_CLOEXEC` discipline is what makes it work. Both pipe ends are opened @@ -92,7 +92,7 @@ struct ExecPipe { ``` If exec succeeds, the kernel closes the child's write end atomically at exec -time and the parent's `read` returns 0 — EOF, no child code required. If exec +time and the parent's `read` returns 0 (EOF, no child code required). If exec fails, the child writes the raw `errno` into the pipe and `_exit(127)`s. The parent reads either nothing or four bytes that name the exact failure. This trick is not pedagogy: the cross-check shows Go and Rust doing precisely the @@ -103,19 +103,19 @@ same dance inside their standard libraries. When a process exits, the kernel cannot simply delete it: the parent may not have asked yet how it died, and the pid must not be recycled while an answer is owed. So the address space, fds, and threads are freed immediately, but -the task record — pid, exit status, rusage — stays as a **zombie** (`Z` in +the task record (pid, exit status, rusage) stays as a **zombie** (`Z` in `ps`, rendered ``) until the parent calls `wait4(2)`/`waitid(2)`. Reaping *is* that call: the kernel copies the status and rusage out and only -then frees the pid. A parent that spawns children and never waits leaks pids -— a real resource, bounded by `pid_max` — which is why every supervisor loop +then frees the pid. A parent that spawns children and never waits leaks pids, +a real resource bounded by `pid_max`, which is why every supervisor loop in this book pairs each spawn with exactly one reap. Orphans travel the other direction: if the *parent* dies first, the child is -not left parentless — the kernel reparents it up the tree to the nearest +not left parentless: the kernel reparents it up the tree to the nearest **subreaper** (`PR_SET_CHILD_SUBREAPER`, a Part 3 recurring guest) or to pid 1. On this Fedora host the reparent target is not init itself: orphaning a `sleep` by letting its shell parent exit immediately re-homed it to the user's -`systemd --user` instance, pid 6145 — session managers register as subreapers +`systemd --user` instance, pid 6145; session managers register as subreapers precisely so they, not pid 1, inherit the duty to reap. Reparenting is also why zombies are curable by killing the negligent parent: its zombies get reparented, and subreapers wait promptly. @@ -127,7 +127,7 @@ called `exit(s)` has `WIFEXITED(status)` true and `WEXITSTATUS(status)` yields the low 8 bits of `s`. A process killed by a signal never reached `exit` at all: `WIFSIGNALED(status)` is true and `WTERMSIG(status)` names the signal. They are different fields of the wait status, and conflating them -loses information — "exited 15" and "killed by SIGTERM" are entirely +loses information: "exited 15" and "killed by SIGTERM" are entirely different deaths. The shell flattens the pair into one byte with the `128+n` convention, and pmon mirrors it exactly so it composes with `&&`, `make`, and CI: status `s` becomes exit `s`, signal `n` becomes exit `128+n` (SIGTERM → @@ -139,7 +139,7 @@ checks all of them against live runs, 27 checks per language. The C++ version spells the lifecycle out syscall by syscall; Go and Rust lean on their runtimes for the spawn and reclaim the low-level facts on the -wait side. Here is the spine, three ways — C++ from `fork` to the mirrored +wait side. Here is the spine, three ways: C++ from `fork` to the mirrored exit, and the equivalent `monitor` function in Go and Rust: {% include codetabs.html langs="C++|Go|Rust" %} @@ -310,27 +310,27 @@ child's whole life, and steady (not system) time so an NTP step cannot make a child appear to run backwards. `fork` splits the world; the child branch resets its copy of the read end (chapter 7's move-only `Fd` makes "close exactly once" a type property, and `ExecPipe` is just two of them), calls -`execvp` — the `p` variant walks `PATH` like a shell — and, only if exec +`execvp` (the `p` variant walks `PATH` like a shell) and, only if exec *returned*, ships `errno` up the pipe and dies with `_exit(127)`. `_exit`, not `exit`: the child still shares the parent's stdio buffers, and running the parent's atexit handlers or flushing its buffered output from the child -would emit duplicate bytes. The parent drops its write end — otherwise its +would emit duplicate bytes. The parent drops its write end; otherwise its own open fd would hold the pipe open and `read` would block forever, a -self-deadlock — then reads the verdict, retrying on `EINTR`. `wait_child` +self-deadlock. It then reads the verdict, retrying on `EINTR`. `wait_child` loops `waitpid` under the same `EINTR` discipline; the reap must survive a stray signal. Only then does the status get decoded through the `WIFEXITED`/`WIFSIGNALED` pair, and the rusage line printed from -`getrusage(RUSAGE_CHILDREN)` — the accumulated cost of all waited-for +`getrusage(RUSAGE_CHILDREN)`: the accumulated cost of all waited-for children, which equals this child's cost because there is exactly one. The Go tab hides the fork behind `cmd.Start`, but nothing else changes shape. `cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, ...` inherits stdio so -the child's output interleaves naturally — the pid-identity check in +the child's output interleaves naturally; the pid-identity check in `verify.lua` depends on it, the child echoing `$$` through the shared descriptor. A `Start` error *is* the self-pipe verdict, surfaced as a value. `waitChild` needs one subtlety: `cmd.Wait` returns an error for a child that ran and failed, but an `exec.ExitError` still carries the `ProcessState` we -want — a child exiting 42 is pmon's *success path* — so only non-`ExitError` +want (a child exiting 42 is pmon's *success path*), so only non-`ExitError` failures propagate. The wait-side facts come back out of the portable wrapper by type assertion: `state.Sys().(syscall.WaitStatus)` is the raw status int with the same `Exited()`/`Signaled()` split, and @@ -339,7 +339,7 @@ filled in at reap time. Rust's `Command::spawn` similarly returns `Err` on exec failure (same status-pipe trick internally), but the wait side goes straight to the -syscall, because `wait4(2)` answers both questions in one call — status +syscall, because `wait4(2)` answers both questions in one call: status *and* rusage for exactly this child: ```rust @@ -371,12 +371,12 @@ The two `SAFETY` comments are the whole cost of dropping below status into a typed `WaitStatus`, giving the `match` in `monitor` the same two-armed shape as `WIFEXITED`/`WIFSIGNALED`. Note the deliberate asymmetry with C++: `RUSAGE_CHILDREN` aggregates every reaped child, `wait4` meters -one — identical numbers here, different interfaces, and the distinction will +one; identical numbers here, different interfaces, and the distinction will matter the moment pmon supervises more than one process. -Two shared fragile bits, stated plainly. The signal names in +Two shared fragile bits follow. The signal names in `killed by signal 15 (TERM)` come from a hand-rolled 32-entry table, -identical in all three sources — `strsignal(3)` returns locale-dependent +identical in all three sources: `strsignal(3)` returns locale-dependent prose ("Terminated") and Go and Rust each have their own spellings, so the table is the only way three runtimes print one string. And the rusage line's `%03d` milliseconds formatting is part of the output contract: @@ -384,13 +384,13 @@ table is the only way three runtimes print one string. And the rusage line's ## Errors, three ways -The contract for a failed spawn is byte-identical output — `pmon: exec +The contract for a failed spawn is byte-identical output: `pmon: exec : No such file or directory` on stderr, exit 127, and *no* fate or rusage lines, because no child ran. Each language must dig the errno out of different wreckage. C++ owns the errno directly: the child ships the raw `int` over the pipe and the parent formats it through `std::error_code::message()`. Rust gets an `io::Error` from `spawn` and -recovers `raw_os_error()`, rendering it with `nix`'s `Errno::desc` — the +recovers `raw_os_error()`, rendering it with `nix`'s `Errno::desc`, the classic `strerror(3)` text without `io::Error`'s "(os error 2)" suffix. Go is the interesting one, from `examples/11-process-lifecycle/go/main.go`: @@ -423,23 +423,23 @@ and exited 127 in this session's runs. `fork` and threads are famously hostile: the child inherits *only the calling thread*, so any lock another thread held at fork time stays locked -forever in the child — which is why the only safe thing a multithreaded +forever in the child, which is why the only safe thing a multithreaded process can do between fork and exec is call async-signal-safe functions. pmon's C++ version can fork casually only because it is single-threaded. Go -cannot make that promise — the runtime is multithreaded before `main` runs — +cannot make that promise (the runtime is multithreaded before `main` runs), so `os/exec` confines the child to a tiny fixed path of raw syscalls, and the strace shows the runtime spawning its scheduler threads (`CLONE_THREAD` clones) before the one process-creating `clone(CLONE_VM|CLONE_PIDFD|CLONE_VFORK|SIGCHLD)`. That `CLONE_PIDFD` flag is this part's chapter 13 arriving early: Go receives a *pidfd* for the -child and waits via `waitid(P_PIDFD, ...)`, immune to pid-reuse races — it +child and waits via `waitid(P_PIDFD, ...)`, immune to pid-reuse races: it identifies the child by fd, not by number. Rust's `CLONE_CLEAR_SIGHAND` (via `posix_spawn`) solves a different race: the child starts with default signal dispositions instead of whatever handlers the parent had installed, exactly the class of inherited-state bug the vfork'd shared address space would otherwise make lethal. One more accounting subtlety: -`getrusage(RUSAGE_CHILDREN)` is process-wide state — if concurrent code -reaped other children between two calls, the aggregate moves — while +`getrusage(RUSAGE_CHILDREN)` is process-wide state (if concurrent code +reaped other children between two calls, the aggregate moves), while `wait4`'s per-child rusage is race-free by construction. pmon v0 is single-threaded in all three languages; these flags are the runtimes defending themselves. @@ -480,18 +480,18 @@ yields `wall=200ms` on the nose (`sleep 0.2` → `maxrss=2120KB user=0.000s sys=0.000s wall=200ms`), and the pid in the fate line is the child's own: `run -- sh -c 'echo "child: my pid is $$"'` printed `child: my pid is 2607401` through the inherited stdout, followed by pmon's -`pid 2607401 exited status 0` — same number, one from inside, one from the +`pid 2607401 exited status 0`: same number, one from inside, one from the wait. The runner (`python3 scripts/test-all-examples.py --only 11-process-lifecycle`) -reports `PASS PASS PASS` for cpp, go, and rust — 3 passed, 0 failed, 0 -skipped — with `verify.lua` asserting 27 behavioral checks per language. +reports `PASS PASS PASS` for cpp, go, and rust (3 passed, 0 failed, 0 +skipped) with `verify.lua` asserting 27 behavioral checks per language. ## Cross-check, three ways **The clone flavors, under `strace -f`.** The claim was that three runtimes spell "make a child" three ways. Tracing each binary running `/bin/true` with `strace -f -e trace=clone,clone3,vfork,execve,wait4,waitid`, the -process-creating call and the reap, verbatim per runtime — C++ (glibc +process-creating call and the reap, verbatim per runtime: C++ (glibc `fork`): ``` @@ -502,7 +502,7 @@ process-creating call and the reap, verbatim per runtime — C++ (glibc 2600465 <... wait4 resumed>, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2600466 ``` -A full copy-on-write fork — `child_stack=NULL` means "same stack, COW" — +A full copy-on-write fork (`child_stack=NULL` means "same stack, COW") then `wait4` with a null rusage pointer (pmon's C++ uses `getrusage` instead). Go issues: @@ -515,11 +515,11 @@ instead). Go issues: ``` vfork semantics (`CLONE_VM|CLONE_VFORK`: shared address space, parent -suspended until exec), *plus* `CLONE_PIDFD` — the reap is +suspended until exec), *plus* `CLONE_PIDFD`: the reap is `waitid(P_PIDFD, 6, ...)`, identifying the child by the returned pidfd, and -the trace even shows Go probing pidfd support first — a +the trace even shows Go probing pidfd support first, a `waitid(P_PIDFD, 4, ...) = -1 ECHILD` self-check, then a throwaway -`CLONE_PIDFD` clone waited on fd 5 — before the real spawn. Rust is the +`CLONE_PIDFD` clone waited on fd 5, before the real spawn. Rust is the third flavor: ``` @@ -530,8 +530,8 @@ third flavor: 2600701 <... wait4 resumed>, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={tv_sec=0, tv_usec=0}, ru_stime={tv_sec=0, tv_usec=656}, ...}) = 2600702 ``` -`clone3(2)` — the modern extensible spelling, here via glibc's -`posix_spawn` — with the same vfork optimization plus +`clone3(2)` (the modern extensible spelling, here via glibc's +`posix_spawn`) with the same vfork optimization plus `CLONE_CLEAR_SIGHAND`, and then pmon's own explicit `wait4` with a real rusage out-parameter, exactly as `wait4()` in `rust/src/main.rs` claims. Three clone flavors, one lifecycle. @@ -554,20 +554,20 @@ Tgid: 2601641 2601641 (sleep) Z 2601639 2601637 2601637 0 -1 4227084 103 0 0 0 0 0 0 0 25 5 1 0 64173304 0 0 ... ``` -Walk the `stat` fields (the format is `pid (comm) state ...` — parse from +Walk the `stat` fields (the format is `pid (comm) state ...`; parse from the right of the closing parenthesis, since `comm` may contain spaces): field 3 is `Z`, the zombie state `ps` renders as ``; field 4 is the -ppid `2601639` — stopped pmon, which `ps f`'s tree draws as the parent of -its own corpse; field 10, `minflt=103`, is the copy-on-write bill discussed -earlier; field 19 is nice `5` (the `N` in both STAT columns — this session -ran niced); and fields 23–24, `vsize` and `rss`, are both `0` — the address +ppid `2601639` (stopped pmon, which `ps f`'s tree draws as the parent of +its own corpse); field 10, `minflt=103`, is the copy-on-write bill discussed +earlier; field 19 is nice `5` (the `N` in both STAT columns; this session +ran niced); and fields 23–24, `vsize` and `rss`, are both `0`: the address space is already gone, which is why killing a zombie is meaningless: there is nothing left to kill, only data waiting to be read. `kill -CONT 2601639` ended the window: pmon's `wait_child` returned at last and it printed -`pmon: pid 2601641 exited status 0` with `wall=4343ms` — wall time measures +`pmon: pid 2601641 exited status 0` with `wall=4343ms`: wall time measures until the *reap*, not the death, and the stopped window is in the bill. -**An orphan, reparented.** The inverse experiment — parent dies first — +**An orphan, reparented.** The inverse experiment (parent dies first) takes one line: `sh -c 'sleep 7 >/dev/null 2>&1 & echo $!'` prints the background child's pid and exits, orphaning it. Half a second later: @@ -581,10 +581,10 @@ background child's pid and exits, orphaning it. Half a second later: ``` The dead shell's pid is gone from the pair; the child's `PPid` is now 6145, -the user's `systemd --user` — a registered subreaper, which will `waitid` +the user's `systemd --user`, a registered subreaper, which will `waitid` the sleep the moment it exits. No zombie ever forms on this path. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the fleet-wide view of this chapter is `execsnoop` and `exitsnoop` from > bcc-tools: every exec and every exit on the box, with the same > status/signal split pmon decodes, no per-process strace needed. bcc-tools @@ -594,20 +594,20 @@ the sleep the moment it exits. No zombie ever forms on this path. ## What you learned -- A process is born in two steps — `fork` (COW: page tables now, pages only +- A process is born in two steps: `fork` (COW: page tables now, pages only on write; 103 minor faults for a whole spawn) and `exec` (new image, same - pid and fds) — and the gap between them is where fd plumbing and privilege + pid and fds), and the gap between them is where fd plumbing and privilege dropping live; a CLOEXEC self-pipe is how exec failure reports back. - Death is data: the kernel holds exit status and rusage as a zombie until `wait4`/`waitid` reaps it, `ps` state `Z`/`` with `rss=0`; orphans are reparented to a subreaper (`systemd --user` here), never abandoned. - One wait status, two disjoint deaths: `WIFEXITED`+`WEXITSTATUS` vs - `WIFSIGNALED`+`WTERMSIG`, flattened by the shell convention pmon mirrors — + `WIFSIGNALED`+`WTERMSIG`, flattened by the shell convention pmon mirrors: `s`, `128+n`, or 127 when exec itself failed. - The tidy APIs are the same machinery: glibc `fork` is `clone(SIGCHLD)`, Go spawns with `clone(CLONE_VM|CLONE_PIDFD|CLONE_VFORK)` and reaps by pidfd via `waitid(P_PIDFD)`, Rust's `Command` rides - `posix_spawn`'s `clone3(CLONE_VM|CLONE_VFORK|CLONE_CLEAR_SIGHAND)` — + `posix_spawn`'s `clone3(CLONE_VM|CLONE_VFORK|CLONE_CLEAR_SIGHAND)`: three flavors of one lifecycle, all read from `strace -f`. Next, **safe signals**: what a handler is allowed to do (almost nothing), @@ -616,7 +616,7 @@ signals to a child without racing its death. --- -

verified — every number and +

verified: every number and output line above was produced on the Fedora 44 reference host this session: the runner printed 11-process-lifecycle PASS PASS PASS (3 passed, 0 failed, 0 skipped; 27 verify.lua checks per language); the fate diff --git a/_docs/12-signals-and-safe-handling.md b/_docs/12-signals-and-safe-handling.md index b08c0fd..4890013 100644 --- a/_docs/12-signals-and-safe-handling.md +++ b/_docs/12-signals-and-safe-handling.md @@ -2,20 +2,20 @@ title: "Signals and safe handling" order: 12 part: "Processes, Signals, Privilege" -description: "Dispositions, per-thread masks, and the pending set — then three race-free ways to consume signals as pmon v1 becomes a restarting supervisor: signalfd in C++, runtime channels in Go, the self-pipe trick in Rust, with strace catching Go's 114-call rt_sigaction storm and SIGURG preemption in the act." +description: "Dispositions, per-thread masks, and the pending set, then three race-free ways to consume signals as pmon v1 becomes a restarting supervisor: signalfd in C++, runtime channels in Go, the self-pipe trick in Rust, with strace catching Go's 114-call rt_sigaction storm and SIGURG preemption in the act." duration: 50 minutes --- Chapter 11 taught `pmon` to run one child and report how it died. Version 1 makes it a real supervisor: restart a crashing child with a doubling backoff, shut down cleanly on SIGTERM or SIGINT, and replace the child on SIGHUP. That -sounds like three `signal()` calls — and that is exactly the trap. The one new +sounds like three `signal()` calls, and that is exactly the trap. The one new idea of this chapter is that a signal handler is close to lawless territory where almost no code is safe to run, so well-built programs run *no* meaningful -code there at all: they convert signals into ordinary data — a file descriptor -read, a channel receive, a byte in a pipe — and consume that data in a plain +code there at all: they convert signals into ordinary data (a file descriptor +read, a channel receive, a byte in a pipe) and consume that data in a plain loop. Each language implements a different one of the three classic -conversions, on purpose, because together they are the lesson. +conversions, because together they make the lesson. The code is in `examples/12-signals-and-safe-handling/`. `./demo.sh` builds all three implementations and drives a crash-restart-give-up cycle; the @@ -25,9 +25,9 @@ three languages share. {% include excalidraw.html file="12-signal-delivery-paths" alt="Two bands. Kernel band: kill and child-exit sources feed a per-process disposition table and a per-thread mask, converging on an amber pending set that coalesces one bit per signal, then a dashed delivery box gated on unblocked. User band: three columns for C++ signalfd read, Go runtime handler into a signal.Notify channel, and Rust self-pipe handler setting an AtomicBool and writing a doorbell byte, each ending in the main loop reading plain data." - caption="Figure 12.1 — every signal passes the same three kernel words — disposition, mask, pending — and pmon hears it three ways, none of them in signal context" %} + caption="Figure 12.1: every signal passes the same three kernel words (disposition, mask, pending) and pmon hears it three ways, none of them in signal context" %} -> **Tools used** — `strace`, `kill`, `sh`, `python3`, `/proc` (host); +> **Tools used**: `strace`, `kill`, `sh`, `python3`, `/proc` (host); > `bpftrace` (lab VM, exercised in Part 8). Everything here is checked by > `scripts/check-host.sh`, ships with Fedora, or is preinstalled in the lab > VMs. @@ -38,18 +38,18 @@ The kernel's whole signal model fits in three words, and every bug in this area comes from mixing them up. The **disposition** is per-process: for each signal, one of `SIG_DFL` (the -default — terminate, ignore, stop, or dump core, depending on the signal), +default: terminate, ignore, stop, or dump core, depending on the signal), `SIG_IGN`, or a handler function installed with `sigaction(2)`. Dispositions survive `fork`; across `execve` a handler resets to `SIG_DFL` (the function pointer would be meaningless in the new image) while `SIG_IGN` survives. The **mask** is per-*thread*: the set of signals whose delivery is postponed. -A blocked signal is not lost — it waits. The mask survives both `fork` *and* +A blocked signal is not lost; it waits. The mask survives both `fork` *and* `exec`, which is a supervisor-shaped landmine: `pmon supervise` blocks SIGTERM, and a forked child inherits that mask straight through `execvp`, so the supervised program would start life unable to be terminated. This is why `spawn` in all three implementations resets the mask in the child, between -`fork` and `exec` — the C++ spelling, from +`fork` and `exec`: the C++ spelling, from `examples/12-signals-and-safe-handling/cpp/src/main.cpp`: ```cpp @@ -71,19 +71,19 @@ of SIGCHLD semantics: the signal means "one **or more** children changed state", never "exactly one". The robust consumer treats SIGCHLD as a doorbell and asks `waitpid(…, WNOHANG)` what actually happened; `pmon` has exactly one child at a time, so a targeted `waitpid(child, …, WNOHANG)` is the whole -answer — the general reap-until-empty loop arrives with pmon's later growth. +answer; the general reap-until-empty loop arrives with pmon's later growth. Delivery itself happens only when a signal is pending *and* unblocked in some -thread, at an arbitrary instant between two instructions — which is the +thread, at an arbitrary instant between two instructions, which is the problem the next section is about. ## Why handlers are a minefield -A handler interrupts your program wherever it happens to be — possibly +A handler interrupts your program wherever it happens to be, possibly halfway through `malloc`, holding the allocator's lock. If the handler then calls `malloc` (or `printf`, or anything that might), the same thread waits on a lock it already holds: deadlock, rarely, under load, in production. That is why POSIX defines **async-signal-safety**: the short list of -functions guaranteed re-entrant against this kind of interruption — +functions guaranteed re-entrant against this kind of interruption: `write`, `_exit`, `sigprocmask`, `kill`, `waitpid`, and a few dozen friends (`man 7 signal-safety`). Not `malloc`. Not `printf`. Not most of any language's runtime: locking a mutex, growing a Go stack, or formatting a @@ -94,7 +94,7 @@ The chapter's three implementations all draw the same conclusion: get out of signal context immediately, running either nothing there (C++), or only handler code someone already proved safe (Go's runtime, Rust's `signal-hook`). The only place `pmon` itself must obey the list is the child -branch of `spawn` above — after `fork` in a program that may have threads +branch of `spawn` above: after `fork` in a program that may have threads (Go always does), the child runs with only async-signal-safe calls until `execvp`: `sigprocmask`, `execvp`, `write`, `_exit`. All four are on the list, and nothing else happens there. @@ -102,12 +102,12 @@ list, and nothing else happens there. ## The self-pipe trick, and signalfd as its resolution The classic escape is the **self-pipe trick**: at startup, create a pipe; -the handler does exactly one safe thing — `write` one byte to it — and the +the handler does exactly one safe thing (`write` one byte to it), and the main loop, already blocked in `poll`, wakes up, drains the pipe, and handles the event as ordinary code. The pipe is the bridge from signal context to loop context, and `write(2)` is async-signal-safe. The Rust implementation is deliberately the historical exhibit, with the unsafe parts delegated to -`signal-hook` — each signal is registered twice, a flag and a doorbell, from +`signal-hook`: each signal is registered twice, a flag and a doorbell, from `examples/12-signals-and-safe-handling/rust/src/main.rs`: ```rust @@ -127,7 +127,7 @@ fn flag_for(signal: i32, doorbell: &PipeWriter) -> Result> { Linux then folded the trick into the kernel: `signalfd(2)` gives you a file descriptor that *is* the pipe, with structured `signalfd_siginfo` records instead of anonymous doorbell bytes. You block the signals with -`sigprocmask` — so normal delivery never happens — and read them like any +`sigprocmask` (so normal delivery never happens) and read them like any other fd. The C++ implementation opens with exactly that pair: ```cpp @@ -148,8 +148,8 @@ other fd. The C++ implementation opens with exactly that pair: No handler is ever installed; nothing asynchronous ever runs. Go is the third design point: the runtime owns real handlers for almost every signal -(you will count them in the cross-check), and those handlers — written once, -by people who proved them safe — forward into the scheduler, where +(you will count them in the cross-check), and those handlers (written once, +by people who proved them safe) forward into the scheduler, where `signal.Notify` turns delivery into a channel send: ```go @@ -169,21 +169,21 @@ everything meaningful happens in a loop that reads plain values. `pmon` forwards shutdown explicitly: `kill(child_pid, SIGTERM)`, one target. But when you press Ctrl-C in a terminal, the kernel sends SIGINT to the -foreground **process group** — pmon *and* its child both receive it, because +foreground **process group**: pmon *and* its child both receive it, because `spawn` leaves the child in pmon's group. That is a deliberate v1 choice, and it composes: the child dies of its default SIGINT disposition, pmon -consumes its copy through its mechanism, reaps, prints the shutdown line — +consumes its copy through its mechanism, reaps, prints the shutdown line: the same path `kill -INT ` exercises alone. The alternative design, `setpgid` the child into its own group and signal `-pgid` to reach a whole tree, is what shells do to implement job control; pmon does not need it yet, and neither spelling can catch a child's *grandchildren* that -daemonize out of the group — the real fix for that is cgroups, in the +daemonize out of the group; the real fix for that is cgroups, in the containers part. {% include excalidraw.html file="12-supervisor-state-machine" alt="State machine: an amber child-running state with infinite poll timeout, a backoff-wait state with deadline poll timeout, an amber transition between them labeled with the doubling backoff, a return transition when the backoff elapses, exits to exit 0 on clean exit or SIGTERM/SIGINT and to exit 1 when restarts reach the maximum, and a dashed SIGHUP reload box that stops the child, resets the budget, and spawns a replacement." - caption="Figure 12.2 — pmon supervise as a two-state machine; every transition is triggered by an ordinary value read in the main loop" %} + caption="Figure 12.2: pmon supervise as a two-state machine; every transition is triggered by an ordinary value read in the main loop" %} ## How the code works @@ -195,7 +195,7 @@ every crash. A crash moves the machine from *running* to *backoff wait*; the backoff elapsing moves it back by spawning a fresh child; a clean exit, SIGTERM, or SIGINT leaves through exit 0; exhausting the budget leaves through exit 1; SIGHUP stays in *running* but swaps the child and resets -both counter and backoff — a reload should not inherit a crashing child's +both counter and backoff: a reload should not inherit a crashing child's penalties. Here is the consumption loop, three ways: {% include codetabs.html langs="C++|Go|Rust" %} @@ -415,13 +415,13 @@ penalties. Here is the consumption loop, three ways: } ``` -Structure first: all three block on *one* wait point — `poll` on the -signalfd, `select` over three channels, `poll` on the doorbell — and the +Structure first: all three block on *one* wait point (`poll` on the +signalfd, `select` over three channels, `poll` on the doorbell), and the backoff is not a `sleep` but that wait's *timeout* (Go uses a timer channel for the same effect). That single decision keeps the supervisor responsive: a SIGTERM arriving mid-backoff interrupts the wait instead of queuing behind a sleep. In C++ and Rust the timeout is recomputed each iteration from an -absolute deadline — poll can return early for unrelated reasons, and +absolute deadline; poll can return early for unrelated reasons, and re-deriving "time left" from a deadline is immune to that; the `+ 1` guards the truncation of a fractional millisecond so the loop cannot spin at zero-timeout just before the deadline. @@ -429,43 +429,43 @@ zero-timeout just before the deadline. Three per-language details deserve a closer look. The C++ SIGCHLD arm checks `reaped != child` and drops the record: on a reload, `stop_child` already reaped the old child synchronously, but the SIGCHLD it generated is -still queued in the signalfd — a *stale notification*. The state machine +still queued in the signalfd: a *stale notification*. The state machine must be driven by `waitpid`'s answer, never by the signal's arrival; the signal is a doorbell, remember, not a fact. Go's version of the same care is `waitCh` having buffer size 1 with one wait-goroutine per child: `stopChild` consumes the exit synchronously from the channel, so a reload cannot leave a goroutine's send racing the next child's. Rust's is the -drain-then-check ordering — read up to 64 doorbell bytes (several signals +drain-then-check ordering: read up to 64 doorbell bytes (several signals may have coalesced into one wakeup, exactly like the pending set), *then* swap each `AtomicBool` to false, then let `waitpid(WNOHANG)` say whether SIGCHLD meant anything; a doorbell byte with nothing behind it (`StillAlive`) just loops. -Fragile bits, stated plainly: the doubling backoff is uncapped — with +Fragile bits: the doubling backoff is uncapped, with `--backoff-ms 1` and `--max-restarts 40` the final wait would be 2³⁹ ms, -about seventeen years — so `--max-restarts` is the only brake, and a +about seventeen years, so `--max-restarts` is the only brake, and a production supervisor would clamp the backoff itself; Rust's `PollTimeout` is a `u16`, so waits past 65.5 s are clamped and re-entered (the `Instant::now() < deadline` check); and `run` (unchanged from v0) reaps with the default mask and dispositions, so Ctrl-C during `pmon run` kills both -processes — supervision semantics belong to `supervise` only. +processes; supervision semantics belong to `supervise` only. ## Errors, three ways -The contract is unchanged from chapter 11 — `pmon: error: …` on stderr and -exit 1 for runtime failures, usage on exit 2 — but signals sharpen the +The contract is unchanged from chapter 11: `pmon: error: …` on stderr and +exit 1 for runtime failures, usage on exit 2, but signals sharpen the `EINTR` policy from chapter 4 into three distinct spellings. C++ retries `poll` on `EINTR` explicitly, and `reap_blocking` loops `waitpid` the same way; under `supervise` the mask makes those interruptions nearly impossible, -but `run` reaps with the default mask, so the loop is load-bearing there. -Rust matches `Errno::EINTR` as a value in both `poll` and `waitpid` — its -handlers really do interrupt syscalls, as the cross-check will show — and +but `run` reaps with the default mask, so the loop still matters there. +Rust matches `Errno::EINTR` as a value in both `poll` and `waitpid`; its +handlers really do interrupt syscalls, as the cross-check will show, and additionally tolerates `Errno::ECHILD` ("no child to reap") as a stale doorbell rather than an error. Go never sees `EINTR` at all: the runtime installs every handler with `SA_RESTART` and retries interrupted waits inside `cmd.Wait`, so user code only ever receives the final verdict. One deliberate acceptance shows up in all three `stop_child`s: `ESRCH` from -`kill` means the child is already gone mid-shutdown — a race you expect, +`kill` means the child is already gone mid-shutdown: a race you expect, handled at the call site, while everything else carries context up to one printer, exactly the chapter-5 policy. @@ -477,7 +477,7 @@ blocked while handling) and `SA_ONSTACK` (an alternate stack, so a handler survives even a blown goroutine stack), and the runtime *sends itself* signals: since Go 1.14, a goroutine stuck in a tight loop is preempted by a `tgkill(…, SIGURG)` from another runtime thread. An idle supervisor never -needs that, but GC pressure forces it — `GOGC=1` makes collections constant, +needs that, but GC pressure forces it: `GOGC=1` makes collections constant, and the trace shows runtime threads interrupting each other: ```bash @@ -490,7 +490,7 @@ and the trace shows runtime threads interrupting each other: 4 ``` -Four SIGURG deliveries in a supervisor that never asked for any signal — +Four SIGURG deliveries in a supervisor that never asked for any signal; your Go programs live in this weather all the time, which is why `os/exec` retries `EINTR` for you and why you must never install a C signal handler into a Go process. The same lens explains a signalfd caveat: the mask is @@ -499,8 +499,8 @@ might otherwise take delivery. The C++ code blocks the mask before any thread could exist, so the invariant holds trivially; in a threaded program you must block-then-spawn so children inherit the mask. Rust's mechanism is the only one where user-adjacent code truly runs in signal context, and its -concurrency budget is exactly one atomic store plus one non-blocking `write` -— a full doorbell drops the byte harmlessly, because bytes, like pending +concurrency budget is exactly one atomic store plus one non-blocking `write`: +a full doorbell drops the byte harmlessly, because bytes, like pending bits, are allowed to coalesce. ## Build, run, observe @@ -509,7 +509,7 @@ bits, are allowed to coalesce. [host]$ cd examples/12-signals-and-safe-handling && ./demo.sh ``` -Each language builds and runs the crash loop. By hand, from this session — +Each language builds and runs the crash loop. By hand, from this session, `run` still propagates verdicts exactly as v0 did: ```bash @@ -535,7 +535,7 @@ pmon: giving up after 2 restarts Three distinct pids, the backoff printed doubling, exit 1. Driving the Go build with signals from another shell produces the reload-then-shutdown -story — note the replacement pid: +story: note the replacement pid: ```bash [host]$ ./go/bin/app supervise -- sleep 300 & @@ -547,8 +547,8 @@ pmon: started pid 2625068 pmon: shutting down (SIGTERM) ``` -`verify.lua` asserts all of this behaviorally — 31 checks per language, -including measured backoff elapsed time and post-reload shutdown — and this +`verify.lua` asserts all of this behaviorally (31 checks per language, +including measured backoff elapsed time and post-reload shutdown), and this session's runner (`python3 scripts/test-all-examples.py --only 12-signals-and-safe-handling`) reports `PASS PASS PASS` for cpp, go, and rust, with a standalone `LSP_LANG=cpp` run printing `PASS 31 / FAIL 0`. @@ -557,7 +557,7 @@ rust, with a standalone `LSP_LANG=cpp` run printing `PASS 31 / FAIL 0`. **The syscall shape, under `strace`.** If the three designs are real, the traces must be unmistakably different. The C++ supervisor's *entire* signal -footprint for a one-restart run is ten lines — zero `rt_sigaction`, one +footprint for a one-restart run is ten lines: zero `rt_sigaction`, one `signalfd4`: ```bash @@ -576,20 +576,20 @@ poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}]) +++ exited with 1 +++ ``` -(The block-all/restore pairs are glibc protecting its own `fork` internals — +(The block-all/restore pairs are glibc protecting its own `fork` internals, not our code.) No `--- SIGCHLD ---` stops ever appear: blocked signals are never delivered, they are *read*, and the 30 ms poll timeout is the backoff. The same command with `-f` on the Go binary tells the opposite story: this session's supervisor pid 2617196 issued exactly **114 `rt_sigaction` -calls** — 58 probes reading old dispositions plus 56 installs of the runtime +calls**: 58 probes reading old dispositions plus 56 installs of the runtime handler (`sa_handler=0x484d40`, `sa_mask=~[]`, -`SA_ONSTACK|SA_RESTART|SA_SIGINFO`) covering SIGHUP through `SIGRT_32` — the +`SA_ONSTACK|SA_RESTART|SA_SIGINFO`) covering SIGHUP through `SIGRT_32`, the identical 114-call storm chapter 4 counted for a program that never touched `os/signal`, because it is runtime initialization, not `signal.Notify`. Zero `signalfd4`, zero `poll`/`ppoll` (the runtime waits in `futex` and its -netpoller). Rust sits between: 13 `rt_sigaction` — probe+install pairs for +netpoller). Rust sits between: 13 `rt_sigaction`, probe+install pairs for our four signals via `signal-hook`, plus `SIGPIPE` set to ignore and a -`SIGSEGV`/`SIGBUS` pair that std installs for stack-overflow reporting — a +`SIGSEGV`/`SIGBUS` pair that std installs for stack-overflow reporting, plus a `pipe2([3, 4], O_CLOEXEC)` for the doorbell, and then real deliveries interrupting the wait, self-pipe mechanics live on the wire: @@ -600,7 +600,7 @@ poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}]) ``` **The kernel's bookkeeping, in `/proc`.** `/proc//status` prints the -three words of this chapter as hex bitmasks — bit *n* is signal *n + 1*. A +three words of this chapter as hex bitmasks: bit *n* is signal *n + 1*. A live C++ supervisor from this session: ```bash @@ -614,9 +614,9 @@ SigCgt: 0000000000000000 ``` Decode `SigBlk` by hand: `0x14003` = bits 0, 1, 14, 16 = signals 1, 2, 15, -17 — SIGHUP, SIGINT, SIGTERM, SIGCHLD, precisely the `sigprocmask` set. And -`SigCgt: 0`: this process catches *nothing* — the whole design in one zero. -(`SigIgn: 6` — SIGINT and SIGQUIT — was inherited from the non-interactive +17: SIGHUP, SIGINT, SIGTERM, SIGCHLD, precisely the `sigprocmask` set. And +`SigCgt: 0`: this process catches *nothing*, the whole design in one zero. +(`SigIgn: 6`, SIGINT and SIGQUIT, was inherited from the non-interactive shell that launched this background job: dispositions cross `fork` and `exec`, as promised.) The other two, decoded with three lines of Python against the same live technique: @@ -632,18 +632,18 @@ go SigCgt -> 56 caught; SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGABRT ... rust SigCgt -> 6 caught; SIGHUP SIGINT SIGBUS SIGSEGV SIGTERM SIGCHLD ... ``` -Go catches 56 of 64 signals — everything except SIGKILL and SIGSTOP (which +Go catches 56 of 64 signals: everything except SIGKILL and SIGSTOP (which no one may catch), the job-control quartet SIGCONT/SIGTSTP/SIGTTIN/SIGTTOU, -and two reserved realtime slots — while its `SigBlk` line +and two reserved realtime slots, while its `SigBlk` line (`fffffffc3bba3a00` on this run) belongs to the main thread only; the mask is per-thread, and the runtime retunes it constantly, which is the `rt_sigprocmask` chatter in its trace. Rust's `0x14443` is our four signals -(`0x14003`, the same bits as the C++ *blocked* set — caught in one design, +(`0x14003`, the same bits as the C++ *blocked* set; caught in one design, blocked in the other) plus bits 6 and 10: SIGBUS and SIGSEGV, the std stack-overflow guard the strace already showed. Three processes, one hex format, and each design legible straight from the kernel's ledger. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the kernel-side view of this chapter is `bpftrace` on > `tracepoint:signal:signal_generate` and `signal_deliver`, watching the > pending-set handoff fleet-wide instead of per-process: which pid raised @@ -654,8 +654,8 @@ format, and each design legible straight from the kernel's ledger. ## What you learned - A signal's life is three kernel words: a per-process **disposition**, a - per-thread **mask** (which survives fork *and* exec — reset it in the - child), and a coalescing **pending set** — which is why SIGCHLD means "one + per-thread **mask** (which survives fork *and* exec; reset it in the + child), and a coalescing **pending set**, which is why SIGCHLD means "one or more" and `waitpid(WNOHANG)` is the only source of truth. - Handlers interrupt arbitrary code, so only the async-signal-safe list may run there; the professional designs shrink signal context to nothing: @@ -666,16 +666,16 @@ format, and each design legible straight from the kernel's ledger. be discarded by checking `waitpid`'s answer, not the signal's arrival. - The three designs are visible from outside: 0 vs 114 vs 13 `rt_sigaction` calls, `signalfd4` vs a runtime `tgkill(…, SIGURG)` storm vs `--- SIGCHLD - ---` interrupting `poll` — and `/proc//status` decodes each one in + ---` interrupting `poll`, and `/proc//status` decodes each one in hex. -Next, **pidfds**: the pid-reuse race this supervisor still carries — a -recycled pid can send our forwarded SIGTERM to a stranger — and the file +Next, **pidfds**: the pid-reuse race this supervisor still carries (a +recycled pid can send our forwarded SIGTERM to a stranger) and the file descriptor that pins a process's identity for good. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host this session: the runner printed 12-signals-and-safe-handling PASS PASS PASS (3 passed, 0 failed, 0 skipped) and a standalone diff --git a/_docs/13-pidfd-modern-process-mgmt.md b/_docs/13-pidfd-modern-process-mgmt.md index 53b5fde..480731b 100644 --- a/_docs/13-pidfd-modern-process-mgmt.md +++ b/_docs/13-pidfd-modern-process-mgmt.md @@ -2,19 +2,19 @@ title: "pidfd: modern process management" order: 13 part: "Processes, Signals, Privilege" -description: "The pid-reuse race constructed step by step, then closed for good: pidfd_open, clone3(CLONE_PIDFD), poll on a process, waitid(P_PIDFD), and pidfd_send_signal become pmon v2's default engine — with strace evidence of what the Go runtime and the Rust stdlib really do with pidfds today." +description: "The pid-reuse race constructed step by step, then closed for good: pidfd_open, clone3(CLONE_PIDFD), poll on a process, waitid(P_PIDFD), and pidfd_send_signal become pmon v2's default engine, with strace evidence of what the Go runtime and the Rust stdlib really do with pidfds today." duration: "45 minutes" --- `pmon` v1 ended the last chapter supervising a crashing child correctly: no async handlers, SIGCHLD arriving as data through a signalfd, a `WNOHANG` reap classifying every death. But it still carries a lie it inherited from 1970s -UNIX — it names its child by an *integer*, and integers get reused. Between +UNIX: it names its child by an *integer*, and integers get reused. Between the moment a child dies and the moment you act on its pid, that number can come to mean a completely different process, and `kill(2)` will cheerfully signal the stranger. The one new idea in this chapter is that a process can be a *file descriptor*: `pidfd_open(2)` hands you an fd that refers to the -process itself, not to its number — pollable in exactly the event-loop style +process itself, not to its number, pollable in exactly the event-loop style Chapter 9 built, reapable with `waitid(2)`, signalable with `pidfd_send_signal(2)`, and immune to reuse by construction. `pmon` v2 makes that its default engine and keeps the v1 sigchld engine beside it, @@ -28,9 +28,9 @@ three languages share. {% include excalidraw.html file="13-pid-reuse-race-pidfd" alt="Two horizontal timeline lanes. Top lane, five numbered steps: spawn remembering pid 4242, child crashes leaving a zombie, a wait reaps it freeing the pid, the kernel recycles 4242 for an unrelated process, and kill 4242 delivers SIGTERM to the stranger, with an amber dashed arrow noting same integer different process. Bottom lane, four numbered steps: spawn plus pidfd_open pinning child 4242 to pidfd 4, the child exit turning the fd readable under poll, waitid P_PIDFD reaping exactly the process behind the fd, and pidfd_send_signal returning ESRCH after death instead of hitting a recycled pid." - caption="Figure 13.1 — the pid-reuse race in five numbered steps, and the pidfd lane that closes it; the fdinfo and strace annotations are from this chapter's real runs" %} + caption="Figure 13.1: the pid-reuse race in five numbered steps, and the pidfd lane that closes it; the fdinfo and strace annotations are from this chapter's real runs" %} -> **Tools used** — `strace`, `pgrep`, `man`, `ls`/`cat` over `/proc`, +> **Tools used**: `strace`, `pgrep`, `man`, `ls`/`cat` over `/proc`, > `gcc`, `rustc` (host); `exitsnoop`/`killsnoop` bcc-tools (lab VM, exercised > in Part 8). Everything here is checked by `scripts/check-host.sh`, ships > with Fedora, or is preinstalled in the lab VMs. @@ -38,26 +38,26 @@ three languages share. ## The pid-reuse race, in five steps Build the failure concretely, because every step is ordinary. (1) You spawn -a worker and remember its pid — say 4242. A pid is an index into a kernel +a worker and remember its pid, say 4242. A pid is an index into a kernel table, nothing more; on this host the table wraps at `/proc/sys/kernel/pid_max` = 4194304. (2) The worker crashes. It becomes a zombie: the pid stays reserved *only* until someone collects the exit -status. (3) Someone does — and here is the trap: in any program bigger than +status. (3) Someone does, and here is the trap: in any program bigger than a toy, "someone" is not always you. A library thread that spawns helpers, a language runtime that reaps everything (Chapter 11 showed Go's runtime -holding exactly this power), a signal handler someone else installed — any +holding exactly this power), a signal handler someone else installed: any `wait()` in the process can free the pid. (4) The kernel recycles 4242 for an unrelated process. Under default settings that needs the counter to wrap, but a busy build server wraps in hours, and a container with a small -`pid_max` wraps in minutes. (5) You finally act on your saved integer — -`kill(4242, SIGTERM)` — and the signal lands on the stranger. Steps 3→5 are +`pid_max` wraps in minutes. (5) You finally act on your saved integer +(`kill(4242, SIGTERM)`), and the signal lands on the stranger. Steps 3→5 are the window in Figure 13.1; nothing you hold distinguishes old 4242 from new 4242, because an integer carries no identity. -One subtlety makes supervisors *seem* safe and is worth stating precisely: a +One subtlety makes supervisors only *seem* safe: a zombie's pid cannot be recycled, so as long as *you* are the parent and *you* have not reaped yet, your pid is still good. The race needs a reap you -don't control — which is exactly what runtimes, PAM modules, and "helpful" +don't control, which is exactly what runtimes, PAM modules, and "helpful" libraries do. And any pid-based operation on a process you are *not* the parent of (a `kill` from a shell script, a `/proc/` poke from a monitoring agent) has the race with no mitigation at all. @@ -65,33 +65,33 @@ monitoring agent) has the race with no mitigation at all. ## Four syscalls that make a process a file The fix arrived in a burst across Linux 5.1–5.4, and this host's manpages -(kernel `7.1`) document each piece. `pidfd_open(2)` — HISTORY: -"Linux 5.3" — "creates a file descriptor that refers to the process whose +(kernel `7.1`) document each piece. `pidfd_open(2)`'s manpage (HISTORY: +"Linux 5.3") says it "creates a file descriptor that refers to the process whose PID is specified in pid"; the close-on-exec flag is set automatically, so the Chapter 7 discipline costs nothing here. The fd's superpower is stated under NOTES: it "can be monitored using poll(2), select(2), and epoll(7). When the process that it refers to terminates, these interfaces indicate the -file descriptor as readable" — a process exit becomes an ordinary readiness +file descriptor as readable": a process exit becomes an ordinary readiness event, composable with every other fd your Chapter 9 event loop owns. There is nothing to actually read: `read(2)` on a pidfd fails with `EINVAL`. The -manpage also warns that an fd obtained by opening `/proc/` — the older -trick — "is not pollable and can't be waited on with waitid(2)". +manpage also warns that an fd obtained by opening `/proc/` (the older +trick) "is not pollable and can't be waited on with waitid(2)". For your *own* children there is a spelling with no window at all: `clone(2)`/`clone3(2)` with `CLONE_PIDFD` (since Linux 5.2) delivers the pidfd atomically at spawn time, via `parent_tid` for `clone` or -`cl_args.pidfd` for `clone3` — the same "set the property at creation, not +`cl_args.pidfd` for `clone3`, the same "set the property at creation, not after" move as `O_CLOEXEC` in Chapter 7. Acting through the fd: `pidfd_send_signal(2)` (Linux 5.1) "sends the signal sig to the target process referred to by pidfd", and `waitid(2)` grew an ID type `P_PIDFD` (Linux 5.4) to "wait for the child referred to by the PID file descriptor". Once the process behind the fd has been reaped, signalling it -yields `ESRCH` — a clean error instead of a delivered signal to a recycled +yields `ESRCH`: a clean error instead of a delivered signal to a recycled pid, which is the whole point of the bottom lane of Figure 13.1. One documentation trap, caught live: this host's `pidfd_open` manpage still says "glibc provides no wrapper for pidfd_open(), necessitating the use of -syscall(2)" — but a one-line probe built with `gcc` against the installed +syscall(2)", but a one-line probe built with `gcc` against the installed glibc 2.43 (`#include ` … `pidfd_open(1,0)`) compiles *and links*. Manpages lag libraries. The C++ example keeps the raw @@ -103,11 +103,11 @@ and because seeing the syscall spelled out is the point of the chapter. Claims about runtime internals rot fast, so here is only what `strace -f` showed on *this* host, running *these* binaries. Go 1.26.5 is a pidfd enthusiast: before its first spawn, the runtime probes the whole feature -set — `pidfd_open` on its own pid, a `waitid(P_PIDFD, …)` that must fail +set: `pidfd_open` on its own pid, a `waitid(P_PIDFD, …)` that must fail `ECHILD`, a `pidfd_send_signal(fd, 0, NULL, 0)` no-op, and then a throwaway no-op child cloned with `CLONE_PIDFD` and reaped through its fd. Every subsequent `exec.Cmd` child is spawned as -`clone(CLONE_VM|CLONE_PIDFD|CLONE_VFORK|SIGCHLD, parent_tid=[6])` — the +`clone(CLONE_VM|CLONE_PIDFD|CLONE_VFORK|SIGCHLD, parent_tid=[6])`: the runtime keeps its *own* pidfd for every child it starts, acquired atomically at clone time, and `os.Process` documents using it for `Wait` and `Signal` when available. Rust 1.97.1's stdlib, by contrast, spawns with a plain @@ -127,17 +127,17 @@ The structure is shared across the three implementations. A `ChildStatus` formats it identically everywhere and computes the mirrored exit code (`status` as-is, `128+signo` for a signal death). An `Engine` selects one of two wait strategies behind one restart policy; an `Outcome` says which of -the two things a wait produced — a reaped status, or a stop request +the two things a wait produced: a reaped status, or a stop request (`signal` or `timeout`). C++ spells that as -`std::variant`, Rust as an `enum Outcome`, and Go — with -`select` as its native "whichever happens first" — doesn't need the type at +`std::variant`, Rust as an `enum Outcome`, and Go, with +`select` as its native "whichever happens first", doesn't need the type at all. The C++ `Fd` owner is the same class Chapter 7 built; Rust gets `OwnedFd` straight from `rustix::process::pidfd_open`, and Go holds the raw integer from `unix.PidfdOpen` with an explicit `unix.Close`. Spawning carries one landmine the sigchld engine planted: the supervisor blocks SIGINT/SIGTERM (and SIGCHLD, in the sigchld engine) process-wide so -its signalfd can deliver them as data — and a blocked mask is *inherited* +its signalfd can deliver them as data, and a blocked mask is *inherited* across fork and exec. A supervised `sleep` that inherits a blocked SIGTERM can never be stopped. Each language defuses it differently: C++ `posix_spawnp` with `POSIX_SPAWN_SETSIGMASK` resets the child's mask @@ -265,16 +265,16 @@ fn wait_pidfd(pidfd: &impl AsFd, sigfd: &SignalFd, deadline: Instant) -> Result< ``` Walk the C++ and Rust versions first, because they are the same function. -`poll` watches two fds — the pidfd and the signalfd — with a timeout derived +`poll` watches two fds (the pidfd and the signalfd) with a timeout derived from the supervision deadline. `remaining_ms`/`remaining` recompute the budget on every iteration precisely because `poll` can return early (`EINTR`, a signalfd wakeup that turns out to be SIGCHLD-free); a fixed timeout would stretch the deadline every retry. A return of 0 is the deadline itself. The signalfd is checked *before* the pidfd deliberately: if the operator's Ctrl-C and the child's exit arrive together, we prefer to -report a stop — the reap still happens on the stop path, so nothing is lost. +report a stop; the reap still happens on the stop path, so nothing is lost. Only then does readability on `fds[0]` mean "the child terminated", and -`reap_pidfd` collects the status with `waitid(P_PIDFD, fd, …, WEXITED)` — +`reap_pidfd` collects the status with `waitid(P_PIDFD, fd, …, WEXITED)`, naming the child *by the fd*, so even in a process where some library reaps promiscuously, this call can never collect a stranger. `status_from` classifies `si_code == CLD_EXITED` versus killed/dumped, the same siginfo @@ -283,30 +283,30 @@ plumbing as Chapter 11. Go cannot put a channel in a `poll` set, so it inverts the bridge: one goroutine parks in `unix.Poll` on the pidfd alone, infinite timeout, and `close(exited)` converts readability into channel-readability. The `select` -then merges three event sources — exit, `signal.Notify` delivery, deadline — +then merges three event sources (exit, `signal.Notify` delivery, deadline) exactly as the C++/Rust `poll` merged two fds and a timeout. On a stop, the -`<-exited` after `PidfdSendSignal` is load-bearing twice over: it waits for +`<-exited` after `PidfdSendSignal` does double duty: it waits for the child to actually die (the pidfd goes readable), and it guarantees the -goroutine has finished touching `pidfd` before `unix.Close(pidfd)` runs — +goroutine has finished touching `pidfd` before `unix.Close(pidfd)` runs: close an fd a goroutine is still polling and the slot number could be recycled mid-poll, which is Chapter 7's double-close lesson wearing a concurrency hat. The stop path is the syscall the diagram promised: `pidfd_send_signal(pidfd, SIGTERM, …)` in all three languages, result -deliberately ignored. If the child died between the deadline and the signal, +ignored by design. If the child died between the deadline and the signal, the target is a zombie (signal discarded, returns 0) or already reaped -(`ESRCH`) — both harmless, and `reap_pidfd` right after collects the truth +(`ESRCH`), both harmless, and `reap_pidfd` right after collects the truth either way. The restart policy above all this is unchanged from v1: a clean `status=0` ends supervision, anything else burns one of `--max-restarts`, and the counter lines (`restart 1/2` … `giving up after 2 restarts`) are -byte-identical across engines and languages — `verify.lua` literally diffs +byte-identical across engines and languages; `verify.lua` literally diffs the sequences against each other. One fragile bit, documented rather than hidden: `pidfd_open` after spawn has a theoretical window that `CLONE_PIDFD` at clone time would close. It is safe *here* because `pmon` is the parent and reaps only through the fd, so -the child's pid is pinned (zombie at worst) until after the pidfd exists — +the child's pid is pinned (zombie at worst) until after the pidfd exists, but the moment you take pidfds of processes you did not spawn, insist on verifying identity (the cross-check below shows `fdinfo`'s `Pid:` line) or get the fd at clone time the way Go's runtime does. @@ -320,32 +320,32 @@ goroutine's `err == unix.EINTR`), and Rust (`Err(rustix::io::Errno::INTR) => continue`). Expected failures are policy at the call site: `pidfd_send_signal` on the stop path ignores its result (zombie or `ESRCH`, both fine), and the sigchld engine treats a `WNOHANG` -reap that returns nobody as "coalesced SIGCHLD, keep waiting" — Chapter 12's +reap that returns nobody as "coalesced SIGCHLD, keep waiting": Chapter 12's lesson encoded in all three languages. Everything unexpected carries context up to one printer with the raw error preserved: C++ wraps `errno` in `std::error_code` inside `std::expected`, Go wraps with `%w` so `errors.Is`/`errors.As` still see the errno underneath, Rust chains `.context("waitid(P_PIDFD)")` onto the `rustix` errno with `anyhow`. And one error is a *feature test*: Go's runtime probe expects `waitid(P_PIDFD)` on -itself to fail `ECHILD` — using a precise errno as a yes/no answer about +itself to fail `ECHILD`: using a precise errno as a yes/no answer about kernel support, the same trick your own code can use on kernels older than 5.4. ## Concurrency lens C++ and Rust run this whole chapter on one thread: one `poll`, two fds, no -shared state, no reentrancy — the safest concurrency is none. Go runs the +shared state, no reentrancy; the safest concurrency is none. Go runs the same logic as a small concurrent system: the runtime's own reaper machinery, a poller goroutine per spawn, and a `select` merging channels. That shape has two sharp edges the code steps around. First, fd lifetime across -goroutines — covered above; the `<-exited` handshake is what makes +goroutines (covered above); the `<-exited` handshake is what makes `unix.Close` safe. Second, process-wide signal state: the blocked mask that powers the C++/Rust signalfd is shared by *all* threads, which is why the child-side mask reset (`POSIX_SPAWN_SETSIGMASK`, `pre_exec`) is -non-negotiable — any thread may be the one that spawns. Note also what the +non-negotiable: any thread may be the one that spawns. Note also what the strace shows about fd-table behavior under concurrency: every spawn prints `pidfd=4` because the previous pidfd was closed before the next -`pidfd_open`, and Chapter 7's lowest-free-slot rule hands the number back — +`pidfd_open`, and Chapter 7's lowest-free-slot rule hands the number back, while in Go the runtime's *own* pidfd for the same child sits one slot over. Two fds, one process, no interference: identity multiplied is still identity. @@ -357,7 +357,7 @@ identity. ``` Each language builds and supervises a short-lived child. The chapter's two -canonical runs, from this host — a crash loop that exhausts its restart +canonical runs, from this host: a crash loop that exhausts its restart budget: ```bash @@ -373,7 +373,7 @@ pmon: child=2621529 exited status=1 pmon: giving up after 2 restarts ``` -— exit code 1, three spawns, `pidfd=4` every time. And a timeout stop that +This run shows exit code 1, three spawns, `pidfd=4` every time. And a timeout stop that must not leak the child: ```bash @@ -387,10 +387,10 @@ pgrep exit=1 The `pgrep` exit 1 is the observable that matters: no supervised `sleep` outlived its supervisor. (The `[.]` keeps `pgrep -f` from matching its own -command line — a pid-namespace-sized footgun in miniature.) The Rust binary +command line: a pid-namespace-sized footgun in miniature.) The Rust binary produces the same crash-loop shape (children 2622026–2622028 on this host), -and the runner — `python3 scripts/test-all-examples.py --only -13-pidfd-modern-process-mgmt` — reports `PASS PASS PASS` with `verify.lua`'s +and the runner (`python3 scripts/test-all-examples.py --only +13-pidfd-modern-process-mgmt`) reports `PASS PASS PASS` with `verify.lua`'s behavioral checks at `PASS 37 / FAIL 0` per language. ## Cross-check, three ways @@ -407,7 +407,7 @@ poll([{fd=4, events=POLLIN}, {fd=3, events=POLLIN}], 2, 9999) = 1 ([{fd=4, reven waitid(P_PIDFD, 4, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2622912, si_uid=1000, si_status=1, ...}, WEXITED, NULL) = 0 ``` -The stop path adds the last syscall of the set — +The stop path adds the last syscall of the set: `poll(..., 299) = 0 (Timeout)`, then `pidfd_send_signal(4, SIGTERM, NULL, 0) = 0`, then the same `waitid(P_PIDFD, 4, …)` now reporting `CLD_KILLED` with `si_status=SIGTERM`. Go shows the runtime's pidfd life underneath ours @@ -424,7 +424,7 @@ The stop path adds the last syscall of the set — `parent_tid=[6]` is the runtime's own pidfd for the child, delivered atomically by `CLONE_PIDFD`; fd 4 is ours. Rust spawns with a plain -`clone(CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD)` — no stdlib pidfd — +`clone(CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD)` (no stdlib pidfd) followed by our `pidfd_open(2623708, 0) = 4` and the identical `waitid(P_PIDFD, …)`. Three runtimes, three spawn spellings, one kernel API. @@ -448,11 +448,11 @@ NSpid: 2624390 The `Pid: 2624390` line is the payoff quoted directly from the kernel: the open file description *itself* records which process it refers to, and it matches the `child=2624390` that `pmon` announced. `flags: 02000002` is -`O_RDWR|O_CLOEXEC` — the close-on-exec bit the manpage promised, set with no +`O_RDWR|O_CLOEXEC`: the close-on-exec bit the manpage promised, set with no code on our side. (`NSpid` previews Part 10: inside a pid namespace the same process has a different number, but the same pidfd.) -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the fleet-wide view of this chapter is bcc-tools: `exitsnoop` timestamping > every process exit and `killsnoop` logging who signalled whom, which would > show `pmon`'s SIGTERM (and would *not* show a pid-reuse mishit, because @@ -466,26 +466,26 @@ process has a different number, but the same pidfd.) steps, and the dangerous reap in step 3 is often performed by a runtime or library you don't control. - `pidfd_open` (or `CLONE_PIDFD` at clone time) pins process identity to an - fd: exit becomes `poll` readability — Chapter 9's event-loop model now - covers processes — `waitid(P_PIDFD)` reaps exactly that process, and + fd: exit becomes `poll` readability (Chapter 9's event-loop model now + covers processes), `waitid(P_PIDFD)` reaps exactly that process, and `pidfd_send_signal` yields `ESRCH` rather than signalling a stranger; `/proc//fdinfo/`'s `Pid:` line names the pinned process. - The runtimes diverge today, verified by strace on this host: Go 1.26 probes and adopts `CLONE_PIDFD` for every child it spawns, while stable Rust 1.97 spawns with fork-style `clone` and keeps its pidfd API behind - the unstable `linux_pidfd` feature — so our Rust and C++ engines call the + the unstable `linux_pidfd` feature, so our Rust and C++ engines call the syscalls themselves. - Signal masks are inherited across exec: a supervisor that blocks signals for a signalfd must reset the child's mask (`POSIX_SPAWN_SETSIGMASK`, `pre_exec`) or its children become unstoppable. -Next, **identity and privilege**: who a process *is* — real, effective, and +Next, **identity and privilege**: who a process *is*, real, effective, and saved uids, capability sets, and how `pmon` drops root without dropping the ability to do its job. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host (kernel 7.1) this session: the runner printed 13-pidfd-modern-process-mgmt PASS PASS PASS (3 passed, 0 diff --git a/_docs/14-identity-and-privilege.md b/_docs/14-identity-and-privilege.md index a794f64..9acc627 100644 --- a/_docs/14-identity-and-privilege.md +++ b/_docs/14-identity-and-privilege.md @@ -2,16 +2,16 @@ title: "Identity and privilege" order: 14 part: "Processes, Signals, Privilege" -description: "Real, effective, and saved uids as one state machine that setresuid finally makes explicit; umask and the file bits; then capabilities done properly — file sets vs process sets, why ambient exists, and the KEEPCAPS ordering trap — proven live as pmon v3 binds :80 at uid 65534 carrying exactly one capability." +description: "Real, effective, and saved uids as one state machine that setresuid finally makes explicit; umask and the file bits; then capabilities done properly: file sets vs process sets, why ambient exists, and the KEEPCAPS ordering trap, proven live as pmon v3 binds :80 at uid 65534 carrying exactly one capability." duration: "50 minutes" --- `pmon` can spawn a child, shepherd it through signals, and supervise it race-free through a pidfd. Version 3 faces the question every real supervisor -meets next: it starts as root — it has to, to become another user — and +meets next: it starts as root (it has to, to become another user) and everything it launches must *not* be root. The one new idea is that identity -on Linux is not one number but a handful of small state machines — three uids -and five capability sets — and the transitions between them are +on Linux is not one number but a handful of small state machines (three uids +and five capability sets), and the transitions between them are order-sensitive. The same five syscalls in the wrong order either leave the child privileged (you proved nothing) or strip the one capability it needed (the child cannot do its job). `pmon drop` performs the transition correctly, @@ -20,17 +20,17 @@ carrying exactly `CAP_NET_BIND_SERVICE` across a full identity change and an possible if the drop worked: an unprivileged uid bound to port 80. The code is in `examples/14-identity-and-privilege/`. This is a **VM -example** — the drop path needs real root, so `./demo.sh run` deploys +example**: the drop path needs real root, so `./demo.sh run` deploys to the `systems-target` guest when `TARGET` is set; the `README.md` there specifies the CLI, the exit-code contract, and the paired positive/negative verification. {% include excalidraw.html file="14-credential-sets-caps" - alt="Two bands. The top band shows the three uids — real, effective, saved — each moving 0 to 65534, with a note that setresuid moves all three at once and the saved uid is the way back that the drop deliberately destroys. The bottom band shows pmon's kept-cap path as six boxes: pmon as root with full permitted and effective sets, prctl PR_SET_KEEPCAPS, setresuid to 65534 with permitted kept, capset pinning 0x400 into permitted inheritable and effective, prctl AMBIENT_RAISE putting 0x400 into ambient, and execve into bindprobe bound to port 80 at uid 65534. A dashed negative-control row underneath skips the prctl and capset steps: setresuid scrubs permitted to zero and execve reaches bindprobe with every set zero and bind denied, exit 3. Notes carry the real /proc status values: the kept-cap child reads 0000000000000400 in CapInh CapPrm CapEff CapAmb with CapBnd 000001ffffffffff." - caption="Figure 14.1 — the three-uid state machine and the five capability sets across pmon's drop; every hex value comes from /proc/<pid>/status on systems-target, read during the cross-check below" %} + alt="Two bands. The top band shows the three uids: real, effective, saved, each moving 0 to 65534, with a note that setresuid moves all three at once and the saved uid is the way back that the drop destroys for good. The bottom band shows pmon's kept-cap path as six boxes: pmon as root with full permitted and effective sets, prctl PR_SET_KEEPCAPS, setresuid to 65534 with permitted kept, capset pinning 0x400 into permitted inheritable and effective, prctl AMBIENT_RAISE putting 0x400 into ambient, and execve into bindprobe bound to port 80 at uid 65534. A dashed negative-control row underneath skips the prctl and capset steps: setresuid scrubs permitted to zero and execve reaches bindprobe with every set zero and bind denied, exit 3. Notes carry the real /proc status values: the kept-cap child reads 0000000000000400 in CapInh CapPrm CapEff CapAmb with CapBnd 000001ffffffffff." + caption="Figure 14.1: the three-uid state machine and the five capability sets across pmon's drop; every hex value comes from /proc/<pid>/status on systems-target, read during the cross-check below" %} -> **Tools used** — `ssh`/`scp` and the Python runner +> **Tools used**: `ssh`/`scp` and the Python runner > (`scripts/test-all-examples.py`) on the host; `umask`, `touch`, `ls` > (host); `getpcaps`, `capsh`, `getcap`, `sysctl`, `pgrep`, `bpftrace` > (systems-target VM). Everything here is checked by @@ -39,7 +39,7 @@ positive/negative verification. ## Three uids, one state machine -Every process carries three user ids, and `/proc` shows them side by side — +Every process carries three user ids, and `/proc` shows them side by side: this is the dropped child from the cross-check later in the chapter: ```bash @@ -52,28 +52,28 @@ Linux relic that shadows the effective uid; NFS servers use it, nothing in this book will). The *real* uid is who you are for accounting and for who may signal you. The *effective* uid is the one that matters: every permission check the kernel makes reads it. The *saved* uid is the escape -hatch — it exists so a set-uid program can drop its effective uid to do +hatch: it exists so a set-uid program can drop its effective uid to do unprivileged work and later climb back to the saved copy. That escape hatch is also the source of thirty years of confusion, because the classic calls mutate the machine implicitly. `setuid(2)` sets all three -ids if you are privileged but only the effective uid if you are not — the +ids if you are privileged but only the effective uid if you are not; the same call is either a permanent drop or a temporary one depending on who runs it. `seteuid(2)` moves only the middle value. `setreuid(2)` updates the saved uid as a *side effect*, under rules baroque enough that the manual page documents them as conditions. Daemons of the 1990s performed a "setuid dance" across these calls and regularly got it wrong in one -direction — the direction where root comes back. +direction: the direction where root comes back. `setresuid(2)` ends the dance by making the state machine explicit: you name all three values in one call (`-1` means leave that one alone), the kernel applies them atomically, and the call either fully succeeds or -changes nothing. `pmon` calls `setresuid(uid, uid, uid)` — all three to -65534 — which is deliberately irreversible: with no root left in *any* of +changes nothing. `pmon` calls `setresuid(uid, uid, uid)` (all three to +65534), which is irreversible by design: with no root left in *any* of the three slots, there is no privileged state left to return to, and a later `setresuid(0,0,0)` fails. One ordering rule already applies here: groups first. `setgroups(2)` and `setresgid(2)` are themselves privileged -operations, so they must run while the effective uid is still 0 — drop the +operations, so they must run while the effective uid is still 0; drop the uid first and you are locked out of fixing the gids, leaving the child in root's groups. @@ -97,45 +97,45 @@ Same `touch`, same requested 0666: umask 022 clears group/other write (0644), umask 077 clears everything but owner (0600). The umask is inherited across `fork` and `execve` like the uids are, which is why the verify script `chmod 755`s the binary it stages on the guest instead of -trusting whatever mode survived `scp` — permissions you did not state +trusting whatever mode survived `scp`; permissions you did not state explicitly are permissions someone else chose. And mode bits are exactly why this chapter stages that copy in `/tmp` at all: `/home/fedora` is mode -`700`, so once `pmon` becomes `nobody`, a binary under it is unreachable — +`700`, so once `pmon` becomes `nobody`, a binary under it is unreachable: the drop is real enough to lock the child out of its own image. ## Capabilities: five process sets, three file sets Root's power is not one bit; since Linux 2.2 it is ~41 distinct -**capabilities** — `CAP_NET_BIND_SERVICE` is the one that lets a process +**capabilities**: `CAP_NET_BIND_SERVICE` is the one that lets a process bind ports below `net.ipv4.ip_unprivileged_port_start` (1024 on the guest, confirmed below). Each *process* carries five sets, visible as the `Cap*` lines of `/proc//status`: -- **Permitted** — the ceiling: capabilities the process may use or raise. -- **Effective** — the subset the kernel actually consults in checks. -- **Inheritable** — offered across `execve`, but only into binaries whose +- **Permitted**, the ceiling: capabilities the process may use or raise. +- **Effective**, the subset the kernel actually consults in checks. +- **Inheritable**, offered across `execve`, but only into binaries whose *file* inheritable set accepts them. -- **Bounding** — a mask over what `execve` can ever grant. On the guest it +- **Bounding**, a mask over what `execve` can ever grant. On the guest it reads `000001ffffffffff` for every process, root or not: it masks, it never grants. -- **Ambient** — the modern one, added in Linux 4.3: capabilities that +- **Ambient**, the modern one, added in Linux 4.3: capabilities that survive an `execve` by a non-root process into a plain, uncapped binary. Each *file* can carry three pieces in its `security.capability` xattr -(permitted, inheritable, and a one-bit effective flag — set with +(permitted, inheritable, and a one-bit effective flag, set with `setcap(8)`, read with `getcap(8)`), and `execve` combines the two sides: the new permitted set is the file's permitted masked by bounding, plus the intersection of the process and file inheritable sets, plus ambient. Why does ambient exist? Because the inheritable set is a trap. Before 4.3, a supervisor that wanted to pass one capability to an unprivileged child -had to intersect its inheritable set with the *file's* inheritable set — +had to intersect its inheritable set with the *file's* inheritable set, and ordinary binaries have no file capabilities at all, so the intersection was empty and the capability died at `execve`. Your options were `setcap` on every binary you might run (a fleet-management problem and useless for scripts) or keeping root. Ambient is the bridge: a -capability raised into the ambient set — legal only while it is in both -permitted and inheritable — flows through a non-root `execve` of an +capability raised into the ambient set (legal only while it is in both +permitted and inheritable) flows through a non-root `execve` of an uncapped binary intact. The cross-check proves the child's capability came this way and not from the file: `getcap` on the staged binary prints nothing. @@ -152,25 +152,25 @@ The whole lesson of `pmon drop` is that five correct syscalls in the wrong order fail silently. The correct sequence, as the C++ header comment states it: -1. `prctl(PR_SET_KEEPCAPS, 1)` — **before** the uid change. When a +1. `prctl(PR_SET_KEEPCAPS, 1)`: **before** the uid change. When a process's uids go from 0 to all-nonzero, the kernel scrubs the permitted and effective sets; `KEEPCAPS` preserves permitted (effective is cleared regardless). -2. `setgroups` / `setresgid` / `setresuid` — become the target user, +2. `setgroups` / `setresgid` / `setresuid`: become the target user, groups and gid while still privileged. -3. `capset(2)` — pin `CAP_NET_BIND_SERVICE` into permitted **and** +3. `capset(2)`: pin `CAP_NET_BIND_SERVICE` into permitted **and** inheritable, re-raise it into effective (cleared by step 2), and drop every other capability KEEPCAPS retained. -4. `prctl(PR_CAP_AMBIENT_RAISE, CAP_NET_BIND_SERVICE)` — legal only +4. `prctl(PR_CAP_AMBIENT_RAISE, CAP_NET_BIND_SERVICE)`: legal only because step 3 put the cap in both permitted and inheritable. -5. `execvp(3)` — the ambient set is the only one that survives this for a +5. `execvp(3)`: the ambient set is the only one that survives this for a non-root process running an uncapped binary. Each step is a fence the others depend on. Skip step 1 and step 2 scrubs -permitted — steps 3 and 4 then have nothing to work with. Run step 3 +permitted; steps 3 and 4 then have nothing to work with. Run step 3 before step 2 and you have dropped `CAP_SETUID`/`CAP_SETGID`, so the uid change itself fails. Try step 4 without step 3 and `prctl` returns `EPERM` -because root's inheritable set is empty — the raise requires permitted ∩ +because root's inheritable set is empty; the raise requires permitted ∩ inheritable. Skip step 4 and everything is technically held right up to `execve`, where permitted and effective are recomputed from file caps the binary does not have, and the capability evaporates. The negative control @@ -180,7 +180,7 @@ steps 1, 3, and 4 never happen and the child arrives with every set zero. ## How the code works The error plumbing comes first because every step uses it. C++ and Rust -declare the same two-field failure record — `SysErr { what, errno }` — and +declare the same two-field failure record (`SysErr { what, errno }`) and a `checked(rc, what)` adapter that turns a negative syscall return into `std::expected` / `Result` carrying the captured errno. All five steps thread through it with `?`-style propagation, and exactly one printer at @@ -377,27 +377,27 @@ fn arrange_and_exec( } ``` -C++ and Rust do the dance **in-process** with raw syscalls — no libcap on -purpose, so every step is visible. The `capset` ABI deserves a close look +C++ and Rust do the dance **in-process** with raw syscalls, skipping libcap +entirely, so every step is visible. The `capset` ABI deserves a close look because glibc does not even wrap it (the "public" API is libcap's; we call `syscall(SYS_capset, …)` directly). The header names ABI version 3 (`0x20080522`) and pid 0, meaning "me"; version 3 represents the 64-bit capability space as **two 32-bit words**, which is why `data` is a two-element array. `CAP_NET_BIND_SERVICE` is capability number 10, so its -bit lives in word 0 and word 1 is deliberately all zero — writing zeros +bit lives in word 0 and word 1 stays all zero: writing zeros there is what drops the *other* nine retained-by-KEEPCAPS capabilities above 31. Setting `effective = bit` in the same call does double duty: the uid change cleared the effective set, and this is what re-raises it. Rust mirrors the C++ byte for byte, down to `#[repr(C)]` on the two structs so their layout matches the kernel's, and defines the two constants the `libc` crate omits (`LINUX_CAPABILITY_VERSION_3`, `CAP_NET_BIND_SERVICE`) -locally — both are stable kernel ABI. +locally; both are stable kernel ABI. Go is the deliberate inversion, and the reason is a rule worth memorizing: **credentials are per-thread at the kernel level**, and a Go program is multithreaded before `main` runs. Instead of attempting the dance -in-process, the Go build describes the drop declaratively — -`SysProcAttr.Credential` for step 2, `AmbientCaps` for steps 1, 3, and 4 — +in-process, the Go build describes the drop declaratively: +`SysProcAttr.Credential` for step 2, `AmbientCaps` for steps 1, 3, and 4, and the runtime's fork/exec child, where exactly one thread exists, replays the same kernel-correct order between `fork` and `exec`. Same observable behavior; the ordering lives in the standard library instead of @@ -406,17 +406,17 @@ propagate CMD's own exit status, which is how the negative control's exit 3 travels back through `pmon` to the verify script intact. `bindprobe` is short by design: socket, `SO_REUSEADDR` (so rapid repeat -runs do not trip over `TIME_WAIT`), bind on `INADDR_ANY`, then one line — -`bindprobe: uid= bound :` — and exit 0, or the bind failure with +runs do not trip over `TIME_WAIT`), bind on `INADDR_ANY`, then one line +(`bindprobe: uid= bound :`) and exit 0, or the bind failure with `strerror` text on stderr and exit 3. It prints `getuid()` precisely so one line carries both facts the verification needs: *who* the process was and *what* it was allowed to do. -Fragile bits, stated plainly: `--keep-cap` accepts only -`net_bind_service` — the cap number and its word-0 position are hardcoded, +A few fragile points: `--keep-cap` accepts only +`net_bind_service`; the cap number and its word-0 position are hardcoded, and generalizing to caps ≥ 32 means touching word 1; the staged binary must live somewhere the dropped user can read (`/tmp`, not `/home/fedora`); -and `pmon` does not call `setsid` or close inherited fds — real daemons +and `pmon` does not call `setsid` or close inherited fds; real daemons layer those on, and chapter 7's `O_CLOEXEC` discipline is half of why they can skip the fd audit. @@ -425,15 +425,15 @@ can skip the fd audit. The output contract is byte-identical diagnostics across languages, and capabilities add a twist: the interesting failure happens in the *child*, after `execve`, where `pmon`'s error plumbing no longer exists. So the -contract splits in two. Setup failures — not root, unknown user, a failed -step — flow through `SysErr`: C++ formats `std::expected` errors with +contract splits in two. Setup failures (not root, unknown user, a failed +step) flow through `SysErr`: C++ formats `std::expected` errors with `std::strerror`, Rust calls `libc::strerror` through a safe wrapper, and both exit 1 from a single printer. The child's bind failure is reported by `bindprobe` itself with exit 3, and Go's `drop` forwards that code via `errors.As(err, &ee)` → `ee.ExitCode()`. The subtle one is Go's `errnoText`: Go spells errno strings lower-case (`permission denied`) while `strerror(3)` capitalizes (`Permission denied`), so the Go build -re-capitalizes the first letter — without it, `verify.lua`'s +re-capitalizes the first letter; without it, `verify.lua`'s byte-for-byte match on `bindprobe: bind :80: Permission denied` would fail in one language out of three. Usage errors exit 2 everywhere, before any privileged call is attempted. @@ -446,14 +446,14 @@ notice; glibc papers over it for threaded ones with its setxid machinery, signaling every thread to run the same `setresuid` so the process moves in lockstep. But `pmon`'s C++ and Rust builds bypass glibc for `capset` (raw `syscall`), so their safety argument is structural instead: both are -**deliberately single-threaded** — no thread pool, no async runtime — +**kept single-threaded** (no thread pool, no async runtime) because a privilege drop must cover every task, and `execve` replaces the whole process anyway. Go cannot make that promise: the runtime spawns threads before `main`, and a raw in-process dance would leave sibling -threads still root — a real vulnerability class, not a theoretical one. +threads still root, a real vulnerability class, not a theoretical one. That is exactly why the Go build pushes the entire sequence into the fork/exec child. The general rule: change identity either before threads -exist or in a fresh child — never "around" a running thread pool. +exist or in a fresh child, never "around" a running thread pool. ## Build, run, observe @@ -489,7 +489,7 @@ bindprobe: uid=65534 bound :80 ``` That single line is the positive proof: uid 65534 (`nobody`) is not root, -yet the bind on :80 succeeded. And the negative control — the same drop +yet the bind on :80 succeeded. And the negative control: the same drop with `--keep-cap` removed: ```bash @@ -503,7 +503,7 @@ bindprobe: bind :80: Permission denied (An ssh pseudo-terminal warning line is trimmed from both transcripts.) The Go and Rust binaries produce these exact same two lines and exit -codes — that is what `verify.lua` asserts per language. The pair matters +codes; that is what `verify.lua` asserts per language. The pair matters more than either half: without the negative control, a lowered `ip_unprivileged_port_start` or a stray root uid would make the positive run pass while proving nothing about capabilities. @@ -512,7 +512,7 @@ run pass while proving nothing about capabilities. `bindprobe` exits as soon as it has bound, so to inspect the dropped child's credential state directly, run the same drop around a `sleep` and -read the child while it lives — three independent tools against the same +read the child while it lives: three independent tools against the same process: ```bash @@ -533,7 +533,7 @@ CapAmb: 0000000000000400 Read the layers against Figure 14.1: `getpcaps` names the one capability with `=eip` (effective, inheritable, permitted); the raw `/proc` view -shows the same bit — `0x400` is `1 << 10` — in all four transferable sets +shows the same bit (`0x400` is `1 << 10`) in all four transferable sets including ambient, which is the set that carried it through `execve`; and `capsh --decode` closes the loop by translating the hex back to the name. `CapBnd` is still the full mask because `pmon` never touched the bounding @@ -553,16 +553,16 @@ CapAmb: 0000000000000000 ``` Two more readings pin down where the capability did *not* come from. -`getcap /tmp/pmon-verify` prints nothing — no file capabilities, so the +`getcap /tmp/pmon-verify` prints nothing: no file capabilities, so the ambient path is the only route the bit had. And `sysctl net.ipv4.ip_unprivileged_port_start` reports `1024`, ruling out the sysctl shortcut that would have made :80 free for everyone. For contrast, a root shell on the same guest reads -`CapPrm`/`CapEff` `000001ffffffffff` with `CapInh` and `CapAmb` zero — -full power, nothing inheritable, nothing ambient: the default state the +`CapPrm`/`CapEff` `000001ffffffffff` with `CapInh` and `CapAmb` zero: +full power, nothing inheritable, nothing ambient, the default state the drop starts from. -> **On the lab VM** — you can also watch the kernel make the decision +> **On the lab VM**: you can also watch the kernel make the decision > itself. `bpftrace` on the `cap_capable` hook, filtered to the probe > process, during both runs: > @@ -573,37 +573,37 @@ drop starts from. > ``` > > Capability 10 is `CAP_NET_BIND_SERVICE`: the kept-cap run's check -> returns 0 (granted) and the negative control's returns -1 (`EPERM`) — +> returns 0 (granted) and the negative control's returns -1 (`EPERM`): > the same verdict the bind lines reported, observed at the kernel hook -> that issued it. (Both runs also emit a burst of `cap=21` — -> `CAP_SYS_ADMIN` — denials from unrelated kernel probing; only the +> that issued it. (Both runs also emit a burst of `cap=21` +> (`CAP_SYS_ADMIN`) denials from unrelated kernel probing; only the > `cap=10` verdicts differ. bcc-tools' `capable` would show the same > stream with symbolic names, but it failed to compile against kernel > 6.19 on this guest; Part 8 works through that toolchain properly.) One deliberate non-observation: you cannot follow the drop with `strace` from outside, because ptrace attachment is severed at a privilege -transition. The port bind *is* the observation surface here — and there is +transition. The port bind *is* the observation surface here, and there is no dashboard panel by design; this example emits no telemetry. ## What you learned - The three uids are a state machine, and `setresuid(u, u, u)` is the one transition that says what it does: all three slots move atomically, the - saved-uid escape hatch is destroyed, and the drop is irreversible — + saved-uid escape hatch is destroyed, and the drop is irreversible; groups and gid must move first, while you are still privileged. - Capabilities live in five process sets and three file sets; `execve` recombines them, and the ambient set exists so a non-root process can - carry a capability into an ordinary uncapped binary — the case the + carry a capability into an ordinary uncapped binary, the case the inheritable set was never able to serve. - Ordering is the whole game: `PR_SET_KEEPCAPS` before `setresuid`, `capset` into permitted *and* inheritable after it, `PR_CAP_AMBIENT_RAISE` - last — and in Go the same sequence must run in the fork/exec child, + last, and in Go the same sequence must run in the fork/exec child, because credentials are per-thread and the runtime is threaded before `main`. - A privilege claim needs a negative control: `uid=65534 bound :80` proves nothing until the same drop without the cap yields - `Permission denied` — paired evidence, checked by `getpcaps`, + `Permission denied`: paired evidence, checked by `getpcaps`, `/proc//status` + `capsh --decode`, and the kernel's own `cap_capable` verdict. @@ -613,7 +613,7 @@ fault costs. --- -

verified — every transcript +

verified: every transcript and number above was produced this session against the systems-target guest (Fedora 44, kernel 6.19.10-300.fc44): the runner printed 14-identity-and-privilege PASS PASS PASS diff --git a/_docs/15-virtual-memory.md b/_docs/15-virtual-memory.md index 15dc2c6..1bf6482 100644 --- a/_docs/15-virtual-memory.md +++ b/_docs/15-virtual-memory.md @@ -2,7 +2,7 @@ title: "Virtual memory" order: 15 part: "Memory" -description: "The address space read line by line from /proc/self/maps, brk versus mmap in each language's allocator under strace, minor and major page faults measured with getrusage and perf, RSS versus VSZ versus PSS from a live smaps_rollup, overcommit and OOM-killer policy, and a tour of madvise — with memmap, a self-inspecting memory tool in all three languages." +description: "The address space read line by line from /proc/self/maps, brk versus mmap in each language's allocator under strace, minor and major page faults measured with getrusage and perf, RSS versus VSZ versus PSS from a live smaps_rollup, overcommit and OOM-killer policy, and a tour of madvise, with memmap, a self-inspecting memory tool in all three languages." duration: 45 minutes --- @@ -10,11 +10,11 @@ Part 3 closed with `pmon` dropping privileges in chapter 14; this part turns inward, to the memory every one of those processes lives in. The one new idea is that **an address is a promise, not a page**: when `malloc`, `make`, or `Vec` hands you 64 MiB, the kernel has recorded a range in a bookkeeping -structure (a *virtual memory area*) and given you nothing else — no RAM moves +structure (a *virtual memory area*) and given you nothing else: no RAM moves until you touch a page and the resulting fault forces the kernel to make good -on the promise, one 4 KiB installment at a time. Everything in this chapter — -the `/proc/self/maps` layout, the brk/mmap split, minor versus major faults, -RSS versus VSZ versus PSS, overcommit — is a different view of that +on the promise, one 4 KiB installment at a time. Everything in this chapter +(the `/proc/self/maps` layout, the brk/mmap split, minor versus major faults, +RSS versus VSZ versus PSS, overcommit) is a different view of that promise-then-pay machinery, and this part's artifact, `shmkv`, will spend the next chapters building a shared key-value store directly on top of it. @@ -27,9 +27,9 @@ the `README.md` pins the CLI, the output contract, and the exit codes. {% include excalidraw.html file="15-address-space-map" alt="A vertical stack of address-space regions from high to low: vsyscall, the stack growing down, an unmapped gap, the mmap region containing vdso and vvar pages, ld-linux and libc segments, and an accented 64 MiB anonymous rw-p mapping, another gap, the brk-grown heap, data and relro segments, and the executable text at the bottom, each labeled with a real address from this chapter's runs." - caption="Figure 15.1 — one x86-64 process address space, high addresses at the top; every address label comes from the /proc maps output walked below" %} + caption="Figure 15.1: one x86-64 process address space, high addresses at the top; every address label comes from the /proc maps output walked below" %} -> **Tools used** — `strace`, `perf`, `dd`, `ps`, `pgrep`, and `cat`/`grep` +> **Tools used**: `strace`, `perf`, `dd`, `ps`, `pgrep`, and `cat`/`grep` > over `/proc` and `/sys` (host); `oomkill`/`bpftrace` (lab VM, exercised in > Part 8). Everything here is checked by `scripts/check-host.sh` or ships > with Fedora. @@ -37,7 +37,7 @@ the `README.md` pins the CLI, the output contract, and the exit codes. ## The address space, line by line `/proc/self/maps` lists every promise the kernel is currently holding for a -process — one line per VMA. The cleanest specimen is `cat` reading its own +process: one line per VMA. The cleanest specimen is `cat` reading its own maps, because whatever process opens that file sees *itself*: ```bash @@ -69,16 +69,16 @@ ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] (Whitespace compressed; the columns are start–end, permissions, file offset, device, inode, and path.) The fourth permission character is the one people -miss: `p` means *private* — writes trigger copy-on-write and never reach the -file — while `s` would mean *shared*, the mode chapter 16 is built on. Read +miss: `p` means *private*, writes trigger copy-on-write and never reach the +file; `s` would mean *shared*, the mode chapter 16 is built on. Read the file top to bottom and the classic layout falls out. The binary itself -appears four times — `r-xp` text, `r--p` read-only data, a second `r--p` that +appears four times: `r-xp` text, `r--p` read-only data, a second `r--p` that is the relro region (writable during relocation, sealed read-only before -`main`), and `rw-p` data/bss — all file-backed at increasing offsets, all +`main`), and `rw-p` data/bss, all file-backed at increasing offsets, all shareable between every process running `cat`. Above it sits `[heap]`, the one region the legacy `brk(2)` syscall grows. Then a jump to the high `7f7c…` range: the **mmap region**, where the kernel hands out addresses -top-down — the locale archive `cat` mapped read-only, `libc` in the same +top-down: the locale archive `cat` mapped read-only, `libc` in the same four-segment pattern as the binary, anonymous `rw-p` slabs with inode 0 (allocator arenas and thread machinery), and the kernel-provided `[vvar]` + `[vdso]` pages that let chapter 4's `clock_gettime` skip the syscall @@ -86,14 +86,14 @@ entirely. Near the top, `[stack]` grows downward toward the mmap region; the `[vsyscall]` page at the very top is a frozen compatibility relic. The gaps between regions are the point, not an accident: unmapped gigabytes that turn stray pointers into segfaults instead of silent corruption, with layout -randomized per-exec (ASLR) — rerun the command and every address shifts. +randomized per-exec (ASLR); rerun the command and every address shifts. ## brk or mmap? Ask strace Two syscalls create writable anonymous memory: `brk`, which moves the single high-water mark of `[heap]`, and `mmap`, which drops an independent VMA anywhere in the mmap region. Which one does a 64 MiB allocation use? The -brief answer for all three languages is *mmap* — but the trace is more +brief answer for all three languages is *mmap*, but the trace is more instructive than the answer: ```bash @@ -105,17 +105,17 @@ mmap(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0 (Loader mmaps trimmed.) glibc's malloc probes `brk(NULL)` for the current break, grows it by 204 KiB (`0x2504b000` → `0x2507e000`) for its startup -arena — that is small allocations living in `[heap]` — and then satisfies the +arena (small allocations living in `[heap]`) and then satisfies the 64 MiB `std::vector` with a single `mmap` of 67,112,960 bytes: our 67,108,864 plus one 4 KiB page for the chunk header. The dividing line is malloc's `M_MMAP_THRESHOLD`, 128 KiB by default (dynamically adjusted): below it, carve from the arena; above it, a dedicated mapping that `free` can return to the kernel instantly with `munmap` instead of hoping the arena top shrinks. -Rust tells the same story with the same syscalls — its default global +Rust tells the same story with the same syscalls: its default global allocator *is* glibc malloc, and its trace shows the identical `brk` pair (`0x55cf01269000` → `0x55cf0128a000`) followed by the same 67,112,960-byte `mmap`. Go is the outlier: zero `brk` calls, ever. Its runtime reserves heap -arenas with `PROT_NONE` mappings and commits pages as needed — the +arenas with `PROT_NONE` mappings and commits pages as needed; the cross-check below shows that dance. This is why `memmap --mode heap` prints a target line with *no* `[heap]` tag: the region backing a big allocation is an anonymous mapping in the mmap region, in every language. The `[heap]` name in @@ -126,33 +126,33 @@ maps is a fossil; the working heap is wherever the allocator put it. {% include excalidraw.html file="15-fault-flow" alt="A left-to-right flow: a user write to buf, then an MMU miss with a CPU trap, then the kernel page-fault handler looking up the VMA, fanning out to four outcomes: no VMA or bad permissions raising SIGSEGV, an anonymous first write getting a new zero-filled frame as a minor fault, a file page found in the page cache mapped with fault-around as a minor fault, and an uncached file page requiring a disk read as a major fault. Notes list the measured counts from this chapter's runs." - caption="Figure 15.2 — one touched page's journey through the fault handler; the counts in the notes are this chapter's getrusage measurements" %} + caption="Figure 15.2: one touched page's journey through the fault handler; the counts in the notes are this chapter's getrusage measurements" %} The touch is where the promise gets paid. The MMU finds no page-table entry, the CPU traps, and the kernel's fault handler asks: is this address inside a VMA, with compatible permissions? No → `SIGSEGV`. Yes → resolve it. If the -page can be produced without I/O — a zero-filled frame for a first anonymous -write, or a file page already sitting in the page cache — that is a **minor +page can be produced without I/O (a zero-filled frame for a first anonymous +write, or a file page already sitting in the page cache), that is a **minor fault**: microseconds, bookkeeping only. If the kernel must read the disk, the process sleeps through a **major fault**: the same event, three to six orders of magnitude slower. `memmap` measures both with `getrusage(2)` deltas -around the touch loop. Writing 64 MiB of heap costs `minor=16385 major=0` — +around the touch loop. Writing 64 MiB of heap costs `minor=16385 major=0`: one fault per touched 4 KiB page, plus one for the chunk-header page. The read-side is cheaper than you would guess: `fault-walk` writes a 16 MiB temp -file, maps it read-only, and touches its 4,096 pages in eight steps of 512 — +file, maps it read-only, and touches its 4,096 pages in eight steps of 512, and every step reports `minor=32`, not 512. That is **fault-around**: for file pages already in the page cache, one fault maps a batch of 16 surrounding pages, so sequential reads of cached files fault 16× less often than anonymous writes (which must give each page its own private zeroed frame). And `major=0` throughout, because the walk file was written moments -earlier and still sits in the page cache — a major fault would mean the +earlier and still sits in the page cache; a major fault would mean the machine actually went to disk. ## Three sizes and a policy: VSZ, RSS, PSS, and overcommit Promise-then-pay means every process has at least three sizes. **VSZ** is promises: the sum of all VMAs. **RSS** is payments: pages currently resident. -The gap is routinely enormous — +The gap is routinely enormous, for example: ```bash [host]$ ps -o pid,vsz,rss,comm $$ @@ -160,14 +160,14 @@ The gap is routinely enormous — 2620509 233356 4808 zsh ``` -— a 227 MiB address space with 4.7 MiB resident. But RSS double-counts: +That's a 227 MiB address space with 4.7 MiB resident. But RSS double-counts: every process using `libc` counts its shared pages at full price. **PSS** (proportional set size, from `/proc//smaps_rollup`) divides each shared page by the number of processes sharing it, so PSS sums to something meaningful across a fleet; the cross-check below watches `memmap`'s PSS jump from 403 kB to 65,948 kB as it dirties private pages no one else can share. -Unbacked promises need a policy, and it lives in `/proc/sys/vm` — read these, +Unbacked promises need a policy, and it lives in `/proc/sys/vm`: read these, do **not** write them: ```bash @@ -183,26 +183,26 @@ Mode `0` is the default heuristic: refuse only absurd single allocations, otherwise assume most promises are never fully touched. The proof is in the second read: this 64 GiB host has *already* promised 48.5 GiB against a nominal `CommitLimit` of 41.2 GiB (swap + 50% of RAM, per -`overcommit_ratio`), and nothing is wrong — in mode 0 the limit is -advisory. Mode `2` enforces it (`malloc` starts returning `NULL` for real); +`overcommit_ratio`), and nothing is wrong: in mode 0 the limit is +advisory. Mode `2` enforces it (`malloc` actually starts returning `NULL`); mode `1` promises anything. The price of overcommit is paid at touch time: if a fault needs a frame and there is nothing left to reclaim, the **OOM -killer** picks the process with the highest badness score — roughly its +killer** picks the process with the highest badness score: roughly its share of RAM, adjusted by `/proc//oom_score_adj` (−1000 exempts, +1000 -volunteers). This shell runs at `100` — `cat /proc/self/oom_score_adj` — a +volunteers). This shell runs at `100` (`cat /proc/self/oom_score_adj`), a nudge from systemd making user sessions die before system services. The lesson for a systems programmer: on mode-0 Linux, allocation failure mostly does not arrive as a clean error at the call site; it arrives as `SIGKILL` at the worst touch. Chapter 22's cgroup work sharpens this with per-service memory limits. -The last piece of the VMA toolkit is `madvise(2)` — hints on a range you +The last piece of the VMA toolkit is `madvise(2)`: hints on a range you already own. `MADV_WILLNEED` starts readahead; `MADV_SEQUENTIAL` and `MADV_RANDOM` tune it; `MADV_DONTNEED` drops pages *now* (anonymous memory -reads back as zeros — it is destructive, not advisory, despite the name); +reads back as zeros; it is destructive, not advisory, despite the name); `MADV_FREE` marks pages reclaimable-when-convenient, cheaper because the kernel may never bother; `MADV_HUGEPAGE`/`MADV_NOHUGEPAGE` opt a range in or -out of transparent huge pages — which matters here because this host's THP +out of transparent huge pages, which matters here because this host's THP mode is `madvise` (`cat /sys/kernel/mm/transparent_hugepage/enabled` prints `always [madvise] never`), meaning 2 MiB pages only where explicitly requested. This is not exotic: the Go runtime in the cross-check below calls @@ -212,20 +212,20 @@ to an allocator actually returns to the kernel. ## How the code works -Three small structures carry `memmap`. A `Baseline` — `{rss_kb, minor, -major}` — is captured *before* each allocation because `/proc/self/status`'s +Three small structures carry `memmap`. A `Baseline` (`{rss_kb, minor, +major}`) is captured *before* each allocation because `/proc/self/status`'s `VmRSS` and `getrusage`'s `ru_minflt`/`ru_majflt` are cumulative since process start; only deltas isolate the window we care about. A `Mapping` -owns an mmap'd range exactly the way chapter 7's `Fd` owns a descriptor — +owns an mmap'd range exactly the way chapter 7's `Fd` owns a descriptor: move-only in C++, `Drop` calls `munmap` in Rust, `defer unix.Munmap(m)` in -Go — so no error path leaks a VMA. A `TempFile` unlinks its path on +Go, so no error path leaks a VMA. A `TempFile` unlinks its path on destruction, which is why `verify.lua` can assert the walk file is gone even when a run fails halfway. -The probes are deliberately boring text-parsing: `vmrss_kb()` scans +The probes stick to boring text-parsing: `vmrss_kb()` scans `/proc/self/status` for the `VmRSS:` line; `fault_counts()` wraps `getrusage(RUSAGE_SELF)`; and the maps excerpt re-reads `/proc/self/maps`, -hex-parses each line's range, and prints those overlapping the target — the +hex-parses each line's range, and prints those overlapping the target; the entire selection logic is one interval test, C++ spelling: ```cpp @@ -240,7 +240,7 @@ actively defend that loop against its optimizer, because a loop whose stores are never read is dead code: C++ uses an empty `asm volatile` barrier taking the pointer, Rust `std::hint::black_box`, Go a package-level `sink` variable. Delete the barrier and `--mode heap` still *allocates*, but release-build -touch loops vanish and the fault counts collapse — the tool would silently +touch loops vanish and the fault counts collapse: the tool would silently measure nothing. The mode worth reading three times is `stack`, because each language's @@ -312,23 +312,23 @@ fn run_stack(bytes: usize, page: usize) -> Result<()> { The shared discipline: the worker is `noinline` so the 4 MiB frame really exists as a frame, and the baseline is taken by the *caller*, because the compiler (stack probing in C++/Rust, stack growth in Go) touches frame pages -at function entry — measure from inside and you miss the very faults you came +at function entry; measure from inside and you miss the very faults you came for. The divergence: C++ and Rust run the worker on the OS thread, so their excerpt shows the `[stack]` VMA growing; Go runs it on a fresh goroutine whose stack is runtime-managed heap spans, so its excerpt shows an anonymous -region — same contract, different mechanism, and the accurate place to see +region: same contract, different mechanism, and the accurate place to see that difference is the output itself, which is why the maps line is part of the contract. The wiring is a strict flag parser (exit 2 with a usage line on anything malformed), a per-mode header computing `bytes`/`pages` (with `stack` clamped to 4 MiB to stay inside the 8 MiB rlimit, and `mmap-file` sized by `fstat` instead of `--mb`), and the -common `report()` tail. Fragile bits, stated plainly: the page size comes +common `report()` tail. A few fragile points: the page size comes from `sysconf` at runtime (4096 here; 16 KiB Apple-style kernels would change every count); `fault-walk`'s step quota is `pages/8` with the remainder folded into step 8, so `--mb` values whose page count is not divisible by 8 print an uneven last step; and the fault window unavoidably includes the -probe's own work — one page of malloc header here, a few hundred runtime -pages in Go — which is why the verifier asserts *floors*, not exact counts. +probe's own work (one page of malloc header here, a few hundred runtime +pages in Go), which is why the verifier asserts *floors*, not exact counts. ## Errors, three ways @@ -345,11 +345,11 @@ memmap: error: /no/such/file: No such file or directory (os error 2) (Rust, exi ``` Three renderings of the same `ENOENT`, because each rides its language's -native machinery — `std::error_code::message()` inside `std::expected` +native machinery: `std::error_code::message()` inside `std::expected` plumbing, a `%w`-wrapped `*os.PathError`, `anyhow`'s `{:#}` chain over `io::Error`. Unlike chapter 5's `fwatch`, this contract pins the prefix and the exit code but leaves message text native, and `verify.lua` asserts -exactly that split — while errors the *program* generates, like mapping an +exactly that split, while errors the *program* generates, like mapping an empty file (`mmap` of length 0 would fail anyway, so `open_for_map` checks first), produce the identical `memmap: error: empty.bin: file is empty` in all three, exit 1, byte for byte. @@ -357,21 +357,21 @@ all three, exit 1, byte for byte. ## Concurrency lens All three `memmap`s are logically single-threaded, but the measurements say -otherwise, and the noise is the lesson. Rust reports `minor=16384` — exact, +otherwise, and the noise is the lesson. Rust reports `minor=16384`: exact, one fault per page, because `vec![0; n]` allocates through `calloc`, which knows fresh mmap pages are already zero and skips touching them; the touch loop pays for every page itself. C++ reports `16385`: `std::vector` zero-fills eagerly during construction, so the faults land in the `memset` (plus the header page), and the later touch loop faults nothing. Go reports -`16598`: the same 16,384 plus a few hundred faults from runtime machinery — -GC metadata, spans, a second OS thread growing something — because +`16598`: the same 16,384 plus a few hundred faults from runtime machinery +(GC metadata, spans, a second OS thread growing something) because `getrusage(RUSAGE_SELF)` sums *every thread in the process*, and a Go program is never really alone. The same reasoning explains why `perf stat` whole-process counts (below) always exceed the in-window deltas, and why `verify.lua` asserts `minor > 0` and RSS floors instead of equalities. One more thread-shaped trap from the stack tabs: Go's worker goroutine could be *rescheduled onto another OS thread* mid-measurement, which is fine for -fault counting precisely because `RUSAGE_SELF` is process-wide — the +fault counting precisely because `RUSAGE_SELF` is process-wide; the per-thread `RUSAGE_THREAD` would be the wrong tool here. ## Build, run, observe @@ -386,7 +386,7 @@ memmap: faults minor=16385 major=0 ``` Read it against the concepts: the target is an anonymous `rw-p` region (not -`[heap]`), spanning `0x4001000` bytes — 64 MiB plus the header page — and +`[heap]`), spanning `0x4001000` bytes (64 MiB plus the header page), and the RSS delta is 65,540 kB, the *same* number, because every promised page was touched. Go printed 2508 → 68728 (`minor=16598`) and Rust 2128 → 67668 (`minor=16384`) on the same host. Then contrast the modes: `--mode stack` @@ -405,7 +405,7 @@ memmap: vmrss_before=4052KB vmrss_after=6100KB memmap: faults minor=32 major=0 ``` -512 pages read, 32 faults — fault-around's 16× batching, live. `--mode +512 pages read, 32 faults: fault-around's 16× batching, live. `--mode fault-walk --mb 16` shows the same ratio held across all eight steps (`step=1/8 pages=512 minor=32 major=0` … eight times), with a final RSS delta of 16,384 kB: 4,096 resident pages bought with 256 faults. The full @@ -417,7 +417,7 @@ matrix is one command: `python3 scripts/test-all-examples.py --only **The allocators' syscalls, across all three.** The C++ trace appeared above; the other two confirm the split. Rust, under `strace -f -e -trace=mmap,brk,madvise`, shows glibc's fingerprint — `brk(NULL)`, a 132 KiB +trace=mmap,brk,madvise`, shows glibc's fingerprint: `brk(NULL)`, a 132 KiB break growth, then `mmap(NULL, 67112960, …)`. Go shows a different species entirely: @@ -429,7 +429,7 @@ madvise(0x7fee9f000000, 33554432, MADV_NOHUGEPAGE) = 0 ``` (Trimmed to the shape; pid columns dropped.) Reserve arenas with -`PROT_NONE` — promises so weak even a *read* faults fatally — then commit +`PROT_NONE` (promises so weak even a *read* faults fatally), then commit the needed span read-write with `MAP_FIXED`, and `madvise` runtime metadata out of transparent huge pages. Zero `brk` calls in the entire trace. @@ -452,11 +452,11 @@ Pss: 65948 kB Private_Dirty: 65768 kB ``` -Before: 3.7 MB resident but only 403 kB PSS — nearly everything is `libc` +Before: 3.7 MB resident but only 403 kB PSS: nearly everything is `libc` and loader pages shared with the whole system, billed proportionally. After: the deltas are private dirty pages nobody can share, so RSS and PSS converge. -`VmSize` in `/proc/$APP/status` moved 6860 → 72400 kB across the same pause -— a delta of 65,540 kB, agreeing to the kilobyte with the 67,112,960-byte +`VmSize` in `/proc/$APP/status` moved 6860 → 72400 kB across the same pause, +a delta of 65,540 kB, agreeing to the kilobyte with the 67,112,960-byte `mmap` strace showed and with the tool's own printed RSS growth. **`perf stat`, against the tool's own numbers.** An independent counter over @@ -471,27 +471,27 @@ memmap: faults minor=16385 major=0 Go: 16,879 against its in-window 16,606; Rust: 16,458 against 16,384. In each pair perf sits a stable one-to-three hundred faults above the -`getrusage` delta — precisely the loader, runtime startup, and probe pages +`getrusage` delta, precisely the loader, runtime startup, and probe pages that fall outside the measured window. Three observers (the kernel's rusage accounting, `/proc` VMA accounting, the perf software event), one consistent story. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the fleet-wide view of this chapter is bcc-tools territory: `oomkill` > logging every OOM-killer decision with the victim's badness score, and > `bpftrace` histograms over the `page_fault_user` tracepoint replacing > per-process `getrusage`. Those need root on a kernel you can afford to -> pressure — chapter 30, on the `systems-target` VM, does exactly that. +> pressure; chapter 30, on the `systems-target` VM, does exactly that. ## What you learned - An address is a promise: allocation creates VMAs (visible line by line in - `/proc/self/maps`, gaps and all), and pages arrive only at first touch — + `/proc/self/maps`, gaps and all), and pages arrive only at first touch, which is why VSZ, RSS, and PSS are three different truths, and why `smaps_rollup`'s PSS is the number worth summing across processes. - All three languages satisfy a 64 MiB allocation with `mmap`, not `brk`: glibc (C++ and Rust alike) via `M_MMAP_THRESHOLD`, Go via `PROT_NONE` - arena reservations committed with `MAP_FIXED` — the `[heap]` VMA is where + arena reservations committed with `MAP_FIXED`; the `[heap]` VMA is where small glibc allocations live, nothing more. - Minor faults are bookkeeping (one per anonymous page written: 16,384–16,385 measured; 16 pages per fault for cached file reads, thanks to @@ -503,11 +503,11 @@ story. telling the kernel what a range is really for. Next, **mmap and shared mappings**: `MAP_SHARED`, `msync`, and two processes -seeing one page — the machinery `shmkv` v0 is built from. +seeing one page, the machinery `shmkv` v0 is built from. --- -

verified — every number and +

verified: every number and excerpt above was produced on the Fedora 44 reference host (kernel 7.1, THP=madvise) this session: the runner printed 15-virtual-memory PASS PASS PASS (3 passed, 0 failed, 0 diff --git a/_docs/16-mmap-and-shared-mappings.md b/_docs/16-mmap-and-shared-mappings.md index 7bd10ab..22745e9 100644 --- a/_docs/16-mmap-and-shared-mappings.md +++ b/_docs/16-mmap-and-shared-mappings.md @@ -2,16 +2,16 @@ title: "mmap and shared mappings" order: 16 part: "Memory" -description: "File-backed vs anonymous and MAP_SHARED vs MAP_PRIVATE copy-on-write, msync guarantees against page-cache writeback, the ftruncate+mmap lifecycle, and memfd sealing — proven by shmkv v0, one byte-exact store format that the C++, Go, and Rust binaries create, write, and read interchangeably." +description: "File-backed vs anonymous and MAP_SHARED vs MAP_PRIVATE copy-on-write, msync guarantees against page-cache writeback, the ftruncate+mmap lifecycle, and memfd sealing: proven by shmkv v0, one byte-exact store format that the C++, Go, and Rust binaries create, write, and read interchangeably." duration: 50 minutes --- -The previous chapter took the address space apart — VMAs, page tables, faults — +The previous chapter took the address space apart (VMAs, page tables, faults) without ever asking where a mapping's bytes come from. This chapter answers that with the book's next artifact: `shmkv`, a key/value store whose entire database is one `mmap`'d file. The one new idea is that `MAP_SHARED` collapses -three things you have treated as separate — your memory, the page cache, and -the file on disk — into a single set of physical pages. A store instruction in +three things you have treated as separate (your memory, the page cache, and +the file on disk) into a single set of physical pages. A store instruction in one process is a load result in another and a dirty page the kernel will eventually write out, with no `read`, no `write`, and no copy in between. Version 0 leans on that hard: three binaries in three languages agree on every @@ -25,9 +25,9 @@ format, the CLI, and the exit codes all three languages share. {% include excalidraw.html file="16-mapping-page-cache" alt="Two user-space process bands above a kernel band and a disk band. Process A holds an mmap MAP_SHARED node marked rw-s and a dashed MAP_PRIVATE node; process B holds its own MAP_SHARED node. Amber and grey arrows from both mappings converge on one page-cache node for store.kv in the kernel band; a dashed arrow back to the private node is labeled first write copies the page; an amber msync MS_SYNC arrow and a dashed writeback-eventually arrow lead down to the store.kv-on-disk node." - caption="Figure 16.1 — one file, one set of page-cache pages, many mappings: A's store and B's load meet in the same physical memory, msync forces those pages to disk, and a MAP_PRIVATE view forks its own copy on first write" %} + caption="Figure 16.1: one file, one set of page-cache pages, many mappings: A's store and B's load meet in the same physical memory, msync forces those pages to disk, and a MAP_PRIVATE view forks its own copy on first write" %} -> **Tools used** — `strace`, `od`, `cmp`, `stat`, `python3` (host); +> **Tools used**: `strace`, `od`, `cmp`, `stat`, `python3` (host); > `bpftrace` (lab VM, exercised in Part 8). Everything here is checked by > `scripts/check-host.sh`, ships with Fedora, or is preinstalled in the lab > VMs. @@ -37,13 +37,13 @@ format, the CLI, and the exit codes all three languages share. Every `mmap` call picks one cell of a 2×2. The first axis is **what backs the pages**: a *file-backed* mapping names an fd and an offset, so its pages are literally page-cache pages of that file; an *anonymous* mapping -(`MAP_ANONYMOUS`) is backed by nothing but zero-fill — this is what your +(`MAP_ANONYMOUS`) is backed by nothing but zero-fill: this is what your allocator asks for when it wants a fresh arena, which is chapter 17's whole subject. The second axis is **who sees your writes**. `MAP_SHARED` means writes go to the backing object itself: for a file, straight into the shared page cache, visible immediately to every other process mapping the same range. `MAP_PRIVATE` means copy-on-write: your mapping starts out referencing the -very same page-cache pages — reads are free and shared — but the page-table +very same page-cache pages (reads are free and shared), but the page-table entries are write-protected, and the first store to a page faults, at which point the kernel copies that page and points only *your* mapping at the copy. Your writes never reach the file, and after the copy, other processes' writes @@ -71,23 +71,23 @@ private view : b'XXXXX' file on disk : b'SHKV1' ``` -The write landed in a COW copy; the file — and every `MAP_SHARED` mapper — +The write landed in a COW copy; the file, and every `MAP_SHARED` mapper, still sees `SHKV1`. `shmkv` therefore uses `MAP_SHARED` everywhere: the page cache *is* the database. -## What `msync` promises — and what writeback already does +## What `msync` promises, and what writeback already does Two guarantees get conflated here, and they have different owners. **Visibility** between processes needs no syscall at all: two `MAP_SHARED` mappings of the same file range resolve to the same physical pages, so a store in one is a load in the other at ordinary memory speed. **Durability** is a separate question: your store dirtied a page-cache page, and the kernel's -writeback machinery will get it to disk *eventually* — on Fedora's defaults, +writeback machinery will get it to disk *eventually*: on Fedora's defaults, dirty pages start aging out after 30 seconds. If the machine loses power before that, the store never happened. `msync(addr, len, MS_SYNC)` closes the gap: it blocks until the dirty pages in that range are written to the backing file, the mapped-memory analogue of `fsync`'s promise from the durability -chapter (`MS_ASYNC`, by contrast, is close to a no-op on modern Linux — it +chapter (`MS_ASYNC`, by contrast, is close to a no-op on modern Linux; it just nudges writeback). `shmkv` calls `msync(MS_SYNC)` at the end of every `create` and `set`, so "`set alpha`" on stdout means the bytes are on disk, not merely in the cache. The trade-off is the same one `fsync` bought: each @@ -99,10 +99,10 @@ A file-backed mapping does not size itself. `create` performs a fixed ritual: `open(O_RDWR|O_CREAT|O_TRUNC)` so a reused path starts at zero bytes, then `ftruncate(fd, 10 + slots*256)` to extend the file, then `mmap` for exactly that length. Order matters twice. First, the mapping's length is fixed at map -time — mapping a still-empty file and truncating afterwards would leave pages +time: mapping a still-empty file and truncating afterwards would leave pages you can touch but a file too short to back them, and touching a page past EOF delivers `SIGBUS`, not an error return. Second, `ftruncate` extension is -guaranteed zero-fill, and the format defines `key[0] == 0` as "empty slot" — +guaranteed zero-fill, and the format defines `key[0] == 0` as "empty slot", so a fresh store needs no initialization loop at all; every slot is born empty because the kernel says extended bytes are zero. The header write and one `msync` finish the job. `open` for an existing store is the mirror image: @@ -113,8 +113,8 @@ single slot. ### Aside: `memfd_create` and sealing, live Sometimes you want the mapping semantics without any filesystem name. -`memfd_create(2)` returns an fd to an anonymous tmpfs file — mappable, -`ftruncate`-able, passable over a Unix socket — and, uniquely, **sealable**: +`memfd_create(2)` returns an fd to an anonymous tmpfs file (mappable, +`ftruncate`-able, passable over a Unix socket) and, uniquely, **sealable**: ```python fd = os.memfd_create("shmkv-demo", os.MFD_CLOEXEC | os.MFD_ALLOW_SEALING) @@ -133,12 +133,12 @@ seals: 22 new writable map: [Errno 1] Operation not permitted ``` -Run this session: the fd's `/proc` link shows `/memfd:shmkv-demo (deleted)` — +Run this session: the fd's `/proc` link shows `/memfd:shmkv-demo (deleted)`, a file with no name anywhere; `F_GET_SEALS` reads back 22 (`F_SEAL_SHRINK`=2 + `F_SEAL_GROW`=4 + `F_SEAL_FUTURE_WRITE`=16); the existing mapping keeps writing, but a *new* writable `MAP_SHARED` attempt fails with `EPERM`. Sealing is what makes shared memory safe to accept from an untrusted -peer — the `SIGBUS`-via-truncation attack below becomes impossible once +peer: the `SIGBUS`-via-truncation attack below becomes impossible once `F_SEAL_SHRINK` is set. The IPC part passes exactly such fds between processes. @@ -146,19 +146,19 @@ processes. Interop works because the format is defined in bytes, not in types: magic `"SHKV1\0"` at offset 0, a **u32 little-endian** slot count at offset 6, then -256-byte slots — key NUL-padded in `[0..64)`, value in `[64..256)`. Note that -offset 6 is deliberately *misaligned* for a 4-byte integer, and none of the +256-byte slots: key NUL-padded in `[0..64)`, value in `[64..256)`. Note that +offset 6 is *misaligned* for a 4-byte integer, and none of the three implementations care, because none of them ever casts a pointer into the mapping: C++ assembles the u32 from four bytes with shifts, Go uses -`binary.LittleEndian.PutUint32`, Rust `u32::to_le_bytes` — byte moves have no +`binary.LittleEndian.PutUint32`, Rust `u32::to_le_bytes`: byte moves have no alignment requirements and no host-endianness dependence. -The tempting alternative — declare a struct and cast the mapping to it — +The tempting alternative (declare a struct and cast the mapping to it) is where every language would break differently. A C++ `struct { char magic[6]; uint32_t slots; }` inserts two padding bytes to align `slots`, silently making the header 12 bytes (and the unaligned cast would be -undefined behavior besides); `repr(Rust)` makes no layout promise at all — -Rust is free to reorder fields — so only `#[repr(C, packed)]` would be +undefined behavior besides); `repr(Rust)` makes no layout promise at all: +Rust is free to reorder fields, so only `#[repr(C, packed)]` would be defensible; Go specifies no struct padding contract and has no packed annotation, which is precisely why `encoding/binary` exists. And all three would still inherit the CPU's byte order. Stating "u32 LE at offset 6" in the @@ -167,13 +167,13 @@ stores identical. {% include excalidraw.html file="16-shmkv-layout" - alt="A byte-layout band shows store.kv as 1034 bytes: a dark magic node with bytes 53 48 4b 56 31 00, an amber u32 little-endian slot_count node with bytes 04 00 00 00, and four 256-byte slot boxes. A zoom band below splits one slot into a key field covering bytes 0 to 64 and a value field covering 64 to 256, both NUL-padded. Three language nodes — C++ writes it, Go reads it, Rust reads it — point up into the layout." - caption="Figure 16.2 — the shmkv v0 format, byte for byte; the header bytes shown are the real od output from this chapter's run, and cmp proves a Rust-built and a C++-built store file identical" %} + alt="A byte-layout band shows store.kv as 1034 bytes: a dark magic node with bytes 53 48 4b 56 31 00, an amber u32 little-endian slot_count node with bytes 04 00 00 00, and four 256-byte slot boxes. A zoom band below splits one slot into a key field covering bytes 0 to 64 and a value field covering 64 to 256, both NUL-padded. Three language nodes (C++ writes it, Go reads it, Rust reads it) point up into the layout." + caption="Figure 16.2: the shmkv v0 format, byte for byte; the header bytes shown are the real od output from this chapter's run, and cmp proves a Rust-built and a C++-built store file identical" %} ## How the code works -Each implementation is one `Store` abstraction over two resources — the fd and -the mapping — plus four commands. The mapping is where the languages differ +Each implementation is one `Store` abstraction over two resources (the fd and +the mapping) plus four commands. The mapping is where the languages differ most, so that is the excerpt: {% include codetabs.html langs="C++|Go|Rust" %} @@ -282,15 +282,15 @@ fn map_shared(file: &std::fs::File) -> Result { ``` Read the three as one argument about **who admits the danger**. A shared -mapping is memory whose contents can be changed — or whose backing can be -yanked — by something the compiler cannot see. Rust's `memmap2` makes you sign +mapping is memory whose contents can be changed, or whose backing can be +yanked, by something the compiler cannot see. Rust's `memmap2` makes you sign for that: `MmapMut::map_mut` is `unsafe`, and the boundary comment is the -signature — the promises (in-bounds slices, nobody shrinks the file, writers +signature: the promises (in-bounds slices, nobody shrinks the file, writers are sequenced) are exactly the conditions under which treating the mapping as `&mut [u8]` is sound, and everything outside that one function is safe Rust. C++ silently allows the same thing: `mmap` hands back `void*`, and nothing distinguishes a pointer into a shared, externally mutable, `SIGBUS`-capable -region from a pointer into a private heap block — the `Mapping` class adds +region from a pointer into a private heap block; the `Mapping` class adds lifetime discipline (move-only, `munmap` exactly once, in `Store` declared *after* `Fd` so it unmaps before the fd closes) but the type system never learns what the pointer means. Go hides it behind familiarity: @@ -300,21 +300,21 @@ moment `Munmap` runs, and the one-line discipline comment in `close` is all that stands between you and a use-after-unmap fault the runtime will report as a plain `SIGSEGV`/`SIGBUS`. -Above that boundary the three programs are deliberately the same. `create` +Above that boundary the three programs behave the same. `create` runs the truncate-extend-map-stamp ritual and reports `created FILE: N slots, SIZE bytes`. `open` validates before trusting: -size ≥ 10, magic bytes equal, and `size == 10 + slots*256` exactly — the last +size ≥ 10, magic bytes equal, and `size == 10 + slots*256` exactly: the last check is what turns "some file that starts with SHKV1" into "a store whose every slot index is in bounds", so the slot arithmetic (`offset = 10 + i*256`) can never leave the mapping. `set` is a linear probe that prefers the slot already holding the key (overwrite burns no new slot), else the first empty one, else fails with exit 5; it clears the whole 256-byte slot before copying key and value so a shorter value never leaves a longer -predecessor's tail bytes behind — without that `memset`, the two "identical" +predecessor's tail bytes behind; without that `memset`, the two "identical" stores in the cross-check would differ in the padding and `cmp` would say so. `get` and `dump` scan slots through a `field()` helper that stops at the first -NUL, and `dump` sorts pairs bytewise by key so slot order — an artifact of -insertion history — never leaks into output. One fragile bit is worth naming: +NUL, and `dump` sorts pairs bytewise by key so slot order (an artifact of +insertion history) never leaks into output. One fragile detail applies here: C++ flushes stdout before printing `dump`'s stderr summary, because stdout to a pipe is fully buffered and the combined-stream interleaving must match Go and Rust byte for byte. @@ -328,34 +328,34 @@ store. C++ carries `CliError{code, msg}` through `std::expected`, and `main` is the single printer. Rust's `Fail` is the same shape with one twist: an empty message *means* "print the usage line", so `--slots 0`, `--slots 12x`, and `--slots 99999999999` all funnel through `parse_slots` -(digits only, range [1, u32 max] — identical rules in all three languages) +(digits only, range [1, u32 max], identical rules in all three languages) into the same exit-2 usage path. Go is the most instructive: `cliError` pins `code` and `msg` but also wraps an underlying cause, so `errors.As` routes it -in `main` while the `%w` chain keeps `mmap: ...` diagnosable — and a sentinel +in `main` while the `%w` chain keeps `mmap: ...` diagnosable, and a sentinel `errUsage` plays Rust's empty-message role via `errors.Is`. Reproduced this session: a junk file gets `shmkv: bad.kv: not a shmkv v0 store` and exit 1, a missing file `shmkv: cannot open missing.kv` and exit 1, the third `set` into a 2-slot store `shmkv: store full (2 slots)` and exit 5, a 64-byte key -`shmkv: key too long (max 63 bytes)` and exit 2 — identically from all three +`shmkv: key too long (max 63 bytes)` and exit 2, identically from all three binaries, which is what 48 `verify.lua` checks per language assert. ## Concurrency lens `MAP_SHARED` gives you visibility, and visibility is not synchronization. Two `set` processes running *concurrently* could both probe, both find slot 2 -empty, and both write it — last `memcpy` wins, one key silently lost; worse, a +empty, and both write it: last `memcpy` wins, one key silently lost; worse, a `get` racing a `set` can read a slot mid-`memset` and see a torn key. Nothing in this chapter prevents that: `msync` is a durability barrier, not a memory barrier, and the kernel orders nothing between mappers. shmkv v0's defense is -its usage model — commands are whole processes run sequentially, so every -write is ordered by process exit — and the Rust unsafe-boundary comment is +its usage model: commands are whole processes run sequentially, so every +write is ordered by process exit, and the Rust unsafe-boundary comment is where that assumption is written down rather than left implicit. This is the cliff-hanger by design: chapter 20 puts atomics and futexes *inside* a shared mapping to make concurrent access actually safe. Two language-specific hazards round it out. Go's `[]byte` view means the race detector and GC offer no help -across processes — and the slice must never be touched after `Munmap`, which +across processes, and the slice must never be touched after `Munmap`, which `close` enforces by nil-ing it. And any mapper can be killed by a peer that -truncates the file — in-bounds loads become `SIGBUS` — which is exactly the +truncates the file (in-bounds loads become `SIGBUS`), which is exactly the hole `F_SEAL_SHRINK` sealed in the memfd aside. ## Build, run, observe @@ -364,7 +364,7 @@ hole `F_SEAL_SHRINK` sealed in the memfd aside. [host]$ cd examples/16-mmap-and-shared-mappings && ./demo.sh ``` -The full matrix is one runner line — from this session: +The full matrix is one runner line, from this session: ```console [host]$ python3 scripts/test-all-examples.py --only 16-mmap-and-shared-mappings @@ -418,7 +418,7 @@ munmap(0x7ff82e437000, 1034) = 0 Exactly the lifecycle the code promises: map 1034 bytes `MAP_SHARED`, one `MS_SYNC` over the whole store before success is reported, unmap before the -fd closes. The Go binary's trace shows the identical four lines (its fd is 4 — +fd closes. The Go binary's trace shows the identical four lines (its fd is 4; the runtime holds extras). **The mapping, in `/proc`.** `set` exits too fast to inspect, so inject a 3 s @@ -431,20 +431,20 @@ delay into `msync` and read the paused process's map table: 7ff342434000-7ff342435000 rw-s 00000000 00:24 24538261 .../demo.kv ``` -The permissions column is the whole chapter in four characters: `rw-s` — -readable, writable, **s**hared. A `MAP_PRIVATE` mapping shows `p` in that +The permissions column is the whole chapter in four characters: `rw-s` +means readable, writable, **s**hared. A `MAP_PRIVATE` mapping shows `p` in that position (look at your own binary's lines in the same file). The 1034-byte store occupies one 4096-byte page range: mappings are page-granular, and the file's last partial page is zero-padded in memory. **The format, by `cmp`.** A second store built by the *C++* binary replaying the identical operations (`create --slots 4`, then the same four `set`s) is -byte-identical to the Rust-created, Go-amended one — `cmp demo.kv twin.kv` +byte-identical to the Rust-created, Go-amended one: `cmp demo.kv twin.kv` prints nothing and exits 0, zero padding included. That, plus the three byte-identical `dump` outputs, is what `verify.lua` mechanizes: 48 checks per language, `PASS 48 / FAIL 0` for cpp, go, and rust in this session's run. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the eBPF view of this chapter is watching page faults and writeback > fleet-wide (`bpftrace` on the `page-faults` and `writeback` tracepoints) > instead of per-process `strace`; bcc-tools are not runnable on this host @@ -453,20 +453,20 @@ language, `PASS 48 / FAIL 0` for cpp, go, and rust in this session's run. ## What you learned -- `mmap` is a 2×2 — file-backed vs anonymous, `MAP_SHARED` vs `MAP_PRIVATE` — +- `mmap` is a 2×2 (file-backed vs anonymous, `MAP_SHARED` vs `MAP_PRIVATE`) and `MAP_PRIVATE`'s copy-on-write means your first store to a page forks it: the file keeps `SHKV1` while your view says `XXXXX`. - Visibility and durability are separate: shared mappers see each other's stores instantly through common page-cache pages, but only - `msync(MS_SYNC)` — visible in the strace as one call over the store — + `msync(MS_SYNC)` (visible in the strace as one call over the store) guarantees the bytes reached disk before success is reported. - The `ftruncate`-then-`mmap` ritual sizes the mapping, gets zero-filled - "empty" slots for free, and avoids `SIGBUS` past EOF; `memfd_create` plus + "empty" slots automatically, and avoids `SIGBUS` past EOF; `memfd_create` plus `F_SEAL_SHRINK`/`F_SEAL_GROW`/`F_SEAL_FUTURE_WRITE` (seals read back as 22) makes an anonymous, truncation-proof variant. - Cross-language interop comes from defining bytes, not structs: stated offsets, NUL padding, and a u32 LE assembled bytewise dodge every repr, - padding, and endianness rule — and `cmp` proving two stores byte-identical + padding, and endianness rule, and `cmp` proving two stores byte-identical is the test that format survived three implementations. - The unsafe boundary is the same fact three ways: Rust makes you write down the promises (`unsafe MmapMut::map_mut`), C++ lets a `void*` into shared @@ -478,7 +478,7 @@ Next, **allocators and GC runtimes**: what actually happens between your --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host this session: the runner printed 16-mmap-and-shared-mappings PASS PASS PASS (3 passed, 0 failed, 0 skipped) and verify.lua diff --git a/_docs/17-allocators-and-gc-runtimes.md b/_docs/17-allocators-and-gc-runtimes.md index b539247..110b89c 100644 --- a/_docs/17-allocators-and-gc-runtimes.md +++ b/_docs/17-allocators-and-gc-runtimes.md @@ -2,20 +2,20 @@ title: "Allocators and GC runtimes" order: 17 part: "Memory" -description: "What actually happens between your allocation and the kernel's pages: glibc malloc's arenas and tcache, what pmr and bumpalo really change, sync.Pool as Go's arena substitute, and the Go GC's pacer driven live with GOGC and GOMEMLIMIT — cross-checked with massif, /usr/bin/time, ltrace, and pprof." +description: "What actually happens between your allocation and the kernel's pages: glibc malloc's arenas and tcache, what pmr and bumpalo really change, sync.Pool as Go's arena substitute, and the Go GC's pacer driven live with GOGC and GOMEMLIMIT; cross-checked with massif, /usr/bin/time, ltrace, and pprof." duration: 50 minutes --- The last two chapters dealt in wholesale memory: pages from the kernel, mapped and shared with `mmap`. This chapter is retail. Between your `std::string`, -your `strconv.Itoa`, your `format!` and those pages sits an allocator — glibc +your `strconv.Itoa`, your `format!` and those pages sits an allocator: glibc malloc for C++ *and* (this surprises people) for Rust, and a garbage-collected -runtime heap for Go — and the one new idea here is that the axis that +runtime heap for Go, and the one new idea here is that the axis that separates them is not allocation but **reclamation**: free every object individually, free a whole batch at once, or free nothing and let a collector trace what survived. `allocbench` runs the same allocation-heavy workload through all three strategies in all three languages and prints the two numbers -that let you compare them: the process's peak RSS and the wall time — plus, +that let you compare them: the process's peak RSS and the wall time, plus, for Go only, how many GC cycles the churn cost. The code is in `examples/17-allocators-and-gc-runtimes/`. `./demo.sh` builds @@ -26,9 +26,9 @@ the three binaries share. {% include excalidraw.html file="17-arena-vs-gc" alt="Three columns compare reclamation strategies for the same 200,000 short-lived strings. Left, free one by one: objects flow into glibc's main arena and tcache with 64 cached chunks per size class, pages stay with the process (mallinfo2 reports arena 6340KB after everything is freed), peak RSS 3908KB for C++ and 2548KB for Rust with one free per object. Middle, arena and pool: a reusable 256KiB slab with a bump pointer, survivors copied out, the whole batch reclaimed in O(1) by reset, peak RSS 4240KB C++ and 2544KB Rust with zero per-object frees. Right, tracing GC: the churn becomes garbage the collector must find, the pacer triggers on GOGC growth or the GOMEMLIMIT soft limit, concurrent mark and sweep runs 9 cycles by default for 8064KB peak, or 47 cycles at 5676KB with GOMEMLIMIT=8MiB." - caption="Figure 17.1 — three ways to take memory back: per-object free through glibc malloc, wholesale arena release, and traced collection under a pacer; every number is from an allocbench run on the reference host" %} + caption="Figure 17.1: three ways to take memory back: per-object free through glibc malloc, wholesale arena release, and traced collection under a pacer; every number is from an allocbench run on the reference host" %} -> **Tools used** — `valgrind --tool=massif` + `ms_print`, `/usr/bin/time`, +> **Tools used**: `valgrind --tool=massif` + `ms_print`, `/usr/bin/time`, > `ltrace`, `nm`, `gcc`, `go tool pprof`, `go doc` (host); `memleak` from > bcc-tools (lab VM, exercised in Part 8). All host tools are checked by > `scripts/check-host.sh` or ship with the pinned toolchains. @@ -36,15 +36,15 @@ the three binaries share. ## What malloc does with your free `free(3)` almost never gives memory back to the kernel. glibc's allocator -carves the pages it got from `brk`/`mmap` into an **arena** — a heap region -plus the bookkeeping to subdivide it — and a freed chunk goes onto a list to +carves the pages it got from `brk`/`mmap` into an **arena** (a heap region +plus the bookkeeping to subdivide it), and a freed chunk goes onto a list to be handed out again: first into the calling thread's **tcache** (a tiny per-thread stash, by default up to 64 chunks in each of 64 size classes, touchable without any lock), then into the arena's small/large bins. The pages themselves stay with the process, which is exactly what RSS measures and why "my program freed everything but RSS didn't drop" is normal, not a leak. -You can watch this bookkeeping directly with `mallinfo2(3)` — a ten-line probe +You can watch this bookkeeping directly with `mallinfo2(3)`: a ten-line probe (allocate 100,000 48-byte chunks, free every other one, then the rest) prints, on this host: @@ -57,12 +57,12 @@ all freed: arena=6340KB in-use=5KB free-on-list=6334KB ``` The `holey` line is **fragmentation** in one row: half the bytes are free, but -they are interleaved hole-by-hole with live chunks, so the arena cannot shrink -— 6340KB of address space held to service 3130KB of live data. This recycling +they are interleaved hole-by-hole with live chunks, so the arena cannot shrink: +6340KB of address space held to service 3130KB of live data. This recycling is also why the C++ default variant below stays so flat: 200,000 short-lived strings churn through the *same* few chunks, tcache-fast, and peak RSS barely notices. (An aside: `mallinfo2` is glibc-specific and its counters are -process-wide approximations — use it for intuition, `getrusage` for numbers +process-wide approximations: use it for intuition, `getrusage` for numbers you report.) ## Arenas: pay once, free once @@ -70,18 +70,18 @@ you report.) The arena idea removes per-object reclamation entirely: allocate by bumping a pointer through a slab, never free individuals, release the whole slab in one move when the batch of work is done. C++23 spells it -`std::pmr::monotonic_buffer_resource` — a memory resource whose `deallocate` -is deliberately a **no-op**; everything comes back when the resource is +`std::pmr::monotonic_buffer_resource` (a memory resource whose `deallocate` +is a **no-op**); everything comes back when the resource is destroyed. What pmr actually changes is therefore two things and only two things: **allocation locality** (the batch's strings sit contiguously in one slab instead of scattered across malloc chunks) and **free-at-once** (batch -teardown is one release, not a thousand `free` calls). It does not magically -lower your high-water mark — the slab is a fixed cost you always touch, and +teardown is one release, not a thousand `free` calls). It does not lower +your high-water mark on its own: the slab is a fixed cost you always touch, and the numbers below show it. Rust's story starts with a fact worth proving rather than asserting: the default `GlobalAlloc` on Linux **is malloc**. The release binary imports it -from glibc — +from glibc: ```console [host]$ nm -D rust/target/release/app | grep -wE 'malloc|free|realloc|posix_memalign' @@ -91,13 +91,13 @@ from glibc — U realloc@GLIBC_2.2.5 ``` -— and `ltrace -x malloc -c` counts 415 live `malloc` calls for a 100-iteration +`ltrace -x malloc -c` counts 415 live `malloc` calls for a 100-iteration default run against 317 for the arena run (at tiny N the two copies into the long-lived index dominate; the churn savings grow with N). For the arena variant the example uses `bumpalo 3.20.3`: a `Bump` is the same bump-pointer design, `bump.reset()` is the O(1) wholesale release, and the borrow checker statically guarantees nothing bump-allocated survives the reset. Go gets -neither pmr nor bumpalo — the language's `arena` experiment stalled — so the +neither pmr nor bumpalo (the language's `arena` experiment stalled), so the idiomatic third spelling is `sync.Pool`: churn objects are *recycled* rather than batch-freed, which attacks the same cost (garbage pressure) from the other side. @@ -128,14 +128,14 @@ allocbench: variant=default allocs=200000 peak_rss=37164KB ms=50 gc_cycles=0 Read it as a supply curve: 9 cycles at 8064KB by default; four times the cycles (32) buys the peak down to 5516KB at a wall-time cost; with pacing off, an 8MiB soft limit alone drives 47 cycles to stay under it; and a 64MiB limit -the workload never approaches means the collector *never fires* — zero cycles, +the workload never approaches means the collector *never fires*: zero cycles, and the garbage piles up to 37164KB of RSS. That last line is the sharpest lesson: a GC's memory use is a policy outcome, not a property of your code. One note on *which* collector: go1.26.5's Green Tea GC. On this host the toolchain accepts `GOEXPERIMENT=greenteagc` and `nogreenteagc` (and rejects made-up names), and the version stamps tell you the default: a -`nogreenteagc` build is marked `go1.26.5-X:nogreenteagc` — a deviation — +`nogreenteagc` build is marked `go1.26.5-X:nogreenteagc` (a deviation), while a `greenteagc` build stamps plain `go1.26.5`, meaning Green Tea *is* this toolchain's default state. `go doc runtime` here documents GOGC and GOMEMLIMIT but nothing Green-Tea-specific, so that observable stamp is as far @@ -144,8 +144,8 @@ as this chapter's claims go. ## How the code works Two structures anchor all six builds. The long-lived survivor is a plain hash -map — `std::unordered_map`, `map[string]string`, -`HashMap` — deliberately ordinary, because the chapter is +map (`std::unordered_map`, `map[string]string`, +`HashMap`), kept ordinary because the chapter is about the churn, not the index. The churn carrier is what varies: C++ holds each batch's intermediates in a `std::pmr::vector` wired to a `monotonic_buffer_resource` over one reusable 256KiB slab (a @@ -153,7 +153,7 @@ a `monotonic_buffer_resource` over one reusable 256KiB slab (a `bumpalo::collections::String`s in a `BumpVec` inside a `Bump`; Go pools a `scratch` struct of three `bytes.Buffer`s plus a `[]byte` for `strconv.AppendInt`. The scratch containers exist so the *default* variant -also keeps a batch's fragments alive to batch end — both variants hold the +also keeps a batch's fragments alive to batch end: both variants hold the same live set, so their peaks are comparable and only reclamation differs. Here is the arena-side builder in each language, verbatim from @@ -250,19 +250,19 @@ Walk the C++ one first. The slab lives *outside* the batch loop and the `slab.data()/slab.size()`: each batch gets a fresh bump pointer into the same already-touched pages, so batch teardown (`~arena` at the closing brace) frees 1000 iterations of strings without a single `free` call, and the next batch -pays no new page faults. The `scratch.reserve` is load-bearing in a way it -usually isn't: a monotonic resource never reclaims, so letting the vector grow +pays no new page faults. The `scratch.reserve` matters more here than it +usually does: a monotonic resource never reclaims, so letting the vector grow geometrically would leave every outgrown copy as dead weight *inside the slab*. Same reason `make_kv` builds numbers with `std::to_chars` via -`append_num` — `std::to_string` would hand back an ordinary heap +`append_num`: `std::to_string` would hand back an ordinary heap `std::string`, silently routing churn around the arena. The survivors are copied out explicitly (`std::string{key}`) because the index must outlive the resource; forgetting that copy is the classic pmr lifetime bug, and it is the exact discipline in one line: *bulk-free the churn, copy out what survives.* The Rust version encodes the same rules in types instead of comments. -`BumpString::new_in(&bump)` borrows the arena, so the inner block must end — -every borrow dropped — before `bump.reset()` will compile. Where the C++ +`BumpString::new_in(&bump)` borrows the arena, so the inner block must end +(every borrow dropped) before `bump.reset()` will compile. Where the C++ version would let you keep a `pmr::string` past `~arena` and crash, here the borrow checker rejects the program. `reset()` keeps bumpalo's largest chunk, so after the first batch the steady state is the same reused-slab pattern as @@ -270,31 +270,31 @@ C++. The Go version has no batch boundary at all: `Get` → `reset` → build in place → copy out via `String()` → `Put`, per iteration. `writeNum` reuses one `[]byte` through `strconv.AppendInt` so digit formatting allocates nothing; the only per-iteration allocations left are the two `String()` copies that the -index keeps — visible below in the heap profile, and the reason `gc_cycles` +index keeps: visible below in the heap profile, and the reason `gc_cycles` drops from 9 to 3. The wiring is the book's standard shape: a hand-rolled flag loop (`--allocs`, `--variant`) so all three usage diagnostics are byte-identical; the build+query phase timed with a monotonic clock; `getrusage(RUSAGE_SELF)` -read afterwards for `ru_maxrss` (kilobytes on Linux — it is bytes on the -BSDs, one of the fragile bits stated plainly); and Go additionally reads +read afterwards for `ru_maxrss` (kilobytes on Linux; it is bytes on the +BSDs, one of the fragile bits in this accounting); and Go additionally reads `runtime.MemStats` before and after, reporting the `NumGC` delta so the GC -cost of *this workload* — not the process's whole life — is what prints. The +cost of *this workload* (not the process's whole life) is what prints. The query phase re-derives every key and fails on a missing key or a zero byte sum, so neither compiler nor collector can quietly optimize the work away. Two more fragile bits: the 256KiB slab is sized to a batch, and a monotonic resource that overflows silently falls through to its upstream -(the default resource — malloc), correct but no longer arena-fast; and +(the default resource, malloc), correct but no longer arena-fast; and `ru_maxrss` is a high-water mark, so it can never show the arena *releasing* -memory — that is what massif is for below. +memory: that is what massif is for below. ## Errors, three ways The contract: bad usage (unknown flag, non-positive `--allocs`, unknown variant) prints a diagnostic plus the usage line on stderr and exits 2; runtime failures print `allocbench: ...` and exit 1. Running all three with -`--variant marble` on this host produced byte-identical stderr — confirmed -with `cmp` across the three captures — and exit 2 from each: +`--variant marble` on this host produced byte-identical stderr (confirmed +with `cmp` across the three captures) and exit 2 from each: ```console [host]$ ./cpp/build/release/app --variant marble @@ -303,7 +303,7 @@ usage: allocbench [--allocs N] [--variant default|arena] ``` The mechanisms differ by language, the policy doesn't. C++ parses into -`std::expected` — a usage error is a *value* carried to +`std::expected`: a usage error is a *value* carried to `main`, which owns both stderr lines and the exit code; the one true runtime error path (`getrusage` failing) converts `errno` into `std::error_code` first. Go returns wrapped errors (`fmt.Errorf` with `%w`) from `parseArgs` @@ -318,18 +318,18 @@ errors are boring values, runtime errors carry their cause. Everything in this chapter is single-threaded, and almost none of it is thread-neutral. glibc malloc scales by giving *threads* their own arenas (up -to 8x the core count) plus the lock-free tcache — which is why a +to 8x the core count) plus the lock-free tcache, which is why a multi-threaded allocation-heavy program's RSS can multiply: each arena fragments separately. `monotonic_buffer_resource` is explicitly *not* thread-safe; the pattern that survives contact with threads is one arena per thread or per task, never one shared bump pointer. bumpalo says the same thing in the type system: `Bump` is not `Sync`, so two threads cannot even -share a reference. `sync.Pool` is the one built for concurrency — it shards +share a reference. `sync.Pool` is the one built for concurrency: it shards per-P (per scheduler processor) so `Get`/`Put` on different cores don't contend, and the GC empties it between cycles, which is precisely why it suits churn-recycling and not long-term caching. And the Go collector itself is concurrent: those 9 (or 47) cycles ran on other cores while `buildDefault` -kept allocating, so `gc_cycles` costs less wall time than it suggests — but +kept allocating, so `gc_cycles` costs less wall time than it suggests, but `GOMEMLIMIT` is global, so in a real service every goroutine shares the same bargain you tuned above. @@ -359,10 +359,10 @@ allocbench: variant=arena allocs=200000 peak_rss=2424KB ms=45 Three readings. The C++ arena's peak is *higher* (4240 vs 3908KB): malloc recycled the default variant's churn through the same tcache chunks, while -the always-touched slab is a fixed cost — the arena's win is deterministic +the always-touched slab is a fixed cost: the arena's win is deterministic wholesale release, not a lower high-water mark at this size. Go shows the GC account directly: recycling via `sync.Pool` cut cycles 9 → 3 and time 47 → -33 ms. Rust's arena win is wall time — on repeated runs the default settles +33 ms. Rust's arena win is wall time: on repeated runs the default settles around 39–45 ms and the arena at 26–27 ms (the tied 45/45 pair above caught a cold first run). The runner (`python3 scripts/test-all-examples.py --only 17-allocators-and-gc-runtimes`) @@ -393,18 +393,18 @@ Number of snapshots: 74 The default peak is 273,800 bytes of many small blocks, the largest tree being the index's hashtable nodes out of `build_default`. The arena run's -peak (snapshot 4 of 59) is 504,000 bytes — *larger*, matching the RSS story — +peak (snapshot 4 of 59) is 504,000 bytes (*larger*, matching the RSS story) and 52.01% of it is a single 262,144-byte block: `_M_create_storage` → `build_arena (main.cpp:151)`, the 256KiB slab, allocated once. The per-string churn has vanished from malloc's ledger entirely, because inside the slab it -is pointer arithmetic — which is the pmr claim (locality plus free-at-once) +is pointer arithmetic, which is the pmr claim (locality plus free-at-once) made visible by an independent tool. **RSS agreement, with `/usr/bin/time -v`.** The kernel's accounting of the same runs: the Go binary's self-reported `peak_rss=8320KB` matched `Maximum resident set size (kbytes): 8320` exactly (both read the same `ru_maxrss`, one via `getrusage`, one via `wait4`). The C++ pair read 3916 self vs 4388 -from time, and 4312 vs 4356 for arena — the binary samples before printing +from time, and 4312 vs 4356 for arena: the binary samples before printing and process teardown, time observes the whole life; same ordering, sub-500KB skew, no surprises. @@ -423,13 +423,13 @@ Showing nodes accounting for 41474.84kB, 100% of 41474.84kB total ``` The default variant allocated 41.5MB total, 93.82% of it flat in -`buildDefault` — the churn. The arena profile totals 13.3MB, topped by +`buildDefault`: the churn. The arena profile totals 13.3MB, topped by `bytes.(*Buffer).String` at 10752.55kB (80.76%): almost exactly the two surviving copies per iteration, with the churn recycled through the pool. -Three tools, three vantage points — allocator ledger, kernel RSS, runtime -profile — one consistent story. +Three tools, three vantage points (allocator ledger, kernel RSS, runtime +profile), one consistent story. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the eBPF view of this chapter is allocator tracing without recompiling: > `[vm]$ /usr/share/bcc/tools/memleak -p ` samples outstanding > allocations by stack, kernel-side, on any binary. bcc-tools need root and @@ -438,21 +438,21 @@ profile — one consistent story. ## What you learned -- `free` returns chunks to glibc's arena and tcache, not pages to the kernel - — `mallinfo2` showed 6340KB of arena still held with 5KB in use, and a +- `free` returns chunks to glibc's arena and tcache, not pages to the kernel: + `mallinfo2` showed 6340KB of arena still held with 5KB in use, and a half-freed heap is fragmentation you can read out (`in-use 3130KB` interleaved with `free-on-list 3209KB`). -- An arena changes two things — allocation locality and free-at-once — and +- An arena changes two things (allocation locality and free-at-once), and massif proved both: the churn collapsed into one 262,144-byte slab allocation, while peak RSS stayed *comparable* (4240 vs 3908KB) because malloc recycles churn well too. - Rust's default allocator is glibc malloc (`nm -D` shows the imports; `ltrace` counts the calls), so everything in the malloc story applies until - you opt into `bumpalo` — where `reset()` is O(1) and the borrow checker + you opt into `bumpalo`, where `reset()` is O(1) and the borrow checker enforces the copy-out-what-survives rule that pmr leaves to discipline. - Go's GC is a negotiable policy: the same binary ran 9 cycles at 8064KB, 47 cycles at 5676KB under `GOMEMLIMIT=8MiB`, and 0 cycles at 37164KB when the - limit was never approached — and `sync.Pool` is the arena-shaped tool that + limit was never approached, and `sync.Pool` is the arena-shaped tool that cut the default workload's cycles from 9 to 3. Next, **pipes, FIFOs, and splice**: the oldest IPC primitive on the system, @@ -461,7 +461,7 @@ userspace at all. --- -

verified — every number above +

verified: every number above was produced on the Fedora 44 reference host (kernel 7.1) this session: the runner printed 17-allocators-and-gc-runtimes PASS PASS PASS (3 passed, 0 failed, 0 skipped); the six variant lines, the four diff --git a/_docs/18-pipes-fifos-splice.md b/_docs/18-pipes-fifos-splice.md index 105fccc..08ed21d 100644 --- a/_docs/18-pipes-fifos-splice.md +++ b/_docs/18-pipes-fifos-splice.md @@ -2,21 +2,21 @@ title: "Pipes, FIFOs, and splice" order: 18 part: "IPC" -description: "pmon v4 turns the supervisor into a log pipeline: the pipe as a ring of kernel pages with its capacity read live via F_GETPIPE_SZ, what PIPE_BUF atomicity does and does not promise, FIFO open semantics and the ENXIO trick, SIGPIPE/EPIPE as the writer's contract, and splice moving log bytes into a FIFO without a userspace copy — proven with strace, /proc, and a 5.3 GB/s dd run." +description: "pmon v4 turns the supervisor into a log pipeline: the pipe as a ring of kernel pages with its capacity read live via F_GETPIPE_SZ, what PIPE_BUF atomicity does and does not promise, FIFO open semantics and the ENXIO trick, SIGPIPE/EPIPE as the writer's contract, and splice moving log bytes into a FIFO without a userspace copy, proven with strace, /proc, and a 5.3 GB/s dd run." duration: "45 minutes" --- This part of the book is about processes talking to each other, and it opens with the primitive every shell has used since 1973. `pmon` ended the process -part supervising a crashing child with two byte-identical engines — pidfd and -sigchld — but it threw the child's output away, and a supervisor that +part supervising a crashing child with two byte-identical engines, pidfd and +sigchld, but it threw the child's output away, and a supervisor that discards logs is half a supervisor. Version 4 captures the child's stdout and stderr into a framed log file and then republishes that log through a named pipe to any reader that wants it, surviving readers that come and go. The one new idea is that a pipe is not a file with two ends: it is a bounded ring of -kernel pages, and every behavior this chapter meets — the 65536-byte +kernel pages, and every behavior this chapter meets, the 65536-byte capacity, `PIPE_BUF` atomicity, EOF as "last write end closed", `SIGPIPE` on -a readerless write, even `splice(2)`'s zero-copy trick — falls straight out +a readerless write, even `splice(2)`'s zero-copy trick, falls straight out of that ring. The code is in `examples/18-pipes-fifos-splice/`. `./demo.sh` there builds @@ -27,22 +27,22 @@ languages share. {% include excalidraw.html file="18-pipe-internals" alt="Three panels: a writer process box holding fd 1 the write end, a kernel band labelled struct pipe_inode_info inode 16462630 containing a row of eight page slots from the sixteen-slot ring with head and tail pointers marked, and a reader process box holding fd 5 the read end with fdinfo flags 02000000; notes record the 65536-byte capacity read via F_GETPIPE_SZ, the F_SETPIPE_SZ rounding of 1 up to 4096, the 1048576 pipe-max-size ceiling, and the warning that PIPE_BUF atomicity is not message framing." - caption="Figure 18.1 — one pipe from this chapter's live supervise run: the child's fd 1 and the supervisor's fd 5 share the ring behind pipe inode 16462630; every number is from /proc or a live fcntl" %} + caption="Figure 18.1: one pipe from this chapter's live supervise run: the child's fd 1 and the supervisor's fd 5 share the ring behind pipe inode 16462630; every number is from /proc or a live fcntl" %} -> **Tools used** — `strace`, `mkfifo`, `dd`, `stat`, `getconf`, `ls`/`cat` +> **Tools used**: `strace`, `mkfifo`, `dd`, `stat`, `getconf`, `ls`/`cat` > over `/proc`, `python3` (host); `bpftrace`/bcc-tools (lab VM, exercised in > Part 8). Everything here is checked by `scripts/check-host.sh`, ships with > Fedora, or is preinstalled in the lab VMs. ## A pipe is a ring of pages -`pipe2(2)` allocates a `pipe_inode_info` in the kernel — an array of sixteen -`pipe_buffer` slots, each holding one page — and returns two fds onto it: +`pipe2(2)` allocates a `pipe_inode_info` in the kernel: an array of sixteen +`pipe_buffer` slots, each holding one page, and returns two fds onto it: slot 0 reads from the ring's tail, slot 1 writes at its head. Nothing about this is a file. There is no offset (`lseek` fails with `ESPIPE`), no name, and no storage: a full ring blocks the writer, an empty ring blocks the reader, and the kernel wakes each side as the other makes progress. That -back-pressure is the feature — a slow log consumer stalls the producer +back-pressure is the feature: a slow log consumer stalls the producer instead of eating unbounded memory. The capacity is not folklore; you can ask for it. `fcntl(F_GETPIPE_SZ)` @@ -69,14 +69,14 @@ rounds up to one page; unprivileged growth stops at `pipe` for the same reason chapter 7 put `O_CLOEXEC` in every open: the flag lands atomically, with no window for a concurrently spawned child to inherit the fd. `strace -f -e trace=pipe2` on the C++ supervisor shows exactly two -calls per attempt — `pipe2([5, 6], O_CLOEXEC) = 0` and -`pipe2([7, 8], O_CLOEXEC) = 0` — one pipe for the child's stdout, one for +calls per attempt: `pipe2([5, 6], O_CLOEXEC) = 0` and +`pipe2([7, 8], O_CLOEXEC) = 0`, one pipe for the child's stdout, one for its stderr. ## `PIPE_BUF`, and what it does not promise -POSIX guarantees that a single `write(2)` of at most `PIPE_BUF` bytes — 4096 -on Linux, read out above with `pathconf` — is atomic: its bytes land in the +POSIX guarantees that a single `write(2)` of at most `PIPE_BUF` bytes (4096 +on Linux, read out above with `pathconf`) is atomic: its bytes land in the ring contiguously, never interleaved with another writer's bytes. That guarantee is narrower than it sounds, and the gap is why v4's capture architecture exists. Atomicity is *not framing*: the reader can still `read` @@ -85,27 +85,27 @@ about large writes*: one `write` of 8000 bytes may be split anywhere, with another writer's data landing in the middle. And it is *not a promise about your language's buffering*: a child that "prints a line" may reach the kernel as several small writes or one giant flush, at the whim of its stdio -library — a supervised process is exactly the kind of writer you cannot +library; a supervised process is exactly the kind of writer you cannot retrain. So `pmon` never relies on write atomicity to keep `[out]` and `[err]` -records apart. The tempting shortcut — `dup2` both child streams onto *one* -pipe and let `PIPE_BUF` sort it out — loses the out/err distinction forever +records apart. The tempting shortcut, `dup2` both child streams onto *one* +pipe and let `PIPE_BUF` sort it out, loses the out/err distinction forever and still interleaves mid-line whenever a write exceeds 4096 bytes or a line crosses two writes. Instead each stream gets its own pipe, and a userspace relay loop reassembles byte chunks into complete lines before stamping the -prefix. Interleaving still happens — but only at line granularity, chosen by +prefix. Interleaving still happens, but only at line granularity, chosen by the relay, never mid-record. That relay loop is the real price of framed capture: the kernel moves bytes; meaning is your job. ## FIFOs: a pipe with a name `supervise` and its child share ancestry, so inheriting pipe ends works. A -log *consumer* started an hour later from another terminal shares nothing — +log *consumer* started an hour later from another terminal shares nothing; it needs a rendezvous point with a name. `mkfifo(3)` creates one: an inode of type `S_IFIFO` whose only content is a pipe ring materialized in the kernel when both ends open it. `stat` reports our created FIFO as `fifo 644 -live2.fifo` — a filesystem name, zero bytes of storage. +live2.fifo`: a filesystem name, zero bytes of storage. Opening a FIFO has its own choreography, because a pipe with one end is useless. Who blocks when: @@ -118,39 +118,39 @@ useless. Who blocks when: | `O_WRONLY\|O_NONBLOCK` | **fails with `ENXIO`** | returns at once | | `O_RDWR` | returns at once (Linux behavior; POSIX leaves it undefined) | returns at once | -The asymmetry in row four is deliberate kernel policy — a nonblocking -writer with no reader would otherwise instantly fill the ring and stall — +The asymmetry in row four is deliberate kernel policy: a nonblocking +writer with no reader would otherwise instantly fill the ring and stall, and `pmon tail` turns it into a feature: it opens the FIFO `O_WRONLY|O_NONBLOCK|O_CLOEXEC`, treats `ENXIO` as "no reader yet, sleep 50 ms, retry", and on success clears `O_NONBLOCK` with `fcntl(F_SETFL)` so the relay gets blocking back-pressure. The cleared bit is visible in `/proc//fdinfo/4`: `flags: 02100001` decodes as `O_WRONLY` + -`O_LARGEFILE` + `O_CLOEXEC` — no `O_NONBLOCK` remaining. +`O_LARGEFILE` + `O_CLOEXEC`, no `O_NONBLOCK` remaining. ## `SIGPIPE`/`EPIPE`: the writer's contract The reader side of a pipe has a gentle end-of-conversation signal: when the last write end closes, `read` returns 0, EOF. The writer side gets a harsher one. Writing to a pipe whose last read end has closed raises -`SIGPIPE`, whose default action *terminates the process* — the design that -makes `head -1 < bigfile-producer` shut the producer down for free in a +`SIGPIPE`, whose default action *terminates the process*: the design that +makes `head -1 < bigfile-producer` shut the producer down automatically in a shell pipeline. A long-lived relay must not die because one `cat` was killed, so `cmd_tail` ignores `SIGPIPE` in all three languages, converting the event into an `EPIPE` error return from `write`/`splice` at the exact call that noticed. That is the writer's contract: handle `EPIPE` at every write site, or inherit a default that kills you. `pmon` treats it as a -state transition — print `pmon: tail reader detached`, close the writer fd, +state transition: print `pmon: tail reader detached`, close the writer fd, and go back to waiting for the next reader. {% include excalidraw.html file="18-log-topology" alt="Three horizontal bands. Top, pmon supervise pid 2633754: the sh child pid 2633756 writes fd 1 and fd 2 into two kernel pipe nodes with inodes 16462630 and 16462631, which feed a poll loop node listing signalfd 4, pidfd 6, and pipe read ends 5 and 7, appending framed out and err lines into a live.log node opened append-only as fd 3. Middle, pmon tail pid 2634759: a log fd 3 node with fdinfo pos 10 connects by an amber splice arrow labelled with the real splice call returning 22 to a FIFO writer fd 4 node whose flags 02100001 show O_NONBLOCK cleared. Bottom, kernel FIFO plus reader: the live.fifo inode of type S_IFIFO flows into a cat reader holding fd 3, with a note that a detached reader causes EPIPE and the relay re-splices undelivered bytes to the next reader." - caption="Figure 18.2 — pmon v4's log pipeline end to end; every pid, fd number, pipe inode, flag value, and splice return comes from the live runs traced in this chapter" %} + caption="Figure 18.2: pmon v4's log pipeline end to end; every pid, fd number, pipe inode, flag value, and splice return comes from the live runs traced in this chapter" %} ## How the code works Two structures carry the capture. `Capture` owns one child's two pipe read -ends plus per-stream open/closed state; `LineRelay` owns the framing — a +ends plus per-stream open/closed state; `LineRelay` owns the framing: a byte buffer per stream that accumulates chunks, emits every completed line to the log as `[out] line\n` or `[err] line\n` immediately (line-buffered, so the log is useful while the child lives), bounds the partial-line buffer @@ -177,7 +177,7 @@ The wiring is where the invariants live. The C++ supervisor, per attempt: ``` `spawn_child` wires the write ends to the child with -`posix_spawn_file_actions_adddup2(fa, out.get(), STDOUT_FILENO)` — and +`posix_spawn_file_actions_adddup2(fa, out.get(), STDOUT_FILENO)`, and `dup2` semantics do the O_CLOEXEC bookkeeping for us: the *duplicate* created in the child has the close-on-exec bit clear, so the child's fds 1 and 2 survive `exec` while every original supervisor-side fd vanishes from @@ -190,14 +190,14 @@ the child. Two lines later comes the sentence this chapter exists to teach: EOF on a pipe means "*every* write-end fd is closed". The supervisor holds copies of the write ends between `pipe2` and `posix_spawnp`; if it kept -them, the child could die and the read ends would stay open forever — the +them, the child could die and the read ends would stay open forever: the supervisor would be holding its own pipe open, waiting on itself. Closing the parent's copies immediately after spawn makes child-exit imply pipe-EOF, which is what lets `drain_to_eof` terminate. From there the pipes simply join the multiplexing each engine already had. -C++ polls four fds — signalfd, both pipe read ends, and (pidfd engine) the -pidfd — marking a drained pipe's slot as `fd = -1`, which `poll(2)` +C++ polls four fds: signalfd, both pipe read ends, and (pidfd engine) the +pidfd, marking a drained pipe's slot as `fd = -1`, which `poll(2)` ignores. Rust builds the same four-slot `rustix::event::poll` set beside its nix `SignalFd`, parking closed pipes with an empty event mask. Go inverts idiomatically: `capture` wraps each pipe in a `bufio.Scanner` @@ -205,12 +205,12 @@ goroutine feeding one channel of prefixed `logLine`s into the same `select` that already multiplexes the pidfd poller, `signal.Notify`, and the deadline; a `sync.WaitGroup` closes the channel when both scanners hit EOF, so channel-closed is the Go spelling of drained-to-EOF. In every language -the reap path calls the drain *before* printing the exit observation — the +the reap path calls the drain *before* printing the exit observation: the log is complete before `pmon: child=… exited status=…` appears, an ordering `verify.lua` asserts. `tail` is the second half: create the FIFO (`EEXIST` is tolerated only if -the existing inode really is a FIFO — anything else is error, exit 1), open +the existing inode really is a FIFO; anything else is error, exit 1), open the log read-only at offset 0 so a late reader gets the backlog, then relay forever. The C++ and Rust relays move bytes with `splice(2)`; Go uses `io.Copy` per reader session. The heart of each: @@ -294,7 +294,7 @@ fn relay_splice(log: &File, w: &OwnedFd) -> Result { `splice` exists because a pipe *is* a ring of page references: when one fd of the pair is a pipe, the kernel can move data by wiring the file's -page-cache pages into `pipe_buffer` slots — no copy into userspace, no +page-cache pages into `pipe_buffer` slots: no copy into userspace, no copy back. Passing `NULL`/`None` for the file's offset pointer means "use the fd's own offset and advance it", which is precisely the no-loss mechanism: the offset moves only for bytes the pipe accepted, so `EPIPE` @@ -303,19 +303,19 @@ reader's first `splice` resends exactly from there. Go has no `splice` in its portable API, so it makes the same promise arithmetically: remember `start`, let `io.Copy` report `n` bytes delivered before the `EPIPE`, seek back to `start+n`. (`io.Copy` between an `*os.File` pair actually tries -`copy_file_range`/`sendfile` internally — but to a FIFO it settles on +`copy_file_range`/`sendfile` internally, but to a FIFO it settles on plain read/write, as the strace below confirms only for C++ and Rust.) The C++/Rust fallback `relay_rw` keeps the identical rewind contract for filesystems where `splice` returns `EINVAL`; a `tee(2)` aside completes -the family — it duplicates pipe contents to a second pipe without +the family: it duplicates pipe contents to a second pipe without consuming them, which is how you would split this log stream to two FIFOs. -The fragile bits, stated plainly: the 50 ms `sleep`-poll for both "log has -no new bytes" and "no reader yet" is deliberate simplicity — chapter 9's +The fragile bits: the 50 ms `sleep`-poll for both "log has +no new bytes" and "no reader yet" is deliberate simplicity; chapter 9's inotify machinery could make both event-driven, and a real system would use it; the FIFO relay serves one reader at a time (a second simultaneous -opener would interleave records with the first — FIFOs have one ring, not +opener would interleave records with the first, since FIFOs have one ring, not one per reader); and log rotation is unhandled: `tail` follows an fd, not a name, so a renamed-away log keeps feeding the old inode. Chapter 19 replaces "one anonymous byte stream" with a real control plane. @@ -326,12 +326,12 @@ The chapter's errno vocabulary is small and every entry is *expected*, so the interesting part is which errors are policy at the call site versus fatal. `ENXIO` from a nonblocking FIFO open means "no reader yet": C++ checks `fd.error().value() != ENXIO`, Go `errors.Is(err, unix.ENXIO)`, -Rust matches `Err(rustix::io::Errno::NXIO)` — all three sleep and retry. +Rust matches `Err(rustix::io::Errno::NXIO)`: all three sleep and retry. `EPIPE` means "reader left": mapped to the `detached` state, never propagated. `EEXIST` from `mkfifo` is acceptance-with-verification: stat the path, require `S_ISFIFO` (C++), `fs.ModeNamedPipe` (Go), `file_type().is_fifo()` (Rust), and only a non-FIFO squatter is fatal. -`EINTR` everywhere returns to the loop — `tail`'s stop handlers install +`EINTR` everywhere returns to the loop: `tail`'s stop handlers install with no `SA_RESTART` precisely so a signal can interrupt a blocking call and let the loop observe the stop flag. Everything unexpected carries its context up to one printer and exit 1, same as every pmon before it: three @@ -341,7 +341,7 @@ results), one policy. ## Concurrency lens The capture is a small concurrency study. C++ and Rust stay -single-threaded by *adding fds to an existing poll set* — the same event +single-threaded by *adding fds to an existing poll set*: the same event loop that watches for child exit watches for child output, so there is no locking anywhere; the log write happens on the only thread. Go runs three goroutines per attempt (two scanners, one WaitGroup-closer), and the @@ -349,11 +349,11 @@ channel is what serializes: only the `select` loop's goroutine touches the log file, so `[out]`/`[err]` records cannot tear even though the scanners run concurrently. The deadlock the design must dodge is the classic pipe-capture one: a child that fills both pipes while the supervisor waits -only for exit would stall forever — which is why output fds live in the +only for exit would stall forever, which is why output fds live in the same poll set/select as the exit source, never behind it. Signal-wise, `SIGPIPE` disposition is process-wide state: C++/Rust set `SIG_IGN` explicitly; Go's `signal.Ignore(syscall.SIGPIPE)` does the same job with a -runtime subtlety — Go would otherwise re-raise a fatal SIGPIPE only for +runtime subtlety: Go would otherwise re-raise a fatal SIGPIPE only for writes to fds 1 and 2, and the FIFO writer is fd 3, but the explicit ignore documents intent and covers the stdout prints too. And the supervisor still resets the child's signal mask at spawn (chapter 12's @@ -366,7 +366,7 @@ lesson): a captured child is still a child. ``` Each language builds and runs a capture self-demo. Driving it by hand the -way this chapter's evidence was produced — first the capture: +way this chapter's evidence was produced, first the capture: ```bash [host]$ APP=examples/18-pipes-fifos-splice/cpp/build/release/app @@ -409,8 +409,8 @@ Every line above is from the captured run: the first reader got `[out] one`, the line appended while no reader existed arrived intact at the second reader, and SIGTERM ended the relay with exit 0. The runner (`python3 scripts/test-all-examples.py --only 18-pipes-fifos-splice`) -asserts this whole choreography — plus the v2 engine contract unchanged -under capture — and printed `PASS PASS PASS` for cpp, go, and rust on this +asserts this whole choreography, plus the v2 engine contract unchanged +under capture, and printed `PASS PASS PASS` for cpp, go, and rust on this host. ## Cross-check, four ways @@ -433,11 +433,11 @@ descriptor in Figure 18.2 (paths shortened): ``` The child's fd 1 and the supervisor's fd 5 name the *same* pipe inode -16462630 — one ring, two processes — and fd 2/fd 7 pair up on 16462631; +16462630 (one ring, two processes), and fd 2/fd 7 pair up on 16462631; the relay and the reader meet at the FIFO. `fdinfo` reads out the flags: the supervisor's pipe read end shows `flags: 02000000` (`O_CLOEXEC`, from `pipe2`), and the relay's FIFO writer shows `02100001` with `pos: 10` on -its log fd — ten bytes, exactly `[out] one\n`, delivered. +its log fd: ten bytes, exactly `[out] one\n`, delivered. **The capacity, from the fd itself.** The `F_GETPIPE_SZ` probe above printed `capacity: 65536` and `PIPE_BUF: 4096` from a live pipe, agreeing @@ -453,12 +453,12 @@ splice(3, NULL, 4, NULL, 65536, 0) = 0 splice(3, NULL, 4, NULL, 65536, 0) = -1 EPIPE (Broken pipe) ``` -fd 3 (the log) to fd 4 (the FIFO), 65536-byte requests: 22 bytes moved — -one `[out] tick 1784434137\n` record — then 0 for "caught up", then +fd 3 (the log) to fd 4 (the FIFO), 65536-byte requests: 22 bytes moved, +one `[out] tick 1784434137\n` record, then 0 for "caught up", then `EPIPE` at the instant the `cat` died, answered on stdout by `pmon: tail reader detached`. The Rust binary's trace is the same shape: `splice(3, NULL, 4, NULL, 65536, 0) = 16` for its 16-byte record, then -`= 0`, then `= -1 EPIPE (Broken pipe)` — 22 splice calls total in that +`= 0`, then `= -1 EPIPE (Broken pipe)`: 22 splice calls total in that session and not one `read` of log data in between. Go, by design, shows plain `read`/`write` instead. @@ -470,11 +470,11 @@ bytes through one should run at memory speed, not disk speed: 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.202097 s, 5.3 GB/s ``` -One GiB through the FIFO in a fifth of a second on this host — 5.3 GB/s +One GiB through the FIFO in a fifth of a second on this host: 5.3 GB/s through two processes and one ring, with `bs=64k` matching both the ring capacity and pmon's `kChunk`. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the fleet-wide view of this chapter is tracing pipe traffic and splice > calls system-wide with `bpftrace` kprobes and bcc-tools instead of > per-process `strace`; that tooling is not runnable on this host without @@ -483,30 +483,30 @@ capacity and pmon's `kChunk`. ## What you learned -- A pipe is a bounded ring of sixteen kernel pages — capacity readable and - resizable through `fcntl(F_GETPIPE_SZ/F_SETPIPE_SZ)` in whole-page units - — and its blocking behavior in both directions is back-pressure, not a +- A pipe is a bounded ring of sixteen kernel pages, capacity readable and + resizable through `fcntl(F_GETPIPE_SZ/F_SETPIPE_SZ)` in whole-page units, + and its blocking behavior in both directions is back-pressure, not a defect. - `PIPE_BUF` (4096) makes small single writes atomic but promises nothing - about framing, large writes, or reads — which is why framed `[out]`/`[err]` + about framing, large writes, or reads, which is why framed `[out]`/`[err]` capture needs two pipes and a userspace line relay, and why the parent must close its write-end copies or EOF never comes. - A FIFO is a pipe with an inode: `open` blocks until both ends exist, `O_WRONLY|O_NONBLOCK` turns "no reader" into a pollable `ENXIO`, and `SIGPIPE`-ignored plus `EPIPE`-handled is the contract every long-lived writer must sign. -- `splice(2)` moves file bytes into a pipe kernel-side — `strace` showed - the relay running on nothing but `splice` — and advancing the file +- `splice(2)` moves file bytes into a pipe kernel-side: `strace` showed + the relay running on nothing but `splice`, and advancing the file offset only for delivered bytes is what makes reader churn lossless in all three languages. -Next, **UNIX sockets and fd passing**: pmon grows a real control plane — a +Next, **UNIX sockets and fd passing**: pmon grows a real control plane, a socket a stranger can find, credentials the kernel vouches for, and open file descriptors sent across process boundaries. --- -

verified — every number and +

verified: every number and output excerpt above was reproduced on the Fedora 44 reference host (kernel 7.1) this session: the runner printed 18-pipes-fifos-splice  PASS  PASS  PASS diff --git a/_docs/19-unix-sockets-fd-passing.md b/_docs/19-unix-sockets-fd-passing.md index 3504cd5..9388583 100644 --- a/_docs/19-unix-sockets-fd-passing.md +++ b/_docs/19-unix-sockets-fd-passing.md @@ -2,22 +2,22 @@ title: "UNIX sockets and fd passing" order: 19 part: "IPC" -description: "pmon grows a control plane on a UNIX stream socket: filesystem vs abstract addresses under ss -x, SO_PEERCRED as kernel-vouched local auth, and SCM_RIGHTS sending an open log fd across process boundaries — proven shared at the open-file-description layer with /proc fdinfo on both sides." +description: "pmon grows a control plane on a UNIX stream socket: filesystem vs abstract addresses under ss -x, SO_PEERCRED as kernel-vouched local auth, and SCM_RIGHTS sending an open log fd across process boundaries, proven shared at the open-file-description layer with /proc fdinfo on both sides." duration: "50 minutes" --- -Chapter 18 gave `pmon` a log pipeline built from pipes and FIFOs — fast, -kernel-buffered, and strictly byte-shaped: a pipe moves data, and only +Chapter 18 gave `pmon` a log pipeline built from pipes and FIFOs (fast, +kernel-buffered, and strictly byte-shaped): a pipe moves data, and only between processes that inherited its ends. Version 5 needs something pipes -cannot do: a stranger — a `pmctl` process started minutes later, from -another terminal — must find the running supervisor, prove who it is, ask +cannot do: a stranger (a `pmctl` process started minutes later, from +another terminal) must find the running supervisor, prove who it is, ask questions, and get answers. That is a *control plane*, and the Linux primitive for it is the UNIX domain socket: a socket with the full `connect`/`accept` shape of TCP but living entirely inside the kernel, with two abilities no network socket has. It can tell the server exactly which uid and pid is calling (`SO_PEERCRED`), and it can carry *file descriptors* in its messages (`SCM_RIGHTS`). The second one is the headline: `pmctl -logfd` never opens the log file — the supervisor hands it an already-open +logfd` never opens the log file; the supervisor hands it an already-open descriptor, and the chapter closes the loop on chapter 7's three-layer model by proving, with `/proc` on both processes at once, exactly which layer crossed the socket. @@ -29,9 +29,9 @@ one-line wire protocol, and the exit codes all three languages share. {% include excalidraw.html file="19-uds-control-plane" alt="Two process bands over a kernel band: the pmon supervise process holds listener fd 4 bound to /tmp/…/ctl.sock, an accepted fd 5 annotated with the SO_PEERCRED result uid=1000 pid=2624432, and log fd 3 opened O_RDWR|O_APPEND feeding a dashed child box for sleep 300 pid 2622905; the pmctl process holds client fd 3 which connects down to socket inode 16422777 in the kernel band, paired by a bidirectional amber arrow with socket inode 16411996 owned by the supervisor, exactly as ss -xp reported." - caption="Figure 19.1 — pmon v5's control plane during one status call: every pid, fd number, and socket inode in this figure comes from the live run traced later in the chapter" %} + caption="Figure 19.1: pmon v5's control plane during one status call: every pid, fd number, and socket inode in this figure comes from the live run traced later in the chapter" %} -> **Tools used** — `ss`, `strace`, `stat`, `tail`, `ls`, `python3` (host); +> **Tools used**: `ss`, `strace`, `stat`, `tail`, `ls`, `python3` (host); > `bpftrace`/bcc-tools (lab VM, exercised in Part 8). Everything host-side is > checked by `scripts/check-host.sh` or ships with Fedora. @@ -39,20 +39,20 @@ one-line wire protocol, and the exit codes all three languages share. `AF_UNIX` sockets come in the same three types as `AF_INET`, and the choice is about message shape, not speed. `SOCK_STREAM` is a connected, reliable -byte stream — no boundaries, which is why pmon's protocol is line-framed +byte stream: no boundaries, which is why pmon's protocol is line-framed ("one `\n`-terminated command in, one reply out"). `SOCK_DGRAM` is -connectionless datagrams — unlike UDP they are reliable and ordered +connectionless datagrams; unlike UDP they are reliable and ordered kernel-local, but there is no connection, and without a connection there is no accepted socket to ask `SO_PEERCRED` about, which rules it out here. `SOCK_SEQPACKET` is the hybrid TCP never had: connection-oriented *and* boundary-preserving, a good fit for record protocols. pmon uses a stream -because the framing is one trivial line and because the connection itself is -load-bearing: it scopes authentication (credentials are checked per accepted +because the framing is one trivial line and because the connection itself +does real work: it scopes authentication (credentials are checked per accepted connection) and delivery (the reply belongs to whoever wrote the command). `socketpair(2)` is the degenerate case worth knowing: it returns two already-connected `AF_UNIX` ends with no address at all, the right tool when -the two parties are parent and child and nobody needs to *find* anybody — +the two parties are parent and child and nobody needs to *find* anybody; it is how container runtimes hand fds to the processes they spawn. pmon needs the opposite: a rendezvous point a stranger can name. That means an address, and UNIX sockets have two address namespaces. @@ -61,14 +61,14 @@ address, and UNIX sockets have two address namespaces. A **filesystem** socket is what `bind` creates when `sun_path` starts with a normal character: a socket-type inode at that path. It obeys directory -permissions — parking `ctl.sock` in a `0700` directory *is* the access -policy — and it outlives its creator, which is why `listen_ctl` starts with +permissions: parking `ctl.sock` in a `0700` directory *is* the access +policy, and it outlives its creator, which is why `listen_ctl` starts with `unlink()`: a crashed supervisor leaves a stale inode that would make the next `bind` fail with `EADDRINUSE`. An **abstract** socket, a Linux extension, is what you get when `sun_path[0]` is a NUL byte: no inode, no permissions, automatically gone when the last fd closes, and scoped to the network namespace. Both kinds show up in `ss -x`, and the rendering -distinguishes them — on this host, with pmon's socket listening and a +distinguishes them: on this host, with pmon's socket listening and a one-line `python3` peer bound to the abstract name `\0pmon-demo`: ```console @@ -82,14 +82,14 @@ uses the filesystem kind: the path *is* the discovery mechanism and the permission check, and `stop` unlinking it is a clean "no longer accepting" signal. The classic trap is capacity, not permissions: `sun_path` is 108 bytes, so a control socket under a deep CI workspace path fails to bind at -all — every implementation checks this (the C++ one returns +all; every implementation checks this (the C++ one returns `socket path too long`), and `verify.lua` runs from `mktemp -d` for exactly this reason. ## Who is calling? Ask the kernel, not the packet -The first thing the supervisor does with an accepted connection — before -reading a single command byte — is: +The first thing the supervisor does with an accepted connection, before +reading a single command byte, is: ```cpp ucred cred{}; @@ -101,7 +101,7 @@ if (::getsockopt(conn.get(), SOL_SOCKET, SO_PEERCRED, &cred, &cred_len) == 0) { `SO_PEERCRED` returns the pid, uid, and gid of the peer *as recorded by the kernel at `connect` time*. The client cannot forge it, because the client -never supplies it — this is the difference between authentication and a +never supplies it: this is the difference between authentication and a claim. On the live run every control connection produced a line like: ```console @@ -112,7 +112,7 @@ and `verify.lua` asserts that the uid printed equals `id -u` in all three languages. pmon only logs the identity; the one-line change to *enforce* (compare `cred.uid` against `getuid()` and hang up) is the pattern every socket-activated daemon on your machine uses. Two sharp edges: the pid is -the connector's pid at connect time — the peer may have exited or the pid +the connector's pid at connect time; the peer may have exited or the pid been reused by the time you act on it (chapter 13's pidfd argument, again), and stronger per-message guarantees exist via `SCM_CREDENTIALS` if you need them. @@ -120,43 +120,43 @@ them. ## What actually crosses the socket `SCM_RIGHTS` is routinely described as "sending a file descriptor", and that -phrase hides the mechanism. The number does not travel — fd 3 in the sender +phrase hides the mechanism. The number does not travel: fd 3 in the sender arrived as fd 4 in the receiver on the traced run below. What the kernel does is precise in chapter 7's terms: it takes the sender's fd-table slot, follows it to the **open file description**, allocates a *new slot in the receiver's fd table* (lowest free, as always), and points it at that same -description. The result is exactly what `dup(2)` produces — shared offset, -shared status flags, independent close — except the two references live in +description. The result is exactly what `dup(2)` produces (shared offset, +shared status flags, independent close) except the two references live in different processes. The inode layer never moves either; it was always system-wide. {% include excalidraw.html file="19-scm-rights-transfer" alt="Two fd-table bands over a shared kernel band: sender slot 3 with close-on-exec set at open, and receiver slot 4 labeled new lowest free with close-on-exec set by MSG_CMSG_CLOEXEC, both pointing at one dark open-file-description box holding pos 47 and flags 02102002, which points down to inode 95934 for pmon.log; sendmsg with cmsg_data 3 names the sender slot, an amber dashed arrow shows the kernel installing the new receiver slot, and recvmsg reports cmsg_data 4." - caption="Figure 19.2 — SCM_RIGHTS in chapter 7's three layers: two per-process slots, one open file description, one inode; pos, flags, inode, and both fd numbers are from the held-open run below" %} + caption="Figure 19.2: SCM_RIGHTS in chapter 7's three layers: two per-process slots, one open file description, one inode; pos, flags, inode, and both fd numbers are from the held-open run below" %} That layer choice has practical consequences pmon is built around. The log -is opened `O_RDWR | O_APPEND`, not `O_WRONLY` — flags live on the +is opened `O_RDWR | O_APPEND`, not `O_WRONLY`: flags live on the description, so a write-only description would arrive write-only and `pmctl`'s reads would fail `EBADF`. And `pmctl` reads with `pread` at explicit offsets instead of `read`, because the *offset* is shared too: a `lseek`+`read` would move the supervisor's append position out from under -it. (`O_APPEND` writes ignore the offset, so the writer is safe regardless — +it. (`O_APPEND` writes ignore the offset, so the writer is safe regardless, but the discipline costs nothing and the point generalizes.) ## How the code works The supervision half is chapter 11–13 machinery matured: `spawn_child` forks/execs the command with `setpgid(0,0)` so `stop` can `SIGTERM` the -whole group, `dup2`s the log fd over stdout and stderr, and a reaper — -`std::jthread`, goroutine, or `std::thread` — `wait`s, backs off 300 ms, and +whole group, `dup2`s the log fd over stdout and stderr, and a reaper +(`std::jthread`, goroutine, or `std::thread`) `wait`s, backs off 300 ms, and respawns, guarding a small `State{child, started, restarts, stopping}` with a mutex (C++/Rust condvar, Go channels). What is new is the socket plumbing. `listen_ctl` is the canonical four-step: `unlink` the stale path, `socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0)`, `bind` a `sockaddr_un` (after the 108-byte check), `listen`. The accept loop takes each connection -with `accept4(…, SOCK_CLOEXEC)` — chapter 7's rule that the cloexec bit must -be set atomically applies to *every* fd source, and `accept` is one — +with `accept4(…, SOCK_CLOEXEC)` (chapter 7's rule that the cloexec bit must +be set atomically applies to *every* fd source, and `accept` is one), logs `SO_PEERCRED`, reads one line, and dispatches: `status` formats the shared state, `stop` tears down, `logfd` calls the interesting function. Sending, in the C++ spelling, verbatim from @@ -183,10 +183,10 @@ void send_fd(int sock, int fd) { } ``` -Every line is load-bearing. Control messages ride in an *ancillary buffer* +Every line here matters. Control messages ride in an *ancillary buffer* alongside the normal payload, and the `CMSG_*` macros exist because that buffer has alignment rules you should never hand-compute: `CMSG_SPACE(n)` -sizes the buffer (header + padded data — 24 bytes here), `CMSG_LEN(n)` is +sizes the buffer (header + padded data, 24 bytes here), `CMSG_LEN(n)` is the length recorded *inside* the header (20: header + 4 unpadded), and `CMSG_DATA` locates the aligned payload slot where the fd is `memcpy`ed. Getting `msg_controllen` and `cmsg_len` confused is the classic way to send @@ -196,7 +196,7 @@ and sending at least one byte is the portable way to guarantee the cmsg is picked up by the receiver's next `recvmsg`. The receive side is where the three languages diverge most, so it gets the -tabs — each function verbatim: +tabs, each function verbatim: {% include codetabs.html langs="C++|Go|Rust" %} @@ -274,42 +274,42 @@ fn recv_log_fd(conn: &UnixStream) -> Result { Three details deserve emphasis. First, `MSG_CMSG_CLOEXEC`: the received fd is installed by the kernel, not by any `open` you control, so this flag is -the only atomic way to give it the cloexec bit — the recvmsg-then-`fcntl` +the only atomic way to give it the cloexec bit; the recvmsg-then-`fcntl` window is chapter 7's fork/exec race all over again. Note the layering in Figure 19.2: both slots end up cloexec, but each got the bit *its own way*, because close-on-exec is per-slot, never shared. Second, ownership: the C++ code moves the raw int into the RAII `Fd` immediately, and Rust wraps -it in `OwnedFd::from_raw_fd` behind the crate's only `unsafe` block — the +it in `OwnedFd::from_raw_fd` behind the crate's only `unsafe` block: the comment is the ownership proof that makes it sound; Go documents intent with `defer unix.Close(fd)` at the call site. Third, all three iterate the -control messages rather than assuming the first one is the rights message — +control messages rather than assuming the first one is the rights message, and all three return the byte-identical diagnostic `no SCM_RIGHTS control message in reply` when a plain reply arrives, which is what a `status` response would look like to `recvLogFd`. From there -`log_tail_via_fd` is deliberately mundane: `pread` from offset 0 in 4 KiB -chunks (1 MiB cap — a fragile bound, stated in the source), split lines, +`log_tail_via_fd` is mundane by design: `pread` from offset 0 in 4 KiB +chunks (1 MiB cap, a fragile bound, stated in the source), split lines, print the last three prefixed `via-fd: `. Mundane is the point: the fd works like any other, because it *is* any other fd. ## Errors, three ways The contract is the same as every pmon version: exit 0, runtime failures -exit 1 with one diagnostic line, usage errors exit 2 — and `verify.lua` +exit 1 with one diagnostic line, usage errors exit 2, and `verify.lua` asserts all three per language, 27 behavioral checks each. The interesting split is *which* errors each side owns. `pmctl` treats a failed `connect` as fatal (`pmctl: error: connect /tmp/…/absent.sock: No such file -or directory` — `std::error_code` text in C++, wrapped `%w` chain in Go, +or directory`: `std::error_code` text in C++, wrapped `%w` chain in Go, `anyhow` context in Rust; same errno underneath). The supervisor, by contrast, must survive *its* clients: an unknown command gets an `err unknown command` reply line, `pmctl` unwraps the `err ` prefix and exits 1, and the supervisor keeps serving. A peer that hangs up mid-reply -just ends that connection — `write_all` swallows the error deliberately, +just ends that connection: `write_all` swallows the error by design, because there is nobody left to report it to, and the loop `accept`s the next caller. `EINTR` is retried at every blocking call in the C++ (`read_command`, `write_all`, `waitpid`), which the runtimes of Go and Rust handle beneath their standard libraries. And the address-length failure is caught before the syscall in all three, because a 108-byte truncation by -`strncpy` — the traditional C bug — would *bind to the wrong path* rather +`strncpy` (the traditional C bug) would *bind to the wrong path* rather than fail. ## Concurrency lens @@ -319,17 +319,17 @@ loop and the reaper. The subtle race is `stop` versus respawn: `stop` sets `stopping` and SIGTERMs the current group, but the reaper may already be past its backoff and inside `spawn_child`. All three implementations re-check `stopping` *after* the spawn returns and, if the flag flipped, -undo — SIGTERM and reap the child they just created — before signaling +undo (SIGTERM and reap the child they just created) before signaling `child_done`. Without that re-check, `stop` leaks exactly one freshly respawned process, a bug that only bites when the child dies within 300 ms -of a stop. The languages phrase the wait differently — C++ and Rust block +of a stop. The languages phrase the wait differently: C++ and Rust block on a condvar predicate, Go closes a `stopCh` the reaper `select`s against -the backoff timer — but the state machine is identical, which is what makes +the backoff timer, but the state machine is identical, which is what makes the single `verify.lua` possible. Two quieter points: connections are handled serially by design (one accepted fd at a time, so state locking stays trivial and `SO_PEERCRED` lines interleave with nothing), and the log -file is shared by three writers — supervisor and each child generation via -`dup2` — safely, because every line is a single `write` on an `O_APPEND` +file is shared by three writers (supervisor and each child generation via +`dup2`) safely, because every line is a single `write` on an `O_APPEND` description, which the kernel makes an atomic append. One Go footnote: `listener.SetUnlinkOnClose(false)` keeps Go's helpfulness from unlinking the socket before the stop path does it explicitly, and @@ -343,10 +343,10 @@ the socket before the stop path does it explicitly, and ``` The runner (`python3 scripts/test-all-examples.py --only -19-unix-sockets-fd-passing`) reports `PASS PASS PASS` for cpp, go, rust — +19-unix-sockets-fd-passing`) reports `PASS PASS PASS` for cpp, go, rust: 27 behavioral checks per language. Driving the C++ build by hand the way this chapter's evidence was produced (`$WD` is a short `mktemp -d` -directory — remember `sun_path`): +directory, remember `sun_path`): ```bash [host]$ ./cpp/build/release/app supervise --ctl $WD/ctl.sock --log $WD/pmon.log -- \ @@ -369,7 +369,7 @@ Each `pmctl` call also produced a `pmon: ctl connect uid=1000 pid=…` line on the supervisor's stderr; the kill produced `pmon: child pid=2622905 exited status=143` and `pmon: restart 1 child pid=2624872`; and after `stop` the socket file was gone (`test -S` fails) and the child group with it. The -`via-fd:` lines matched `tail -3 $WD/pmon.log` byte for byte — but output +`via-fd:` lines matched `tail -3 $WD/pmon.log` byte for byte, but output equality could still be a program quietly opening the path, so the cross-checks watch the mechanism itself. @@ -385,7 +385,7 @@ u_str ESTAB 0 0 /tmp/p19.t9j0/ctl.sock 16411996 * 16422777 users:(("app",pid=2 u_str ESTAB 0 0 * 16422777 * 16411996 users:(("app",pid=2624432,fd=3)) ``` -Supervisor pid 2622904 holds fd 5 (the accepted socket — Figure 19.1), +Supervisor pid 2622904 holds fd 5 (the accepted socket, Figure 19.1), client pid 2624432 holds fd 3, and the peer columns pair the two inodes. This is the same accounting the supervisor's `SO_PEERCRED` line reported from the other direction: `pmon: ctl connect uid=1000 pid=2624432`. @@ -406,13 +406,13 @@ pread64(4, "", 4096, 47) = 0 ``` (Fields trimmed; the full lines are in the trace files.) The sender named -fd **3**; the receiver got fd **4** — the number is per-process, so it did +fd **3**; the receiver got fd **4**: the number is per-process, so it did not travel. `cmsg_len=20` and `msg_controllen=24` are `CMSG_LEN(4)` and `CMSG_SPACE(4)` from `send_fd`, live. And the very next syscalls show -`pmctl` reading all 47 bytes of log *through fd 4* at explicit offsets — +`pmctl` reading all 47 bytes of log *through fd 4* at explicit offsets, no `openat` anywhere in the trace. -**The shared description, in `/proc` — both processes at once.** `pmctl` +**The shared description, in `/proc`: both processes at once.** `pmctl` normally exits in milliseconds, so to catch both sides holding the fd, the run above injected a 15-second delay after `recvmsg` (`strace -e inject=recvmsg:delay_exit=15000000`). During the window, with @@ -436,18 +436,18 @@ ino: 95934 ``` Field for field equal, and each field pins a layer of the chapter 7 model. -`ino: 95934` on both fds matches `stat` on the path — one inode. `pos: 47` -on both — one *offset*, which only makes sense if there is one open file +`ino: 95934` on both fds matches `stat` on the path: one inode. `pos: 47` +on both: one *offset*, which only makes sense if there is one open file description; 47 is the file's size because every `O_APPEND` write advanced -the shared position, and `pread` never moved it. `flags: 02102002` on both — +the shared position, and `pread` never moved it. `flags: 02102002` on both: one set of status flags, decoding as `O_RDWR` (02) + `O_APPEND` (02000) + `O_LARGEFILE` (0100000) + `O_CLOEXEC` (02000000): the description's flags -crossed intact, while the cloexec bit — per-slot, remember — reads set on +crossed intact, while the cloexec bit (per-slot, remember) reads set on both sides only because each side set its own, one at `open` time, one via `MSG_CMSG_CLOEXEC`. Two fd tables, one description, one inode: each claim in Figure 19.2 observed on a live pair of processes. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the fleet-wide view of this chapter is tracing fd passing across *all* > processes with bcc-tools (`sofdsnoop`) and bpftrace probes on > `unix_stream_sendmsg`, instead of per-process `strace`; bcc-tools are not @@ -457,7 +457,7 @@ in Figure 19.2 observed on a live pair of processes. ## What you learned - `AF_UNIX` gives you three conversation shapes (stream, datagram, - seqpacket) and two address namespaces — a filesystem inode governed by + seqpacket) and two address namespaces: a filesystem inode governed by directory permissions and unlink, or an abstract NUL-prefixed name that `ss -x` renders with `@` and the kernel garbage-collects; `socketpair` skips addressing entirely for related processes. @@ -466,20 +466,20 @@ in Figure 19.2 observed on a live pair of processes. the client claiming anything. - `SCM_RIGHTS` transfers a *reference to the open file description*, not a number: sender fd 3 arrived as receiver fd 4, and `/proc` fdinfo on both - processes simultaneously showed one `pos`, one `flags`, one `ino` — + processes simultaneously showed one `pos`, one `flags`, one `ino`, which is why the log must be `O_RDWR`, why `pmctl` uses `pread`, and why `MSG_CMSG_CLOEXEC` exists. - The cmsg API is exact bookkeeping: `CMSG_SPACE` sizes the buffer, - `CMSG_LEN` fills the header, at least one real payload byte rides along — + `CMSG_LEN` fills the header, at least one real payload byte rides along, and `strace` will show you all of it, decoded, when you get it wrong. Next, **shared memory coordination**: `shmkv` stops copying bytes through -the kernel entirely — one mapped file, a seqlock for consistent reads, and +the kernel entirely: one mapped file, a seqlock for consistent reads, and a futex so readers learn about writes without polling. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host this session: the runner printed 19-unix-sockets-fd-passing PASS PASS PASS (3 passed, 0 failed, 0 skipped; verify.lua 27/27 per language); diff --git a/_docs/20-shared-memory-coordination.md b/_docs/20-shared-memory-coordination.md index f32435d..922ae44 100644 --- a/_docs/20-shared-memory-coordination.md +++ b/_docs/20-shared-memory-coordination.md @@ -2,7 +2,7 @@ title: "Shared-memory coordination: futexes and seqlocks" order: 20 part: "IPC" -description: "The coordination ladder from a 1 ms poll loop through POSIX message queues to the futex — the kernel's one real primitive, sleep/wake on a shared word — plus a seqlock for read-mostly snapshots, process-shared atomics in all three languages, and a measured IPC latency matrix, as shmkv v1 gains watchers that never poll." +description: "The coordination ladder from a 1 ms poll loop through POSIX message queues to the futex, the kernel's one real primitive, sleep/wake on a shared word, plus a seqlock for read-mostly snapshots, process-shared atomics in all three languages, and a measured IPC latency matrix, as shmkv v1 gains watchers that never poll." duration: 60 minutes --- @@ -11,14 +11,14 @@ processes share a page of key-value slots, but a reader who wants to *notice* a change has no option except reading the page again, and again, and again. This chapter fixes that, and the one new idea is smaller than you expect: for coordination between processes, the kernel provides essentially **one -primitive** — `futex(2)`, "sleep until this 32-bit word changes; wake the +primitive**: `futex(2)`, "sleep until this 32-bit word changes; wake the sleepers queued on it." Everything you have ever called a mutex, condition variable, semaphore, or `WaitGroup` is userspace protocol wrapped around that word. `shmkv` v1 builds the protocol in the open: a **seqlock** makes reads of the shared page consistent without ever blocking the writer, and a shared futex turns "something changed" from a question you keep asking into an event -you sleep through. Along the way we climb the whole coordination ladder — -polling, POSIX message queues, futex — and measure every rung. +you sleep through. Along the way we climb the whole coordination ladder +(polling, POSIX message queues, futex) and measure every rung. The code is in `examples/20-shared-memory-coordination/`. `./demo.sh` builds all three implementations and runs the serve/watch/bench demo; the `README.md` @@ -28,11 +28,11 @@ languages share. {% include excalidraw.html file="20-futex-wait-wake" alt="Three lanes: watcher process, kernel, writer process. The watcher's seqlock read finds nothing new and calls futex FUTEX_WAIT on word value 3; the kernel re-checks the word equals 3 and sleeps the task on a wait queue keyed by inode and offset 12. The writer publishes k4 under the seqlock bumping seq 6 to 7 to 8, stores futex word 4 with release ordering, and calls FUTEX_WAKE INT_MAX which returns 1; an amber arrow marks the watcher runnable again, and it back-fills ids from the 8-slot ring. A dashed note explains the EAGAIN lost-wakeup guard." - caption="Figure 20.1 — one shmkv wake end to end: the watcher parks on the mapped word, the writer publishes and wakes, the kernel's wait queue keyed by (inode, offset) connects them; every syscall in this figure appears verbatim in the strace cross-check below" %} + caption="Figure 20.1: one shmkv wake end to end: the watcher parks on the mapped word, the writer publishes and wakes, the kernel's wait queue keyed by (inode, offset) connects them; every syscall in this figure appears verbatim in the strace cross-check below" %} -> **Tools used** — `strace`, `xxd`, `perf`, `/usr/bin/time`, `awk` (over +> **Tools used**: `strace`, `xxd`, `perf`, `/usr/bin/time`, `awk` (over > `/proc//status`), `gcc` (for the comparison harness), `python3` (the -> example runner) — all host; `offcputime`/`futexctn` (bcc-tools, lab VM, +> example runner), all host; `offcputime`/`futexctn` (bcc-tools, lab VM, > exercised in Part 8). Everything host-side is checked by > `scripts/check-host.sh` or ships with Fedora. @@ -42,8 +42,8 @@ Suppose process A has changed shared memory and process B needs to know. The options form a ladder, and this chapter's `bench` subcommand times every rung. **Rung 0: poll.** B re-reads the page in a loop, sleeping between looks so it -does not burn a CPU. Simple, portable — and the sleep *is* the latency. Our -poll channel deliberately sleeps 1 ms *before* checking, so by construction +does not burn a CPU. Simple, portable, and the sleep *is* the latency. Our +poll channel sleeps 1 ms *before* checking by design, so by construction every observation costs at least one full nap; the measured p50 of 1,056 µs is the sleep plus scheduler overhead, and no tuning removes the trade-off: a shorter nap trades latency for wasted wakeups, and every nap is a voluntary @@ -52,14 +52,14 @@ context switch whether or not anything happened. **Rung 1: a kernel object with a queue.** POSIX message queues (`mq_open(2)`, `mq_timedsend(2)`, `mq_timedreceive(2)`) give you a named kernel channel with priorities and blocking receive. They carry *data*, not -just "something happened" — our bench sends the 8-byte update id through the +just "something happened"; our bench sends the 8-byte update id through the queue. The cost is a syscall on every send and every receive, and a second copy of information the shared page already holds. **Rung 2: the futex.** `futex(addr, FUTEX_WAIT, expected, timeout)` says: *if* the 32-bit word at `addr` still holds `expected`, put me to sleep on a kernel wait queue attached to that address; `futex(addr, FUTEX_WAKE, n)` -wakes up to `n` sleepers. The check-then-sleep is atomic against wakers — +wakes up to `n` sleepers. The check-then-sleep is atomic against wakers; that is the entire reason the primitive exists. If the word already moved, `FUTEX_WAIT` returns `EAGAIN` immediately and the caller re-reads shared state: the lost-wakeup race that plagues homemade "check a flag, then sleep" @@ -77,9 +77,9 @@ void futex_wait(std::uint32_t* addr, std::uint32_t expected, } ``` -Two details are load-bearing. There is no glibc wrapper — the raw +Two details matter here. There is no glibc wrapper: the raw `syscall(SYS_futex, …)` *is* the interface, in all three languages. And the -call deliberately omits `FUTEX_PRIVATE_FLAG`: a private futex is keyed by +call omits `FUTEX_PRIVATE_FLAG`: a private futex is keyed by process-local virtual address, which is faster but invisible to other processes; a **shared** futex is keyed by `(inode, offset)` of the mapped file, so two unrelated processes that `mmap` the same `/tmp/kv.shm` sleep and @@ -92,7 +92,7 @@ That is why glibc builds every pthread mutex and condvar on this exact call. The futex says *when* to look; the seqlock makes what you see *consistent*. `shmkv`'s page has one writer and many readers, and the readers vastly -outnumber the writes — the classic seqlock shape (the kernel uses one for +outnumber the writes, the classic seqlock shape (the kernel uses one for `gettimeofday`'s clock data). The writer's side is four lines: ```cpp @@ -107,9 +107,9 @@ void seqlock_write(const Shm& shm, Mutate&& mutate) { ``` The writer bumps the sequence word **twice** per update, and both bumps are -essential. The first makes the word *odd* — a flag meaning "mutation in +essential. The first makes the word *odd*: a flag meaning "mutation in progress; anything you copy now may be torn," so readers that see an odd -value do not even bother copying. The second makes it even again — but at a +value do not even bother copying. The second makes it even again, but at a *different* even value than before. That difference is the whole detection scheme: a reader records the word before copying and compares it after; if a writer ran at any point in between, the word has moved (odd mid-flight, or a @@ -119,7 +119,7 @@ before the writer could see the same value on both sides of a torn copy. The fences pair up across processes: the writer's release fence ensures the slot bytes are visible before the closing bump, and the reader's acquire fence ensures its copies happened before it re-reads the word. Writers never wait, -never take a lock, never care how many readers exist — readers pay with +never take a lock, never care how many readers exist; readers pay with occasional retries, which is exactly the right trade for read-mostly data. The retry loop is bounded at 10⁶ attempts so a corrupt file (a stuck-odd word from a crashed writer) produces `seqlock read livelocked` instead of a hang. @@ -129,7 +129,7 @@ from a crashed writer) produces `seqlock read livelocked` instead of a hang. All of this only works if atomic operations on *mapped* memory are really atomic, and each language has its own rules for saying so. In C++, a `std::atomic` must be constructed, but our words live in a file another -process formatted — so `Shm` binds `std::atomic_ref` onto the raw bytes: +process formatted, so `Shm` binds `std::atomic_ref` onto the raw bytes: ```cpp std::atomic_ref seq() const { @@ -148,10 +148,10 @@ mutates underneath you. Casting `*mut u8` into `&u32` and racing it would be undefined behavior; casting to `&AtomicU32` is the sanctioned spelling, provided the pointer is aligned, live for the lifetime, and every access to those four bytes goes through atomics. Go has no blessed story at all: -`sync/atomic` on a `*uint32` pointing into an `unix.Mmap` slice works — +`sync/atomic` on a `*uint32` pointing into an `unix.Mmap` slice works: since Go 1.19 the memory model documents atomics as sequentially consistent, they compile to the same `lock`-prefixed instructions, the mapping is outside -the GC-managed heap so nothing moves it — but *you* carry the alignment +the GC-managed heap so nothing moves it, but *you* carry the alignment burden (64-bit atomics fault on unaligned addresses on some platforms; our counter at offset 16 of a page is safe), and nothing stops ordinary code from racing the same bytes non-atomically. The comment block at the top of @@ -160,20 +160,20 @@ from racing the same bytes non-atomically. The comment block at the top of > **Sidebar: where is eventfd?** `eventfd(2)` is a lighter kernel counter > that chapter 9's event loops already waited on, and it measures well (2 µs > p50 on this host, same harness). But an eventfd is named by *file -> descriptor*, not by anything in the shared page — to coordinate two +> descriptor*, not by anything in the shared page; to coordinate two > unrelated processes with one you must inherit it across `fork` or pass it > through a UNIX socket with `SCM_RIGHTS`, exactly the machinery chapter 19 > (UNIX sockets and fd passing) built for pmon's `logfd`. The futex needs > none of that: its rendezvous *is* the mapped file. That is the deep -> difference — fd-based channels need an fd-passing bootstrap; a shared +> difference: fd-based channels need an fd-passing bootstrap; a shared > mapping is its own bootstrap. ## How the code works -One page carries everything. The SHKV2 layout — magic at offset 0, the +One page carries everything. The SHKV2 layout (magic at offset 0, the seqlock `u32` at 8, the futex `u32` at 12, a `u64` update counter at 16, and -eight 64-byte slots (24-byte key, 40-byte value, NUL-padded) from offset 64 — -is deliberately two counters, not one. The `u64` is the truth: total updates +eight 64-byte slots (24-byte key, 40-byte value, NUL-padded) from offset 64) +carries two counters by design, not one. The `u64` is the truth: total updates ever published, compared and back-filled from. The `u32` at offset 12 exists because `FUTEX_WAIT` can only watch 32 bits; it holds the low half of the counter and is *only* used as the thing to sleep on. The slots form a ring @@ -187,7 +187,7 @@ the per-language atomic views described above. The publish path in `cmd_serve` is the writer's half of Figure 20.1: under `seqlock_write`, store the slot bytes and the `u64` counter; after the closing bump, store the `u32` futex word with release ordering; then -`FUTEX_WAKE` with `INT_MAX` — wake *every* watcher, since any number of +`FUTEX_WAKE` with `INT_MAX`: wake *every* watcher, since any number of processes may be parked. The order matters: data first (under the seqlock), then the word, then the wake, so a woken watcher is guaranteed to find the word already changed and the data already consistent. @@ -206,7 +206,7 @@ nothing new does it arm the sleep: Read the guard carefully: it re-checks the word *after* deciding the snapshot was stale, and passes the exact observed value as `expected`. If a publish lands between those two lines, the kernel's own compare fails, `FUTEX_WAIT` -returns `EAGAIN`, and the loop re-reads — a wake can be early, late, or +returns `EAGAIN`, and the loop re-reads; a wake can be early, late, or spurious, and the loop is correct anyway, because sleeping is only ever an optimization over re-checking. The 2-second timeout is the same humility applied to crashes: a dead writer costs a watcher at most 2 s per lap until @@ -279,38 +279,38 @@ Same protocol, three idioms: acquire-load the word, refuse odd, copy the counter and all 512 slot bytes with a *plain* copy that is allowed to race, fence, re-check. Go leans on its sequentially-consistent atomics instead of explicit fences and yields to the goroutine scheduler rather than the OS. -The copy being torn sometimes is not a bug — it is the design; the re-check +The copy being torn sometimes is not a bug, it is the design; the re-check is what makes it safe to have raced. `cmd_bench` is the measurement harness for the ladder. A consumer thread (`std::jthread`, a goroutine, a `thread::scope` spawn) blocks on one of the -three channels; the producer publishes and — this is the part that keeps the -numbers meaningful — **spins until the consumer acknowledges round `i-1` +three channels; the producer publishes and, this is the part that keeps the +numbers meaningful, **spins until the consumer acknowledges round `i-1` before sending round `i`**, through an `ack` atomic. Without that serialization, a fast producer queues ahead and you measure queue depth, not wakeup latency. The mq channel carries the id through a real POSIX queue -(created `O_EXCL`, unlinked by RAII/defer on every path — `verify.lua` +(created `O_EXCL`, unlinked by RAII/defer on every path; `verify.lua` checks `/dev/mqueue` is clean afterwards); Go builds `mq_open`, `mq_timedsend`, `mq_timedreceive`, and `mq_unlink` from raw `Syscall6` -because glibc's wrappers live in librt — and the kernel, it turns out, wants +because glibc's wrappers live in librt, and the kernel, it turns out, wants the queue name *without* the leading slash glibc requires, which the strace below confirms. -Fragile bits, stated plainly: one writer per file is an unchecked assumption -(two servers would interleave seqlock bumps and corrupt each other — a +A few fragile points: one writer per file is an unchecked assumption +(two servers would interleave seqlock bumps and corrupt each other; a `flock` would fix it and chapter 25's futex-built locks are the deeper answer); the futex word is 32 bits, so a watcher blind for exactly 2³² updates would miss a wake until the 2 s timeout catches it; the ring back-fills at most 8 ids, and a watcher asleep for more sees only the latest -8; and the bench's `p50_us=0` for futex is truncation, not magic — the +8; and the bench's `p50_us=0` for futex is truncation, not magic: the serialized handshake means the consumer often observes the word before it ever reaches the syscall, so the fast path is a sub-microsecond atomic load. ## Errors, three ways The shared contract is exit 2 for usage, exit 1 with a `shmkv:`-prefixed -line on stderr for runtime failures — and unlike fwatch's byte-identical -diagnostics, this contract deliberately pins only the prefix and the codes, +line on stderr for runtime failures, and unlike fwatch's byte-identical +diagnostics, this contract pins only the prefix and the codes, so each language's errno formatting shows through: C++ formats `strerror(errno)` into `std::expected`'s error string (`shmkv: open /tmp/nope.shm: No such file or directory`), Go's wrapped `%w` chain @@ -319,7 +319,7 @@ lower-cases it (`…: no such file or directory`), and Rust's error 2)`). All three were exercised this session and all exited 1. Domain errors the kernel cannot detect get one shared spelling: a file that exists but lacks the magic prints `bad magic (want SHKV2)` identically everywhere, -and the futex-specific errnos — `EAGAIN`, `ETIMEDOUT`, `EINTR` — are not +and the futex-specific errnos (`EAGAIN`, `ETIMEDOUT`, `EINTR`) are not errors at all here; every implementation swallows them and re-reads shared state, because the loop, not the syscall, owns correctness. @@ -329,14 +329,14 @@ Everything in this chapter is the concurrency lens, but the Go caveat deserves its own light. `FUTEX_WAIT` is a blocking syscall: it parks the calling **OS thread** in the kernel, where the goroutine scheduler cannot reach it. The runtime's sysmon notices the stall and hands the thread's P to -another thread, so the program keeps running — but every concurrent futex +another thread, so the program keeps running, but every concurrent futex waiter pins one kernel thread for its whole sleep. One watcher per process is fine; a thousand goroutines each parked in `FUTEX_WAIT` would be a thousand OS threads, which is precisely the degenerate case channels and the netpoller exist to avoid. The comment at the top of `go/main.go` documents this trade so the pattern is not copied blindly. The second thing worth -naming: the memory-ordering vocabulary of chapter 6 — acquire, release, -fences — crossed a process boundary in this chapter without changing at all. +naming: the memory-ordering vocabulary of chapter 6 (acquire, release, +fences) crossed a process boundary in this chapter without changing at all. Cache coherence does not know what a process is; orderings pair between any two threads that touch the same physical page, whichever address spaces they live in. That is why `seqlock_write`'s release fence in a Go server orders @@ -364,7 +364,7 @@ observed update 6: k6=value-6 watch: complete events=6 ``` -The contract is the page plus the futex, not the runtime — the Go process's +The contract is the page plus the futex, not the runtime: the Go process's `FUTEX_WAKE` lands in the Rust process's `FUTEX_WAIT`. Afterwards the file itself reads out the whole protocol: @@ -378,10 +378,10 @@ itself reads out the whole protocol: 00000050: 0000 0000 0000 0000 7661 6c75 652d 3100 ........value-1. ``` -Little-endian, offset 8: the seqlock word is `0x0c` = 12 — exactly two bumps +Little-endian, offset 8: the seqlock word is `0x0c` = 12, exactly two bumps for each of six publishes, resting even. Offset 12: futex word 6. Offset 16: `u64` counter 6. Offset 64: slot 0 still holds `k1`/`value-1`, id 1 having -landed at `(1-1) % 8`. Then the ladder, measured — 2,000 serialized rounds +landed at `(1-1) % 8`. Then the ladder, measured: 2,000 serialized rounds per channel, C++ bench: ```bash @@ -393,9 +393,9 @@ bench: channel=poll rounds=2000 p50_us=1056 p99_us=1083 The Go and Rust benches bracket the same values (Go: 0/4, 3/6, 1003/1021; Rust: 0/0, 2/5, 1061/1153 µs). Folding in the fd-based channels from -chapters 18 and 19 — pipe, UNIX stream socketpair, and the eventfd from the -sidebar, re-measured this session with the identical serialized-round -harness rebuilt around `read(2)` — gives the matrix: +chapters 18 and 19 (pipe, UNIX stream socketpair, and the eventfd from the +sidebar), re-measured this session with the identical serialized-round +harness rebuilt around `read(2)`, gives the matrix: | Channel | Kernel object | p50 (µs) | p99 (µs) | Waiter blocks in | Chapter | |---|---|---|---|---|---| @@ -409,17 +409,17 @@ harness rebuilt around `read(2)` — gives the matrix: {% include excalidraw.html file="20-ipc-matrix" alt="A ladder of six boxes ordered left to right by latency: futex wake p50 under 1 microsecond p99 4, eventfd 2 and 6, pipe 2 and 5, POSIX mq 2 and 14, UNIX socket 3 and 12, then a dark box for the 1 millisecond poll loop at p50 1056 and p99 1083 microseconds with an amber arrow labeled roughly 350x. Below, two summary boxes: the scheduler evidence of 8 versus 3774 voluntary context switches in a 4 second window, and rusage counts of 120 versus 2002 voluntary switches over 2000 rounds." - caption="Figure 20.2 — the coordination ladder as measured on this host: every blocking channel lands within one order of magnitude, and the poll loop pays its nap" %} + caption="Figure 20.2: the coordination ladder as measured on this host: every blocking channel lands within one order of magnitude, and the poll loop pays its nap" %} -Two takeaways, plainly bounded by this host (an idle 8-core/16-thread Fedora +Two takeaways, bounded by this host (an idle 8-core/16-thread Fedora 44 laptop; treat the rankings as portable and the absolutes as local). The -fast one: *every* blocking channel — futex, pipe, eventfd, mq, UDS — lands +fast one: *every* blocking channel (futex, pipe, eventfd, mq, UDS) lands in single-digit microseconds at p50; within that band, choose by shape (named vs fd-based, data-carrying vs pure signal), not speed, though the futex is alone in costing zero syscalls when nothing is contended. The slow one: the poll loop is ~350× worse at p50 than the slowest blocking channel and over 1,000× worse than the futex, and it cannot be fixed, because its -latency is its sleep — the only knob trades latency against wasted wakeups. +latency is its sleep; the only knob trades latency against wasted wakeups. ## Cross-check, three ways @@ -435,11 +435,11 @@ own private-futex noise trimmed), during a three-update serve: 2622774 +++ exited with 0 +++ ``` -The address ends in `0x00c` — offset 12 in the mapped page, exactly where -the layout puts the futex word — the op is plain `FUTEX_WAIT` (shared, no +The address ends in `0x00c` (offset 12 in the mapped page, exactly where +the layout puts the futex word), the op is plain `FUTEX_WAIT` (shared, no `_PRIVATE`), and `expected` climbs 0, 1, 2 as each observed update re-arms the next sleep. The serve side, traced in a separate run, is the mirror -image: `futex(0x7f79491d300c, FUTEX_WAKE, 2147483647) = 1` three times — wake +image: `futex(0x7f79491d300c, FUTEX_WAKE, 2147483647) = 1` three times: wake `INT_MAX`, one waiter actually woken. And the mq bench under `-e trace=mq_open,mq_timedsend,mq_timedreceive,mq_unlink`: @@ -451,11 +451,11 @@ image: `futex(0x7f79491d300c, FUTEX_WAKE, 2147483647) = 1` three times — wake 2623348 mq_unlink("shmkv-bench-2623348") = 0 ``` -The kernel-level name really has no leading slash — the Go implementation's -raw-syscall comment, confirmed against the C++ binary going through glibc. +The kernel-level name really has no leading slash, confirming the Go +implementation's raw-syscall comment against the C++ binary going through glibc. Producer tid 2623348 sends, consumer tid 2623349 receives the same 8 bytes, and the unlink runs on exit. (That traced 5-round run reported -`p50_us=19` — tracing overhead is real; never quote numbers from under +`p50_us=19`: tracing overhead is real; never quote numbers from under `strace`.) **The scheduler's ledger, in `/proc`.** `voluntary_ctxt_switches` counts how @@ -468,23 +468,23 @@ voluntary_ctxt_switches: 2 voluntary_ctxt_switches: 10 ``` -Eight sleeps in four seconds — one per update observed, nothing else. The +Eight sleeps in four seconds: one per update observed, nothing else. The poll-channel bench over the same window (summing all threads via `/proc//task/*/status`): 285 → 4,059, i.e. **3,774** voluntary switches -in 4 s, ~943/s — the 1 ms nap plus overhead, matching the measured 1,056 µs +in 4 s, ~943/s, the 1 ms nap plus overhead, matching the measured 1,056 µs p50 almost exactly. Whole-process rusage agrees: `/usr/bin/time -v` on 2,000 rounds reports 120 voluntary switches for the futex channel (the serialized -handshake usually publishes before the consumer reaches the syscall — the +handshake usually publishes before the consumer reaches the syscall, the p50 = 0 fast path, counted) against 2,002 for poll: one nap per round, plus startup, by construction. `perf stat -e context-switches` was exercised on both runs but on this host counted zero: stock Fedora ships `kernel.perf_event_paranoid = 2`, which restricts unprivileged perf to -user-space counting, and context switches happen in the kernel — the +user-space counting, and context switches happen in the kernel; the elapsed-time line still tells the story (0.004 s futex vs 2.125 s poll for the same 2,000 rounds), and the chapter 28 debugging toolbelt revisits perf with `CAP_PERFMON`. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the eBPF view of this chapter is `offcputime` proving the watcher's stack > is parked in `futex_wait_queue` (not spinning) and `futexctn` summarizing > contention fleet-wide; bcc-tools need privileges this host's lane does not @@ -493,13 +493,13 @@ with `CAP_PERFMON`. ## What you learned -- The kernel's one coordination primitive is the futex — atomically - check-then-sleep on a 32-bit word, wake its queue — and a *shared* futex +- The kernel's one coordination primitive is the futex: atomically + check-then-sleep on a 32-bit word, wake its queue; and a *shared* futex keyed by `(inode, offset)` lets unrelated processes rendezvous through a mapped file with zero syscalls on the uncontended path. - A seqlock suits read-mostly shared data: the writer bumps the sequence word twice (odd = in progress, and the closing bump makes any overlap - detectable), readers copy optimistically and retry on mismatch — torn + detectable), readers copy optimistically and retry on mismatch: torn copies are designed-for, discarded, and bounded. - Process-shared atomics have per-language rules: `std::atomic_ref` and `AtomicU32::from_ptr` bind atomic operations (and, in Rust, the aliasing @@ -508,7 +508,7 @@ with `CAP_PERFMON`. OS thread per waiting goroutine. - Measured on this host, every blocking IPC channel notifies in single-digit microseconds at p50 while a 1 ms poll loop pays three orders of magnitude - more — and `/proc`'s `voluntary_ctxt_switches` (8 vs 3,774 in the same + more, and `/proc`'s `voluntary_ctxt_switches` (8 vs 3,774 in the same 4 s) is the scheduler's own testimony as to why. Next, **TCP servers**: the socket leaves the machine, and listen/accept, @@ -516,7 +516,7 @@ backlogs, and `SO_REUSEPORT` decide who gets the connection. --- -

verified — every number and +

verified: every number and excerpt above was produced on the Fedora 44 reference host this session: the runner printed 20-shared-memory-coordination PASS PASS PASS (3 passed, 0 failed, 0 skipped); the Go-serve/Rust-watch transcript, the @@ -529,4 +529,4 @@ runs performed now; the strace excerpts are real trimmed output (shared poll 285→4,059 over 4 s; rusage 120 vs 2,002) was sampled live; and the perf_event_paranoid = 2 limitation was observed, not assumed. The bcc-tools callout is unverified as marked, and all latency absolutes are -plainly bounded by this host.

+bounded by this host.

diff --git a/_docs/21-tcp-sockets.md b/_docs/21-tcp-sockets.md index c9185ed..18064e3 100644 --- a/_docs/21-tcp-sockets.md +++ b/_docs/21-tcp-sockets.md @@ -2,21 +2,21 @@ title: "TCP sockets from first principles" order: 21 part: "Networking" -description: "chatterd v0 opens Part 6 with a thread-per-connection TCP chat room: socket/bind/listen/accept4 and sockaddr_in by hand, SO_REUSEADDR and why a fresh serve needs it, TCP as a boundaryless byte stream that forces the length-prefix frame reused for the rest of the book, and the RAII Socket that owns every fd — cross-checked with ss, strace, and the wire bytes themselves." +description: "chatterd v0 opens Part 6 with a thread-per-connection TCP chat room: socket/bind/listen/accept4 and sockaddr_in by hand, SO_REUSEADDR and why a fresh serve needs it, TCP as a boundaryless byte stream that forces the length-prefix frame every later chapter reuses, and the RAII Socket that owns every fd; cross-checked with ss, strace, and the wire bytes themselves." duration: "50 minutes" --- Part 5 spent nine chapters moving bytes between processes that already knew -each other — pipes an ancestor set up, UNIX sockets a stranger found by a +each other: pipes an ancestor set up, UNIX sockets a stranger found by a path on the local machine. `chatterd` opens Part 6 by dropping the last assumption: the peer is on another machine, reachable only by an address and a port, over a transport that can lose, reorder, and re-split everything you -send. This first cut is deliberately small — a thread-per-connection TCP +send. This first cut stays small: a thread-per-connection TCP **chat room**, one binary that is both the daemon (`serve`) and the client -(`chatctl`) — so that every primitive underneath a network server is visible -with nothing hidden: `socket`, `bind`, `listen`, `accept4`, a `sockaddr_in` +(`chatctl`), so every primitive underneath a network server stays visible, +with `socket`, `bind`, `listen`, `accept4`, a `sockaddr_in` filled in by hand, and the one idea that TCP forces on every protocol built -on it — that a byte stream has no message boundaries, so *you* must add them. +on it: that a byte stream has no message boundaries, so *you* must add them. The code is in `examples/21-tcp-sockets/`. `./demo.sh` there builds all three implementations; its `README.md` fixes the CLI, the exit codes, and the wire @@ -24,10 +24,10 @@ frame that chapters 22–24 and 27 grow but never replace. {% include excalidraw.html file="21-server-lifecycle" - alt="A top band labeled serve — one listener thread holds five boxes left to right: socket() AF_INET SOCK_STREAM|CLOEXEC, setsockopt SO_REUSEADDR = 1 rebind past TIME_WAIT, bind() sockaddr_in 127.0.0.1:9000, listen(128) passive socket fd 3, and an amber accept4() loop SOCK_CLOEXEC producing new fd 4, 5. Two amber spawn-thread arrows fall from the accept loop into a lower band, one std::jthread or goroutine or std::thread per accepted connection, holding serve_client fd 4 (bob) and serve_client fd 5 (alice). An amber note reads SIGINT arrow g_stop = 1; poll(2) wakes arrow stop accepting; listener fd closes; threads join; exit 0." - caption="Figure 21.1 — chatterd serve's lifecycle: the four setup calls, the accept4 loop, and one thread per accepted fd; every fd number here reappears in the ss and strace captures later" %} + alt="A top band labeled serve: one listener thread holds five boxes left to right: socket() AF_INET SOCK_STREAM|CLOEXEC, setsockopt SO_REUSEADDR = 1 rebind past TIME_WAIT, bind() sockaddr_in 127.0.0.1:9000, listen(128) passive socket fd 3, and an amber accept4() loop SOCK_CLOEXEC producing new fd 4, 5. Two amber spawn-thread arrows fall from the accept loop into a lower band, one std::jthread or goroutine or std::thread per accepted connection, holding serve_client fd 4 (bob) and serve_client fd 5 (alice). An amber note reads SIGINT arrow g_stop = 1; poll(2) wakes arrow stop accepting; listener fd closes; threads join; exit 0." + caption="Figure 21.1: chatterd serve's lifecycle: the four setup calls, the accept4 loop, and one thread per accepted fd; every fd number here reappears in the ss and strace captures later" %} -> **Tools used** — `ss`, `strace`, `python3`, `mktemp`, `grep` (host); +> **Tools used**: `ss`, `strace`, `python3`, `mktemp`, `grep` (host); > `tcpdump`/`perf` (host, where privileged), and bcc-tools > (`tcplife`/`tcpconnect`/`offcputime`) on the lab VM, exercised in Part 8. > Everything host-side is checked by `scripts/check-host.sh` or ships with @@ -65,26 +65,26 @@ Result listen_tcp(const std::string& host, std::uint16_t port) { ``` `socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)` asks for an IPv4 -(`AF_INET`) reliable byte stream (`SOCK_STREAM` — TCP), and folds close-on-exec +(`AF_INET`) reliable byte stream (`SOCK_STREAM`, TCP), and folds close-on-exec into the creating call the same way chapter 7 insisted every fd source must: an `accept4` a moment later has the same discipline, because a listener that leaks into a forked child is a port you can never rebind. The returned fd is -not yet an address — `bind` gives it one. `sockaddr_in` is the IPv4 address -struct, and its two fields carry a trap worth stating once: `sin_port` and +not yet an address: `bind` gives it one. `sockaddr_in` is the IPv4 address +struct, and its two fields carry one trap: `sin_port` and `sin_addr` are **network byte order** (big-endian), which is why the port goes through `htons` (host-to-network-short) and the dotted-quad string goes through `inet_pton` (presentation-to-network) rather than being assigned raw. Skip `htons` on a little-endian x86 box and you bind port 9000 as -0x2823 — 10275 — and spend an afternoon wondering why nobody can connect. +0x2823 (10275) and spend an afternoon wondering why nobody can connect. `inet_pton` here also does double duty as validation: it returns 1 only for a real IPv4 literal, so a hostname or garbage gets the `not an IPv4 address` error instead of a mysterious `bind` failure. Turning a *name* like -`chat.example.com` into a `sockaddr` is a different job — `getaddrinfo(3)`, -which also picks IPv4-vs-IPv6 for you; this example stays numeric on purpose +`chat.example.com` into a `sockaddr` is a different job: `getaddrinfo(3)`, +which also picks IPv4-vs-IPv6 for you. This example stays numeric and leaves resolution to chapter 23. `bind` attaches the address; `listen(fd, 128)` flips the socket from active -to **passive** — it will never `connect` out, only accept in — and sizes the +to **passive** (it will never `connect` out, only accept in) and sizes the *accept backlog*: the queue of connections the kernel completes the handshake for while your code is between `accept` calls. 128 is the number of clients that can be mid-connect before the kernel starts refusing; for a chat room it @@ -98,7 +98,7 @@ socket in `TIME_WAIT` for a couple of minutes, holding the `(local addr, local port, remote addr, remote port)` tuple so a late straggler packet from the old connection cannot be mistaken for the new one. For a server that means: kill `serve`, restart it within that window, and `bind` fails with -`EADDRINUSE` — "Address already in use" — because a lingering client +`EADDRINUSE` ("Address already in use") because a lingering client connection still pins the port. `SO_REUSEADDR` tells the kernel that a `bind` to a port in `TIME_WAIT` is fine, and it is the reason you can Ctrl-C this server and relaunch it immediately. Go sets it for you on every TCP @@ -108,7 +108,7 @@ before `bind`, exactly as the C++ does. ## TCP is a byte stream, so we frame it ourselves -Here is the idea the rest of the book is built on. A UNIX pipe and a TCP +Here is the idea every chapter from here on is built on. A UNIX pipe and a TCP connection are both `SOCK_STREAM`: a reliable, ordered sequence of bytes with **no message boundaries**. If a client sends `build_join("alice")` and then `build_msg("hi")` back to back, the server's next `read` may return all of @@ -128,16 +128,16 @@ opens with a 2-byte magic `0x43 0x48` (ASCII `"CH"`), a 1-byte version {% include excalidraw.html file="21-frame-format" alt="A row of 21 byte cells for the server's DELIVER broadcast of alice's message to bob. The first six cells, amber, hold 43 48 01 03 00 0f labeled magic CH, version 0x01, type 0x03 DELIVER, length (u16BE) = 15. The next five cells hold 61 6c 69 63 65 labeled name = alice. One dark cell holds 00 labeled NUL. Nine cells hold 68 65 6c 6c 6f 20 62 6f 62 labeled text = hello bob. Below: 00 0f = 0x000f = 15 = len(alice) + 1 (NUL) + len(hello bob). Then the frames that got here: alice's own MSG frame (type 0x02) carried just hello bob as its payload, no name; JOIN (type 0x01) payload is name alone, sent once at connect. A footnote notes length is u16BE, 0..65535, counts payload bytes only, and the 6 header bytes are never counted; WELCOME (0x04) and PING (0x05) are declared and reserved but neither sent nor expected in this chapter." - caption="Figure 21.2 — the wire frame byte for byte: the DELIVER broadcast of alice's message to bob; 43 48 01 03 00 0f is the exact magic+version+type+length header captured off the socket in the cross-check below" %} + caption="Figure 21.2: the wire frame byte for byte: the DELIVER broadcast of alice's message to bob; 43 48 01 03 00 0f is the exact magic+version+type+length header captured off the socket in the cross-check below" %} The `type` byte is what makes the NUL separator mostly disappear: a **JOIN** frame's payload is just the sender's `name`; a **MSG** frame's payload is the chat `text` alone, nothing else; the server relays every MSG as a **DELIVER** -frame whose payload is `name` `0x00` `text` — the *only* frame that still +frame whose payload is `name` `0x00` `text`, the *only* frame that still needs a NUL, because it is the only one carrying two fields to a reader who never sent them itself. `WELCOME` and `PING` are declared and reserved for chapters 22 and 24 but never sent or read here. The length prefix means the -reader never scans for a delimiter — it reads exactly the 6 header bytes, +reader never scans for a delimiter: it reads exactly the 6 header bytes, learns the type and size from them, reads exactly that many payload bytes, and is done. There is no separate "reject an oversized frame" check the way an unbounded length would need: `length` is 16 bits, so 65535 bytes is the @@ -220,27 +220,27 @@ fn read_frame(r: &mut R) -> Result)>> { } ``` -The three are the same state machine in three vocabularies. The load-bearing -call is still *read-exactly-n*: C++ hand-rolls it in `read_full`, a loop that +The three are the same state machine in three vocabularies. The call that has +to be right is still *read-exactly-n*: C++ hand-rolls it in `read_full`, a loop that keeps calling `read` until `n` bytes have accumulated, retrying `EINTR` and returning false on EOF; Go and Rust get it from the standard library -(`io.ReadFull`, `read_exact`). This loop is the whole point — a single +(`io.ReadFull`, `read_exact`). This loop is the whole point: a single `read` returning fewer bytes than asked is not an error, it is Tuesday, and the reader that treats a short read as a complete message is the bug the length prefix exists to prevent. The 6-byte header is read whole, then checked before anything in it is trusted: `hdr[0]`/`hdr[1]` must be `'C'`/`'H'` and `hdr[2]` must be the version this build speaks (`0x01`), or the frame is -rejected outright — a stray non-chatterd byte stream, or a future version this +rejected outright: a stray non-chatterd byte stream, or a future version this build doesn't understand, is refused before its length field is ever read as a byte count. Only then are `type` (`hdr[3]`) and the big-endian 16-bit `length` (`hdr[4..6]`, decoded with explicit shifts in C++, `binary.BigEndian` in Go, `u16::from_be_bytes` in Rust) trusted, and the -payload allocated to exactly `length` bytes and read in full — capped at -65535 simply because the field is 16 bits, with no separate size check to +payload allocated to exactly `length` bytes and read in full, capped at +65535 because the field is 16 bits, with no separate size check to remember. None of the three distinguishes a bad magic/version from a short read with a diagnostic in this chapter: a truncated header, a wrong magic, and a truncated payload all end the connection the same quiet way a clean EOF -does — `read_frame` returns `std::nullopt`/an `error`/(mostly) `Ok(None)`, and +does: `read_frame` returns `std::nullopt`/an `error`/(mostly) `Ok(None)`, and the caller simply stops reading that connection. Rust's return type is the one place the distinction survives structurally (`Ok(None)` is EOF, `Err(...)` is a bad magic/version via `bail!`), but `serve_client`'s @@ -254,23 +254,23 @@ id, and hand the connection to a `std::jthread` running `serve_client`. Each connection thread reads frames forever; a JOIN logs `chatterd: joined` and broadcasts a `server\0 joined` DELIVER notice to **all** members, while a MSG is relayed as a DELIVER carrying `\0` to every client -**except** the sender — you already see what you typed. A `Hub` (a +**except** the sender: you already see what you typed. A `Hub` (a `std::vector` under a `std::mutex`, a `map[int]*client` in Go, a `Mutex` in Rust) is the shared roster every thread reads and writes. -The RAII `Socket` is the quiet hero. It is a move-only handle that `close`s -its fd in the destructor and nowhere else — there is not a single naked +The RAII `Socket` handles cleanup. It is a move-only handle that `close`s +its fd in the destructor and nowhere else; there is not a single naked `close` in the C++ program. When `serve_client` returns, its `Socket` parameter dies and the connection closes; when `cmd_serve` returns, the listener `Socket` dies last and the port frees. Ownership *is* lifetime, so -"who closes this fd, and when" is answered by scope rather than by vigilance -— the same lesson chapter 7 drew for files, now load-bearing for a server +"who closes this fd, and when" is answered by scope rather than by vigilance, +the same lesson chapter 7 drew for files, now true for a server that opens and closes a socket per client. ## Errors, three ways The contract is the book's usual triple: exit 0 clean, exit 1 for a runtime -failure with one diagnostic line, exit 2 for a usage error — and one +failure with one diagnostic line, exit 2 for a usage error, and one `verify.lua` asserts all three across the languages (19 behavioral checks each). The instructive case is a client against a dead port. All three report it and exit 1, and the errno underneath is identical (`ECONNREFUSED`) even as @@ -290,7 +290,7 @@ the `std::error_code`; Go wraps with `%w` so the `dial tcp` context chains in; Rust attaches `anyhow` context and prints the bare `os error 111`. A server-side `bind` that loses (a privileged port, say) is the same shape from the other side: `chatterd: error: bind 127.0.0.1:80: Permission denied`, exit 1. Usage -slips — no subcommand, `serve` without `--port`, `chatctl` without `--name` — +slips (no subcommand, `serve` without `--port`, `chatctl` without `--name`) all short-circuit to the two-line usage text and exit 2 before any socket is opened. Inside the connection threads, errors are handled by *ending that one connection*, never the server: a peer that vanishes mid-write makes `write_all` @@ -301,21 +301,21 @@ next caller. `chatterd` v0 is **thread-per-connection**: one OS thread (or goroutine) blocked in `read_frame` per client, plus the listener thread in `accept`. It is the -easiest server model to reason about — each connection is a straight-line -blocking program, and the only shared state is the `Hub` behind a mutex — and +easiest server model to reason about: each connection is a straight-line +blocking program, and the only shared state is the `Hub` behind a mutex, and it is exactly the model that stops scaling, which is why chapter 22 exists. Two cracks are already visible. First, **cost per connection**: a thread is a megabyte or two of stack plus a scheduler entry, so ten thousand idle chat -clients is gigabytes of RAM and a scheduler run queue that thrashes — the +clients is gigabytes of RAM and a scheduler run queue that thrashes: the classic C10K wall. Second, and more subtle, **head-of-line blocking in the broadcast**: the C++ and Rust `broadcast` hold the `Hub` mutex while writing to every client in turn, so one client whose kernel send buffer is full (a slow reader) stalls `write` and freezes *every other* broadcast behind the -lock. Go sidesteps it structurally — each client owns a 64-deep buffered +lock. Go sidesteps it structurally: each client owns a 64-deep buffered channel and a writer goroutine, and `broadcast` does a non-blocking `select { case c.out <- frame: case <-c.done: }`, so a wedged client is -skipped rather than allowed to block the room. That contrast — a mutex held -across I/O versus a bounded queue per consumer — is the seam chapter 22 opens +skipped rather than allowed to block the room. That contrast, a mutex held +across I/O versus a bounded queue per consumer, is the seam chapter 22 opens up when it moves the server onto one epoll loop and a thread pool. For now, the three shutdown strategies all reach the same observable end: C++ interrupts `poll` with SIGINT and lets each `std::jthread`'s `stop_callback` shut its @@ -332,7 +332,7 @@ three close the listener and exit 0. `python3 scripts/test-all-examples.py --only 21-tcp-sockets` builds all three and reports `PASS PASS PASS` (3 passed, 0 failed, 0 skipped; `verify.lua` 19/19 per language). To drive it by hand the way this chapter's evidence was -produced, start a room and connect two clients — bob first, so he witnesses +produced, start a room and connect two clients, bob first, so he witnesses alice arrive: ```bash @@ -373,9 +373,9 @@ LISTEN 0 128 127.0.0.1:9000 0.0.0.0:* users:(("app",pid=2816579,fd=3)) ``` `LISTEN`, the `128` backlog we passed to `listen`, the bound `127.0.0.1:9000`, -and `fd=3` in process `app` — Figure 21.1's listener box, live. **Second, -`ss -tnp` during a two-client session** shows four established sockets — two -in the server, one in each client — and the peer columns pair them by the +and `fd=3` in process `app`: Figure 21.1's listener box, live. **Second, +`ss -tnp` during a two-client session** shows four established sockets, two +in the server, one in each client, and the peer columns pair them by the ephemeral ports the kernel assigned the clients: ```console @@ -386,13 +386,13 @@ ESTAB 0 0 127.0.0.1:47504 127.0.0.1:9001 users:(("app",pid=2818649,fd=3)) ESTAB 0 0 127.0.0.1:47512 127.0.0.1:9001 users:(("app",pid=2818655,fd=3)) ``` -Server pid 2818525 holds `fd 4` and `fd 5` — the two `accept4` results from -Figure 21.1 — each paired with a client's ephemeral port (47504 for bob, +Server pid 2818525 holds `fd 4` and `fd 5`, the two `accept4` results from +Figure 21.1, each paired with a client's ephemeral port (47504 for bob, 47512 for alice), while each client holds its `fd 3` connecting side. **Third, the frame bytes themselves.** `tcpdump -i lo` reads them off the wire, but the capture socket needs `CAP_NET_RAW`; on this unprivileged host the identical bytes are visible one layer down, at the `read`/`write` syscalls, under -`strace` — the exact sequence the socket handed to (and took from) TCP during +`strace`: the exact sequence the socket handed to (and took from) TCP during alice's `hello bob`: ```console @@ -407,26 +407,26 @@ write(4, "CH\1\3\0\17alice\0hello bob", 21) = 21 ``` Every claim in Figure 21.2 is on those lines (bob connected first onto `fd 4`; -his own JOIN read and the join-notice broadcasts are elided above for space — +his own JOIN read and the join-notice broadcasts are elided above for space; they carry the same shape, just a different type and length). `accept4` returns fds 4 and 5, each `SOCK_CLOEXEC` as the source asked. The reader -takes the fixed 6-byte header first — `read(5, "CH\1\1\0\5", 6)` is `43 48` +takes the fixed 6-byte header first: `read(5, "CH\1\1\0\5", 6)` is `43 48` (`"CH"`), `01` (version), `01` (type = JOIN), `00 05` (length = 5, -big-endian) — then exactly that many payload bytes (`"alice"`), a strict +big-endian), then exactly that many payload bytes (`"alice"`), a strict two-phase read that is the length-prefix discipline in the raw. alice's chat line repeats the shape with a different type: `"CH\1\2\0\t"` is -`43 48 01 02 00 09` — type `02` = MSG, length 9 — followed by the 9-byte +`43 48 01 02 00 09`, type `02` = MSG, length 9, followed by the 9-byte payload `"hello bob"`, with no name in it at all; the connection already told the server who she is. The server's relayed DELIVER, `write(4, "CH\1\3\0\17alice\0hello bob", 21)`, is the whole broadcast in one write: header `43 48 01 03 00 0f` (type `03` = DELIVER, length 15 = `len("alice")` -`+ 1 (NUL) + len("hello bob")`), then the 15-byte payload `alice\0hello bob` -— the NUL still separating name from text, because DELIVER is the only type -that carries both. That length prefix, octal `\17`, is `0f` in hex — 15 — +`+ 1 (NUL) + len("hello bob")`), then the 15-byte payload `alice\0hello bob`, +the NUL still separating name from text, because DELIVER is the only type +that carries both. That length prefix, octal `\17`, is `0f` in hex, 15, the same 6-byte magic/version/type/length header that would show up in a `tcpdump` capture off `lo`, byte for byte the amber header of Figure 21.2. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the fleet view of these connections is bcc-tools, not `strace`: > `tcpconnect`/`tcplife` trace every TCP session on the box with its bytes and > lifetime, and `offcputime` shows the connection threads parked in @@ -437,21 +437,21 @@ the same 6-byte magic/version/type/length header that would show up in a ## What you learned -- A server socket is four ordered calls — `socket` → `bind` → `listen` → - `accept4` — with `sockaddr_in` in network byte order (`htons`, `inet_pton`) +- A server socket is four ordered calls (`socket` → `bind` → `listen` → + `accept4`) with `sockaddr_in` in network byte order (`htons`, `inet_pton`) and `SO_REUSEADDR` set before `bind` so a restart can rebind a port still held in `TIME_WAIT`; `ss -tlnp` showed the passive socket with backlog 128 on `fd 3`, and `ss -tnp` the four established halves of a two-client session with their paired ephemeral ports. - TCP is a boundaryless byte stream: a `read` can return part of a frame or - several, so every message opens with a fixed 6-byte header — magic `CH`, - version, type, then a big-endian `u16` length — and a reader that loops + several, so every message opens with a fixed 6-byte header (magic `CH`, + version, type, then a big-endian `u16` length) and a reader that loops until it has exactly the header, then exactly the payload; `strace` caught that two-phase read and the `43 48 01 03 00 0f` (DELIVER, length 15) header on the wire. - The RAII `Socket` owns every fd by scope, so there is no naked `close`; and - thread-per-connection is the simple model that chapter 22 must leave behind - — a thread and a stack per client, plus a broadcast that holds the roster + thread-per-connection is the simple model that chapter 22 must leave behind: + a thread and a stack per client, plus a broadcast that holds the roster lock across I/O and stalls behind one slow reader. Next, **scaling the server**: chapter 22 takes the same `chatterd` frame off @@ -460,12 +460,12 @@ thousand idle clients cost fds instead of stacks. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host (kernel 7.1, strace 7.1) this session. The runner printed 21-tcp-sockets PASS PASS PASS (3 passed, 0 failed, 0 skipped; verify.lua 19/19 per language). The hand-driven two-client -session produced the exact transcripts shown — bob received +session produced the exact transcripts shown: bob received alice: hello bob, alice saw no echo of her own line, both clients saw a server-sender join notice, the server logged each join and chatterd: shutting down, and SIGINT exited 0. ss -tlnp @@ -473,12 +473,12 @@ showed LISTEN 0 128 127.0.0.1:9000 … fd=3; ss -tnp showed the four established sockets with server pid 2818525 holding fd 4/5 and clients on ephemeral ports 47504/47512. The connect-refused, bind-denied, and usage error strings and exit codes (1/1/2) are real output. The -strace excerpt is real trimmed output re-captured against the +strace excerpt shows trimmed output re-captured against the canonical CH frame: accept4 … = 4 and = 5, the two-phase header-then-payload reads for alice's JOIN (CH\1\1\0\5 then alice) and MSG (CH\1\2\0\t then hello bob), and the server's relayed DELIVER, -write(4, "CH\1\3\0\17alice\0hello bob", 21) — the +write(4, "CH\1\3\0\17alice\0hello bob", 21): the \17 (0f in hex, 15) big-endian length half of the 6-byte magic/version/type/length header. tcpdump on lo was not run (needs CAP_NET_RAW, unavailable in diff --git a/_docs/22-scaling-the-server.md b/_docs/22-scaling-the-server.md index ea104ec..51665c5 100644 --- a/_docs/22-scaling-the-server.md +++ b/_docs/22-scaling-the-server.md @@ -2,13 +2,13 @@ title: "Scaling the server" order: 22 part: "Networking" -description: "chatterd v1 holds many connections from one thread: nonblocking accept and per-connection read/write state machines, an outbound queue with EPOLLOUT backpressure reusing chapter 9's level-triggered epoll — then the chapter's real comparison, Go's goroutine-per-connection model with the runtime netpoller running epoll_pwait underneath, proven with per-process Threads counts, ss -tp, and strace." +description: "chatterd v1 holds many connections from one thread: nonblocking accept and per-connection read/write state machines, an outbound queue with EPOLLOUT backpressure reusing chapter 9's level-triggered epoll, then the chapter's central comparison, Go's goroutine-per-connection model with the runtime netpoller running epoll_pwait underneath, proven with per-process Threads counts, ss -tp, and strace." duration: "55 minutes" --- Chapter 21 gave `chatterd` a length-prefixed frame protocol and the obvious server: one OS thread per connection, each blocked in `recv`. That design is -correct and it is easy to reason about — right up to the point a decade of +correct and it is easy to reason about, right up to the point a decade of network programmers named the **C10K problem**: ten thousand idle clients means ten thousand kernel threads, ten thousand stacks, and a scheduler run queue that spends more time context-switching than working. This chapter keeps the threaded @@ -17,9 +17,9 @@ connections from a *single* thread. The one new idea is the **readiness model from chapter 9, aimed at sockets**: put every fd in nonblocking mode, let one `epoll_wait` tell you which are ready, and make each connection a little state machine with an inbound buffer and an outbound queue so a partial read or a -blocked write is just *state*, never a stalled thread. Then the payoff — Go -writes the naive goroutine-per-connection code and gets the epoll engine for -free, because its runtime hides exactly this machinery underneath. +blocked write is just *state*, never a stalled thread. Then comes the payoff: +Go writes the naive goroutine-per-connection code and the epoll engine comes +with it, because its runtime hides exactly this machinery underneath. The code is in `examples/22-scaling-the-server/`. `./demo.sh` there builds all three implementations and runs an end-to-end demo; its `README.md` specifies the @@ -29,9 +29,9 @@ languages share. {% include excalidraw.html file="22-connection-state-machine" alt="Two bands. The top READ PATH band runs left to right: a recv() loop draining nonblocking until EAGAIN feeds an inbuf that reassembles partial frames across reads, into take_frame which asks whether one whole 6+LEN frame is ready, into MSG to broadcast which appends the frame to every connection's outbuf. The bottom WRITE PATH and BACKPRESSURE band: broadcast drops for each conn into queue() plus flush() which send()s what fits now; on send() EAGAIN the remainder stays in outbuf (the backpressure queue) because the kernel socket buffer is full; an amber arrow labelled full: arm EPOLLOUT runs down-left to an epoll_wait node that later reports EPOLLOUT for this conn, and an amber arrow labelled EPOLLOUT wakeup runs back up-right to flush(), closing the loop. A note reads: level-triggered, EPOLLIN always armed, EPOLLOUT armed only while outbuf is non-empty." - caption="Figure 22.1 — chatterd's epoll engine as one per-connection read/write state machine; the EPOLLOUT backpressure loop is the whole reason a slow client cannot stall the server" %} + caption="Figure 22.1: chatterd's epoll engine as one per-connection read/write state machine; the EPOLLOUT backpressure loop is the whole reason a slow client cannot stall the server" %} -> **Tools used** — `strace`, `ss`, `python3` (host); `perf`, bcc-tools +> **Tools used**: `strace`, `ss`, `python3` (host); `perf`, bcc-tools > `offcputime`/`tcplife`/`tcpconnect` (lab VM, exercised in Part 8). Everything > host-side is checked by `scripts/check-host.sh` or ships with Fedora. @@ -39,16 +39,16 @@ languages share. A blocking server couples one thread to one connection because `recv` *waits*. Break that coupling and a single thread can service everyone, but the price is -that every operation now returns "not yet" instead of blocking — and "not yet" +that every operation now returns "not yet" instead of blocking, and "not yet" has a name: `EAGAIN` (equivalently `EWOULDBLOCK`). The listener is created nonblocking (`SOCK_NONBLOCK` in the `socket` type), so `accept4` returns `EAGAIN` when the backlog is drained; each client socket is nonblocking, so `recv` returns `EAGAIN` when the kernel receive buffer is empty and `send` -returns `EAGAIN` when its *send* buffer is full. None of those are errors — they +returns `EAGAIN` when its *send* buffer is full. None of those are errors; they are the readiness model telling you to go wait in `epoll_wait` and come back. -Because a nonblocking `recv` can hand you *any* number of bytes — half a frame, -three frames, one frame plus a header — a connection cannot be a straight-line +Because a nonblocking `recv` can hand you *any* number of bytes (half a frame, +three frames, one frame plus a header), a connection cannot be a straight-line function anymore. It has to be **state that survives across `epoll_wait` iterations**. That is what `EConn` is: @@ -63,8 +63,8 @@ struct EConn { }; ``` -`inbuf` accumulates whatever `recv` returns until `take_frame` — the same -6-byte-header parser from chapter 21 — can lift a whole frame off the front; +`inbuf` accumulates whatever `recv` returns until `take_frame` (the same +6-byte-header parser from chapter 21) can lift a whole frame off the front; leftover bytes stay for next time. `outbuf` is the mirror image and the heart of the chapter: bytes we *want* to send but the kernel would not take. `want_write` records whether we currently have `EPOLLOUT` armed for this connection (so we @@ -72,7 +72,7 @@ only call `epoll_ctl` on a real transition), and `dead` marks a connection to reap after the current event batch rather than deleting it mid-iteration. The connections live in `std::unordered_map` keyed by an epoll **token**, `next_tok_` starting at 2 because tokens 0 and 1 are reserved for the -listener and the `signalfd` — the same "let the kernel echo back a small integer +listener and the `signalfd`, the same "let the kernel echo back a small integer so dispatch is a `switch`, not an fd lookup" trick chapter 9 used. ## How the code works @@ -118,13 +118,13 @@ can), and acting on a fd we just decided to drop is a use-after-free waiting to happen; deferring the actual erase to `reap_closed` at the end of the batch is how a single-threaded loop safely mutates the container it is iterating events from. `accept_ready` loops `accept4(…, SOCK_CLOEXEC | SOCK_NONBLOCK)` until -`EAGAIN` — one `EPOLLIN` on the listener can mean *many* pending connections, and +`EAGAIN`: one `EPOLLIN` on the listener can mean *many* pending connections, and draining to `EAGAIN` is the level-triggered discipline from chapter 9 applied to the backlog. Each new connection is registered `EPOLLIN`-only and immediately handed a `WELCOME` frame via `queue`, which is the client's proof it is in the fan-out set. `on_readable` drains `recv` into `inbuf` to `EAGAIN`, then pulls every complete frame: a `JOIN` sets the nick, a `MSG` bumps the counter and -`broadcast`s a `DELIVER` frame — which is just `queue` called for every +`broadcast`s a `DELIVER` frame, which is just `queue` called for every connection. `queue` and `flush` are where backpressure lives, and where the three languages @@ -212,15 +212,15 @@ The C++ and Rust `flush` are the same state machine: drain until the socket says `EAGAIN`/`WouldBlock`, then toggle write-interest to match. When a client stops reading, its kernel send buffer fills, `send` returns `EAGAIN`, the remaining bytes sit in `outbuf`, and `epoll_ctl`/`reregister` arms `EPOLLOUT` so the next -`epoll_wait` wakes us to try again — the amber loop in Figure 22.1. That is +`epoll_wait` wakes us to try again: the amber loop in Figure 22.1. That is backpressure with no thread blocked and no frame dropped: a slow reader grows its own `outbuf` and nobody else's delivery is affected. The only asymmetry is the -library layer — mio's `reregister` takes an `Interest` set where raw epoll takes -a bitmask — but `EPOLLIN|EPOLLOUT` *is* `Interest::READABLE | Interest::WRITABLE` +library layer: mio's `reregister` takes an `Interest` set where raw epoll takes +a bitmask, but `EPOLLIN|EPOLLOUT` *is* `Interest::READABLE | Interest::WRITABLE` underneath. Go's excerpt looks nothing alike, and that is the whole point of the next section: its backpressure is a *blocking channel send*. `c.send` is a buffered channel (capacity 64); when a client's `writePump` can't keep up the -channel fills, the send blocks, and the hub slows down — the same "slow reader +channel fills, the send blocks, and the hub slows down, the same "slow reader slows the writer, never drops" invariant, expressed as goroutines instead of `EPOLLOUT`. @@ -228,36 +228,36 @@ slows the writer, never drops" invariant, expressed as goroutines instead of {% include excalidraw.html file="22-netpoller-vs-epoll" - alt="Two side-by-side bands. Left, titled C++/Rust — you write the epoll loop: one OS thread (Threads: 1) calling epoll_wait (4 calls for a 50-conn flood) connected by a bidirectional amber arrow labelled EPOLLIN/EPOLLOUT + token to a kernel eventpoll interest list holding tok 0 listener, tok 1 signalfd, tok 2..N every client socket, one process; a note reads backpressure = outbuf + EPOLLOUT (explicit). Right, titled Go — you write goroutines; the runtime hides the epoll: a goroutine per conn (read + write) plus hub with channels fanning out over ~7 OS threads total, a bidirectional amber arrow park/resume goroutine down to a dark runtime netpoller (hidden) box running epoll_pwait (165 calls, edge-triggered EPOLLET), then down to the same kernel sockets, one process, N conns (ss -tp); a note reads backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads." - caption="Figure 22.2 — the chapter's central comparison: the epoll loop you write by hand in C++/Rust is the epoll loop Go's runtime writes for you, driven by epoll_pwait one layer down" %} + alt="Two side-by-side bands. Left, titled C++/Rust, you write the epoll loop: one OS thread (Threads: 1) calling epoll_wait (4 calls for a 50-conn flood) connected by a bidirectional amber arrow labelled EPOLLIN/EPOLLOUT + token to a kernel eventpoll interest list holding tok 0 listener, tok 1 signalfd, tok 2..N every client socket, one process; a note reads backpressure = outbuf + EPOLLOUT (explicit). Right, titled Go, you write goroutines; the runtime hides the epoll: a goroutine per conn (read + write) plus hub with channels fanning out over ~7 OS threads total, a bidirectional amber arrow park/resume goroutine down to a dark runtime netpoller (hidden) box running epoll_pwait (165 calls, edge-triggered EPOLLET), then down to the same kernel sockets, one process, N conns (ss -tp); a note reads backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads." + caption="Figure 22.2: the chapter's central comparison: the epoll loop you write by hand in C++/Rust is the epoll loop Go's runtime writes for you, driven by epoll_pwait one layer down" %} -Go's server has no `epoll_create1`, no `epoll_ctl`, no `epoll_wait` — and it +Go's server has no `epoll_create1`, no `epoll_ctl`, no `epoll_wait`, and it still multiplexes thousands of sockets on a handful of threads. Each accepted connection gets two goroutines (a `readPump` and a `writePump`) plus a shared `hub` goroutine that owns the client set and fans `broadcast` out over channels. The `readPump` calls `fr.next()`, which calls `conn.Read`, which *looks* blocking; but `net.Conn` fds are registered with the runtime's **network -poller**, so a would-be-blocking `Read` parks the *goroutine* — not the OS -thread — until the netpoller's `epoll_pwait` reports the socket readable, then +poller**, so a would-be-blocking `Read` parks the *goroutine*, not the OS +thread, until the netpoller's `epoll_pwait` reports the socket readable, then resumes it. The `--engine threaded|epoll` flag on the Go binary changes only the label it prints; both values run this exact code, because for Go there is no second design to write. This is the payoff Figure 22.2 draws: the hand-rolled epoll interest list on the left and the runtime netpoller on the right are the same kernel object, and the `strace` cross-check below shows Go issuing the -`epoll_pwait` calls our C++ loop makes explicit — plus the edge-triggered +`epoll_pwait` calls our C++ loop makes explicit, plus the edge-triggered `EPOLLET` registration, because the runtime, unlike our level-triggered loop, always drains to `EAGAIN` and parks on the transition. ## Errors, three ways The contract is chapter 21's: exit 0 on success, 1 on a runtime failure with one -diagnostic line, 2 on a usage error — and `verify.lua` asserts all three per +diagnostic line, 2 on a usage error, and `verify.lua` asserts all three per language (24 checks each, 72 total). The interesting split is which side owns which error. A client against a dead port fails `connect` and exits 1 with -`chatctl: cannot connect to 127.0.0.1:1: Connection refused` — `std::error_code` +`chatctl: cannot connect to 127.0.0.1:1: Connection refused`: `std::error_code` text in C++, a `%w` chain in Go, `anyhow` context in Rust, the same errno underneath. The *server* must survive its clients: a peer that hangs up mid-write -turns `send` into an error, which just marks that `EConn` dead and reaps it — +turns `send` into an error, which just marks that `EConn` dead and reaps it. `MSG_NOSIGNAL` on every `send` is what stops a dead peer from killing the whole process with `SIGPIPE`. A foreign speaker whose bytes don't start with the `CH` magic desyncs one connection; `take_frame` clears its buffer and the stream is @@ -271,16 +271,16 @@ nonblocking servers spin at 100% CPU or drop frames. Count the threads doing your work, and the three engines separate cleanly. The epoll engine is **one** thread: `conns_` is touched by nobody else, so there are no locks, no atomics, and the event loop serializes every mutation by -construction — the single-threaded design *is* the concurrency strategy. The +construction: the single-threaded design *is* the concurrency strategy. The threaded engine is **1 + N**: a `Registry` behind a mutex, a per-socket write mutex so two broadcasters can't interleave bytes on one wire, and `std::atomic` counters; simple per connection, but every message wakes N reader threads. Go is a bounded handful of runtime threads regardless of connection count, with `pending`-style state confined to the hub goroutine and everything else -communicating by channel — chapter 6's "share memory by communicating" applied to +communicating by channel: chapter 6's "share memory by communicating" applied to I/O. Measured on this host with 50 connections held open: the C++ epoll daemon reports `Threads: 1`, the C++ threaded daemon `Threads: 51`, and the Go daemon -`Threads: 7` (near, but not pinned to, this box's 16 CPUs — the runtime grows +`Threads: 7` (near, but not pinned to, this box's 16 CPUs; the runtime grows threads on demand up to `GOMAXPROCS` runnable). One process, one design decision, three very different rows in `/proc//status`. @@ -304,7 +304,7 @@ alice: hello, chatterd ``` `bob`'s terminal prints the same `alice: hello, chatterd` line, because every -`MSG` is delivered to every connection including the sender — seeing your own +`MSG` is delivered to every connection including the sender; seeing your own echo is how `send` knows it was accepted. On `SIGTERM` the daemon prints `chatterd: stopped engine=epoll messages=1 peak_conns=2`. The load case is `flood`, which opens N concurrent connections, waits for all N `WELCOME`s, has @@ -316,7 +316,7 @@ flood: connected 50 flood: delivered 50 ``` -Fifty connections, one broadcast, fifty deliveries, no dropped frames — and the +Fifty connections, one broadcast, fifty deliveries, no dropped frames, and the daemon's summary reads `messages=1 peak_conns=50`. The threaded engine and all three languages produce byte-identical output; that identity is what lets one `verify.lua` cover them. @@ -353,8 +353,8 @@ Tracing a two-party exchange, one thread does everything: 2825978 epoll_ctl(5, EPOLL_CTL_DEL, 6, NULL) = 0 ``` -`data=0` is the listener token, `data=0x2`/`0x3` the two client tokens — exactly -the integers `accept_ready` assigned — and `accept4` loops to `EAGAIN` before the +`data=0` is the listener token, `data=0x2`/`0x3` the two client tokens, exactly +the integers `accept_ready` assigned, and `accept4` loops to `EAGAIN` before the next `epoll_wait`. Under a 50-connection flood, `strace -c` counts just **4** `epoll_wait` calls covering 51 `accept4`, 100 `sendto`, and 100 `epoll_ctl` for 457 syscalls total: one wait multiplexes an entire batch. **Third, Go's hidden @@ -379,17 +379,17 @@ engine=epoll : clone3 0 epoll_wait 40 accept4 510 total 4570 (0.0 engine=threaded : clone3 500 accept4 500 total 8546 (1.85s in syscalls) ``` -The threaded engine pays **one `clone3` per connection** — 500 thread creations — +The threaded engine pays **one `clone3` per connection**, 500 thread creations, and two orders of magnitude more time in the kernel; the epoll engine creates zero threads and multiplexes the lot through 40 `epoll_wait` calls. That single counter, `clone3: 500` versus `clone3: 0`, *is* the C10K problem in one line. -> **On the lab VM** unverified — -> the kernel-side cost view — `perf stat` context-switches under load (this +> **On the lab VM** unverified: +> the kernel-side cost view, `perf stat` context-switches under load (this > host's `perf_event_paranoid=2` restricts the counter to userspace, reading 0 > without privilege), `offcputime` flame graphs showing threaded workers parked > off-CPU, and `tcplife`/`tcpconnect` tracing every connection's lifetime across -> the fleet — needs root and a disposable kernel, so it runs on the +> the fleet, needs root and a disposable kernel, so it runs on the > `systems-target` VM in Part 8, not here. ## What you learned @@ -398,25 +398,25 @@ counter, `clone3: 500` versus `clone3: 0`, *is* the C10K problem in one line. error but "go wait", and a connection becomes *state* (`inbuf`, `outbuf`) that survives across wakeups rather than a thread blocked in `recv`. - Backpressure is the outbound queue plus `EPOLLOUT`: a slow reader fills its own - `outbuf`, we arm write-interest, and the next `epoll_wait` drains it — no thread + `outbuf`, we arm write-interest, and the next `epoll_wait` drains it: no thread blocks, no frame drops, no other client is affected. Level-triggered epoll from chapter 9 makes this safe with only "consume what woke you" discipline. - Go's goroutine-per-connection code is the same epoll engine with the loop hidden: the netpoller runs `epoll_pwait` (edge-triggered) underneath and parks - goroutines, not threads — 1 vs 51 vs 7 threads in `/proc`, all serving 50 + goroutines, not threads: 1 vs 51 vs 7 threads in `/proc`, all serving 50 connections from one process. - The mechanism is externally visible: `ss -tp` shows every connection on one pid, `strace` shows `epoll_wait` (or the runtime's `epoll_pwait`) driving everything, and `clone3: 0` versus `clone3: 500` measures the cost the whole design exists to avoid. -Next, **two hosts**: `chatterd` v2 leaves loopback for a real network — UDP -multicast discovery to find peers, then a TCP bridge carrying the same frames +Next, **two hosts**: `chatterd` v2 leaves loopback for a real network, using +UDP multicast discovery to find peers, then a TCP bridge carrying the same frames between two VMs on one L2 segment. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host (kernel 7.1, GCC 16.1.1, strace 7.1, 16 CPUs) this session: the runner printed 22-scaling-the-server PASS PASS PASS (3 passed, 0 failed, 0 @@ -428,8 +428,8 @@ peak_conns=2 summary; flood 50 printed flood: connecte 50 connections held open, /proc/<pid>/status read Threads: 1 (C++ and Rust epoll), 51 (C++ threaded), and 7 (Go), while ss -tnp showed all 50 established sockets -owned by the one daemon pid; the C++ epoll strace excerpt is real -trimmed output (single thread, tokens 0/0x2/0x3, accept4 to +owned by the one daemon pid; the C++ epoll strace excerpt shows +trimmed output captured verbatim (single thread, tokens 0/0x2/0x3, accept4 to EAGAIN, 4 epoll_wait calls for a 50-conn flood); the Go trace showed epoll_ctl … EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET and 165 epoll_pwait calls across several threads; and the 500-connection diff --git a/_docs/23-udp-and-peer-discovery.md b/_docs/23-udp-and-peer-discovery.md index aa61afd..470e1f2 100644 --- a/_docs/23-udp-and-peer-discovery.md +++ b/_docs/23-udp-and-peer-discovery.md @@ -2,22 +2,22 @@ title: "UDP and peer discovery" order: 23 part: "Networking" -description: "chatterd learns to find peers it was never told about: UDP multicast beacons on 239.23.7.1, the IP_ADD_MEMBERSHIP / IP_MULTICAST_TTL setsockopt dance, a small gossip protocol deduped by name, and the bridge from a discovered datagram to the canonical chatterd TCP chat frame (introduced ch21) — proven across two libvirt guests with tcpdump, ss -uap, and both transcripts." +description: "chatterd learns to find peers it was never told about: UDP multicast beacons on 239.23.7.1, the IP_ADD_MEMBERSHIP / IP_MULTICAST_TTL setsockopt dance, a small gossip protocol deduped by name, and the bridge from a discovered datagram to the canonical chatterd TCP chat frame (introduced ch21); proven across two libvirt guests with tcpdump, ss -uap, and both transcripts." duration: "55 minutes" --- Chapters 21 and 22 built `chatterd` on TCP: the canonical chatterd chat -frame — magic `"CH"`, version, type, a big-endian length, and a payload — -carried between two endpoints you already knew — you typed the peer's +frame (magic `"CH"`, version, type, a big-endian length, and a payload) +carried between two endpoints you already knew, since you typed the peer's address on the command line. This chapter removes that assumption. `chatterd discover` finds peers it was never told about, and the primitive that makes that possible is the one TCP cannot offer: **UDP multicast**, a single `sendto` that reaches every listener in a group without the sender knowing -who — or how many — they are. The shape of the chapter is a bridge: a +who (or how many) they are. The shape of the chapter is a bridge: a connectionless UDP beacon announces "I exist, dial me here," a receiver hears strangers and dedupes them, and the first time it meets a new name it opens an ordinary TCP connection and speaks the *same canonical chat frame ch21 -introduced* — the header never changes; each version only adds message +introduced*: the header never changes; each version only adds message *types*. Discovery is UDP; the conversation stays TCP. The star of the run is real: two libvirt guests, `systems-target` and `systems-peer`, each discovering the other across the `virbr0` bridge. @@ -29,9 +29,9 @@ beacon and frame formats, and the exit codes all three languages share. {% include excalidraw.html file="23-two-vm-topology" alt="A libvirt virbr0 bridge band on 192.168.124.0/24 holds a central amber multicast-group box for 239.23.7.1 port 51824 TTL 1, with systems-target at 192.168.124.7 on the left and systems-peer at 192.168.124.95 on the right, each joined to the group on enp1s0 and holding udp fd 4 and tcp 9241; two dashed amber bidirectional beacon arrows (35 bytes from target, 34 bytes from peer) reach the group, and a solid bidirectional TCP arrow labelled one ch21 chat frame each way joins the two guests directly." - caption="Figure 23.1 — the two-guest lab topology: both VMs join 239.23.7.1, beacon to the group, and bridge each discovery to a direct TCP :9241 chat frame; every address, port, and byte-length here is from the two-host run later in the chapter" %} + caption="Figure 23.1: the two-guest lab topology: both VMs join 239.23.7.1, beacon to the group, and bridge each discovery to a direct TCP :9241 chat frame; every address, port, and byte-length here is from the two-host run later in the chapter" %} -> **Tools used** — `ss`, `tcpdump`, `ip`, `strace` (host); `tcpdump`, `ss`, +> **Tools used**: `ss`, `tcpdump`, `ip`, `strace` (host); `tcpdump`, `ss`, > `ip maddr` (systems-target / systems-peer VMs, run live in this chapter); > `tcplife`/`tcpconnect`/`offcputime` bcc-tools (lab VM, exercised in Part 8). > Everything host-side is checked by `scripts/check-host.sh` or ships with @@ -49,12 +49,12 @@ many; and you do not care if a single beacon is lost, because another one follows in 200 ms. That is UDP: connectionless, message-oriented, best-effort. `sendto` hands the kernel one datagram, it goes on the wire as one IP packet (or a fragmented set), and the receiver's `recv` returns it whole or not at -all — **message boundaries are preserved**, unlike a TCP stream where you must +all: **message boundaries are preserved**, unlike a TCP stream where you must re-frame yourself. Loss and reorder are possible and, for a beacon, harmless: a dropped beacon just means "discovered one round later." -The datagram stays small on purpose. `chatterd`'s beacon is one ASCII line — -`CHATTERD1 ` — 34 or 35 bytes on the wire in the +The datagram stays small: `chatterd`'s beacon is one ASCII line +(`CHATTERD1 `), 34 or 35 bytes on the wire in the run below, far under the ~1472-byte non-fragmenting payload of a 1500-byte MTU. Keeping a beacon inside one un-fragmented packet is a real discipline: UDP has no reassembly guarantee beyond IP's, so a beacon that fragments turns @@ -69,20 +69,20 @@ only to sockets that have *joined* that group, via `IP_ADD_MEMBERSHIP`. Three more socket options shape the send: `IP_MULTICAST_IF` pins which interface the beacon leaves by (it matters on a multi-homed host), `IP_MULTICAST_TTL = 1` keeps it on the local link so no router forwards it, and `IP_MULTICAST_LOOP` -controls whether *this host's* other sockets also receive it — the switch that +controls whether *this host's* other sockets also receive it, the switch that makes the local two-process demo work over loopback and is irrelevant across two machines, where the packet is on the wire regardless. {% include excalidraw.html file="23-discovery-sequence" alt="Two horizontal bands. The upper blue band, UDP multicast discovery, holds three boxes left to right: announce loop sendto to group emitting CHATTERD1 target 9241; receive loop recv one datagram splitting four fields and checking magic; and an amber parse-and-dedup box that skips its own name and calls seen.first before dialing. An amber arrow labelled new peer bridge to TCP drops into the lower grey band, TCP bridge, whose boxes are connect with retry, write_frame slash read_frame with a 6-byte header (magic, version, type, 2-byte big-endian length) plus payload, and a dark stdout box printing discovered peer and peer says. A note reads loss and reorder tolerated." - caption="Figure 23.2 — one peer's path from a connectionless beacon to a connection-oriented frame: recv, dedup by name, then dial and speak the canonical chat frame ch21 introduced (JOIN out, DELIVER back)" %} + caption="Figure 23.2: one peer's path from a connectionless beacon to a connection-oriented frame: recv, dedup by name, then dial and speak the canonical chat frame ch21 introduced (JOIN out, DELIVER back)" %} ## How the code works The multicast socket is the heart of the chapter, and all three languages run -the same four-option dance around it — join the group, pin the outgoing -interface, enable loopback, set TTL 1 — differing only in how thick the +the same four-option dance around it (join the group, pin the outgoing +interface, enable loopback, set TTL 1), differing only in how thick the library wrapper is. Here it is verbatim from `examples/23-udp-and-peer-discovery/`: @@ -163,49 +163,49 @@ Three spellings, one syscall sequence. The C++ version is the raw form and shows every step: bind to `INADDR_ANY` on the group port (you bind the *port*, not the group address, so every group joined on that port arrives), then the four `setsockopt` calls. `SO_REUSEADDR` matters because two -`chatterd` processes on one host must both bind the same UDP port — without it +`chatterd` processes on one host must both bind the same UDP port; without it the second `bind` fails `EADDRINUSE`. The `ip_mreq` carries two addresses: `imr_multiaddr` is the group to join, `imr_interface` is the local interface -address to join it *on* — get that wrong on a multi-homed box and the kernel +address to join it *on*: get that wrong on a multi-homed box and the kernel drops group traffic before your socket ever sees it. Rust's `socket2` collapses the same calls into named methods (`join_multicast_v4`, `set_multicast_ttl_v4`) -and then `into()`s the socket to a plain `std::net::UdpSocket` — an `OwnedFd` +and then `into()`s the socket to a plain `std::net::UdpSocket`, an `OwnedFd` it can share across threads. Go splits the roles: the *receiver* is built by `net.ListenMulticastUDP` (which does the join for you), while the beacon *sender* is this separate raw `unix.Socket` with the three send-side options -set explicitly — the standard library has no one-call multicast sender, so it +set explicitly: the standard library has no one-call multicast sender, so it drops to `golang.org/x/sys/unix`. Note the `SO_RCVTIMEO` / `set_read_timeout` of 200 ms on every receive socket: it is not about beacons, it is the escape hatch that lets each blocking `recv` wake periodically to re-check the stop flag at shutdown. The rest is small. The **announce loop** (in `main`/`discover`) formats the -beacon once — `CHATTERD1 ` — and `sendto`s it to the +beacon once (`CHATTERD1 `) and `sendto`s it to the group `--rounds` times, `--announce-ms` apart. The **receive loop** reads a datagram, splits on whitespace, requires exactly four fields with `tok[0] == "CHATTERD1"`, ignores any beacon whose name equals its own, parses the peer's port and IP, and consults a dedup set so a peer is announced *once* no matter how many of its beacons arrive. On first sight it prints `discovered peer at :`, then bridges to TCP: `dial` opens a connection to the -advertised `host-ip:tcp-port` — with a bounded retry, because the beacon can -arrive a hair before the peer's listener is up — `write_frame` sends a +advertised `host-ip:tcp-port` (with a bounded retry, because the beacon can +arrive a hair before the peer's listener is up), `write_frame` sends a canonical **JOIN** frame whose payload is its own name, `read_frame` reads -the peer's **DELIVER** reply (payload `\0` — name, a NUL byte, +the peer's **DELIVER** reply (payload `\0`: name, a NUL byte, then the greeting text), splits it on the NUL, and prints `peer says: `. The **accept loop** on the TCP listener is the mirror image: read the dialer's JOIN frame, reply with a DELIVER frame carrying its own name + NUL + greeting text, close. Both peers discover each other independently, so each dials once and each prints exactly one `peer … says:` line. The `` field is the one fragile -convention worth flagging: a peer advertises the address it wants to be dialed +convention to flag: a peer advertises the address it wants to be dialed at (its `--iface`), so behind NAT or on a different subnet it would advertise -an unreachable address — a limitation a real gossip protocol solves with +an unreachable address, a limitation a real gossip protocol solves with observed source addresses, out of scope here. ## Errors, three ways The contract matches every `chatterd` version: exit 0 on success, 1 on a -runtime failure with one diagnostic line, 2 on a usage error — and +runtime failure with one diagnostic line, 2 on a usage error, and `verify.lua` asserts all three per language. The split is about *when* a bad input is caught. A malformed `--group`/`--iface`, an unknown flag, or a missing subcommand is a pure usage error: parsing rejects it before any socket @@ -213,9 +213,9 @@ exists, prints `usage: chatterd discover …`, and exits 2 (C++ returns `std::unexpected(2)`, Go returns `ok=false` with code 2, Rust returns `Err(2)`). But a *syntactically valid* `--iface` that no interface owns is only discoverable at `bind` time, so it fails later as a runtime error, exit -1 — `chatterd: error: bind(tcp): Cannot assign requested address` and its Go/ -Rust equivalents. The dial path is deliberately forgiving: it retries up to 40 -times at 50 ms before giving up with `connect … giving up` (exit-neutral — it +1: `chatterd: error: bind(tcp): Cannot assign requested address` and its Go/ +Rust equivalents. The dial path stays forgiving: it retries up to 40 +times at 50 ms before giving up with `connect … giving up` (exit-neutral: it logs to stderr and the loop continues), because a beacon legitimately races its sender's own listener. And `send`/`recv` guard `EINTR` with a retry in the C++ frame I/O, which the Go and Rust runtimes handle beneath their standard @@ -226,8 +226,8 @@ libraries. Three flows of control run per process: the announce loop (on `main`), the TCP accept loop, and the UDP receive loop. The receive loop is the only writer of the dedup set, and each language reflects that differently. C++ wraps the set -in a `Seen` struct with a `std::mutex` — defensive, since a future second -reader would need it — while Go confines its `map[string]bool` to the +in a `Seen` struct with a `std::mutex` (defensive, since a future second +reader would need it), while Go confines its `map[string]bool` to the `recvLoop` goroutine and Rust confines its `HashSet` to the `recv_loop` thread, so neither needs a lock at all: the state is owned by one flow, which is Chapter 6's "share memory by communicating" reflex applied to sockets. @@ -264,7 +264,7 @@ peer alice says: hello from alice and `alice`'s stdout is the mirror: `discovered peer bob at 127.0.0.1:9232` / `peer bob says: hello from bob`. The socket is a joined, unconnected UDP -endpoint — `ss -uap` on the host during the run shows exactly that, one +endpoint: `ss -uap` on the host during the run shows exactly that, one `UNCONN` socket bound to all addresses on the group port, owned by our binary: ```console @@ -284,7 +284,7 @@ network, so the real proof is on two machines watching the wire. The star run is `systems-target` (192.168.124.7) and `systems-peer` (192.168.124.95), each launched with its own `--iface`. First, the beacons -themselves — `tcpdump` on the peer guest sees both guests' multicast, decoded +themselves: `tcpdump` on the peer guest sees both guests' multicast, decoded byte for byte: ```console @@ -302,7 +302,7 @@ byte for byte: Everything in the chapter is on the wire. The destination is the group `239.23.7.1.51824`; the ASCII column reads `CHATTERD1 peer 9241 192.168.124.95` (34 bytes) and `CHATTERD1 target 9241 192.168.124.7` (35 -bytes) — the length difference is exactly the shorter name and IP. In the IP +bytes); the length difference is exactly the shorter name and IP. In the IP header, the byte after `4000` is `01`: TTL 1, our `IP_MULTICAST_TTL`, the router-stop that keeps beacons link-local. Second, the joined socket and group membership on the peer guest, while `chatterd` runs: @@ -314,8 +314,8 @@ UNCONN 0 0 0.0.0.0:51824 0.0.0.0:* users:(("chatterd",pid=2105,fd=4)) inet 239.23.7.1 ``` -`ip maddr` proves the kernel actually joined the group on `enp1s0` — the -observable effect of `IP_ADD_MEMBERSHIP` — and `ss -uap` shows the same +`ip maddr` proves the kernel actually joined the group on `enp1s0` (the +observable effect of `IP_ADD_MEMBERSHIP`), and `ss -uap` shows the same `UNCONN` fd-4 UDP socket as on the host. Third, the payoff: both guests' stdout, quoted verbatim from the two-host run: @@ -331,17 +331,17 @@ discovered peer target at 192.168.124.7:9241 peer target says: hello from target ``` -Each guest discovered the other exactly once and exchanged one TCP frame — the -message genuinely crossed the network. No libvirt workaround was needed: both +Each guest discovered the other exactly once and exchanged one TCP frame: the +message crossed the network between two separate hosts. No libvirt workaround was needed: both guests sit on the same `virbr0` L2 segment, so a TTL-1 multicast beacon is delivered by the bridge as ordinary link-local traffic; libvirt's NAT only governs traffic *leaving* the host toward the internet, which discovery never does. -> **On the lab VM** unverified — -> the kernel-side view of the UDP→TCP bridge — watching the dial land with +> **On the lab VM** unverified: +> the kernel-side view of the UDP→TCP bridge (watching the dial land with > bcc-tools `tcpconnect`, the short-lived chat connection appear and close in -> `tcplife`, and the receive loops' off-CPU parking in `offcputime` — needs +> `tcplife`, and the receive loops' off-CPU parking in `offcputime`) needs > root and the bcc stack, which are not run on this host; the Debugging part > (Part 8) exercises exactly that on `systems-target`. @@ -349,18 +349,18 @@ does. - UDP is connectionless, message-oriented, best-effort: `sendto` emits one datagram whose boundaries `recv` preserves, and loss/reorder are tolerable - for a beacon that repeats — which is why discovery is UDP and the + for a beacon that repeats, which is why discovery is UDP and the conversation is TCP. - Multicast is opt-in delivery to a joined group: `IP_ADD_MEMBERSHIP` joins, `IP_MULTICAST_IF` pins the send interface, `IP_MULTICAST_TTL = 1` keeps it - link-local, and `IP_MULTICAST_LOOP` controls local delivery — verified live + link-local, and `IP_MULTICAST_LOOP` controls local delivery: verified live by `ip maddr show` and the TTL-1 byte in `tcpdump`. - Beacons stay small and self-describing (34–35 bytes here) to fit one un-fragmented packet, and a receiver dedupes by name so one peer is announced once regardless of beacon count. - The bridge from discovery to conversation is a plain `connect` to the advertised endpoint, reusing the canonical chat frame ch21 introduced (a - JOIN out, a DELIVER back) — proven across two guests, each printing one + JOIN out, a DELIVER back): proven across two guests, each printing one `discovered peer` and one `peer … says:` line. Next, **time, timers, and deadlines**: the clocks a networked program must @@ -369,7 +369,7 @@ accurate about *when* it fires. --- -

verified — evidence produced +

verified: evidence produced this session on the Fedora 44 reference host (kernel 7.1) and the two lab guests systems-target (192.168.124.7) / systems-peer (192.168.124.95): the runner printed @@ -386,6 +386,6 @@ IP-header TTL byte 01); ss -uanp showed the joined chatterd fd-4 UDP socket and ip maddr show enp1s0 the 239.23.7.1 membership; and both guests' stdout printed one discovered peer and one peer … says: line as quoted. -No libvirt NAT workaround was required — the guests share one virbr0 +No libvirt NAT workaround was required: the guests share one virbr0 L2 segment. The bcc-tools tcpconnect/tcplife/offcputime callout is unverified as marked and is exercised in Part 8.

diff --git a/_docs/24-time-timers-deadlines.md b/_docs/24-time-timers-deadlines.md index 3202a6f..2d3ad29 100644 --- a/_docs/24-time-timers-deadlines.md +++ b/_docs/24-time-timers-deadlines.md @@ -2,19 +2,19 @@ title: "Time, timers, and deadlines" order: 24 part: "Networking" -description: "chatterd v3 learns to survive a peer that vanishes: the POSIX clock taxonomy, clock_gettime's vDSO fast path (no syscall — proven by its absence under strace), timerfd as a pollable heartbeat, the per-language deadline idiom, why liveness must be measured on a monotonic clock, and jittered exponential backoff." +description: "chatterd v3 learns to survive a peer that vanishes: the POSIX clock taxonomy, clock_gettime's vDSO fast path (no syscall, proven by its absence under strace), timerfd as a pollable heartbeat, the per-language deadline idiom, why liveness must be measured on a monotonic clock, and jittered exponential backoff." duration: "45 minutes" --- Chapters 21–23 built `chatterd` into a peer-to-peer chat daemon that links two hosts over TCP and passes framed messages. It has one fatal naïveté: it assumes the peer stays. Pull the cable, kill the process, or let a switch drop the -flow, and a `chatterd` peer sits forever in a `read` that will never return — +flow, and a `chatterd` peer sits forever in a `read` that will never return: TCP alone will not tell you the far end is gone in any useful window. Version 3 fixes this with *time*: a periodic **heartbeat** proves the link is alive, a **deadline** measured on silence declares it dead, and a **jittered exponential backoff** brings it back without hammering. The one new idea under -all three is that not all clocks are the same clock — and picking the wrong one +all three is that not all clocks are the same clock, and picking the wrong one turns a heartbeat into a liability the first time NTP corrects your wall clock. The code is in `examples/24-time-timers-deadlines/`. `./demo.sh` there builds @@ -24,53 +24,53 @@ chapter adds), and the exit codes all three languages share. {% include excalidraw.html file="24-clock-taxonomy" - alt="Top band: the four clocks clock_gettime can read as boxes — CLOCK_REALTIME (dashed, wall time, settable, can step backward), CLOCK_MONOTONIC (amber, since a boot epoch, never steps, steady_clock/Instant), CLOCK_BOOTTIME (monotonic plus suspend time), and PROCESS_CPUTIME_ID (this process's on-CPU time). Bottom band: while a peer is silent and ntpd corrects the wall clock, a dashed REALTIME box shows a wall-clock deadline rewinding so a dead peer looks alive, beside an amber MONOTONIC box where elapsed is always non-negative and the timeout fires on real silence; an amber arrow from CLOCK_MONOTONIC points to it labeled chatterd reads this." - caption="Figure 24.1 — the clock taxonomy and the monotonic-vs-wall bug: a wall-clock deadline slips when the clock is stepped; a CLOCK_MONOTONIC deadline does not" %} + alt="Top band: the four clocks clock_gettime can read as boxes: CLOCK_REALTIME (dashed, wall time, settable, can step backward), CLOCK_MONOTONIC (amber, since a boot epoch, never steps, steady_clock/Instant), CLOCK_BOOTTIME (monotonic plus suspend time), and PROCESS_CPUTIME_ID (this process's on-CPU time). Bottom band: while a peer is silent and ntpd corrects the wall clock, a dashed REALTIME box shows a wall-clock deadline rewinding so a dead peer looks alive, beside an amber MONOTONIC box where elapsed is always non-negative and the timeout fires on real silence; an amber arrow from CLOCK_MONOTONIC points to it labeled chatterd reads this." + caption="Figure 24.1: the clock taxonomy and the monotonic-vs-wall bug: a wall-clock deadline slips when the clock is stepped; a CLOCK_MONOTONIC deadline does not" %} -> **Tools used** — `strace`, `python3` (host); `cyclictest` (systems-target +> **Tools used**: `strace`, `python3` (host); `cyclictest` (systems-target > VM, exercised in Part 8). Everything host-side is checked by > `scripts/check-host.sh` or ships with Fedora. ## The clock taxonomy `clock_gettime(2)` takes a *clock id*, and the choice is the whole chapter. -Four ids matter here. **`CLOCK_REALTIME`** is the wall clock — seconds since the +Four ids matter here. **`CLOCK_REALTIME`** is the wall clock: seconds since the Unix epoch, the thing `date` prints. It is *settable* (`settimeofday`, -`adjtimex`) and *disciplined by NTP*, which means it can jump forward, and — the -part that bites — jump *backward*. **`CLOCK_MONOTONIC`** counts from an +`adjtimex`) and *disciplined by NTP*, which means it can jump forward, and (the +part that bites) jump *backward*. **`CLOCK_MONOTONIC`** counts from an unspecified epoch (near boot) and has one guarantee the wall clock cannot give: it never steps and never goes backward. NTP may gently *slew* its rate, but two successive reads always satisfy `t1 >= t0`. **`CLOCK_BOOTTIME`** is monotonic -*and* keeps counting across system suspend, which `CLOCK_MONOTONIC` does not — +*and* keeps counting across system suspend, which `CLOCK_MONOTONIC` does not: the right clock for "has 30 s of wall time passed even if the laptop slept." **`CLOCK_PROCESS_CPUTIME_ID`** measures only the CPU time *this process* burned, useful for profiling, useless for wall deadlines. The rule that follows is absolute: **any duration or deadline must be measured -on a monotonic clock.** A heartbeat timeout is a duration — "no traffic for -3 seconds" — so it reads `CLOCK_MONOTONIC`, never `CLOCK_REALTIME`. Figure 24.1 +on a monotonic clock.** A heartbeat timeout is a duration ("no traffic for +3 seconds"), so it reads `CLOCK_MONOTONIC`, never `CLOCK_REALTIME`. Figure 24.1 is the bug you avoid by obeying this. Suppose you computed a deadline as `wall_now() + 3s` and stored it. Halfway through, `ntpd` decides the clock is 0.5 s fast and steps it *back*. Now `wall_now()` reads earlier than when you started; your deadline recedes into the future; a peer that died stays "alive" -for an extra half second — or, on a backward step large enough, your `select` +for an extra half second; or, on a backward step large enough, your `select` timeout goes negative and a healthy peer is declared dead instantly. Every language in this chapter routes liveness through a monotonic reading: `std::chrono::steady_clock` in C++, `std::time::Instant` in Rust, and Go's `time.Now`, whose returned value carries a hidden monotonic component precisely so `time.Since` cannot be fooled by a wall-clock step. -## clock_gettime and the vDSO — no syscall at all +## clock_gettime and the vDSO: no syscall at all -Reading the clock happens constantly — on every loop iteration `chatterd` -recomputes how long until the deadline — so it had better be cheap. It is, and +Reading the clock happens constantly (on every loop iteration `chatterd` +recomputes how long until the deadline), so it had better be cheap. It is, and the reason is chapter 4's vDSO. `CLOCK_MONOTONIC` and `CLOCK_REALTIME` reads do not trap into the kernel: the kernel maps a shared page (the vDSO) into every process, updates a timekeeping structure there on each tick, and the libc `clock_gettime` wrapper reads it in userspace with no `syscall` instruction. The proof is negative and we run it later: `strace` a program that reads the monotonic clock four times and you will not see a single `clock_gettime` line, -because there was no syscall to trace. Sleeping is the opposite — you cannot +because there was no syscall to trace. Sleeping is the opposite: you cannot sleep in userspace, so `nanosleep`/`clock_nanosleep` and `timerfd_settime` do trap, and those *do* show up. @@ -78,8 +78,8 @@ trap, and those *do* show up. `clockprobe` is a standalone subcommand that makes the abstractions concrete by measuring them. It reads the monotonic clock's resolution with `clock_getres`, -times a 1 ms `nanosleep`, and times a 1 ms one-shot `timerfd` — each timed on a -steady clock — then prints one line. Here it is in all three languages, +times a 1 ms `nanosleep`, and times a 1 ms one-shot `timerfd` (each timed on a +steady clock), then prints one line. Here it is in all three languages, verbatim from `examples/24-time-timers-deadlines/`: {% include codetabs.html langs="C++|Go|Rust" %} @@ -185,12 +185,12 @@ fn run_clockprobe() -> i32 { ``` The three are the same program in three dialects. `clock_getres` on this host -returns `1ns` — nanosecond nominal resolution — but resolution is not accuracy: +returns `1ns` (nanosecond nominal resolution), but resolution is not accuracy: the two 1 ms sleeps *overshoot*, and that gap is the lesson. A `timerfd` is a timer wrapped in a file descriptor: `timerfd_create(CLOCK_MONOTONIC, …)` makes one, `timerfd_settime` arms it (an `it_value` with no `it_interval` is a one-shot; both set makes it periodic), and it becomes *readable* when it -expires — a `read` returns a `uint64` count of expirations. That last property +expires: a `read` returns a `uint64` count of expirations. That last property is why it belongs in this chapter: a timer that is an fd drops straight into the `poll`/`epoll` loop chapter 9 built, no signal handler required. @@ -210,20 +210,20 @@ that to `poll` as its timeout: ``` `deadline` is a `steady_clock::time_point` set to `now() + timeout` at session -start and *reset to a fresh `now() + timeout` on every received frame* — JOIN, +start and *reset to a fresh `now() + timeout` on every received frame*: JOIN, MSG, or PING, all count as traffic. The `poll` array holds three fds: the `signalfd` shutdown doorbell, the periodic heartbeat `timerfd`, and the peer -socket. When `poll` returns 0, the timeout elapsed with nothing to read — the -peer went silent for `timeout_ms` — and the session returns `TimedOut`. When +socket. When `poll` returns 0, the timeout elapsed with nothing to read (the +peer went silent for `timeout_ms`) and the session returns `TimedOut`. When the heartbeat fd fires, the loop reads its 8-byte tick and sends a `PING`. This is the same one-loop-three-sources shape as chapter 9's `fwatch`, now serving liveness instead of file events. The three languages express the deadline -differently — C++ subtracts two `time_point`s, Rust does the identical +differently: C++ subtracts two `time_point`s, Rust does the identical arithmetic on `Instant`, and Go resets a `time.Timer` and `select`s on its -channel — but the invariant is one sentence: *the clock is monotonic and the +channel, but the invariant is one sentence: *the clock is monotonic and the deadline is data.* -Reconnection is the connector's job, and it is where jitter earns its keep. +Reconnection is the connector's job, and it is where jitter matters most. After a drop, `run_connect` waits out a backoff before redialing: ```cpp @@ -239,17 +239,17 @@ After a drop, `run_connect` waits out a backoff before redialing: backoff = std::min(backoff * 2, opts.max_backoff_ms); ``` -Two ideas are stacked here. **Exponential** — `backoff` doubles per failed +Two ideas are stacked here. **Exponential**: `backoff` doubles per failed attempt (200 → 400 → 800 …) up to `--max-backoff-ms`, so a peer that is down -for a while is polled ever more gently instead of in a hot loop. **Jitter** — +for a while is polled ever more gently instead of in a hot loop. **Jitter**: the actual `delay` is `half + rand(0..half)`, the "equal jitter" strategy: half the interval fixed, half random. Jitter matters because without it, a hundred clients that all lost the same server would redial in lockstep on every doubling boundary, a synchronized thundering herd that knocks the server back down the instant it recovers. Randomizing each client's wait smears the retries across the window. A fresh connection resets `backoff` to the base, so a brief blip is -never punished with a long wait. The wait itself is `sleep_or_shutdown` — a -`poll` on the signalfd with the backoff as its timeout — so a `SIGTERM` +never punished with a long wait. The wait itself is `sleep_or_shutdown` (a +`poll` on the signalfd with the backoff as its timeout), so a `SIGTERM` mid-backoff wakes instantly instead of stranding the process for seconds. ## Errors, three ways @@ -259,32 +259,32 @@ diagnostic, usage errors exit 2. The interesting failures here are temporal. The most important one is a *non-error*: when the peer socket returns EOF (0 bytes) because the far end closed, `chatterd` does **not** immediately declare the peer gone. It sets `sock_dead`, stops polling the socket, and lets the -*deadline* fire — so death is always declared by the same monotonic timeout +*deadline* fire, so death is always declared by the same monotonic timeout whether the peer crashed silently or closed politely, and the observable line is identical. Sending a `PING` into a half-closed socket can raise `EPIPE`; C++ passes `MSG_NOSIGNAL` so it never becomes a fatal `SIGPIPE`, and the failed send -is deliberately ignored — "a broken pipe is left to the deadline," as the +is ignored: "a broken pipe is left to the deadline," as the comment says, because the deadline is the single source of truth about liveness. `EINTR` on `poll` is retried in C++ and Rust and handled beneath the runtime in Go. Bad numeric flags (`--timeout-ms abc`) and missing required -flags exit 2 through each language's parser — `std::expected` chains in C++, -error returns in Go, `Result` in Rust — all printing the same `usage:` block. +flags exit 2 through each language's parser: `std::expected` chains in C++, +error returns in Go, `Result` in Rust, all printing the same `usage:` block. ## Concurrency lens The C++ and Rust sessions are single-threaded: one `poll` loop multiplexes the socket, the heartbeat timerfd, and the shutdown signalfd, so there is no shared -state to lock — the timer and the deadline are just two more fds and one +state to lock: the timer and the deadline are just two more fds and one `time_point`. Go inverts it, exactly as chapter 9 did: a per-connection reader goroutine turns the socket into a `chan frame`, and the session is a `select` -over four cases — `ticker.C` (heartbeat), `deadline.C` (liveness), the frame +over four cases: `ticker.C` (heartbeat), `deadline.C` (liveness), the frame channel, and `ctx.Done()` (shutdown from `signal.NotifyContext`). The subtlety Go forces you to get right is `time.Timer` reset: before `deadline.Reset`, the code drains a possibly-already-fired timer channel, because a `Timer` that expired between `select` iterations leaves a value buffered in `C` that would otherwise make the *next* `select` see a phantom timeout. C++ and Rust sidestep this entirely by recomputing the `poll` timeout from the deadline each -iteration — there is no timer object to get out of sync, just arithmetic on a +iteration: there is no timer object to get out of sync, just arithmetic on a monotonic reading. Both designs reach the same behavior; the single-threaded one has less to synchronize, the Go one reads more like the protocol. @@ -295,7 +295,7 @@ one has less to synchronize, the Go one reads more like the protocol. ``` The runner (`python3 scripts/test-all-examples.py --only -24-time-timers-deadlines`) reports `PASS PASS PASS` for cpp, go, rust — +24-time-timers-deadlines`) reports `PASS PASS PASS` for cpp, go, rust; `verify.lua` asserts observable behavior, not exit 0. First, `clockprobe` on this host: @@ -304,14 +304,14 @@ this host: clockprobe: CLOCK_MONOTONIC res=1ns nanosleep(1ms) actual=1056us timerfd(1ms) actual=1013us ``` -All three languages agree to within noise — Go printed `actual=1066us` / +All three languages agree to within noise: Go printed `actual=1066us` / `1005us`, Rust `1055us` / `1002us`. The number to sit with is the overshoot: a 1 ms sleep took ~1056 µs, ~56 µs long. That gap is not the clock's resolution -(1 ns) — it is the *scheduler's wake-up latency*, the time between the timer +(1 ns); it is the *scheduler's wake-up latency*, the time between the timer expiring and your thread actually getting a CPU. On a general-purpose kernel this is tens of microseconds and variable; the timerfd path is a touch tighter (~1013 µs) because it wakes straight out of `read` rather than through -`nanosleep`'s return. Now the heartbeat scenario, driven by hand — a listener +`nanosleep`'s return. Now the heartbeat scenario, driven by hand: a listener `A` and a connector `B` on loopback, then `A` killed: ```console @@ -334,15 +334,15 @@ chatterd: B shutting down # after SIGTERM The backoff line is the whole design on one screen. Base is 200 ms, so the first `delay` is `100 + rand(0..100)` = 107 ms; then the ceiling doubles and the -window grows — 245 (from base 400), 712 (from 800), 1055 (from 1600) — each a +window grows: 245 (from base 400), 712 (from 800), 1055 (from 1600), each a jittered draw inside its window, never a bare power of two. `SIGTERM` mid-backoff prints `B shutting down` and exits 0, waking out of the `poll` immediately. Figure 24.2 is this sequence. {% include excalidraw.html file="24-heartbeat-timeout-seq" - alt="Two lifelines: connector B (amber) on the left, listener A (grey, ending early) on the right. Top: JOIN 'B' from B to A and JOIN 'A' back, then a note that both print linked and exchange messages. Middle: amber PING arrows every 300 ms in both directions, each resetting B's deadline, with a note that any frame sets deadline = mono_now() + timeout. Then A killed, socket EOF, and a dashed PING from B that is dropped on the broken pipe. Bottom on B's side: an amber box 'deadline fires on silence — chatterd: peer A timed out', then three stacked reconnect boxes 107 ms, 245 ms, 712 ms chained by amber arrows, beside a note explaining equal-jitter backoff delay = B/2 + rand(0..B/2) with B doubling per attempt and capped at max-backoff." - caption="Figure 24.2 — chatterd v3 from link to loss to reconnect: the monotonic deadline fires on silence, then jittered exponential backoff (real 107/245/712 ms from this run) paces the redials" %} + alt="Two lifelines: connector B (amber) on the left, listener A (grey, ending early) on the right. Top: JOIN 'B' from B to A and JOIN 'A' back, then a note that both print linked and exchange messages. Middle: amber PING arrows every 300 ms in both directions, each resetting B's deadline, with a note that any frame sets deadline = mono_now() + timeout. Then A killed, socket EOF, and a dashed PING from B that is dropped on the broken pipe. Bottom on B's side: an amber box 'deadline fires on silence: chatterd: peer A timed out', then three stacked reconnect boxes 107 ms, 245 ms, 712 ms chained by amber arrows, beside a note explaining equal-jitter backoff delay = B/2 + rand(0..B/2) with B doubling per attempt and capped at max-backoff." + caption="Figure 24.2: chatterd v3 from link to loss to reconnect: the monotonic deadline fires on silence, then jittered exponential backoff (real 107/245/712 ms from this run) paces the redials" %} ## Cross-check: the vDSO's absence, and the heartbeat live under strace @@ -357,7 +357,7 @@ monotonic clock four times (`t0`, after the sleep, `t1`, after the timerfd) plus ``` Zero lines. Not because the program never reads the clock, but because those -reads never became syscalls — they were served from the vDSO page in userspace. +reads never became syscalls: they were served from the vDSO page in userspace. What *does* trap is visible when you trace the sleeping calls: ```console @@ -370,10 +370,10 @@ What *does* trap is visible when you trace the sleeping calls: ``` `clock_nanosleep` (glibc implements `nanosleep` on top of it), `timerfd_create`, -`timerfd_settime` — each once, exactly as the source calls them — and no +`timerfd_settime` (each once, exactly as the source calls them) and no `clock_gettime` row at all. The Rust binary shows `clock_nanosleep(CLOCK_MONOTONIC, …)` because it calls that clock explicitly; -the C++ one shows `CLOCK_REALTIME` because that is glibc's `nanosleep` default — +the C++ one shows `CLOCK_REALTIME` because that is glibc's `nanosleep` default, a fine reminder that the *sleep* clock and the *deadline* clock are chosen independently. @@ -393,19 +393,19 @@ poll([{fd=3,...}, {fd=5,...}, {fd=4,...}], 3, 900) = 1 ([{fd=4, revents=POLLIN}] Every claim in the chapter is on those lines. `timerfd_settime` arms the heartbeat with a 300 ms interval. The `sendto(4, "CH\1\1\0\1B", 7, …)` is the -7-byte JOIN frame — `43 48 01 01 00 01 42`, the canonical header (`CH`, -version 1, type 1 = JOIN) plus the name `B` — matching the wire format byte +7-byte JOIN frame, `43 48 01 01 00 01 42`, the canonical header (`CH`, +version 1, type 1 = JOIN) plus the name `B`, matching the wire format byte for byte, the same header chapters 21–23 used; only the TYPE byte and the engine around it differ by chatterd version. When fd 5 (the timerfd) fires, -the loop reads its 8-byte tick (`\1\0\0\0\0\0\0\0` — one expiration) and +the loop reads its 8-byte tick (`\1\0\0\0\0\0\0\0`, one expiration) and sends the 6-byte PING `CH\1\5\0\0` (`43 48 01 05 00 00`). And the `poll` timeout *is* the deadline countdown: `1200` right after receiving a -frame (the deadline was just reset), then `900` after a heartbeat-only wakeup — +frame (the deadline was just reset), then `900` after a heartbeat-only wakeup: the monotonic deadline shrinking exactly as `deadline - steady_clock::now()` says it should. The trace and the printed output tell one story: the timer drives the PINGs, the monotonic deadline drives the timeout. -> **On the lab VM** unverified — +> **On the lab VM** unverified: > the scheduler wake-up latency that `clockprobe` measures as ~56 µs of > overshoot has a proper tool, `cyclictest`, which stresses the timer path and > reports latency percentiles; and comparing a `PREEMPT` versus `PREEMPT_RT` @@ -419,17 +419,17 @@ drives the PINGs, the monotonic deadline drives the timeout. duration or deadline; `CLOCK_BOOTTIME` adds suspend time; `CLOCK_PROCESS_CPUTIME_ID` measures CPU, not wall time. `chatterd` measures silence on the monotonic clock so an NTP correction can never fake a timeout. -- `clock_gettime` on the common clocks is a vDSO read, not a syscall — `strace` +- `clock_gettime` on the common clocks is a vDSO read, not a syscall: `strace` showed **zero** `clock_gettime` lines for a program that read the clock four times, while `clock_nanosleep` and `timerfd_settime` (which must trap) each appeared once. -- A `timerfd(CLOCK_MONOTONIC)` is a timer you can `poll` — it drops the +- A `timerfd(CLOCK_MONOTONIC)` is a timer you can `poll`: it drops the heartbeat into the same event loop as the socket and the shutdown signalfd, and `clockprobe` measured a 1 ms timer landing at ~1013 µs, the ~13 µs overshoot being scheduler wake-up latency, not clock resolution. - Jittered exponential backoff (`delay = B/2 + rand(0..B/2)`, `B` doubling to a cap, reset on a fresh link) paces reconnects and de-synchronizes a fleet of - clients — the live run drew 107, 245, 712, 1055 ms, growing but never a bare + clients: the live run drew 107, 245, 712, 1055 ms, growing but never a bare power of two. Next, **shared state and the futex**: `chatterd`'s peers coordinated by passing @@ -438,7 +438,7 @@ a lock is a userspace atomic that only touches the kernel when it must block. --- -

verified — every number and +

verified: every number and output excerpt above was produced on the Fedora 44 reference host (kernel 7.1, strace 7.1) this session: the runner printed 24-time-timers-deadlines PASS PASS PASS (3 passed, 0 failed, 0 diff --git a/_docs/25-shared-state-and-the-futex.md b/_docs/25-shared-state-and-the-futex.md index ca21980..fc49865 100644 --- a/_docs/25-shared-state-and-the-futex.md +++ b/_docs/25-shared-state-and-the-futex.md @@ -2,7 +2,7 @@ title: "Shared state and the futex" order: 25 part: "Concurrency in Depth" -description: "workq builds a bounded MPMC job queue three ways — std::mutex + condvars, a buffered channel, Mutex + Condvar — and we look under all of them at the one primitive doing the work: the futex, whose uncontended fast path is a userspace CAS with no syscall and whose slow path is FUTEX_WAIT/WAKE, proven by strace scaling from 5,186 to 488,818 calls, then broken on purpose under ThreadSanitizer and go -race while Rust refuses to compile the race at all." +description: "workq builds a bounded MPMC job queue three ways (std::mutex + condvars, a buffered channel, Mutex + Condvar) and we look under all of them at the one primitive doing the work: the futex, whose uncontended fast path is a userspace CAS with no syscall and whose slow path is FUTEX_WAIT/WAKE, proven by strace scaling from 5,186 to 488,818 calls, then broken under ThreadSanitizer and go -race while Rust refuses to compile the race at all." duration: "50 minutes" --- @@ -10,4 +10,4 @@ duration: "50 minutes" > > It maps to **[Shared State and the Futex](https://patterncatalyst.github.io/cpp-concurrency-and-parallelism/docs/05-shared-state-and-the-futex/)** there. -The full, runnable C++ treatment of concurrency and parallelism — threads and `jthread`, mutexes and the memory model, atomics and lock-free structures, futures and coroutines, fibers, Asio, NUMA and mechanical sympathy, and the low-latency patterns — now lives in that book, each chapter with a Docker or KVM-lab demo and a Lua `verify.lua` behind it. This book keeps its Linux systems-programming material; the C++ concurrency compendium was extracted so it could grow on its own. +The full, runnable C++ treatment of concurrency and parallelism (threads and `jthread`, mutexes and the memory model, atomics and lock-free structures, futures and coroutines, fibers, Asio, NUMA and mechanical sympathy, and the low-latency patterns) now lives in that book, each chapter with a Docker or KVM-lab demo and a Lua `verify.lua` behind it. This book keeps its Linux systems-programming material; the C++ concurrency compendium was extracted so it could grow on its own. diff --git a/_docs/26-atomics-and-lock-free.md b/_docs/26-atomics-and-lock-free.md index ff0acb0..f1b58e3 100644 --- a/_docs/26-atomics-and-lock-free.md +++ b/_docs/26-atomics-and-lock-free.md @@ -2,7 +2,7 @@ title: "Atomics and lock-free: memory ordering, the SPSC ring, and false sharing" order: 26 part: "Concurrency in Depth" -description: "spscring drops the lock entirely: a single-producer/single-consumer ring coordinated by two atomic counters, with acquire/release explained on the real head/tail, why a non-atomic ring tears, the ABA problem SPSC sidesteps, and false sharing measured with perf — a padded line beats an unpadded one by ~34% on this host." +description: "spscring drops the lock entirely: a single-producer/single-consumer ring coordinated by two atomic counters, with acquire/release explained on the real head/tail, why a non-atomic ring tears, the ABA problem SPSC sidesteps, and false sharing measured with perf: a padded line beats an unpadded one by ~34% on this host." duration: "55 minutes" --- @@ -10,4 +10,4 @@ duration: "55 minutes" > > It maps to **[Atomics and the Memory Model](https://patterncatalyst.github.io/cpp-concurrency-and-parallelism/docs/08-atomics-and-the-memory-model/)** there. -The full, runnable C++ treatment of concurrency and parallelism — threads and `jthread`, mutexes and the memory model, atomics and lock-free structures, futures and coroutines, fibers, Asio, NUMA and mechanical sympathy, and the low-latency patterns — now lives in that book, each chapter with a Docker or KVM-lab demo and a Lua `verify.lua` behind it. This book keeps its Linux systems-programming material; the C++ concurrency compendium was extracted so it could grow on its own. +The full, runnable C++ treatment of concurrency and parallelism, covering threads and `jthread`, mutexes and the memory model, atomics and lock-free structures, futures and coroutines, fibers, Asio, NUMA and mechanical sympathy, and the low-latency patterns, now lives in that book, each chapter with a Docker or KVM-lab demo and a Lua `verify.lua` behind it. This book keeps its Linux systems-programming material; the C++ concurrency compendium was extracted so it could grow on its own. diff --git a/_docs/27-async-runtimes-and-coroutines.md b/_docs/27-async-runtimes-and-coroutines.md index 3d6183d..1405012 100644 --- a/_docs/27-async-runtimes-and-coroutines.md +++ b/_docs/27-async-runtimes-and-coroutines.md @@ -2,7 +2,7 @@ title: "Async runtimes and coroutines" order: 27 part: "Concurrency in Depth" -description: "chatterd v4 tells the same broadcast three async ways over the same OS threads: hand-rolled C++20 coroutines suspending on an epoll reactor, Go's G-M-P scheduler and netpoller that needed no rewrite, and tokio's tasks-and-Wakers — with /proc//task thread counts (1 vs 17 vs dynamic) and a single-epoll-loop strace showing how each spends threads, plus the framework for when plain threads still win." +description: "chatterd v4 tells the same broadcast three async ways over the same OS threads: hand-rolled C++20 coroutines suspending on an epoll reactor, Go's G-M-P scheduler and netpoller that needed no rewrite, and tokio's tasks-and-Wakers, with /proc//task thread counts (1 vs 17 vs dynamic) and a single-epoll-loop strace showing how each spends threads, plus the framework for when plain threads still win." duration: "60 minutes" --- @@ -10,4 +10,4 @@ duration: "60 minutes" > > It maps to **[Async Runtimes and Event Loops](https://patterncatalyst.github.io/cpp-concurrency-and-parallelism/docs/12-async-runtimes-and-event-loops/)** there. -The full, runnable C++ treatment of concurrency and parallelism — threads and `jthread`, mutexes and the memory model, atomics and lock-free structures, futures and coroutines, fibers, Asio, NUMA and mechanical sympathy, and the low-latency patterns — now lives in that book, each chapter with a Docker or KVM-lab demo and a Lua `verify.lua` behind it. This book keeps its Linux systems-programming material; the C++ concurrency compendium was extracted so it could grow on its own. +The full, runnable C++ treatment of concurrency and parallelism now lives in that book: threads and `jthread`, mutexes and the memory model, atomics and lock-free structures, futures and coroutines, fibers, Asio, NUMA and mechanical sympathy, and the low-latency patterns, each chapter with a Docker or KVM-lab demo and a Lua `verify.lua` behind it. This book keeps its Linux systems-programming material; the C++ concurrency compendium was extracted so it could grow on its own. diff --git a/_docs/28-gdb-and-remote-debugging.md b/_docs/28-gdb-and-remote-debugging.md index a979fb0..2a01f76 100644 --- a/_docs/28-gdb-and-remote-debugging.md +++ b/_docs/28-gdb-and-remote-debugging.md @@ -2,7 +2,7 @@ title: "gdb and Remote Debugging" order: 28 part: "Debugging" -description: "app crashes on purpose, three ways: a real SIGSEGV and a systemd-coredump core in C++, a runtime-caught panic in Go, an Option::unwrap() panic in Rust — and gdb, rust-gdb, coredumpctl, readelf, and objdump read the same crash from four directions, then gdbserver and target remote carry the session from the host to a VM." +description: "app crashes by design, three ways: a real SIGSEGV and a systemd-coredump core in C++, a runtime-caught panic in Go, an Option::unwrap() panic in Rust, and gdb, rust-gdb, coredumpctl, readelf, and objdump read the same crash from four directions, then gdbserver and target remote carry the session from the host to a VM." duration: "55 minutes" --- @@ -10,7 +10,7 @@ Every debugger so far in this book has been a `println` you had to recompile to move. This chapter replaces that habit with the real tool: a crash that already happened, frozen in a core file or a live process, that you can walk backwards through without changing a line of code. `app` is this -chapter's target — a small pmon-like supervisor that behaves identically in +chapter's target: a small pmon-like supervisor that behaves identically in `run` mode across C++, Go, and Rust, then diverges completely in `crash` mode, because the *entire point* of this chapter is that "the same bug" does not mean "the same crash": C++ gets a hardware `SIGSEGV` and a core file, @@ -26,11 +26,11 @@ verification contract. {% include excalidraw.html file="28-stack-frame-anatomy" alt="A column of five stacked frames from a real gdb bt full on app's core file, growing toward lower addresses as they go down: frame 0 std::println<int,int> at print colon 164, the actual crash PC; frame 1 deref at main.cpp:52 showing child equals pid 4242 exit_slot 0x0; frame 2 handle_child at main.cpp:56 with return address 0x4016b2; frame 3 run_supervisor at main.cpp:65 with return address 0x40177c and locals child, guard equals optimized out; frame 4 main at main.cpp:104 with return address 0x400964. Beside it, a DWARF cross-check column shows readelf/objdump decodedline results: main.cpp:52 maps to 0x401680, an exact match to gdb's own breakpoint address; main.cpp:56 maps to 0x4016b2, an exact match to frame 2's return address; main.cpp:65 maps to 0x401772, close to but not exactly frame 3's return address of 0x40177c since that is a post-call return address, not a statement start. A footer note says every frame function is noinline so -O2 cannot fold the call chain away, and ShutdownGuard's destructor never runs because SIGSEGV does not unwind the stack." - caption="Figure 28.1 — the five real frames gdb names on app's core, cross-checked address for address against the DWARF line table readelf and objdump report independently" %} + caption="Figure 28.1: the five real frames gdb names on app's core, cross-checked address for address against the DWARF line table readelf and objdump report independently" %} -> **Tools used** — `gdb` 17.2, `rust-gdb`, `coredumpctl`, `readelf`, `objdump` +> **Tools used**: `gdb` 17.2, `rust-gdb`, `coredumpctl`, `readelf`, `objdump` > (host); `gdb-gdbserver` (preinstalled on the `systems-target` VM image via -> cloud-init, documented but not exercised live in this local lane — no +> cloud-init, documented but not exercised live in this local lane; no > `systems-target` VM was up this session); `dlv` (delve) is referenced for > contrast but is a soft/optional host tool per `scripts/check-host.sh` and > was not installed or run here. Everything else appears in @@ -45,7 +45,7 @@ table to disk before terminating it. On a plain workstation that used to mean a `core` file dropped in the crashing process's working directory; on this host, and on any modern Fedora/systemd machine, `/proc/sys/kernel/core_pattern` instead reads -`|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %d %F` — the +`|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %d %F`: the leading `|` means the kernel *pipes* the core to that program rather than writing a file itself, so "the crash dropped a core" means `coredumpctl` has a compressed copy indexed by pid, not that a bare `core` @@ -54,9 +54,9 @@ prints its own compact backtrace; `coredumpctl dump -o core.app` extracts it as an ordinary file gdb can open directly. gdb's job starts where that photograph does. Opened against a binary that -still carries debug info — this chapter's C++ build uses the `release` +still carries debug info; this chapter's C++ build uses the `release` CMake preset, which is `RelWithDebInfo` (`-O2 -g`, optimized *and* -debuggable) — gdb reconstructs source-level frames, local variables, and +debuggable), gdb reconstructs source-level frames, local variables, and even C++ types from the DWARF the compiler embedded, not from guesswork. That last part is what "pretty-printers" buy you: without them, a `std::string_view` is a two-word struct (`{ _M_str, _M_len }`) and you'd read @@ -75,19 +75,19 @@ type = std::string_view ``` `sub="bogus-thing"` in the frame header and `$1 = "bogus-thing"` are both the -pretty-printer at work — `info pretty-printer` confirms the object file +pretty-printer at work; `info pretty-printer` confirms the object file registering it: `objfile /lib64/libstdc++.so.6 pretty-printers: libstdc++-v6`. The same mechanism turns a struct reference into a readable value: breaking at `deref` and printing `child` (a `const ChildRecord&`) gives `$1 = (const (anonymous namespace)::ChildRecord &) @0x7fff...: {pid = -4242, exit_slot = 0x0}` — field names, not offsets. And because gdb reads +4242, exit_slot = 0x0}`, field names, not offsets. And because gdb reads target memory rather than executing your code's checks, you can trigger the exact fault *before* the program does: `print *child.exit_slot` at that -breakpoint returns `Cannot access memory at address 0x0` — gdb hit the same +breakpoint returns `Cannot access memory at address 0x0`; gdb hit the same null pointer the CPU is about to, just without a signal. `rust-gdb` is the same gdb binary wrapped with a small Python autoload script -that ships with `rustc` and teaches it Rust's type layout — enum +that ships with `rustc` and teaches it Rust's type layout: enum discriminants, `Option`, slices. Breaking at `app::deref` on the Rust build and printing the child record shows the payoff directly: @@ -99,15 +99,15 @@ $1 = app::ChildRecord {pid: 4242, exit_slot: core::option::Option::None} `exit_slot: core::option::Option::None` is the variant *name*, not the tag byte a plain gdb without the Rust autoload would show. Plain `gdb` (no -`rust-gdb` wrapper) can still open the same binary — Rust's DWARF is real -DWARF — but you lose this layout-aware rendering and read raw enum +`rust-gdb` wrapper) can still open the same binary (Rust's DWARF is real +DWARF) but you lose this layout-aware rendering and read raw enum representations by hand instead. -Go breaks this whole story on purpose. There is no core file to open in the +Go breaks this whole story by design. There is no core file to open in the first place: the Go runtime installs its own handler for the fault-causing signals and, on recognizing that the fault came from ordinary user code (not -a bug inside the runtime itself), converts it into a normal — if -unrecovered — `panic`, never re-raising `SIGSEGV` with the default +a bug inside the runtime itself), converts it into a normal, if +unrecovered, `panic`, never re-raising `SIGSEGV` with the default disposition that would dump core. `gdb` *can* still attach to a running Go binary (Fedora's gdb ships "Loading Go Runtime support" goroutine helpers), and on this exact crash it stops cleanly at the fault and even walks a @@ -126,7 +126,7 @@ Thread 1 "app" received signal SIGSEGV, Segmentation fault. That result is flattering and it hides the real problem. gdb's thread list maps to **OS threads** (the `M`s in Go's `G`/`M`/`P` scheduler), not to -**goroutines** (`G`s) — a goroutine that is parked on a channel, blocked in a +**goroutines** (`G`s): a goroutine that is parked on a channel, blocked in a `select`, or simply not currently scheduled onto any `M` has no OS thread for gdb to show you, and gdb has no notion of "goroutine 41" as a selectable frame context the way it has "thread 3". The moment your program has more @@ -134,17 +134,17 @@ than a handful of concurrent goroutines, `info threads` and `bt` stop mapping onto your program's logical concurrency at all. Go's own tooling doesn't have this blind spot because it was built with the scheduler in mind: `dlv exec ./app`, then `goroutines` to list every logical goroutine (parked or -running) and `goroutine bt` to backtrace one specifically — `delve` +running) and `goroutine bt` to backtrace one specifically; `delve` walks the runtime's own goroutine table, not the OS thread list. This host does not have `dlv` installed (it is a soft, optional entry in `scripts/check-host.sh`, not a hard requirement), so that specific `goroutines`/`goroutine bt` session is unverified here — the point stands on the gdb +status--unverified">unverified here; the point stands on the gdb transcript above being accidentally complete only because this crash never gets past goroutine 1. Rust needs neither gdb nor a core for this particular bug class either. -`.unwrap()` on a `None` calls `panic!` directly — no signal is raised at +`.unwrap()` on a `None` calls `panic!` directly; no signal is raised at all, so there is nothing for gdb to catch even if you attached. The idiomatic tool is `RUST_BACKTRACE=1`, which walks the unwinder's own frame records and prints every one by name, entirely in userspace: @@ -165,27 +165,27 @@ exit=101 ``` Without `RUST_BACKTRACE=1` the same panic and the same `exit=101` still -happen, but the frame names are gone — the backtrace is opt-in, the crash +happen, but the frame names are gone; the backtrace is opt-in, the crash is not. ## Remote debugging: gdbserver, target remote, and CLion Everything above assumes gdb and the crashing binary share a filesystem. That stops being true the moment the thing you need to debug lives on a -different machine — this book's `systems-target` lab VM, a container, a +different machine: this book's `systems-target` lab VM, a container, a device you cannot install a full toolchain on. `gdbserver` solves this by splitting gdb in two: a thin stub (`gdbserver`) runs *next to* the target process and does nothing but single-step it, read its memory, and set breakpoints on request; your full gdb runs on the host, keeps all its symbol tables and pretty-printers locally, and drives the stub over gdb's **remote serial protocol**, typically carried over TCP. Only the protocol's -small binary command set crosses the wire — source, DWARF, and the +small binary command set crosses the wire; source, DWARF, and the pretty-printer scripts never leave the host. {% include excalidraw.html file="28-remote-debug-topology" alt="Two stacked bands. The upper host band, marked verified this session, shows the local core-dump path: app crash producing a SIGSEGV, an arrow into systemd-coredump, coredumpctl dump extracting core.app, and gdb app core.app doing a full bt. Beside it, a second host-side node reads gdb (client), target remote systems-target:2345, with an amber bidirectional arrow labeled gdb remote serial protocol, TCP:2345 descending into the lower systems-target VM band, marked documented, not run this session. That band holds a gdbserver :2345 --once app crash node feeding an app (crash target) child process node below it, plus a dashed ghost box off to the side reading CLion, Remote GDB Server run config, unverified. A footer note says gdbserver is preinstalled on the lab VM image via cloud-init but not on this host, and no systems-target VM was up this session." - caption="Figure 28.2 — the local core-dump path this chapter's evidence is built on, beside the gdbserver/target-remote/CLion path documented for the systems-target VM" %} + caption="Figure 28.2: the local core-dump path this chapter's evidence is built on, beside the gdbserver/target-remote/CLion path documented for the systems-target VM" %} The session looks like this in practice: @@ -203,14 +203,14 @@ Remote debugging using systems-target:2345 ``` `--once` tells `gdbserver` to serve exactly one debug session and exit -afterward — the right default for a scripted reproduction, since a stray +afterward; the right default for a scripted reproduction, since a stray `gdbserver` left listening on a lab VM is a debug backdoor with no authentication. `target remote` is the one gdb command that changes everything downstream of it: every `break`, `run` (really `continue`, since the process is already started), `print`, and `bt` you already know behaves identically, dispatched over the wire instead of a local `ptrace(2)`. The binary path you pass to gdb on the host must be the *unstripped*, debug-info -copy — gdbserver on the VM only needs the raw executable, gdb on the host +copy; gdbserver on the VM only needs the raw executable, gdb on the host supplies all the symbols. This exact transcript is unverified -in this chapter — it is real, documented CLion functionality (Settings → +in this chapter; it is real, documented CLion functionality (Settings → Build, Execution, Deployment → Toolchains → a remote toolchain over SSH, then a Remote GDB Server run configuration referencing it), not something run here. ## How the code works -`app` walks the identical four-function call chain in every language — -`main -> run_supervisor -> handle_child -> deref` — and it is written to +`app` walks the identical four-function call chain in every language +(`main -> run_supervisor -> handle_child -> deref`), and it is written to resist the optimizer erasing that chain: every frame function is marked noinline (`[[gnu::noinline]]` / `//go:noinline` / `#[inline(never)]`), and all three builds keep debug info in an otherwise optimized binary (C++'s @@ -247,15 +247,15 @@ all three builds keep debug info in an otherwise optimized binary (C++'s Rust's `Cargo.toml` sets `[profile.release] debug = true`). Rust needed one more trick: `handle_child`'s call to `deref` is its last statement, so without a following `std::hint::black_box(())`, LLVM turned it into a tail -call and reused `handle_child`'s own frame for `deref` — erasing +call and reused `handle_child`'s own frame for `deref`, erasing `handle_child` from every backtrace. Verified directly: removing the `black_box` line made `RUST_BACKTRACE=1`'s trace jump straight from `app::deref` to `app::run_supervisor`. -The data shape is one `ChildRecord` — a pid plus a slot for the child's exit +The data shape is one `ChildRecord`: a pid plus a slot for the child's exit status that a supervisor's reaper is supposed to fill in before anything reads it. In `run` mode that slot is populated first; in `crash` mode -`handle_child` runs *as if it fired before the reaper got there* — the +`handle_child` runs *as if it fired before the reaper got there*, the exact race a real signal-driven supervisor (this book's `pmon`, from Part 3 onward) can hit if a `SIGCHLD` notification and the code consuming it get out of step. Each language expresses "not populated yet" the way its type @@ -330,19 +330,19 @@ fn handle_child(child: &ChildRecord) { ``` C++'s `const int* exit_slot = nullptr` is the raw pointer the whole bug class -is named after — `*child.exit_slot` reads through address `0x0`, the MMU +is named after: `*child.exit_slot` reads through address `0x0`, the MMU raises a page fault on an unmapped page, and the kernel delivers `SIGSEGV` with no C++-level participation at all. Go's `*int` is the same shape at the machine level, but the runtime stands between the fault and the process: its signal handler recognizes a `nil`-pointer-shaped fault from user code and converts it into `runtime.Error`, which surfaces as an ordinary panic. -Rust rejected the raw-pointer option outright — safe Rust has no way to -leave a pointer dangling like this — so the equivalent "not filled in yet" +Rust rejected the raw-pointer option outright: safe Rust has no way to +leave a pointer dangling like this, so the equivalent "not filled in yet" state is `Option::None`, and `.unwrap()` is an explicit, checked operation that panics on `None` rather than reading through anything. `run_supervisor` is the wiring in all three: it builds one `ChildRecord`, chooses `nullptr`/`nil`/`None` versus a real value based on `inject_bug`, and -calls `handle_child` either way — the crash path and the clean path are +calls `handle_child` either way; the crash path and the clean path are *the same code*, differing only in the data, which is the point: this bug is not a different code path, it's a value that should have arrived and didn't. @@ -352,7 +352,7 @@ didn't. The subcommand parsing is this book's usual three-way split and is identical output across languages: no argument or an unrecognized one prints the two-line `usage: app run` / `app crash` block to stderr and -exits 2. `app run` prints three fixed lines and exits 0 in every language — +exits 2. `app run` prints three fixed lines and exits 0 in every language: byte-identical, dual-use, unremarkable. `app crash` is where the three languages part ways, and the table is the chapter in miniature: @@ -364,21 +364,21 @@ languages part ways, and the table is the chapter in miniature: One more thing worth reading off the C++ trace directly: the `ShutdownGuard` RAII type in `run_supervisor`'s clean path prints "supervisor exiting -cleanly" from its destructor, and that line never appears on the crash path -— confirmed in every transcript above. RAII buys you cleanup on every +cleanly" from its destructor, and that line never appears on the crash path, +confirmed in every transcript above. RAII buys you cleanup on every *C++-level* exit (`return`, a thrown exception, `std::exit` via a registered handler); it buys you nothing against a signal, because a signal does not -unwind the stack — the kernel simply stops the process where it stands. +unwind the stack; the kernel simply stops the process where it stands. ## What each language's tooling gives you This chapter is tooling-heavy rather than concurrency-heavy, so the lens here is what each debugger sees, not a data race: gdb sees raw memory, -registers, and DWARF — it is precise but blind to any concept a language +registers, and DWARF: it is precise but blind to any concept a language runtime invented for itself. That is exactly why it is the *only* tool for the C++ crash (there is no higher-level runtime to ask) and only an *accidentally* adequate one for Go (it happened to catch this crash while -the program was still single-goroutine) — `delve` exists specifically to +the program was still single-goroutine); `delve` exists specifically to close that gap by understanding the `G`/`M`/`P` scheduler gdb has no model of. Rust splits the difference: `rust-gdb` gives gdb's raw power *plus* enum/`Option`-aware rendering, but for an ordinary panic-and-unwind exit @@ -415,8 +415,8 @@ Stack trace of thread 2906298: ``` Notice `coredumpctl`'s own quick trace, which is itself worth reading -carefully: it names only three frames — `println`'s formatting internals, -`run_supervisor`, `main` — silently skipping `deref` and `handle_child`. +carefully: it names only three frames (`println`'s formatting internals, +`run_supervisor`, `main`), silently skipping `deref` and `handle_child`. That trace is a lightweight summary, not a full DWARF-driven unwind; it is *not* the tool to trust for a real backtrace, and gdb's full walk against the extracted core proves it: @@ -437,7 +437,7 @@ All five frames, `deref` and `handle_child` included, with the exact `exit_slot = 0x0` that caused the fault sitting right there in frame #3's locals. The runner agrees end to end: `python3 scripts/test-all-examples.py --only 28-gdb-and-remote-debugging` reports `PASS PASS PASS`, and each -language's `verify.lua` run separately confirms its own crash contract — +language's `verify.lua` run separately confirms its own crash contract: `cpp`'s scripted `gdb --batch -ex run -ex bt` names `deref`, `handle_child`, and `run_supervisor`; `go`'s exit code is 2 with the standard nil-deref panic text; `rust`'s exit code is 101 with @@ -448,7 +448,7 @@ without it. Every address gdb reported above is derived from the same `.debug_line` section `readelf` and `objdump` can decode independently, with no gdb -involved at all — if they agree, gdb isn't inventing anything. `readelf -S` +involved at all; if they agree, gdb isn't inventing anything. `readelf -S` first confirms the binary actually carries full DWARF (it is optimized, not stripped): @@ -476,7 +476,7 @@ breakpoint on line 52 (`Breakpoint 1 at 0x401680: file .../main.cpp, line 52`) is the identical address `objdump` reports for that line, and the core backtrace's frame #2 return address (`0x00000000004016b2`) is the identical address `objdump` reports for line 56. Line 65's row (`0x401772`) is close -to but not identical to frame #3's return address (`0x40177c`) — expected, +to but not identical to frame #3's return address (`0x40177c`), expected, since a frame's *return* address sits a few bytes past a `call` instruction's own encoding, not at the statement's first byte the way a breakpoint address does. `readelf -n` supplies one more independent @@ -492,13 +492,13 @@ debug-info walk: across all three languages: Go's toolchain always emits a `.debug_info` / `.debug_line` pair (Go tooling doesn't offer a way to omit DWARF from a default build), and Rust's release binary carries the same sections because -of this crate's `[profile.release] debug = true` — three different +of this crate's `[profile.release] debug = true`: three different compilers, one DWARF-based line-number contract that `readelf`, `objdump`, and gdb all read the same way. ## What you learned -- A core dump on a modern systemd host is not a bare `core` file — it's a +- A core dump on a modern systemd host is not a bare `core` file; it's a `systemd-coredump` entry `coredumpctl` finds by pid and extracts on request; opening that extracted core in gdb (`gdb app core.app`) gives a full DWARF-driven backtrace that a lightweight summary trace (like @@ -507,13 +507,13 @@ and gdb all read the same way. readable values instead of raw offsets, and `rust-gdb` extends the same idea to Rust's `Option`/enum layout (`exit_slot: core::option::Option::None`, not a bare discriminant byte). -- gdb's thread model is OS threads, not goroutines — it can look complete on +- gdb's thread model is OS threads, not goroutines: it can look complete on a single-goroutine crash and still be blind to the concurrency that matters; `delve`'s `goroutines`/`goroutine bt` exists specifically to close that gap. - `gdbserver` + `target remote` split gdb across a network boundary without - losing any capability — only the remote serial protocol crosses the wire, - every symbol and pretty-printer stays on the host — and `readelf`/`objdump` + losing any capability; only the remote serial protocol crosses the wire, + every symbol and pretty-printer stays on the host; and `readelf`/`objdump` on `.debug_line` are the independent cross-check that confirms gdb's own addresses rather than trusting them blind. @@ -523,7 +523,7 @@ happened. --- -

verified — every transcript +

verified: every transcript above except the two spans marked unverified (the `systems-target` gdbserver/target remote session, the CLion remote-debug configuration, and delve's @@ -555,6 +555,6 @@ exactly; and the runner printed 28-gdb-and-remote-debugging PASS PASS PASS. The gdbserver/ target-remote path is documented from the lab VM's cloud-init (gdb-gdbserver is provisioned there) and gdb's own command -semantics, not from a live session — confirm it by bringing +semantics, not from a live session; confirm it by bringing systems-target up and repeating the two-line transcript in "Remote debugging" against the deployed binary.

diff --git a/_docs/29-valgrind-sanitizers-miri.md b/_docs/29-valgrind-sanitizers-miri.md index 6d8db0a..089643e 100644 --- a/_docs/29-valgrind-sanitizers-miri.md +++ b/_docs/29-valgrind-sanitizers-miri.md @@ -2,21 +2,21 @@ title: "valgrind, Sanitizers, and Miri" order: 29 part: "Debugging" -description: "bugfarm seeds five defects — leak, use-after-free, an uninitialised read, a heap overflow, and a data race — behind one `app ` CLI in C++, Go, and Rust, and this chapter builds a real catch/miss matrix from actual runs: valgrind's memcheck/helgrind/massif against a plain binary (10-50x slower, and outright confused by Go's own runtime), the compile-time-instrumented ASan/UBSan/TSan/LSan against a rebuilt one, and Rust's miri catching the identical use-after-free class by symbolic interpretation of a small `unsafe` snippet instead of either." +description: "bugfarm seeds five defects: leak, use-after-free, an uninitialised read, a heap overflow, and a data race, behind one `app ` CLI in C++, Go, and Rust, and this chapter builds a catch/miss matrix from actual runs: valgrind's memcheck/helgrind/massif against a plain binary (10-50x slower, and outright confused by Go's own runtime), the compile-time-instrumented ASan/UBSan/TSan/LSan against a rebuilt one, and Rust's miri catching the identical use-after-free class by symbolic interpretation of a small `unsafe` snippet instead of either." duration: "60 minutes" --- Chapter 28 read a crash after it happened. This chapter catches five bugs -*before* they get the chance to crash anything — or, in one case, watches a +*before* they get the chance to crash anything; or, in one case, watches a tool fail to catch one at all. `bugfarm` is a single CLI, `app `, seeding one specific memory or concurrency defect per subcommand, built three times over. C++ has no compile-time protection against any of the five, so it is the only language that can express all of them; Go's garbage collector and runtime checks rule three out by construction; Rust's borrow checker rules out four. What is -left, in every language, is watched by a different kind of tool — a runtime +left, in every language, is watched by a different kind of tool: a runtime binary translator, a compile-time instrumentation pass, or a symbolic -interpreter — and this chapter runs all of them, for real, against the same +interpreter. This chapter runs all of them against the same five bugs, so the catch/miss matrix below is built entirely from commands executed this session, not from what each tool's documentation claims it does. @@ -29,14 +29,14 @@ directly, since instrumented builds are outside every chapter's normal {% include excalidraw.html file="29-tool-coverage-matrix" alt="A grid with defect down the rows -- leak, uninit, uaf, overflow, race -- and C++, Go, Rust across the columns. C++/leak: valgrind memcheck caught it, definitely lost. C++/uninit: valgrind memcheck caught it, uninitialised value(s); ASan alone missed it, needs MemorySanitizer. C++/uaf: valgrind caught it (invalid write, freed) and ASan caught it (heap-use-after-free). C++/overflow: valgrind caught it (0 bytes after a block) and ASan caught it (heap-buffer-overflow). C++/race: memcheck is not built for this and was not the tool tried; helgrind caught it (possible data race) and ThreadSanitizer caught it. Go/leak: self-detected via goroutines before and after; valgrind was confused, 630 errors and 0 real signal. Go/uninit, uaf, overflow: prevented by the runtime, exit 64, no tool needed. Go/race: go -race caught it, WARNING DATA RACE, exit 66. Rust/leak: valgrind caught it, 65536 bytes lost, and miri on nightly additionally reported memory leaked. Rust/uninit, uaf, overflow, race: prevented at compile time by the borrow checker, exit 64; a note says unsafe Rust plus miri catches the use-after-free class separately, shown elsewhere in the chapter. A footer legend says amber cells mean a tool run this session caught the defect, dark cells mean the language prevents it outright, and plain cells mean a real blind spot this session measured; every cell traces to a command in Build run observe or Cross-check." - caption="Figure 29.1 — the five bugfarm defects against three languages, every cell backed by a command this chapter actually ran" %} + caption="Figure 29.1: the five bugfarm defects against three languages, every cell backed by a command this chapter actually ran" %} -> **Tools used** — `valgrind` 3.27.1 (`--tool=memcheck`, `--tool=helgrind`, +> **Tools used**: `valgrind` 3.27.1 (`--tool=memcheck`, `--tool=helgrind`, > `--tool=massif`), `cmake`/`g++` 16.1.1 with the `asan`/`tsan` CMake presets > (`-fsanitize=address,undefined` / `-fsanitize=thread`), `go build -race`, > `cargo +nightly miri` (all host). `valgrind` and the sanitizer runtimes > (`libasan`, `libtsan`) are hard requirements in `scripts/check-host.sh`; -> `miri` is not — it tracks Rust nightly, and this chapter documents exactly +> `miri` is not: it tracks Rust nightly, and this chapter documents exactly > what happens whether or not a nightly-plus-miri toolchain is installed. ## Two ways to watch a program: translate the binary, or instrument the source @@ -48,11 +48,11 @@ inserting extra instructions around every memory access. `memcheck`, the default tool, gives every byte of memory a shadow bit tracking *addressability* (has this byte been allocated?) and a shadow byte tracking *definedness* (has this byte been written to?), checked in software on every -load and store — no hardware assist, because ordinary CPUs have no such +load and store; no hardware assist, because ordinary CPUs have no such instruction. That software check, repeated for every single memory access of the whole program, is the entire reason for the well-known 10-50x slowdown: this chapter's own `uaf` subcommand took **0.002s** plain, **0.040s** under -ASan, and **0.97-0.99s** under valgrind memcheck — valgrind ran roughly +ASan, and **0.97-0.99s** under valgrind memcheck: valgrind ran roughly **25x slower than the sanitizer build** for the identical bug, on a binary so small that most of that second is JIT-translation and process startup, not steady-state emulation. `helgrind` swaps memcheck's shadow bytes for a @@ -60,21 +60,21 @@ lockset-plus-happens-before model over the same binary-translation core (the `race` subcommand measured 1.22s under it, slightly *heavier* than memcheck's 1.02s on the same input); `massif` swaps in a heap-usage sampler, taking periodic snapshots of live allocation size instead of -checking every access — running it against `leak` produced a real, +checking every access: running it against `leak` produced a real, plottable curve peaking at **80.02 KB** before the process exits with the leaked block still counted. Because valgrind operates on the compiled binary with no notion of what produced it, it also has no notion of runtimes that manage their own memory -outside libc — and Go is exactly that. Pointing valgrind at the Go `leak` +outside libc, and Go is exactly that. Pointing valgrind at the Go `leak` binary produces **630 errors from 411 contexts**, every single one a false "Conditional jump or move depends on uninitialised value(s)" or "Use of uninitialised value" deep inside `runtime.lock2`, `runtime.gopark`, and -`runtime.mcall` — the goroutine scheduler's own stack-switching and lock-free +`runtime.mcall`: the goroutine scheduler's own stack-switching and lock-free bookkeeping look like impossible, undefined states to an instruction-level model built around one C-style stack per OS thread. Worse, the `HEAP SUMMARY` reports **0 allocs, 0 frees, 0 bytes allocated** for the entire run, -even though the program printed a real `leaked=1` on stdout — Go's allocator +even though the program printed a real `leaked=1` on stdout. Go's allocator talks to the kernel directly via `mmap`, bypassing the `malloc`/`free` calls valgrind intercepts, so its leak-tracking machinery has nothing to attach to at all. Neither the 630 false positives nor the invisible real leak is a @@ -84,7 +84,7 @@ chapter's own transcript is the proof. **Sanitizers** take the opposite trade: instrument the *source*, at compile time, and ship a differently-linked binary (`libasan.so`, `libtsan.so`) -whose every load and store already carries its own inline check — no +whose every load and store already carries its own inline check: no separate CPU, no re-translation, just a few extra machine instructions the compiler emitted. ASan's mechanism is **shadow memory**: one shadow byte covers eight bytes of application memory (`shadow_addr = (app_addr >> 3) + @@ -96,23 +96,23 @@ trips instantly. {% include excalidraw.html file="29-shadow-memory" alt="Two stacked bands. The upper application-memory band shows three boxes: a left redzone at addresses ending ff80 through ffff, a 4-byte int* p at address ending 0010 marked as new'd then delete'd with the note that writing through it is undefined behavior, and a right redzone at addresses ending 0018 through 007f. The lower shadow-memory band shows the corresponding three shadow bytes: fa labeled heap left redzone, fd labeled freed heap region and marked as the exact byte the write trips, and fa labeled heap left redzone again for the next block's guard. An arrow labeled shift right by 3 then add offset connects the freed block down to its fd shadow byte. A footer formula reads shadow address equals app address shifted right 3 plus offset, one comparison before every load or store, not a full CPU emulation, and a note gives the real legend from this session's ASan run -- 00 addressable, 01 through 07 partially addressable, fa heap redzone, fd freed heap region, f1 through f8 stack states, f9 global redzone -- plus the measured gap: 0.002s native, 0.046s under ThreadSanitizer, 1.0s under valgrind memcheck, for the same tiny program." - caption="Figure 29.2 — ASan's shadow-memory model, with the real freed-block shadow bytes this chapter's `uaf` run produced" %} + caption="Figure 29.2: ASan's shadow-memory model, with the real freed-block shadow bytes this chapter's `uaf` run produced" %} Running the `uaf` binary under ASan pulls exactly the `fd` byte at `0x7b29be3e0010` and reports `heap-use-after-free` with the freed-at and allocated-at stacks attached; running `overflow` reports `heap-buffer-overflow` the instant the write lands one byte into the `fa` redzone past a 40-byte block. **LSan** is bundled into the ASan runtime by -default on Linux and runs its own reachability walk at process exit — the +default on Linux and runs its own reachability walk at process exit: the same `leak` binary rebuilt under the `asan` preset reports `detected memory leaks` with no separate leak-checking flag needed. **UBSan** rides along in the same build (`-fsanitize=address,undefined`) but none of this chapter's five bugs happen to trip one of its specific checks (signed overflow, invalid enum values, and the like are its domain, not heap safety), so it -stays silent here — worth naming so its absence in these transcripts doesn't -read as a miss. **TSan** needs its own separate build (ASan and TSan cannot -be linked into the same binary) and uses yet another shadow model — a vector -clock per thread recording happens-before relationships — to catch a race +stays silent here: its absence from these transcripts is expected, not a +missed catch. **TSan** needs its own separate build (ASan and TSan cannot +be linked into the same binary) and uses yet another shadow model (a vector +clock per thread recording happens-before relationships) to catch a race *while it actually executes*, meaning it can only report interleavings the run actually hits. The overhead lands close to native either way: `race` measured **0.03-0.046s** under TSan versus **0.002s** plain, roughly the @@ -121,22 +121,22 @@ valgrind's order-of-magnitude software CPU. **Go's `-race`** is the same idea under a different name: `go build -race` links the ThreadSanitizer runtime directly into the Go binary, which is why -its `WARNING: DATA RACE` output — two conflicting accesses, each with a full -goroutine stack, plus the creation site of both goroutines — looks so much +its `WARNING: DATA RACE` output (two conflicting accesses, each with a full +goroutine stack, plus the creation site of both goroutines) looks so much like TSan's C++ report. It is TSan, wearing Go's tooling. **miri** is neither of the above: it never runs your compiled machine code at all. It interprets the program's MIR (Rust's mid-level IR) inside an abstract machine that tracks every allocation's provenance and validity, so it can flag undefined behavior the borrow checker never got a chance to rule -out — specifically, UB reachable only through `unsafe`. This host's pinned +out, specifically, UB reachable only through `unsafe`. This host's pinned toolchain (`rust-toolchain.toml`, `1.97.1`) has no miri component (miri tracks nightly, not stable), but a `nightly-x86_64-unknown-linux-gnu` -toolchain with miri **is** installed here, so this chapter ran it for real +toolchain with miri **is** installed here, so this chapter ran it directly rather than only documenting it. Against the actual `bugfarm` `leak` binary, `cargo +nightly miri run -- leak` reports `error: memory leaked: alloc1026 (Rust heap, size: 24, align: 8)` naming the exact `Box::leak` call -site — the same finding valgrind reached by a completely different route. +site: the same finding valgrind reached by a completely different route. To see miri catch a defect *only* `unsafe` can express, a five-line standalone snippet frees a `Box` through a raw pointer and then writes through it anyway: @@ -149,7 +149,7 @@ unsafe { } ``` -`cargo +nightly run` on this prints `read back: 7` and exits 0 — the +`cargo +nightly run` on this prints `read back: 7` and exits 0: the write "worked," which is exactly the danger UB represents. `cargo +nightly miri run` on the identical source refuses: @@ -162,9 +162,9 @@ help: alloc236 was deallocated here: src/main.rs:11:9 That is the same use-after-free class ASan catches in C++ (right down to naming the allocation and deallocation sites), reached through Rust's -`unsafe` escape hatch instead of a language with no such checks at all — +`unsafe` escape hatch instead of a language with no such checks at all, which is exactly the point: a type system cannot forbid what it never sees, -and `unsafe` is where Rust admits that plainly. +and `unsafe` is where Rust admits it. ## How the code works @@ -176,12 +176,12 @@ idiom (`std::expected`, `(string, error)`, `Result`), and one `do_*` function per variant holding the seeded defect. The enum, not a raw string comparison scattered through `main`, is what makes "five bugs, three languages, one dispatch shape" a closed, exhaustive match instead of an -open-ended set of string checks — C++'s `switch` over `Bug` even gets +open-ended set of string checks. C++'s `switch` over `Bug` even gets `std::unreachable()` after it, telling the compiler (and a reader) that every variant is handled. The C++ `do_leak`, `do_uaf`, `do_uninit`, and `do_overflow` functions each -route their pointer through a `touch()` helper — +route their pointer through a `touch()` helper: `asm volatile("" : : "r,m"(value) : "memory")`, an empty inline-asm block that tells the compiler "something outside this translation unit might read this value," the same trick benchmark libraries call `DoNotOptimize`. It @@ -189,17 +189,17 @@ matters more than it looks: rebuilding `do_leak` with `touch(buf)` removed but its per-element `buf[i] = i` loop left in place *still* reports the leak, because GCC's escape analysis treats a write into heap memory as an observable side effect it won't remove without proof nothing depends on it. -Strip the loop too — an allocation nothing ever writes to or reads — and +Strip the loop too (an allocation nothing ever writes to or reads) and `HEAP SUMMARY` drops straight to **0 allocs, 0 frees, 0 bytes allocated**: GCC proved the entire `new[]` unobservable and deleted it, malloc and all, confirmed directly on this host. `touch()` is what closes that gap; without it (or without a loop that keeps writing), there is no bug left to catch, -which is itself the lesson — an unobserved bug is dead code, and the +which is itself the lesson: an unobserved bug is dead code, and the compiler is entitled to remove dead code. Rust's `leak()` uses `std::hint::black_box(leaked.as_ptr())` for the identical reason: LLVM can see nothing observable depends on the buffer's *contents* (only its statically-known `.len()` gets printed), so without the barrier the whole -64 KiB `vec![0xAAu8; 65536]` — and its `Box::leak` — can vanish too. +64 KiB `vec![0xAAu8; 65536]`, and its `Box::leak`, can vanish too. The bugs themselves are small and literal. `do_uaf` calls `delete p` then writes `*p = 7` through the now-dangling pointer; `do_uninit` @@ -209,7 +209,7 @@ writes `arr[10]` into a 10-element array; `do_race` spawns two tight loop, then joins both (via `jthread`'s own RAII) before printing the result. Go's `leak()` spawns a goroutine blocked forever on an unbuffered channel receive nobody will ever send to, then compares -`runtime.NumGoroutine()` before and after — there is no external tool +`runtime.NumGoroutine()` before and after. There is no external tool watching the Go heap for this the way valgrind watches `malloc`, so the program has to notice its own leak. Rust's `leak()` is the one bug safe Rust can still express: `Box::leak` is a real, safe standard-library API @@ -222,14 +222,14 @@ anything for valgrind or miri to find at all. Bad usage is identical across all three: no subcommand, or an unrecognized one, prints `usage: app ` alone (for no argument) or `app: unknown bug: ` above that same usage line (for an -unrecognized one), and exits **2** — the same shape as every other +unrecognized one), and exits **2**, the same shape as every other chapter's `app`. The interesting three-way split is what each language does with a bug it *cannot* express: Go prints `go: prevented by the runtime — see chapter` for `uaf`/`uninit`/`overflow`, Rust prints `rust: prevented at compile time — see chapter` for `uaf`/`uninit`/`overflow`/`race`, and both exit **64** (`EX_USAGE`) rather than attempting anything. That fixed message and exit code *is* the -finding for those cells — not a placeholder, but the same "no reproducible +finding for those cells, not a placeholder, but the same "no reproducible bug here" result a real investigation would reach chasing the equivalent C++ defect in that language, arrived at by a runtime check or a rejected compilation instead of a crash. @@ -237,24 +237,24 @@ compilation instead of a crash. ## Concurrency lens: TSan vs. helgrind vs. `-race` The race is one unsynchronized `int counter` incremented from two threads -with no lock — the canonical lost-update bug — and this chapter ran all +with no lock (the canonical lost-update bug), and this chapter ran all three of that bug's detectors against it directly. **ThreadSanitizer** (the C++ `tsan` preset) named the exact colliding line, `main.cpp:67` (`++counter;`), printing both threads' stacks and a summary that exits **66**; a plain, uninstrumented three-run sample of the same -binary showed the race is not guaranteed to *manifest* every time — +binary showed the race is not guaranteed to *manifest* every time: `counter=400000` (the mathematically correct total for the workload) came back in one plain sample, while the TSan-instrumented run's altered timing produced `counter=200000`, half the expected total, alongside its report. **helgrind**, valgrind's other thread tool, found the identical line -through a completely different mechanism — a lockset-and-happens-before -model instead of TSan's vector clocks — reporting "Possible data race... +through a completely different mechanism (a lockset-and-happens-before +model instead of TSan's vector clocks), reporting "Possible data race... at (anonymous namespace)::bump() (main.cpp:67)" between the same two threads, both created with "Locks held: none." **`go -race`** found Go's version of the same shape at `main.go:70`, printing `WARNING: DATA RACE` for the conflicting pair and `Found 2 data race(s)` before exiting **66**; a plain Go run in this same session came back `counter=100302` instead of -the expected `200000` — a visibly corrupted count, unlike the C++ plain +the expected `200000`: a visibly corrupted count, unlike the C++ plain run's lucky correct total, which is itself a reminder that "it printed the right number" is not evidence a race is absent. Rust needs none of this: the race subcommand exits 64 with `prevented at compile time`, because @@ -300,7 +300,7 @@ func bump(wg *sync.WaitGroup) { The C++ and Go excerpts are the bug itself, written by hand in both languages; the Rust excerpt is the same `Bug::Race` variant reaching the -`match` arm that refuses to try — there is no racy Rust function to show, +`match` arm that refuses to try: there is no racy Rust function to show, because the compiler never let one exist. ## Build, run, observe @@ -315,8 +315,8 @@ bugfarm: leak: allocated 4096 bytes on the heap, never freed (intentional) ==...== 4,096 bytes in 1 blocks are definitely lost in loss record 1 of 1 ``` -Build the instrumented variants directly — they sit outside `demo.sh`'s -build/run contract on purpose, since this chapter (unlike most) is about +Build the instrumented variants directly: they sit outside `demo.sh`'s +build/run contract because this chapter, unlike most, is about the tool, not a happy-path run: ```console @@ -361,16 +361,16 @@ previously allocated by thread T0 here: ... #1 do_uaf .../main.cpp:84 ``` Both name line 87 as the bad write and lines 84/85 as the allocation and -free — the same defect, independently confirmed. What differs is *how much +free: the same defect, independently confirmed. What differs is *how much work* got them there: valgrind's memcheck took **0.97-0.99s** (repeated twice for stability) against the plain binary with no rebuild required; the ASan build, needing `cmake --preset asan` first, then ran in -**0.040s** — about **25x faster** — and its report additionally carries a +**0.040s**, about **25x faster**, and its report additionally carries a full shadow-byte dump (`fa fa fa fa[fd]fa fa ...`) that pinpoints exactly which byte tripped the check, something memcheck's report describes in words but ASan shows you directly. Neither number should be read as "the" -overhead for real programs — this binary does almost nothing, so both -times are dominated by fixed startup cost — but the *ratio*, valgrind an +overhead for production workloads: this binary does almost nothing, so both +times are dominated by fixed startup cost, but the *ratio*, valgrind an order of magnitude slower than a rebuild-and-relink sanitizer for identical output, is exactly the trade-off the tool-selection question in this chapter's opening comes down to: rebuild if you can, translate the @@ -379,8 +379,8 @@ binary if you cannot. ## What you learned - valgrind instruments the *binary* at runtime (no rebuild, works on - anything) at the cost of a software-emulated CPU — measured at roughly - **25x** the ASan build's time for an identical `uaf` report — and that + anything) at the cost of a software-emulated CPU, measured at roughly + **25x** the ASan build's time for an identical `uaf` report. That same binary-translation model has no concept of a runtime that manages its own stacks and heap outside libc, which is why pointing it at a Go binary produced **630 false-positive errors** and a heap summary blind @@ -388,17 +388,17 @@ binary if you cannot. - Sanitizers instrument the *source* at compile time via shadow memory (one byte per eight application bytes, `>>3 + offset`), trading a required rebuild for overhead close to native (TSan's race run at **0.03-0.046s** - versus plain's **0.002s**) — and LSan/UBSan ride along in the same ASan + versus plain's **0.002s**), and LSan/UBSan ride along in the same ASan build rather than needing separate invocations. - miri interprets MIR symbolically instead of running compiled machine code at all, which lets it catch undefined behavior reachable only - through `unsafe` — demonstrated live on this host's nightly toolchain + through `unsafe`, demonstrated live on this host's nightly toolchain against both the real `Box::leak` bugfarm binary and a standalone use-after-free snippet, naming the exact allocation and deallocation sites the same way ASan does for C++. - The same race bug, checked three ways (TSan, helgrind, `go -race`), gets three different detection mechanisms (vector clocks, lockset/happens-before, - and TSan borrowed wholesale) converging on the identical source line — + and TSan borrowed wholesale) converging on the identical source line, while Rust's `Send`/`Sync` rules mean there is no racy binary for any of them to point at in the first place. @@ -408,7 +408,7 @@ no rebuild, on the lab VM this chapter's tools never needed. --- -

verified — every transcript, +

verified: every transcript, number, and diagnostic above was produced on the Fedora 44 reference host (kernel 7.1, valgrind-3.27.1, gcc 16.1.1, go1.26.5, rustc/cargo 1.97.1, miri 0.1.0 nightly-2026-07-18) this session. Confirmed live: diff --git a/_docs/30-ebpf-observation-toolkit.md b/_docs/30-ebpf-observation-toolkit.md index c0e28f2..e57a38c 100644 --- a/_docs/30-ebpf-observation-toolkit.md +++ b/_docs/30-ebpf-observation-toolkit.md @@ -2,7 +2,7 @@ title: "The eBPF Observation Toolkit" order: 30 part: "Debugging" -description: "app work --seconds N runs an ordinary, unmodified workload on systems-target, and this chapter points the standard eBPF observation stack at it live: bcc-tools' execsnoop/opensnoop/funccount/offcputime/biolatency (and a real, plainly-reported profile failure), a bpftrace crash course from one-liners to a uprobe+uretprobe latency histogram, bpftool naming the loaded programs, c++filt demangling the exact mangled symbol a uprobe attaches to in C++ and Rust, and a live, measured RDI-vs-RAX register mismatch that shows exactly why uprobes on Go need more care than the other two — with SystemTap's kernel-module and dyninst backends compared alongside, one working and one blocked by a real kernel-devel mismatch." +description: "app work --seconds N runs an ordinary, unmodified workload on systems-target, and this chapter points the standard eBPF observation stack at it live: bcc-tools' execsnoop/opensnoop/funccount/offcputime/biolatency (and a real, clearly reported profile failure), a bpftrace crash course from one-liners to a uprobe+uretprobe latency histogram, bpftool naming the loaded programs, c++filt demangling the exact mangled symbol a uprobe attaches to in C++ and Rust, and a live, measured RDI-vs-RAX register mismatch that shows exactly why uprobes on Go need more care than the other two, with SystemTap's kernel-module and dyninst backends compared alongside, one working and one blocked by a real kernel-devel mismatch." duration: "55 minutes" --- @@ -10,14 +10,14 @@ Chapter 28 read a crash gdb already knew about; Chapter 29 caught bugs a compile-time instrumentation pass or a binary-translating emulator watched for. Both chapters needed something extra glued onto the target: debug info gdb could walk, or a sanitizer/valgrind rebuild. This chapter needs -neither. `app work --seconds N` is an ordinary, unmodified binary — no -special build, no `-fsanitize`, not even a debugger attached — and the +neither. `app work --seconds N` is an ordinary, unmodified binary (no +special build, no `-fsanitize`, not even a debugger attached), and the standard eBPF observation stack (bcc-tools, bpftrace, bpftool) watches it entirely from outside, live, on the `systems-target` lab VM this book has -used since Part 3. The workload is deliberately small and legible: every +used since Part 3. The workload is kept small and legible: every ~230ms it opens/writes/closes one fixed file, forks and execs a short-lived `true` every fourth iteration, calls one named hot function `busy_hash()`, -and sleeps through most of the interval — four separate baits for five +and sleeps through most of the interval: four separate baits for five different tools, wired identically into C++23, Go, and Rust so the same observation session works against all three languages' binaries. This chapter also attaches uprobes to *our own* symbols directly, which is where @@ -27,16 +27,16 @@ footnote. The code is in `examples/30-ebpf-observation-toolkit/`. `./demo.sh build` builds all three; `observe.sh [seconds]`, run as root on -the lab guest, is this chapter's actual demo — it starts the workload and +the lab guest, is this chapter's actual demo: it starts the workload and points five tools at it in turn, and the top-level `README.md` documents the full contract and a real run's tail. {% include excalidraw.html file="30-ebpf-tooling-stack" alt="Three stacked bands. The top kernel band holds five small boxes -- kprobe/kretprobe on kernel functions, uprobe/uretprobe on userspace functions, tracepoint as the stable kernel ABI, perf_event as timer/PMU sampling, and BTF/CO-RE for struct layout without headers. Amber arrows descend from three of these into the middle eBPF-tooling band, which holds bcc-tools (execsnoop/opensnoop/biolatency/offcputime/funccount/profile), bpftrace (one-liners and scripts, probe:target /filter/ action), bpftool (prog show/map show), and c++filt (demangle the exact symbol you attach to), with a note that libbpf loads a small verified BPF program per probe that the kernel runs in-kernel on each hit. Below that, a third band holds SystemTap as the sibling technology: its classic kernel-module backend needing matching kernel-devel, beside stap --runtime=dyninst, which patches the target binary directly in userspace with no kernel module. A fourth band at the bottom shows the three target binaries -- C++'s busy_hash under the SysV ABI with a correct arg0, Rust's cmd_work also correct under the SysV ABI once c++filt demangles its v0-mangled name, and Go's cmdWork in a dark box reading arg0/RDI wrong at 1, RAX right at 9, with a dashed amber arrow from it up to the eBPF-tooling band labeled arg0 mismatch. A footer note says every amber box ran live against systems-target this session; SystemTap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, and the dyninst backend attached without erroring." - caption="Figure 30.1 — the eBPF observation stack this chapter exercises live: kernel probe primitives at the bottom, bcc-tools/bpftrace/bpftool/c++filt above them, SystemTap as the sibling technology beside them, and the one real register-ABI mismatch that only shows up on the Go binary" %} + caption="Figure 30.1: the eBPF observation stack this chapter exercises live: kernel probe primitives at the bottom, bcc-tools/bpftrace/bpftool/c++filt above them, SystemTap as the sibling technology beside them, and the one real register-ABI mismatch that only shows up on the Go binary" %} -> **Tools used** — bcc-tools `execsnoop`, `opensnoop`, `funccount`, +> **Tools used**: bcc-tools `execsnoop`, `opensnoop`, `funccount`, > `offcputime`, `biolatency`, `profile` (`systems-target` VM, root); > `bpftrace` v0.24.2 (`systems-target` VM, root); `bpftool` (`systems-target` > VM, root); `c++filt` (host, to demangle before attaching); `nm` (host, to @@ -56,7 +56,7 @@ the **kernel itself** runs a small, verified BPF program on every hit, copying whatever you asked for back to userspace through a map or a ring buffer. No rebuild, because the probe attaches to the compiled code exactly as it stands; no restart even, because a `uprobe` can attach to a process -that is already running. The trade is `CAP_BPF`/`CAP_PERFMON` — on this lab +that is already running. The trade is `CAP_BPF`/`CAP_PERFMON`: on this lab image that means root, which is why every tool in this chapter runs under `sudo` on `systems-target` and never on the host, matching this book's rule that eBPF is tooling we point at our own userspace programs, never kernel @@ -64,17 +64,17 @@ code we write ourselves. Four probe types cover everything this chapter touches. A **kprobe** (and its return-side twin, **kretprobe**) attaches to almost any non-inlined -kernel function by name — `vfs_write`, say — and fires on entry or return. +kernel function by name (`vfs_write`, say) and fires on entry or return. A **uprobe**/**uretprobe** is the identical idea one level up: instead of a kernel symbol, it takes a userspace **binary path plus a symbol name** (or a raw offset), and it needs nothing from the target process except that the -symbol exist in its ELF symbol table — no debug info, no cooperation, no +symbol exist in its ELF symbol table: no debug info, no cooperation, no special build. A **tracepoint** is a stable, kernel-maintained hook baked into specific spots (`syscalls:sys_enter_openat` fires on every `openat(2)` entry, arguments included) that will not move or disappear the way a raw kprobe's target function occasionally does across kernel versions. A **perf_event** probe fires on a timer or a PMU counter rather than a -specific code location — that is what an on-CPU sampling profiler rides on. +specific code location; that is what an on-CPU sampling profiler rides on. BTF (BPF Type Format) is the piece that lets modern tools read kernel `struct` layouts without a matching `kernel-devel` package at all, which matters directly to this chapter's SystemTap comparison below. @@ -83,7 +83,7 @@ matters directly to this chapter's SystemTap comparison below. **bcc** ships as a directory of small, ready-made tools (`/usr/share/bcc/tools/` on this image), each one a purpose-built BPF -program with a Python front end — you do not write BPF C for any of these, +program with a Python front end: you do not write BPF C for any of these, you invoke the tool and read its output. `observe.sh` runs five of them against `app work --seconds N` in sequence, each one confirming a specific bait fired. A real run against the C++ binary, staged at @@ -121,14 +121,14 @@ app_summary: work: done seconds=40 iters=173 opens=173 execs=44 busy_calls=173 b `opensnoop` traces every `open`/`openat` and names the exact path our workload writes, by comm `app`, not an inference from timing. `execsnoop` -shows `true` with **PPID 15320**, our workload's own pid — proof this is +shows `true` with **PPID 15320**, our workload's own pid, proof this is *our* fork/exec, not unrelated system noise. `funccount` attaches a uprobe at the pattern `:*busy_hash*` and just counts hits, reporting **21** for a ~40s window at ~230ms/iteration (173 total iterations, sampled mid-run for a shorter tool budget). `offcputime` samples stacks of threads that are *not* running, bucketed by how long they were off-CPU, and the sleep inside every iteration reliably parks our thread in -`__clock_nanosleep_2` — exactly the bait. `biolatency` is the fifth tool in +`__clock_nanosleep_2`, exactly the bait. `biolatency` is the fifth tool in the family tour, run separately against real block I/O (the lab guest's `/tmp` is tmpfs, so this needs the root filesystem): @@ -142,7 +142,7 @@ Tracing block device I/O... Hit Ctrl-C to end. ``` That histogram came from five `dd ... conv=fsync` writes to `/var/tmp` -completing in the low hundreds of microseconds — a real, measured +completing in the low hundreds of microseconds, a real, measured distribution, not a synthetic example. `profile`, the sixth named tool, is where this tour hit a real wall worth reporting rather than hiding: @@ -157,13 +157,13 @@ Traceback (most recent call last): ``` `profile` compiles a small C snippet against the running kernel's own -headers at load time — even bcc tools with BTF support are not entirely -immune to kernel drift — and on this lab guest's kernel (6.19.10) that +headers at load time; even bcc tools with BTF support are not entirely +immune to kernel drift, and on this lab guest's kernel (6.19.10) that snippet references a `struct ns_common` field the installed headers no longer have under that name. bpftrace's own `profile:hz:N` probe type sidesteps the header dependency and attached cleanly, but on this exact workload it reported **zero samples** at both 49Hz and 99Hz over several -seconds — not a bug, but the real result of an on-CPU sampler pointed at +seconds, not a bug, but the real result of an on-CPU sampler pointed at a process that spends over 99% of its time asleep in `nanosleep`. `offcputime` and `profile` are answering two different questions (where did the *waiting* time go, versus where did the *running* time go), and a @@ -187,7 +187,7 @@ openat comm=app file=/tmp/lsp-ebpf-work-bait.txt ... ``` -A kprobe one-liner aggregating into a histogram — every `vfs_write` call +A kprobe one-liner aggregating into a histogram: every `vfs_write` call from that same pid, bucketed by byte count (`arg2` is `vfs_write`'s third kernel argument, the length): @@ -200,7 +200,7 @@ kernel argument, the length): ``` Every bait line the workload writes (`"iter N\n"`) is 8-15 bytes most of -the time, 16-31 once digits roll over — the histogram is reading the exact +the time, 16-31 once digits roll over: the histogram is reading the exact shape of our own `write(2)` calls. The short script below is the crash course's third piece: a **uprobe/uretprobe pair**, timing one function entry-to-return and bucketing the result, saved as @@ -219,7 +219,7 @@ uretprobe:$1:busy_hash /@start[tid]/ { interval:s:6 { exit(); } ``` -`@start[tid]` is a per-thread-id associative map — the entry probe stashes +`@start[tid]` is a per-thread-id associative map: the entry probe stashes a timestamp keyed by the calling thread, and the matching return probe looks it up, computes the delta, and folds it into a histogram before deleting the entry so a later call cannot read a stale timestamp. Run @@ -235,13 +235,13 @@ Attached 5 probes ``` `busy_hash`'s 300,000-round xorshift loop takes roughly 1-2ms most calls on -this host — a real, measured latency distribution for a function this -book's example code deliberately keeps simple, produced with five lines of +this host, a real, measured latency distribution for a function this +book's example code keeps simple, produced with five lines of bpftrace and no rebuild of `app` at all. ## bpftool: naming what is actually loaded -**bpftool** is the inspection half of the story — it reads back what the +**bpftool** is the inspection half of the story: it reads back what the kernel currently has loaded, independent of whichever tool put it there. With the `busy_hash` uprobe from the previous section still attached in a background shell, a second root shell on the same guest: @@ -255,11 +255,11 @@ background shell, a second root shell on the same guest: Two things are worth reading closely here. First, `bpftrace` compiled the attach point's name straight into the BPF program's own name -(`uprobe__tmp_lsp30_cpp_app_busy_hash_1`) — `bpftool` is reading that name +(`uprobe__tmp_lsp30_cpp_app_busy_hash_1`); `bpftool` is reading that name back from the kernel, not from bpftrace's own bookkeeping, so this is an independent confirmation the probe really loaded. Second, the kernel classifies this program's `type` as plain **`kprobe`** even though we asked -bpftrace for a `uprobe` — uprobes and kprobes share the same underlying BPF +bpftrace for a `uprobe`: uprobes and kprobes share the same underlying BPF program type at the kernel ABI level; the distinction between "kernel function" and "userspace function" lives in how the probe is *attached* (a kernel symbol versus a binary path + symbol), not in a separate program @@ -277,13 +277,13 @@ process: `pids bpftrace(16154)` is the cross-check: this map is held open by the exact bpftrace process running our script, not a leftover from some -earlier session — `bpftool` and `bpftrace` are two independent programs +earlier session; `bpftool` and `bpftrace` are two independent programs agreeing on what is actually resident in the kernel right now. ## SystemTap: the sibling technology **SystemTap** predates eBPF-based tracing by close to a decade and solves -the same problem — attach a probe, run a small script, read the answer — +the same problem (attach a probe, run a small script, read the answer) through a different mechanism: its classic backend compiles your script into a loadable **kernel module**, which means it needs a `kernel-devel` package that matches the *exact* running kernel, not just a recent one. @@ -296,7 +296,7 @@ Incorrect version or missing kernel-devel package, use: dnf install kernel-devel ``` The installed `kernel-devel` on this image is built for `7.1.3-201.fc44`, -one version off the actually-running `6.19.10-300.fc44` — exactly the +one version off the actually-running `6.19.10-300.fc44`, exactly the class of drift bpftrace and bcc's BTF-backed tools were built to avoid. SystemTap also ships a userspace-only **dyninst** backend (`--runtime=dyninst`), which patches the target process's own code directly @@ -310,7 +310,7 @@ Pass 5: run completed in 9810usr/680sys/13771real ms. ``` That attached and ran to completion with no compile or attach error, but it -printed nothing in this session's test window — a real, if inconclusive, +printed nothing in this session's test window, a real, if inconclusive, result reported as such rather than claimed as a hit. The real comparison this callout can make from live evidence: SystemTap's classic path is more sensitive to exact kernel-version matching than the BTF-based bpftrace/bcc @@ -318,8 +318,8 @@ path used everywhere else in this chapter (which needed no matching `kernel-devel` at all), while its dyninst backend trades that dependency for a userspace-patching approach this session did not get a positive result from in the time available. SystemTap has real strengths bpftrace -does not — a proper scripting language and years of production hardening -among them — and Chapter 45 covers it in depth; this callout is only the +does not (a proper scripting language and years of production hardening +among them), and Chapter 45 covers it in depth; this callout is only the comparison this chapter's own live evidence supports. ## How the code works @@ -349,18 +349,18 @@ C++'s fields are `std::atomic` because the workload loop runs on a `std::jthread` while `main` could in principle read them concurrently; Go's equivalent `result` struct has no such requirement (a single goroutine owns it end to end, handed to `main` once, complete, over a channel) and -Rust's `Counters` is a plain `Default`-derived struct for the same reason — +Rust's `Counters` is a plain `Default`-derived struct for the same reason: the concurrency shape, not the language, decides whether the counters need to be atomic. The comment on `busy_hash_val` documents this chapter's own `README.md`-recorded gotcha: the first version of this workload printed `busy_calls` but never the accumulated hash, and GCC's interprocedural analysis proved the entire `busy_hash()` call chain had no observable -effect and deleted it — `funccount` and the bpftrace uprobe both reported +effect and deleted it: `funccount` and the bpftrace uprobe both reported **zero** calls even though `nm`/`objdump` still showed the symbol sitting right there in the binary. `noinline` only forbids folding the function into its caller; it says nothing about deleting a call whose result nothing reads. Printing the final hash in the `done` line, in all three languages, -is what makes the call chain provably necessary again — a fragile-but-named +is what makes the call chain provably necessary again, a fragile-but-named requirement worth restating here: if you trim this program down, keep the final `busy_hash` print or the whole bait disappears. @@ -372,12 +372,12 @@ Rust's `open_bait` returns a `rustix::fs::OwnedFd` and never calls the value goes out of scope at the end of the function; Go relies on `defer f.Close()` right after the successful open, the same "close no matter what" guarantee spelled a third way. All three open the bait file -`O_CREAT|O_WRONLY|O_TRUNC` at mode `0666` — world-writable on purpose, +`O_CREAT|O_WRONLY|O_TRUNC` at mode `0666`, world-writable by design, since the file may be created once by a root-run `observe.sh` and again by an unprivileged `demo.sh run`, and a stricter mode would make the second owner's open fail. `spawn_true`/`spawnTrue` forks (or lets the language runtime fork on its behalf, for Go and Rust's `std::process::Command`) and -execs `true`, then waits for it — the exact shape `execsnoop` watches, with +execs `true`, then waits for it: the exact shape `execsnoop` watches, with the child's real parent pid always the workload's own. The loop itself, `run_workload`/`runWorkload`, is the wiring: on every pass @@ -388,11 +388,11 @@ deadline computed from `--seconds` at start. C++ runs this loop on a `std::jthread` and `main` calls `worker.join()` explicitly, so the summary line always sees the final counters rather than racing the still-running thread; Go starts it as a goroutine and blocks `main` on a buffered result -channel — the channel receive *is* the join; Rust runs it straight-line on +channel: the channel receive *is* the join; Rust runs it straight-line on the main thread, no concurrency at all, since nothing else in the Rust build needs to run alongside it. `cmd_work`/`cmdWork` is the one-line dispatcher tying `main`'s parsed `--seconds` value to this loop and printing -the `start`/`done` lines around it — the exact function this chapter's +the `start`/`done` lines around it: the exact function this chapter's uprobe/mangling section below attaches a probe to directly. The named hot function itself, verbatim in all three languages: @@ -448,7 +448,7 @@ pub extern "C" fn busy_hash(mut x: u64) -> u64 { ## Errors, three ways The CLI contract is this book's usual shape and is byte-identical across -languages — `app work --seconds N` on success, and a bad or missing +languages: `app work --seconds N` on success, and a bad or missing `--seconds` prints a two-line usage/diagnostic message and exits **2** in every language, confirmed live: @@ -476,7 +476,7 @@ ERROR: Unable to attach probe: uprobe:.../app:no_such_function. If this is expec ``` `funccount`, by contrast, treats an unmatched pattern as an **empty result -set**, not an error — asking it for the exact unqualified name `busy_hash` +set**, not an error, asking it for the exact unqualified name `busy_hash` against the *Go* binary, which actually exports `main.busy_hash`: ```console @@ -492,12 +492,12 @@ per-language branch anywhere in the driver script. ## What each binary gives the uprobe tooling -`busy_hash` is deliberately kept unmangled in every language +`busy_hash` is kept unmangled in every language (`extern "C"` in C++, `#[unsafe(no_mangle)]` in Rust, and Go never mangles at all) precisely so one wildcard pattern (`*busy_hash*`) works across all three without a per-language branch anywhere in `observe.sh`. Attaching to any *other* function in the same binaries is where each language's real -symbol story — and one real ABI trap — shows up. `cmd_work`, the function +symbol story, and one real ABI trap, shows up. `cmd_work`, the function that dispatches `--seconds` into the workload loop, carries no such protection in any language, so `nm` shows what each compiler really does with a name by default: @@ -515,7 +515,7 @@ with a name by default: namespace (`(anonymous namespace)::`), which the mangler folds into `_ZN12_GLOBAL__N_1L8cmd_workEi`, and GCC's interprocedural specialization pass appended `.isra.0` (a clone with some arguments replaced by scalars, -"interprocedural scalar replacement of aggregates" — here a no-op clone, +"interprocedural scalar replacement of aggregates", here a no-op clone, but the suffix survives regardless). `c++filt` demangles it without any flag, auto-detecting the Itanium scheme: @@ -525,7 +525,7 @@ flag, auto-detecting the Itanium scheme: ``` **Rust** mangles under its own **v0** scheme (the `_R` prefix), and modern -`c++filt` auto-detects that too — no `--format=rust` flag needed on this +`c++filt` auto-detects that too; no `--format=rust` flag needed on this host's binutils: ```console @@ -534,14 +534,14 @@ app[4db24ee1e6ed4506]::cmd_work ``` **Go never mangles anything.** `main.cmdWork` *is* the linker symbol, -package-qualified and immediately readable — there is no demangling step +package-qualified and immediately readable; there is no demangling step for Go at all, which sounds like the simplest story of the three until the next section shows what it costs. {% include excalidraw.html file="30-uprobe-attachment" alt="Three parallel columns, one per language, each four boxes tall. C++ column: nm reports the mangled symbol t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0; c++filt (auto Itanium ABI) demangles it to (anonymous namespace)::cmd_work(int); a uprobe attaches using the MANGLED string as the attach target; the probe fires with seconds_arg=5 for --seconds 5, and arg0==RDI is correct because C++ follows the plain SysV x86-64 ABI. Rust column: nm reports t _RNvCs6FzKXswKZCc_3app8cmd_work; c++filt auto-detects Rust v0 mangling and demangles it to app[hash]::cmd_work; a uprobe attaches using that same mangled string; the probe fires with seconds_arg=7 for --seconds 7, and arg0==RDI is again correct under the SysV ABI. Go column: nm reports T main.cmdWork, never mangled; no demangling step is needed since the package-qualified name is already the symbol; a uprobe attaches to the plain unmangled name main.cmdWork; the probe fires for --seconds 9 but RDI(arg0) reads 1, which is wrong, while RAX correctly reads 9 -- a dashed amber arrow connects this box back up to the tooling band above, labeled arg0 mismatch. A footer band spans all three columns explaining that Go's ABIInternal (1.17+) passes integer arguments in the order RAX, RBX, RCX, RDI, RSI, R8, R9, ... rather than the SysV order RDI, RSI, RDX, ... that bpftrace's arg0/arg1 assume, and that goroutine stacks also grow by copying, so a captured stack address can move out from under a probe that read it from the stack instead of a register." - caption="Figure 30.2 — the same uprobe-attach recipe (find the symbol, demangle it to read it, attach to the mangled string) across three binaries, ending in the one real, measured argument-register mismatch that shows up only on Go" %} + caption="Figure 30.2: the same uprobe-attach recipe (find the symbol, demangle it to read it, attach to the mangled string) across three binaries, ending in the one real, measured argument-register mismatch that shows up only on Go" %} Attaching directly to the mangled string on the C++ and Rust binaries, with the workload started immediately after so the one-shot call to `cmd_work` @@ -561,7 +561,7 @@ hit cmd_work seconds_arg=5 rust hit cmd_work seconds_arg=7 ``` -Both print exactly the `--seconds` value passed on the command line — +Both print exactly the `--seconds` value passed on the command line: `arg0` (bpftrace's shorthand for the first System V x86-64 integer argument register, `RDI`) is correct in both, because both C++ (this function is not `extern "C"`, but the Itanium C++ ABI still lays out @@ -569,7 +569,7 @@ integer arguments the plain SysV way) and Rust place their first integer argument in `RDI` in practice on this target. **Go does not.** Since Go 1.17, the Go compiler's own **ABIInternal** passes integer and pointer arguments in registers in the order `RAX, RBX, RCX, RDI, RSI, R8, R9, R10, -R11` — a different order than the SysV convention every uprobe tool's +R11`, a different order than the SysV convention every uprobe tool's `arg0`/`arg1` shorthand assumes. The same attach recipe against Go's `main.cmdWork(seconds int) int`, reading both the naive `arg0` and the true first-argument register side by side: @@ -581,7 +581,7 @@ first-argument register side by side: go cmdWork rdi(arg0)=1 rax=9 ``` -`rax=9` is exactly `--seconds 9`; `rdi(arg0)=1` is not — it is whatever +`rax=9` is exactly `--seconds 9`; `rdi(arg0)=1` is not: it is whatever happened to be sitting in `RDI` from a previous call, because Go's compiler never put the argument there in the first place. This is the concrete version of "uprobes on Go are trickier": a tool that reads arguments by the @@ -591,7 +591,7 @@ answer is to know Go's own calling convention for that specific function (which can itself vary by whether the function is inlined, its argument count, and the Go toolchain version) rather than trust a generic `argN`. The second half of "trickier" is one this session did not need to -reproduce to state accurately: Go's goroutine stacks are **movable** — the +reproduce to state accurately: Go's goroutine stacks are **movable**, the runtime grows a goroutine's stack by copying it to a larger allocation and rewriting every pointer *it* tracks into the new location, but a raw stack address a uprobe captured externally is not one of the pointers the runtime @@ -611,7 +611,7 @@ to see coming. `observe.sh` prints the five-tool tour and a `=== summary ===` block deriving one hit count per tool, all nonzero, all tied to the workload's own -pid or bait path — never merely "the tool ran and exited 0." The runner +pid or bait path, never merely "the tool ran and exited 0." The runner drives the same contract for all three languages against the live VM: ```console @@ -622,12 +622,12 @@ drives the same contract for all three languages against the live VM: ## Cross-check: three independent views of one workload agreeing -No single tool in this chapter is trusted alone — every tool's count is +No single tool in this chapter is trusted alone: every tool's count is checked against either another tool watching the same process, or against `bpftool` reading back what the kernel actually loaded. From the same `observe.sh` run quoted above, `execsnoop` (fork/exec) and `bpftrace`'s -uprobe (function calls) are two completely different probe types — a -tracepoint-adjacent syscall tracer versus a userspace-symbol uprobe — +uprobe (function calls) are two completely different probe types (a +tracepoint-adjacent syscall tracer versus a userspace-symbol uprobe) watching the same 40-second window of the same pid, and they agree with each other and with the program's own printed counters: @@ -639,18 +639,18 @@ app_summary: work: done seconds=40 iters=173 opens=173 execs=44 busy_calls=173 b `execsnoop`'s 6 and the program's own `execs=44` differ only because the tool's own capture window (a `timeout`-bounded slice near the start of the -run) is shorter than the workload's full 40 seconds — exactly what +run) is shorter than the workload's full 40 seconds, exactly what `observe.sh`'s design note says to expect, and the ratio (6 execs in roughly 6 seconds of `execsnoop` runtime, at one exec every ~1 second given the 4-iteration cadence) lines up. The second, independent cross-check is `bpftool prog show` naming the exact uprobe program bpftrace loaded while it was still attached (`uprobe__tmp_lsp30_cpp_app_busy_hash_1`, `type kprobe`), read from the kernel's own bookkeeping rather than from anything -bpftrace itself reported — two separate programs reading the same kernel +bpftrace itself reported: two separate programs reading the same kernel state and agreeing. The third is the mangled-symbol pair: `nm` and `c++filt` independently identify and translate `cmd_work`'s real linker name off the binary, and a live uprobe attached to that exact string is -what actually fires and reports the right `--seconds` value back — proof +what actually fires and reports the right `--seconds` value back, proof the demangled name and the attach target are the same symbol, not a coincidence of two unrelated strings. @@ -659,23 +659,23 @@ coincidence of two unrelated strings. - eBPF-based observation is a third way to watch a program, beside binary translation (valgrind) and compile-time instrumentation (sanitizers): the kernel runs a small verified program on a real kprobe/uprobe/tracepoint/ - perf_event hit, live, with no rebuild and no restart — bcc-tools' + perf_event hit, live, with no rebuild and no restart: bcc-tools' `execsnoop`/`opensnoop`/`funccount`/`offcputime`/`biolatency` and bpftrace's one-liners and scripts all ran this way against an unmodified binary on `systems-target`. - `bpftool prog show`/`map show` is the independent check on what the - kernel actually has loaded — it named bpftrace's own uprobe program by + kernel actually has loaded: it named bpftrace's own uprobe program by the exact attach-point string bpftrace compiled into it, confirmed from the kernel's bookkeeping, not bpftrace's. - A uprobe attaches to the ELF symbol table's literal string, mangled or not: `c++filt` auto-detects and demangles both C++'s Itanium scheme and Rust's v0 scheme for reading, but the attach target itself stays the - mangled string in both languages — and both fire correctly (`arg0` + mangled string in both languages, and both fire correctly (`arg0` matching `--seconds`) because both follow the plain SysV x86-64 ABI. - Go never mangles, but its own register-based ABIInternal (`RAX` first, not `RDI`) means the same `arg0` shorthand silently reads the *wrong* - register — measured live as `rdi(arg0)=1` (wrong) beside `rax=9` (right, - matching `--seconds 9`) — and goroutine stacks moving under a copying + register: measured live as `rdi(arg0)=1` (wrong) beside `rax=9` (right, + matching `--seconds 9`), and goroutine stacks moving under a copying stack-grow makes any stack-based argument read riskier still. Next, **per-language toolbelts**: `perf`, `pprof`, `delve`, and @@ -685,7 +685,7 @@ outside. --- -

verified — every transcript +

verified: every transcript above (except the SystemTap dyninst run's silence, reported as inconclusive rather than a positive result) was produced against the running `systems-target` lab VM (kernel 6.19.10-300.fc44.x86_64, @@ -700,7 +700,7 @@ kernel package, stap 5.5) and the Fedora 44 host (binutils busy_calls=173 busy_hash=1836395105209072940; biolatency 1 3 produced a real block-I/O latency histogram against five dd conv=fsync writes; bcc's profile -failed to load with a genuine +failed to load with a real no member named 'ns_id' in 'struct ns_common' compile error on this kernel, while bpftrace's own profile:hz probe attached cleanly but reported zero on-CPU samples against a workload that sleeps @@ -735,7 +735,7 @@ error. The formal three-language runner 30-ebpf-observation-toolkit --mode vm) confirmed the same verify.lua contract end to end against this VM this session, printing 30-ebpf-observation-toolkit PASS PASS PASS (3 -passed, 0 failed, 0 skipped) for cpp/go/rust in turn — each language's own +passed, 0 failed, 0 skipped) for cpp/go/rust in turn, each language's own run independently reproducing opensnoop's bait-path match, execsnoop's PPID-matched true child, funccount's nonzero busy_hash count, offcputime's comm-app stack, and diff --git a/_docs/31-per-language-toolbelts.md b/_docs/31-per-language-toolbelts.md index 7f971d8..fd3a17d 100644 --- a/_docs/31-per-language-toolbelts.md +++ b/_docs/31-per-language-toolbelts.md @@ -2,17 +2,17 @@ title: "Per-Language Toolbelts" order: 31 part: "Debugging" -description: "app hot and app alloc are the same trial-division prime count and the same string-churn index build in C++, Go, and Rust — and each language's own profiler, run for real against the built binary, points at the same widest frame: perf record/report and clang-tidy/-fanalyzer for C++, the pprof ecosystem and delve for Go, cargo-flamegraph and clippy for Rust." +description: "app hot and app alloc are the same trial-division prime count and the same string-churn index build in C++, Go, and Rust, and each language's own profiler, run against the built binary, points at the same widest frame: perf record/report and clang-tidy/-fanalyzer for C++, the pprof ecosystem and delve for Go, cargo-flamegraph and clippy for Rust." duration: "55 minutes" --- Chapter 28 opened this part by reading a crash gdb already knew about; -Chapter 29 broke code on purpose and watched a sanitizer catch it. Both +Chapter 29 broke code by design and watched a sanitizer catch it. Both chapters answer "what is wrong". This one answers a different question that comes up on every program that runs correctly and still feels slow: "where does the time go, and which line is spending it?" That question has no -generic answer — each language ships its own native way of pointing a -profiler at a running binary — but the *skill* of reading what comes back is +generic answer: each language ships its own native way of pointing a +profiler at a running binary, but the *skill* of reading what comes back is the same skill everywhere: find the widest frame in a call-stack sample and that is your hot path. `app` is this chapter's target, one program built three times with one contract: `app [--n N]` runs either an @@ -29,11 +29,11 @@ this chapter draws from. {% include excalidraw.html file="31-flamegraph-reading" alt="A schematic flamegraph with two labeled axes: y is stack depth, bottom the root frame runtime.main and top the leaf running at the instant a sample was taken; x is share of samples, explicitly not time order or call order. A stack of boxes narrows from a full-width root frame up through main/main.run, into a wide accent box for spin_hot/hotSpin covering about 99% of the row's width, into an equally wide inlined divisor-loop frame directly above it, topped by a leaf sliver. A callout box explains that color here just marks which language tab a frame came from, since real profilers usually color by module or at random. A footer note says the way to find the hot path is to scan for the widest unbroken column reaching toward the top." - caption="Figure 31.1 — reading a flamegraph: width is the share of samples, height is stack depth, and the widest column reaching toward the top is the function actually spending the cycles" %} + caption="Figure 31.1: reading a flamegraph: width is the share of samples, height is stack depth, and the widest column reaching toward the top is the function actually spending the cycles" %} -> **Tools used** — `perf` record/report/stat, `clang-tidy`, GCC `-fanalyzer` +> **Tools used**: `perf` record/report/stat, `clang-tidy`, GCC `-fanalyzer` > (C++); `go tool pprof`, `go tool trace`, `dlv` (Go); `perf` record/report, -> `cargo flamegraph`, `cargo clippy`, `cargo fmt --check` (Rust) — all host. +> `cargo flamegraph`, `cargo clippy`, `cargo fmt --check` (Rust), all host. > `perf`, `gdb`, and the Go/Rust toolchains are on `scripts/check-host.sh`; > `dlv` and `cargo-flamegraph` are pulled on demand (`go install .../dlv`, > `cargo install flamegraph`) as this chapter's transcripts show. @@ -43,8 +43,8 @@ this chapter draws from. `perf` is an *external* sampler: it attaches to any ELF binary from the outside, interrupts it on a hardware or software event (by default a fixed number of CPU cycles), and records the instruction pointer and call stack at -each interrupt. It has no idea what language produced the binary — it only -needs a symbol table and, for call graphs, either frame pointers or DWARF — +each interrupt. It has no idea what language produced the binary: it only +needs a symbol table and, for call graphs, either frame pointers or DWARF, so `perf record`/`perf report`/`perf stat` work identically against the C++ and Rust binaries in this chapter. That is also exactly why they *don't* work as cleanly for Go: Go's runtime multiplexes goroutines onto a handful of @@ -60,7 +60,7 @@ but the Rust ecosystem also ships `cargo flamegraph`, a Rust program that *wraps* `perf record` and then renders the result with `inferno`, a Rust reimplementation of Brendan Gregg's original Perl `stackcollapse`/`flamegraph` scripts. Three ecosystems, three different relationships between the -profiler and the profiled program — but the artifact each one hands back is +profiler and the profiled program, but the artifact each one hands back is the same shape: a call stack with a weight attached, which is what a flamegraph draws directly and what `--top` tabulates as text. @@ -74,18 +74,18 @@ first-time readers expecting the x-axis to mean "when". Reading one is a single scan: follow the tallest, widest column of boxes from the bottom to the top, and the name at the top of that column is where the cycles are actually going. Figure 31.1 draws that shape directly from this chapter's -own data — `spin_hot`/`hotSpin` is that column in every one of the three +own data: `spin_hot`/`hotSpin` is that column in every one of the three profiles below. {% include excalidraw.html file="31-toolbelt-map" alt="One shared target box, app hot or alloc with an identical result across languages, feeding three columns. The C++ column lists perf record with call-graph fp as its accent entry naming spin_hot at 99.9%, then perf report/perf stat, clang-tidy, GCC -fanalyzer, and a dashed gdb box marked Chapter 28. The Go column lists runtime/pprof as its accent entry, cpuprofile into go tool pprof top naming main.hotSpin, then memprofile with alloc_space naming main.allocChurn, runtime/trace with go tool trace, delve, and a dashed go vet box. The Rust column lists perf record with call-graph fp as its accent entry naming spin_hot inlined near 100%, then cargo flamegraph producing a real SVG, cargo clippy and cargo fmt check, a dashed tokio-console box marked forward reference to Chapter 27 not applicable here since there is no async runtime, and a dashed gdb/rust-gdb box marked Chapter 28. A note explains the amber boxes are the CPU-profiling step this chapter cross-checks against each other, and the dashed boxes are noted but out of scope." - caption="Figure 31.2 — one shared target, three native toolbelts: the amber boxes are what this chapter's cross-check compares; the dashed boxes point to where gdb (Ch. 28) and tokio-console (Ch. 27) belong instead" %} + caption="Figure 31.2: one shared target, three native toolbelts: the amber boxes are what this chapter's cross-check compares; the dashed boxes point to where gdb (Ch. 28) and tokio-console (Ch. 27) belong instead" %} ## How the code works -The two workloads are deliberately simple, because the point of this chapter -is the tool, not the algorithm — but the two `noinline` functions they route +The two workloads are simple by design, because the point of this chapter +is the tool, not the algorithm, but the two `noinline` functions they route through are the whole reason the profiles below are unambiguous, so they get the tabs, verbatim from `examples/31-per-language-toolbelts/`: @@ -168,8 +168,8 @@ fn spin_hot(n: u64) -> u64 { All three are the same trial-division loop with the same invariant: an `i` in `[2, n)` is prime only if no `d` in `[2, sqrt(i)]` divides it evenly, found by walking `d` up from 2 and stopping the moment `d*d > i`. The loop is -intentionally the slow textbook version — no sieve, no memoized small primes, -no early-exit on evens — because a fast implementation would finish before a +intentionally the slow textbook version (no sieve, no memoized small primes, +no early-exit on evens) because a fast implementation would finish before a sampling profiler collected enough hits to draw a clean picture, and the goal here is a profile, not a fast prime counter. The `noinline` annotation on each (`[[gnu::noinline]]`, `//go:noinline`, `#[inline(never)]`) is not @@ -177,27 +177,27 @@ decoration: at `-O2`/release optimization, all three compilers are entitled to fold a small, single-call-site function directly into its caller, which would erase the frame boundary a profiler needs to name. `spin_hot` also has `extern "C"` linkage in the C++ build specifically so `perf` and `gdb` print -the plain symbol `spin_hot` instead of an Itanium-mangled name — a detail +the plain symbol `spin_hot` instead of an Itanium-mangled name, a detail that matters once you are grepping profiler output for a function name, as every transcript below does. The allocation-heavy sibling (`alloc_churn`/`allocChurn`/`alloc_churn`, not shown in the tabs above but present in every `main.cpp`/`main.go`/`main.rs`) -does the opposite kind of work on purpose: `n` iterations round-robin +does the opposite kind of work by design: `n` iterations round-robin overwrite a 1,000-key string map, and every iteration allocates a fresh key string and a fresh value string before inserting. Where `spin_hot` never -touches the heap, `alloc_churn` never does meaningful arithmetic — its job is +touches the heap, `alloc_churn` never does meaningful arithmetic; its job is to be a clean single allocation site for a heap/allocation profiler to find, the same way `spin_hot` is a clean single CPU sink for a cycle profiler. -The wiring around both functions is intentionally thin. C++ threads +The wiring around both functions is kept thin. C++ threads `std::expected` through argument parsing and the mode dispatch, wraps the timed section in an RAII `ScopedTimer` whose destructor writes the elapsed milliseconds regardless of how the scope is left, and prints with `std::println`. Go's `run(cfg)` additionally opens the profiler -files the CLI's extra flags name — `pprof.StartCPUProfile`/`trace.Start` -before the timed call, `pprof.WriteHeapProfile` after it, each guarded by a -`defer` so a profile still flushes on any return path — and Rust threads +files the CLI's extra flags name (`pprof.StartCPUProfile`/`trace.Start` +before the timed call, `pprof.WriteHeapProfile` after it), each guarded by a +`defer` so a profile still flushes on any return path, and Rust threads `anyhow::Result` through `parse_args`/`run` and returns `ExitCode` from `main` rather than calling `std::process::exit`, so destructors still run on every exit path. None of that wiring changes the algorithm; it only exists @@ -216,14 +216,14 @@ appends its three profiling-only flags (`--cpuprofile`, `--memprofile`, `--trace`); `verify.lua`'s pattern match tolerates that suffix because it only asserts the shared ` [--n N]` prefix. `--n` itself is parsed with `std::from_chars` in C++, `strconv.Atoi` in Go, and `str::parse::` -in Rust — three ways of saying "reject anything that is not cleanly a -positive integer" — and every language rejects the same malformed inputs the +in Rust, three ways of saying "reject anything that is not cleanly a +positive integer," and every language rejects the same malformed inputs the same way, which `verify.lua` checks per language before it ever touches a profiler. ## Concurrency lens: what each language's tooling gives you -This is a pure-tooling chapter — `app` is single-threaded in every language, +This is a pure-tooling chapter: `app` is single-threaded in every language, so there is no race to hunt here the way Chapters 25 and 26 hunted one. What still differs by language is how much of the *concurrency picture* each toolbelt can show once a program does have more than one thread of control. @@ -233,8 +233,8 @@ Chapter 25 and the flamegraph would show each worker thread's stack separately. Go's profilers go further because they are runtime-aware: a `runtime/pprof` CPU profile attributes samples to the goroutine that was running, and `go tool trace` is built specifically to show -goroutine-level concurrency — when one parks, when the scheduler hands its P -to another, when the GC stops the world — none of which a generic external +goroutine-level concurrency: when one parks, when the scheduler hands its P +to another, when the GC stops the world, none of which a generic external sampler can label without help. Rust's forward-referenced tool, `tokio-console`, is the async-native version of that same idea: it shows live per-task poll counts and waker state for an async runtime the way `go @@ -248,7 +248,7 @@ against this chapter's synchronous `app` and belongs instead with Chapter [host]$ cd examples/31-per-language-toolbelts && ./demo.sh build ``` -A hand run confirms the shared contract before any profiler gets involved — +A hand run confirms the shared contract before any profiler gets involved: `result` is the same number in all three languages, because it is the same algorithm over the same input: @@ -287,10 +287,10 @@ app: mode=hot n=3000000 result=216816 ms=871 ---spin_hot ``` -`spin_hot` is 99.95% of 3,461 samples — everything else (`ld-linux`'s +`spin_hot` is 99.95% of 3,461 samples; everything else (`ld-linux`'s startup path) is the loader noise that ran before the timed section even began. `clang-tidy` and GCC's `-fanalyzer` are the static complements: they -never run the binary at all, so they catch a different class of thing — +never run the binary at all, so they catch a different class of thing: patterns that are *suspicious* in the source, not slow at runtime: ```console @@ -302,11 +302,11 @@ main.cpp:143:5: warning: an exception may be thrown in function 'main' which One warning, and it is a known, practical false positive: `bugprone-exception-escape` traces a theoretical path where `std::format`'s `format_error` could propagate out of `main`, reachable only if the format -string itself were malformed — ours are compile-time literals, so the path +string itself were malformed; ours are compile-time literals, so the path is dead code the checker cannot prove unreachable statically. GCC's `-fanalyzer` (its own CMake preset, `cmake --build --preset analyze`) is stricter about interprocedural state and prints nothing at all against this -source — a clean run I reproduced with a fresh `-fanalyzer` build during +source, a clean run reproduced with a fresh `-fanalyzer` build during this chapter's verification pass. **Go: the `pprof` ecosystem, `go tool trace`, `delve`.** @@ -324,7 +324,7 @@ app: mode=hot n=3000000 result=216816 ms=770 ``` `main.hotSpin` owns 98.70% of samples on its own line (`flat`) and 100% once -you count everything under it on the stack (`cum`) — `runtime.asyncPreempt` +you count everything under it on the stack (`cum`); `runtime.asyncPreempt` is the Go scheduler's cooperative preemption check firing inside the tight loop, not separate work. The heap profile needs one extra flag to be useful: @@ -341,7 +341,7 @@ app: mode=alloc n=200000 result=5000 ms=33 ``` By the time `pprof.WriteHeapProfile` runs, the churned map from the -just-returned `allocChurn` call is already garbage — the default *in-use* +just-returned `allocChurn` call is already garbage; the default *in-use* view would show almost nothing live. `--alloc_space` switches to pprof's cumulative-since-start counter, which remembers every byte `mallocgc` ever handed out whether or not it was later freed, which is the view that @@ -364,9 +364,9 @@ app: mode=hot n=800000 result=63951 ms=114 Delve is the interactive debugger this chapter ties back to Chapter 28's `gdb` work: same idea (breakpoints, stack frames, live variables), native to -Go's runtime instead of DWARF-only. It was not preinstalled on this host — +Go's runtime instead of DWARF-only. It was not preinstalled on this host; `go install github.com/go-delve/delve/cmd/dlv@latest` pulled it from the Go -module proxy in a few seconds — and it scripts the same way `gdb --batch` +module proxy in a few seconds, and it scripts the same way `gdb --batch` does, by piping a command list instead of typing at a TUI: ```console @@ -383,8 +383,8 @@ does, by piping a command list instead of typing at a TUI: The breakpoint lands exactly on `hotSpin`'s declaration, `print n` confirms the argument delve stopped with (`500000`, matching `--n 500000`), and `bt` -names the same two frames `go tool pprof` and `go tool trace` already named -— three different Go tools, one function. +names the same two frames `go tool pprof` and `go tool trace` already named: +three different Go tools, one function. **Rust: `perf record`/`perf report`, `cargo flamegraph`, `clippy`.** @@ -406,11 +406,11 @@ app: mode=hot n=3000000 result=216816 ms=476 99.98% again, spelled `app::spin_hot` (Rust's own path-qualified symbol, needing no `extern "C"` trick to stay readable) and reported as inlined at -release optimization — the frame boundary survives as debug-info metadata +release optimization; the frame boundary survives as debug-info metadata even though the call itself was folded away. `cargo-flamegraph` was already on this host (`cargo install flamegraph --locked` is what would put it there -otherwise), so I ran it for real rather than falling back to the tabular -report: +otherwise), so I ran it against the live binary rather than falling back to +the tabular report: ```console [host]$ cargo flamegraph --output rust-hot.svg -- hot @@ -429,34 +429,34 @@ title>spin_hot (301,908,555 samples, 15.57%) Brendan Gregg's classic Perl `stackcollapse-perf.pl`/`flamegraph.pl` pipeline; the SVG it wrote splits `spin_hot` across three adjacent frames (different instruction addresses inside the same inlined loop get -symbolized to the same function name but folded as separate stack shapes) — +symbolized to the same function name but folded as separate stack shapes); summed, `38.41 + 46.02 + 15.57 = 99.998%`, the same number `perf report` gave directly, just rendered as boxes on a page instead of tabulated as text. Where crates.io is unreachable, the `perf record`/`perf report --stdio` pair above is the fallback: same data, no picture. `cargo clippy --release --all-targets` and `cargo fmt --check` are both -clean on this source — no findings, no diff. `tokio-console` is worth naming -and then setting aside: it attaches to a running **async runtime** and shows +clean on this source, no findings, no diff. `tokio-console` deserves a brief +look before being set aside: it attaches to a running **async runtime** and shows live task/waker state, which only exists once a program has `tokio` tasks to -show. `app` has none — no `.await` anywhere in this chapter's source — so -`tokio-console` genuinely has nothing to display here; it belongs with +show. `app` has none; no `.await` anywhere in this chapter's source, so +`tokio-console` has nothing to display here; it belongs with Chapter 27's async runtimes, where `chatterd`'s tokio tasks are the thing worth watching live. -> **CLion's built-in profiler** unverified — +> **CLion's built-in profiler** unverified: > CLion wraps this same `perf record`/`perf report` pipeline (and, on > supported platforms, its own sampler) behind a GUI flamegraph and a > call-tree view, so a reader working in that IDE gets Figure 31.1's shape > without leaving the editor. This chapter's transcripts are all terminal > tools run directly, so the IDE integration itself was not exercised this -> session — confirm it against your own CLion install if that is your +> session; confirm it against your own CLion install if that is your > workflow. ## Cross-check: three profilers, one function name -The claim worth checking independently is not any single number — sample -counts vary run to run with host load — but that **three unrelated +The claim worth checking independently is not any single number (sample +counts vary run to run with host load) but that **three unrelated profiling tools, built by three different projects, agree on which function is hot**. `perf stat` gives the coarsest independent signal, a hardware counter read that needs no symbolication at all: @@ -473,7 +473,7 @@ app: mode=hot n=3000000 result=216816 ms=867 0.870321798 seconds time elapsed ``` -Roughly one instruction retired per cycle — consistent with a tight, +Roughly one instruction retired per cycle, consistent with a tight, branch-heavy integer loop, not a program stalled on cache misses or memory. Then the three per-language profilers, each pointed at the same `hot` workload, each naming its own hot frame in its own vocabulary: @@ -484,10 +484,10 @@ workload, each naming its own hot frame in its own vocabulary: | Go `go tool pprof --top` | `app hot --cpuprofile cpu.prof` | `main.hotSpin` | 98.70% flat / 100% cum | | Rust `perf report --stdio` | `perf record --call-graph fp -- app hot` | `app::spin_hot` | 99.98% | -Three profilers with nothing in common under the hood — a kernel-level +Three profilers with nothing in common under the hood (a kernel-level hardware sampler reading raw ELF symbols, an in-process Go library sampling -its own goroutine stacks, the same kernel sampler again reading Rust's DWARF -— converge on the same name in every language's own spelling of it, at the +its own goroutine stacks, the same kernel sampler again reading Rust's DWARF) +converge on the same name in every language's own spelling of it, at the same overwhelming share of samples. That agreement is the cross-check: if `spin_hot`/`hotSpin` were *not* actually where the time went, three independently-implemented tools would have no reason to all point at the @@ -497,15 +497,15 @@ same place. - Reading a flamegraph is one scan regardless of language: **width is share of samples, height is stack depth**, and the widest column reaching toward - the top names the hot path — `spin_hot`/`hotSpin` at 98.7–99.98% across + the top names the hot path: `spin_hot`/`hotSpin` at 98.7–99.98% across three unrelated profilers in this chapter's own run. - `perf` is external and language-agnostic (it attaches to any ELF binary); Go's `runtime/pprof`/`runtime/trace` are internal libraries the program calls into, which is why Go's CLI carries `--cpuprofile`/`--memprofile`/ `--trace` flags that C++ and Rust do not need. - Static analyzers (`clang-tidy`, `-fanalyzer`, `cargo clippy`) answer a - different question than a profiler — "is this pattern suspicious" instead - of "is this slow" — and both classes of tool are worth running even on + different question than a profiler ("is this pattern suspicious" instead + of "is this slow"), and both classes of tool are worth running even on code with no known defect. - `--alloc_space` (not the default in-use view) is what makes a Go heap profile useful once the allocating call has already returned and its @@ -519,7 +519,7 @@ tracing tools this book has deferred until now. --- -

verified — every command, +

verified: every command, number, and transcript above was run this session on the Fedora 44 reference host (kernel 7.1, GCC 16.1.1, clang 22.1.8, go1.26.5, rustc 1.97.1, perf 7.1.3, gdb 17.2) against the binaries in @@ -542,8 +542,8 @@ analyze) produced zero diagnostics; dlv was installed live with go install .../dlv@latest and its scripted session broke at main.hotSpin, printed n=500000, and backtraced through main.run; cargo flamegraph was -already present, ran for real, and its output SVG contained +already present, ran against the live binary, and its output SVG contained spin_hot frames summing to 99.998% of samples; cargo clippy --release --all-targets and cargo fmt --check were both clean. The CLion profiler-integration callout is -unverified as marked — no CLion session was run this session.

+unverified as marked: no CLion session was run this session.

diff --git a/_docs/32-namespaces-and-cgroups.md b/_docs/32-namespaces-and-cgroups.md index 4e42a8f..268ee5b 100644 --- a/_docs/32-namespaces-and-cgroups.md +++ b/_docs/32-namespaces-and-cgroups.md @@ -2,7 +2,7 @@ title: "Namespaces and cgroups" order: 32 part: "Containers and Virtualization" -description: "The eight Linux namespaces and cgroup v2's unified hierarchy as the two primitives every container runtime assembles from — unshare vs setns vs clone3, the pid-1 reparenting consequence, memory.max enforced by a real OOM kill, and the genuine ENOMEM Go hits calling unshare(CLONE_NEWPID) directly from a threaded runtime." +description: "The eight Linux namespaces and cgroup v2's unified hierarchy as the two primitives every container runtime assembles from: unshare vs setns vs clone3, the pid-1 reparenting consequence, memory.max enforced by an OOM kill, and the ENOMEM Go hits calling unshare(CLONE_NEWPID) directly from a threaded runtime." duration: "55 minutes" --- @@ -10,8 +10,8 @@ Every `pmon` version so far has supervised, signaled, and identified a child, but the child has always lived in the same world as its parent: same process-id space, same hostname, same view of every device and byte of RAM on the box. Real container runtimes give a process a *different* -world — its own pid 1, its own hostname, and a hard ceiling on memory and -CPU — without a hypervisor anywhere in sight. The one new idea this +world: its own pid 1, its own hostname, and a hard ceiling on memory and +CPU, without a hypervisor anywhere in sight. The one new idea this chapter teaches is that "container" is not a kernel object at all: it is two independent primitives, **namespaces** (what a process can *see*) and **cgroups** (what a process is *allowed to use*), wired together by @@ -19,46 +19,46 @@ whatever userspace tool assembles them. `pmon containerize` is that tool, built by hand, with nothing between the syscalls and the terminal. The code is in `examples/32-namespaces-and-cgroups/`. This is a **VM -example** — namespace and cgroup writes need real root and a real kernel, +example**: namespace and cgroup writes need real root and a real kernel, so `./demo.sh run` deploys to `systems-target` when `TARGET` is set; the `README.md` there covers the CLI and the paired positive/negative verification in full. {% include excalidraw.html file="32-namespace-overview" - alt="Two columns, host and pmon containerize's PID-1 child, with eight namespace-type rows: PID, mount, UTS, network, time, IPC, user, cgroup. The first four rows are amber boxes with an arrow marked 'differs' between real /proc/<pid>/ns/* inode numbers — pid 4026531836 vs 4026532530, mount 4026531832 vs 4026532528, UTS 4026531838 vs 4026532529, network 4026531833 vs 4026532531 — read on systems-target. The last four rows are grey dashed boxes marked 'shared with host' with identical inode numbers on both sides — time 4026531834, IPC 4026531839, user 4026531837, cgroup 4026531835 — because pmon containerize only requests CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID and deliberately leaves the other four alone." - caption="Figure 32.1 — the eight Linux namespaces, host vs pmon's PID-1 child, every inode number read live from /proc/<pid>/ns/* on systems-target" %} + alt="Two columns, host and pmon containerize's PID-1 child, with eight namespace-type rows: PID, mount, UTS, network, time, IPC, user, cgroup. The first four rows are amber boxes with an arrow marked 'differs' between real /proc/<pid>/ns/* inode numbers: pid 4026531836 vs 4026532530, mount 4026531832 vs 4026532528, UTS 4026531838 vs 4026532529, network 4026531833 vs 4026532531, read on systems-target. The last four rows are grey dashed boxes marked 'shared with host' with identical inode numbers on both sides: time 4026531834, IPC 4026531839, user 4026531837, cgroup 4026531835, because pmon containerize only requests CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID and leaves the other four alone by design." + caption="Figure 32.1: the eight Linux namespaces, host vs pmon's PID-1 child, every inode number read live from /proc/<pid>/ns/* on systems-target" %} -> **Tools used** — `ssh`/`scp` and the Python runner +> **Tools used**: `ssh`/`scp` and the Python runner > (`scripts/test-all-examples.py`) on the host; `readlink`, `cat`, `pgrep`, > `nsenter`, `dmesg`, `strace` (systems-target VM). Everything here ships > with Fedora or is preinstalled in the lab VM by cloud-init. ## Eight namespaces, one job each -A Linux namespace does not hide anything from the kernel — the kernel +A Linux namespace does not hide anything from the kernel: the kernel always has the complete, global view. It hides things from the *process*: each namespace type wraps one kind of global kernel state so a process inside it sees only its own slice. -- **PID** — a private process-id tree. The first process created inside a +- **PID**: a private process-id tree. The first process created inside a new PID namespace is *always* pid 1 of that namespace, no matter what its "real" (host-visible) pid is. -- **Mount** — a private copy of the mount table. Mounting or unmounting +- **Mount**: a private copy of the mount table. Mounting or unmounting inside the namespace does not touch the host's view, once propagation is detached (below). -- **UTS** — the hostname and NIS domain name. One `sethostname(2)` call, +- **UTS**: the hostname and NIS domain name. One `sethostname(2)` call, fully isolated. -- **Network** — its own loopback, interfaces, routes, and port space; a +- **Network**: its own loopback, interfaces, routes, and port space; a bare network namespace starts with only `lo`, down. -- **IPC** — its own System V IPC objects and POSIX message queues. -- **User** — its own uid/gid mapping, letting an unprivileged host user - become "root" inside the namespace over a mapped range — the one - namespace `pmon containerize` does *not* request, discussed below. -- **Cgroup** — virtualizes which cgroup a process believes is its root, +- **IPC**: its own System V IPC objects and POSIX message queues. +- **User**: its own uid/gid mapping, letting an unprivileged host user + become "root" inside the namespace over a mapped range (the one + namespace `pmon containerize` does *not* request, discussed below). +- **Cgroup**: virtualizes which cgroup a process believes is its root, so `/proc/self/cgroup` reads `/` from inside even when the real path is nested five levels deep on the host. -- **Time** (Linux 5.6+) — lets a namespace offset `CLOCK_MONOTONIC` and +- **Time** (Linux 5.6+): lets a namespace offset `CLOCK_MONOTONIC` and `CLOCK_BOOTTIME`, the one facility built specifically for checkpoint/ restore rather than isolation. @@ -67,15 +67,15 @@ CLONE_NEWNET | CLONE_NEWPID`. Figure 32.1 is the receipt: every inode number in it came from `readlink /proc//ns/*` on `systems-target` during this chapter's own run, and the four requested types read a *different* inode on the child than on the host's pid 1, while the four -left alone — time, IPC, user, cgroup — read the *same* inode on both +left alone (time, IPC, user, cgroup) read the *same* inode on both sides. A namespace you didn't ask for is not a weaker namespace; it is no namespace at all, just the host's, shared. -## unshare, setns, and clone3 — three ways into a namespace +## unshare, setns, and clone3: three ways into a namespace The three syscalls answer three different questions: -- **`unshare(2)`** — "take *me*, right now, out of my current namespaces +- **`unshare(2)`**: "take *me*, right now, out of my current namespaces and into brand-new ones." For most types this applies to the calling process/thread immediately. `CLONE_NEWPID` is the deliberate exception: unshare(2) can never move an already-running process into a new PID @@ -83,13 +83,13 @@ The three syscalls answer three different questions: arranges that the *next* child the caller creates becomes pid 1 of a fresh one. That single asymmetry is why every implementation in this chapter unshares first and forks second, not the other way around. -- **`setns(2)`** — "join a namespace that already exists," given an open +- **`setns(2)`**: "join a namespace that already exists," given an open file descriptor on one of the `/proc//ns/*` links. This is what `nsenter` does under the hood: open the target's namespace fds, `setns` into each, then fork and exec the requested command. Like `unshare`, `setns(CLONE_NEWPID)` only takes effect for children the caller creates - *after* the call — the caller's own pid never moves. -- **`clone3(2)`** — create the child and select its namespaces in the + *after* the call; the caller's own pid never moves. +- **`clone3(2)`**: create the child and select its namespaces in the *same* syscall, via `struct clone_args.flags`. This is the one true atomic option: there is no window where the new task exists but hasn't yet joined its namespaces. `pmon`'s C++ build calls it directly (below); @@ -102,24 +102,24 @@ The three syscalls answer three different questions: Becoming pid 1 of a new namespace is not a label, it is a job with two real obligations the kernel enforces. First, **reparenting**: inside a PID namespace, an orphaned process is reparented to namespace-local pid 1, -not to the host's real init — so pid 1 must be prepared to `wait()` for +not to the host's real init, so pid 1 must be prepared to `wait()` for processes it never directly spawned, exactly the way real init is. -Second, **reaping**: default signal dispositions are altered for pid 1 — +Second, **reaping**: default signal dispositions are altered for pid 1; an unhandled `SIGTERM`/`SIGKILL` that would kill any other process is ignored by pid 1 unless it installs its own handler. `pmon containerize` sidesteps both obligations by design: its pid-1 child immediately -`execve`s CMD, so CMD — not pmon's own code — inherits the pid-1 role for -whatever it does next. That is a real, stated simplification, not an +`execve`s CMD, so CMD, not pmon's own code, inherits the pid-1 role for +whatever it does next. That is a simplification, not an oversight: a supervisor that expects to run *multiple* processes inside one PID namespace (the way `podman run` with more than one process, or a -process manager container, would) needs an init that actually reaps — +process manager container, would) needs an init that actually reaps, which is exactly what chapters 11 and 12 already built, just not wired to this namespace yet. ## Cgroup v2: the unified hierarchy Cgroup v1 let different controllers (cpu, memory, …) organize processes -into *different* tree shapes — a process could be in one cpu group and an +into *different* tree shapes: a process could be in one cpu group and an unrelated memory group. Cgroup v2 unifies this: **one tree**, and every controller you enable sees the same groups. `systems-target` mounts it at the conventional single point: @@ -134,21 +134,21 @@ cpu memory pids ``` Three files matter for `containerize`. **`memory.max`** is a hard byte -ceiling — cross it and the kernel's OOM killer runs *inside this cgroup +ceiling: cross it and the kernel's OOM killer runs *inside this cgroup only*, picking a victim from the cgroup's own processes regardless of system-wide memory pressure. **`cpu.max`** is written as `"$QUOTA -$PERIOD"` microseconds — `"max 100000"` (pmon's default) means unthrottled +$PERIOD"` microseconds: `"max 100000"` (pmon's default) means unthrottled inside a 100 ms accounting period; a real ceiling would read e.g. `"50000 100000"` for half a core. A sibling knob `containerize` doesn't set, worth knowing: **`cpu.weight`** (default 100, range 1–10000) doesn't -cap anything — it only decides who wins when several cgroups *compete* +cap anything; it only decides who wins when several cgroups *compete* for the same CPU, the cgroup analogue of `nice`. `memory.max` and `cpu.max` answer "how much," `cpu.weight` answers "whose turn," and a real container runtime writes all three. The fourth file is the pressure signal: **PSI** (Pressure Stall Information), `memory.pressure`, is not a counter of *how much* memory a -cgroup used — it is how much time tasks in this cgroup spent *stalled* +cgroup used: it is how much time tasks in this cgroup spent *stalled* waiting for memory, per-cgroup, independent of pressure anywhere else on the box: @@ -159,13 +159,13 @@ full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ``` `pmon containerize` reads the `some avg10=` field after CMD exits and -prints it as `pmon: cgroup mem.pressure some=` — a real number, not a -placeholder, confirming the kernel is actually accounting this cgroup, not -just enforcing a limit blindly. +prints it as `pmon: cgroup mem.pressure some=`, a real number rather +than a placeholder, confirming the kernel is actually accounting this +cgroup, not just enforcing a limit blindly. One subtlety governs the whole design: a *non-root* cgroup cannot both hold processes directly and delegate a controller to its children (the -"no internal process" constraint) — but the **root** cgroup is explicitly +"no internal process" constraint), but the **root** cgroup is explicitly exempt. That is why `setup_cgroup()` can write `+memory +cpu` into `/sys/fs/cgroup/cgroup.subtree_control` without first relocating `pmon` itself into a leaf: the root is the one place in the tree allowed to hold @@ -175,36 +175,36 @@ processes and delegate at the same time. Strip away the marketing and a container is five primitives, wired together by one userspace program: **namespaces** (this chapter, minus -user — `containerize` never requests `CLONE_NEWUSER`, so its "root" inside -the container is genuinely uid 0 on the host, which is why the demo needs +user: `containerize` never requests `CLONE_NEWUSER`, so its "root" inside +the container is uid 0 on the host, which is why the demo needs real root rather than rootless mapped uids), **cgroups** (this chapter), -a **rootfs** (a different filesystem view via `pivot_root`/bind mounts — +a **rootfs** (a different filesystem view via `pivot_root`/bind mounts; chapter 34 builds this), **capabilities** (chapter 14's five sets, dropped to a minimal set before `exec`), and a **seccomp/Landlock** filter (chapter 33, next). `pmon containerize` deliberately stops after -the first two, on the same host filesystem CMD would otherwise see — the +the first two, on the same host filesystem CMD would otherwise see; the point is to make namespaces and cgroups *visible* as syscalls you could write yourself, before chapter 34 shows what podman assembles from all -five at once inside a real multi-stage Containerfile. +five at once inside a multi-stage Containerfile. {% include excalidraw.html file="32-cgroup-hierarchy" alt="The cgroup v2 unified hierarchy from root down to a leaf. The root band shows cgroup.controllers listing cpuset, cpu, io, memory, hugetlb, pids, rdma, misc, dmem, and cgroup.subtree_control enabling plus memory plus cpu, delegated to children. An arrow labeled enables leads down into the pmon-oomdemo leaf band, showing memory.max as 67108864 (64 MiB), memory.swap.max as 0 (no swap headroom), cpu.max as max 100000 (unthrottled), cgroup.procs holding pmon's own pid written before fork, and memory.pressure PSI reading some avg10=0.00. A dark box below states the hog process (sh, real anon-rss 64384kB) writes past the 64 MiB ceiling. An arrow labeled 'OOM killer fires' leads into a kernel band showing the real dmesg lines from this session: oom-kill constraint=CONSTRAINT_MEMCG oom_memcg=/pmon-oomdemo task=sh pid=27013, and 'Memory cgroup out of memory: Killed process 27013 (sh) anon-rss:64384kB', with pmon's own report alongside it: signal=9 (KILL), exit 137." - caption="Figure 32.2 — the cgroup v2 unified hierarchy, root to leaf, ending in the real dmesg OOM-kill line this session produced" %} + caption="Figure 32.2: the cgroup v2 unified hierarchy, root to leaf, ending in the real dmesg OOM-kill line this session produced" %} ## How the code works All three languages share one shape: build the cgroup, unshare/clone into fresh namespaces, become pid 1, exec CMD, then the parent waits and reports the cgroup-side evidence. `setup_cgroup`/`setupCgroup` (identical -logic across C++, Go, and Rust — read `cgroup.subtree_control`, add +logic across C++, Go, and Rust: read `cgroup.subtree_control`, add `+memory`/`+cpu` as whole tokens only if missing so `"cpu"` never false-matches inside `"cpuset"`, `mkdir` the leaf, write `memory.max` then `cpu.max` then a best-effort `memory.swap.max=0`, then write pmon's own pid into `cgroup.procs`) runs **before** any namespace call in every -language, on purpose: cgroup membership is inherited across `fork`/ -`clone`/`exec`, so writing pmon's pid first means every later child — the -pid-1 task, and CMD after it execs — is under the limits from its very +language, by design: cgroup membership is inherited across `fork`/ +`clone`/`exec`, so writing pmon's pid first means every later child (the +pid-1 task, and CMD after it execs) is under the limits from its very first instruction, never after. {% include codetabs.html langs="C++|Go|Rust" %} @@ -446,11 +446,11 @@ first instruction, never after. } ``` -C++ and Rust get there through the same shape — `unshare` first, taking +C++ and Rust get there through the same shape: `unshare` first, taking mount/UTS/network effect immediately on the caller; the mount-private remount so nothing propagates back to the host; `sethostname`; then a fork-shaped child creation (`clone3(flags=0)` in C++, `nix::fork` in -Rust — both give "no code runs between clone and exec except what we +Rust, both giving "no code runs between clone and exec except what we wrote," fork(2)'s classic contract) whose child becomes pid 1 of the deferred PID namespace and immediately prints its two proof lines before `execvp`. The parent side is identical in shape across both: `waitpid`, @@ -462,15 +462,15 @@ reused verbatim from every earlier `pmon` chapter) and the `128+n` convention. Go's excerpt is the odd one out and is the subject of the Concurrency lens below. -**Fragile bits, stated plainly**: `containerize` never remounts a fresh +**Fragile bits**: `containerize` never remounts a fresh `/proc` inside the new mount namespace, so tools that read `/proc` from -inside — including `ps` run via `nsenter` — still see the host's process -table (the cross-check below demonstrates this directly); a real rootfs +inside, including `ps` run via `nsenter`, still see the host's process +table (the cross-check below demonstrates this directly); a rootfs container fixes this with `mount -t proc proc /proc` after `pivot_root`, which is chapter 34's job, not this one's. `memory.swap.max` is written best-effort because not every kernel exposes swap accounting identically. And `deploy-to-vm.sh` rebuilds the remote command line with a bare `$*`, -which flattens quoting — every `CMD` in this chapter's runs is therefore +which flattens quoting, so every `CMD` in this chapter's runs is space-free tokens (`/usr/bin/echo hi`, `sh /tmp/hog.sh`), with anything needing real shell logic staged as a script file on the guest first. @@ -478,7 +478,7 @@ needing real shell logic staged as a script file on the guest first. The exit-code contract is identical across all three binaries, and it splits by *when* the kernel or pmon rejects something. A **usage** error -(exit 2) is caught before any privileged call — a missing `-- CMD`: +(exit 2) is caught before any privileged call: a missing `-- CMD`: ```bash [host]$ ./app containerize --hostname x @@ -486,7 +486,7 @@ containerize: missing -- CMD ``` A **setup** error (exit 1) is a syscall that failed before CMD ever ran. -The ordinary case is the privilege check — `containerize: must run as +The ordinary case is the privilege check: `containerize: must run as root` when invoked as a normal user, real output from this session, unprivileged, on the reference host. The instructive case is the one Go's first implementation actually produced on `systems-target`, verbatim: @@ -495,12 +495,12 @@ first implementation actually produced on `systems-target`, verbatim: containerize: exec /usr/bin/echo: fork/exec /home/fedora/app: cannot allocate memory ``` -That is `ENOMEM` from a real `clone(2)` call, captured live — the +That is `ENOMEM` from a real `clone(2)` call, captured live; the Concurrency lens below explains exactly why. Finally, a **child-side** error is CMD's own fate, reported through the same wait-status translation in every language: the OOM negative control's `pmon: child killed signal=9 (KILL)` becomes exit `137` (`128 + 9`), and `pmon` -propagates it unchanged rather than collapsing it to a generic failure — +propagates it unchanged rather than collapsing it to a generic failure: the caller can tell "CMD chose to fail" from "CMD was killed" from the exit code alone, no log-scraping required. @@ -508,24 +508,24 @@ exit code alone, no log-scraping required. {% include excalidraw.html file="32-clone3-flow" - alt="Three lanes. The left lane, C++/Rust, shows unshare taking effect on the caller, then clone3 with flags=0 (fork semantics) creating a child that prints its pid/hostname proof lines and execs CMD — labeled PASS, real run this session. The middle lane, Go's first attempt, shows runtime.LockOSThread pinning the goroutine then a raw unshare call, followed by os/exec.Start issuing an internal vfork/pidfd probe fork, then the real fork for CMD failing with clone returning -1 ENOMEM and the message fork/exec: cannot allocate memory — real, this session. The bottom lane, Go's shipped fix, shows setup_cgroup writing its own pid first same as the other two languages, then exec.Command with SysProcAttr.Cloneflags carrying CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID, one clone(2) call creating the child already inside all four namespaces with no prior unshare and no LockOSThread needed, the child re-execing itself as __ns_init__ to mount / MS_PRIVATE, sethostname, print the proof lines, and exec CMD as PID 1 — labeled PASS 13/13 across cpp, go, rust on systems-target, this session. A note explains the ENOMEM is a real, kernel-documented restriction: unshare(CLONE_NEWPID) allows the caller to create exactly one child in the fresh namespace before further forks there return ENOMEM, and os/exec's own vfork/pidfd fast path performs an internal probe fork that silently spends that one shot." - caption="Figure 32.3 — C++/Rust's unshare-then-clone3 shape beside Go's two attempts: a real ENOMEM, then the SysProcAttr.Cloneflags fix that passes" %} + alt="Three lanes. The left lane, C++/Rust, shows unshare taking effect on the caller, then clone3 with flags=0 (fork semantics) creating a child that prints its pid/hostname proof lines and execs CMD, labeled PASS, a real run this session. The middle lane, Go's first attempt, shows runtime.LockOSThread pinning the goroutine then a raw unshare call, followed by os/exec.Start issuing an internal vfork/pidfd probe fork, then the real fork for CMD failing with clone returning -1 ENOMEM and the message fork/exec: cannot allocate memory, real output from this session. The bottom lane, Go's shipped fix, shows setup_cgroup writing its own pid first same as the other two languages, then exec.Command with SysProcAttr.Cloneflags carrying CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID, one clone(2) call creating the child already inside all four namespaces with no prior unshare and no LockOSThread needed, the child re-execing itself as __ns_init__ to mount / MS_PRIVATE, sethostname, print the proof lines, and exec CMD as PID 1, labeled PASS 13/13 across cpp, go, rust on systems-target, this session. A note explains the ENOMEM is a real, kernel-documented restriction: unshare(CLONE_NEWPID) allows the caller to create exactly one child in the fresh namespace before further forks there return ENOMEM, and os/exec's own vfork/pidfd fast path performs an internal probe fork that silently spends that one shot." + caption="Figure 32.3: C++/Rust's unshare-then-clone3 shape beside Go's two attempts: a real ENOMEM, then the SysProcAttr.Cloneflags fix that passes" %} Namespace changes made by `unshare(2)`/`setns(2)` are per-**thread** -kernel state, not per-process — the same fact chapter 14 hit with +kernel state, not per-process: the same fact chapter 14 hit with capabilities. A Go program is multithreaded before `main` even runs, and the goroutine scheduler is free to migrate any goroutine onto any OS thread between two lines of Go code. The textbook answer is `runtime.LockOSThread()`: pin the calling goroutine to its current OS thread so a subsequent raw `unshare()` and the fork/exec that must land on the *same*, now-unshared thread can't be split across two different ones. -That is exactly what this chapter's Go build tried first — `LockOSThread`, +That is exactly what this chapter's Go build tried first: `LockOSThread`, then `unix.Unshare(CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID)`, then re-exec itself via `os/exec` so the fork would land on the pinned thread. It built. Its pid/hostname proof lines were real. And it failed the memory-hog negative control on real hardware, every time, with exit 1 -and `fork/exec /home/fedora/app: cannot allocate memory` — not a typo, not -a flaky VM, a `strace` on `systems-target` shows the actual syscall: +and `fork/exec /home/fedora/app: cannot allocate memory`, not a typo and +not a flaky VM; a `strace` on `systems-target` shows the actual syscall: ``` [vm]$ sudo strace -f -e trace=clone,unshare,execve -o /tmp/go-trace.log \ @@ -539,13 +539,13 @@ containerize: exec /usr/bin/echo: fork/exec /home/fedora/app: cannot allocate me 24652 clone(child_stack=NULL, flags=CLONE_VM|CLONE_PIDFD|CLONE_VFORK|SIGCHLD, parent_tid=...) = -1 ENOMEM (Cannot allocate memory) ``` -`unshare` genuinely succeeds. The first `clone` after it — a vfork/pidfd +`unshare` succeeds. The first `clone` after it (a vfork/pidfd probe `os/exec` performs internally on first use, unrelated to our -request — also succeeds and exits immediately, and in doing so becomes +request) also succeeds and exits immediately, and in doing so becomes the fresh PID namespace's pid 1. That is the real, kernel-documented restriction: `unshare(CLONE_NEWPID)` lets the caller create exactly *one* child in the new namespace; every fork after that one returns `ENOMEM`. -LockOSThread solved the problem it was built for — thread affinity — and +LockOSThread solved the problem it was built for (thread affinity) and was powerless against this one, because this failure has nothing to do with which thread is running: it is about how many times the *same* thread group is allowed to fork into a namespace it merely unshared @@ -555,7 +555,7 @@ The fix is the same move chapter 14 made for capabilities: stop calling the namespace syscall directly from live, threaded Go code, and describe it declaratively instead. `exec.Cmd.SysProcAttr.Cloneflags` asks the runtime's own fork/exec trampoline to create the namespaces **and** the -real child in one `clone(2)` call — there is no separate `unshare()` +real child in one `clone(2)` call: there is no separate `unshare()` anywhere, so there is no "used up my one child" state to exhaust, and `LockOSThread` becomes unnecessary because there is no longer a namespace-mutating syscall for a goroutine to be scheduled away from. The @@ -563,7 +563,7 @@ general rule, twice proven now across two different kernel primitives: whenever Go needs to change something that is per-thread and irreversible, push the whole operation into the runtime's fork/exec trampoline via `SysProcAttr` rather than performing it by hand on a -pinned goroutine — the trampoline is the one place in a Go binary +pinned goroutine; the trampoline is the one place in a Go binary guaranteed to run alone, between one `clone(2)` and the `execve` right after it. @@ -586,7 +586,7 @@ example cpp go rust ``` Each language's verify run passed all 13 assertions -(`PASS 13 / FAIL 0`). By hand, the positive run — a trivial CMD proves +(`PASS 13 / FAIL 0`). By hand, the positive run, a trivial CMD, proves both the PID and UTS namespaces at once: ```bash @@ -600,10 +600,10 @@ pmon: cgroup mem.pressure some=0.00 pmon: child exited status=0 ``` -And the negative control — a real memory hog (`head -c 200000000 +And the negative control, a memory hog (`head -c 200000000 /dev/zero | tr '\0' x`, held in a shell variable so it can't be swapped or paged away, staged as a guest script since `deploy-to-vm.sh` would mangle -its embedded pipe) under `--mem-max 67108864` (64 MiB, far below the +its embedded pipe), under `--mem-max 67108864` (64 MiB, far below the ~190 MiB it tries to hold): ```bash @@ -617,14 +617,14 @@ pmon: child killed signal=9 (KILL) ``` The hog never reaches its own trailing `echo should-not-print` in any of -the three languages — proof the cgroup terminated it mid-allocation, -not merely slowed it down. Go, Rust, and C++ all produce byte-identical +the three languages, proof the cgroup terminated it mid-allocation +rather than merely slowing it down. Go, Rust, and C++ all produce byte-identical output and the same exit 137; that identity is exactly what `verify.lua`'s 13 assertions per language check. ## Cross-check on systems-target -Three independent tools confirm the same two facts Figure 32.1 claims — +Three independent tools confirm the same two facts Figure 32.1 claims: that the namespaces are real, and that the missing `/proc` remount has a real, visible consequence. First, the namespace identities themselves, read directly while a long-lived `containerize` child is alive: @@ -646,12 +646,12 @@ net:[4026532531] Every one of the four requested namespaces reads a different inode than the host's pid 1. The three namespaces `containerize` never requests read -*identical* inodes on both sides — `time:[4026531834]`, +*identical* inodes on both sides: `time:[4026531834]`, `ipc:[4026531839]`, `user:[4026531837]`, `cgroup:[4026531835]`, on host -and child alike — confirming those are genuinely shared, not silently +and child alike, confirming those are shared, not silently isolated by some other mechanism. -Second, `nsenter` — the `setns(2)`-based tool — demonstrates the missing +Second, `nsenter` (the `setns(2)`-based tool) demonstrates the missing `/proc` remount directly instead of asserting it: ```bash @@ -661,13 +661,13 @@ root 1 0 0 14:42 ? 00:00:06 /usr/lib/systemd/systemd --s root 2 0 0 14:42 ? 00:00:00 [kthreadd] ``` -`nsenter` genuinely `setns`'d into the child's pid/mount/uts/net -namespaces before forking `ps` — but `ps` reads `/proc`, and the mount +`nsenter` `setns`'d into the child's pid/mount/uts/net +namespaces before forking `ps`, but `ps` reads `/proc`, and the mount namespace it entered never got a *fresh* procfs mounted after `unshare`; it still resolves to the host's original procfs instance, so `ps` prints the host's entire process table instead of a scoped one. This is not a bug in `containerize`; it is precisely why chapter 34's Containerfile -build includes a `pivot_root` and a fresh `/proc` mount — a namespace +build includes a `pivot_root` and a fresh `/proc` mount; a namespace without a matching filesystem view is only half a container. Third, the kernel's own OOM log, read independently of anything `pmon` @@ -681,7 +681,7 @@ Memory cgroup out of memory: Killed process 27013 (sh) total-vm:71332kB, anon-rs `CONSTRAINT_MEMCG` and `oom_memcg=/pmon-oomdemo` are the kernel's own words for "this cgroup's `memory.max`, not global memory pressure, is why -this process died" — the same verdict `pmon`'s `signal=9 (KILL)` line +this process died," the same verdict `pmon`'s `signal=9 (KILL)` line reported, confirmed at the source that actually made the decision. ## What you learned @@ -691,21 +691,21 @@ reported, confirmed at the source that actually made the decision. `CLONE_NEWPID`, which only ever affects the *next* child), `setns(2)` joins one that already exists via an open `/proc//ns/*` fd, and `clone3(2)` creates the child and its namespaces atomically in one - syscall — the only option with no in-between state. + syscall, the only option with no in-between state. - A cgroup v2 leaf's `memory.max`/`cpu.max` only prove anything paired with a negative control: a real hog under a 64 MiB ceiling must be - OOM-killed (exit 137, `dmesg`'s `oom_memcg=` naming the cgroup) — a - cgroup that let it finish would be decorative, not enforcing — and PSI + OOM-killed (exit 137, `dmesg`'s `oom_memcg=` naming the cgroup); a + cgroup that let it finish would be decorative, not enforcing. PSI (`memory.pressure`) is the per-cgroup stall signal, not a system-wide one. - Namespace changes are per-thread state, and a Go program is threaded before `main` runs: `unshare(CLONE_NEWPID)` also caps the caller to exactly one child in the fresh namespace, a restriction `os/exec`'s own internal probe fork silently spends, producing a real `ENOMEM` this - chapter reproduced live — the fix is `SysProcAttr.Cloneflags`, pushing + chapter reproduced live; the fix is `SysProcAttr.Cloneflags`, pushing the whole operation into the runtime's own fork/exec trampoline, the same move chapter 14 made for capabilities. - A container is namespaces + cgroups + rootfs + capabilities + seccomp, - assembled by userspace, not one kernel object — and a namespace without + assembled by userspace, not one kernel object, and a namespace without its matching filesystem view is observably incomplete: `nsenter`'s `ps -ef` proved it by showing the host's whole process table through a correctly-`setns`'d pid namespace that never got a fresh `/proc`. @@ -716,7 +716,7 @@ primitives chapter 32 left out of "what a container runtime assembles." --- -

verified — every +

verified: every transcript and number above was produced this session against the systems-target guest (Fedora 44, kernel 6.19.10-300.fc44.x86_64): the runner printed @@ -737,6 +737,6 @@ the Concurrency lens is this session's own strace output against the pre-fix build, kept as evidence of a real, reproduced defect before the Cloneflags fix that now ships and passes. A one-line verify.lua bug this chapter also surfaced and fixed -— an unescaped - in a Lua pattern silently failing the -hostname assertion — is recorded in the example's own history, not +(an unescaped - in a Lua pattern silently failing the +hostname assertion) is recorded in the example's own history, not restated here as a finding about the chapter's subject matter.

diff --git a/_docs/33-seccomp-and-landlock.md b/_docs/33-seccomp-and-landlock.md index 98a05e9..c7ac168 100644 --- a/_docs/33-seccomp-and-landlock.md +++ b/_docs/33-seccomp-and-landlock.md @@ -33,9 +33,9 @@ contract, and the paired positive/negative verification in full. {% include excalidraw.html file="33-seccomp-filter-flow" alt="Two stacked bands. The top userspace band shows fwatch issuing a syscall such as epoll_wait, read, or socket, beside a note that this is classic BPF -- you write the filter, Part 8's bpftrace only observes. The bottom kernel band shows the filter running on every syscall entry: a BPF_LD_ABS load of the syscall number and architecture, a compiled jump table of 26 entries for cpp and rust or 39 for go, a match arrow to SECCOMP_RET_ALLOW that returns control to the caller, and a no-match arrow down to the filter's default action. Two default-action boxes sit side by side: SCMP_ACT_ERRNO(EPERM), the one fwatch actually installs, where the process keeps running and sees EPERM on socket(2); and a dashed, unused SCMP_ACT_KILL_PROCESS box, where a separate probe program's getpid() call ends the whole process with SIGSYS. Footer notes report the real guest transcript: socket(AF_INET,...) = -1 EPERM under fwatch's filter, and the ACT_KILL_PROCESS probe's getpid() ending in killed by SIGSYS (core dumped)." - caption="Figure 33.1 — a syscall's path through fwatch's seccomp-bpf filter: one cBPF program, run again on every entry, deciding ALLOW versus its default action" %} + caption="Figure 33.1: a syscall's path through fwatch's seccomp-bpf filter: one cBPF program, run again on every entry, deciding ALLOW versus its default action" %} -> **Tools used** — `strace` (host and `systems-target` VM, to build and +> **Tools used**: `strace` (host and `systems-target` VM, to build and > confirm the syscall profile), `cmake`+`ninja`/`go build`/`cargo` (the > Python runner's build phase, host, all three toolchains), `ssh`/`scp` > (via `scripts/test-all-examples.py --mode vm` and its staging step onto @@ -112,7 +112,7 @@ sandboxed process regaining privilege through a setuid binary it execs. ## `SCMP_ACT_ERRNO` versus `SCMP_ACT_KILL` -`fwatch` deliberately chose `SCMP_ACT_ERRNO(EPERM)`, not +`fwatch` chose `SCMP_ACT_ERRNO(EPERM)`, not `SCMP_ACT_KILL_PROCESS`, as its default action, and the difference is observable, not cosmetic. With `ACT_ERRNO`, a denied syscall returns `-1` with `errno` set and the calling code continues to run -- the process can @@ -120,7 +120,7 @@ log the denial, clean up, and exit on its own terms; that is what makes `probe --forbidden-syscall`'s own `EPERM`-and-continue behavior possible. With `ACT_KILL_PROCESS`, a denied syscall never returns at all: the kernel delivers an uncatchable `SIGSYS` and the process dies mid-instruction. To -see the contrast for real rather than assert it, a small standalone probe +observe the contrast directly rather than assert it, a small standalone probe (not part of `fwatch`, built only to observe this) installs a filter allowing only `write`/`exit_group`/`rt_sigreturn` under `SCMP_ACT_KILL_PROCESS`, then calls the unlisted `getpid()`: @@ -184,7 +184,7 @@ misbehave the moment it ran on the guest. {% include excalidraw.html file="33-landlock-ruleset" alt="Two stacked bands. The top band builds the ruleset: landlock_create_ruleset called first with NULL/0/VERSION to probe the ABI, returning 9 on the host versus 7 on the systems-target guest despite both being Fedora 44 kernels; then landlock_create_ruleset(attr) with handled_access_fs set to READ_FILE and READ_DIR, feeding into landlock_add_rule binding a PATH_BENEATH rule to one directory's O_PATH file descriptor; then prctl(PR_SET_NO_NEW_PRIVS,1) feeding into landlock_restrict_self, marked irrevocable, with a note that Go's build also passes the TSYNC flag here to cover every OS thread already running. The bottom band shows the filesystem view after restrict_self: the sandboxed directory /tmp/lsp33-watch-sandbox with READ_FILE and READ_DIR still allowed and the watch loop working normally, beside /tmp/lsp33-outside-target/secret.txt where open(O_RDONLY) returns EACCES, with an arrow from the ruleset's restrict_self box down to each outcome, solid to the grant and dashed to the denial. A footer note across the bottom states that handled_access_net was left at zero in this ruleset, and that Landlock's TCP bind/connect rights have existed since ABI 4 and UDP rights only since ABI 10 -- so even at ABI 7 or 9 a Landlock-only sandbox does not cover the network by itself." - caption="Figure 33.2 — the Landlock ruleset/rule model, with the ABI probed live rather than assumed and the real grant/deny split from this chapter's probes" %} + caption="Figure 33.2: the Landlock ruleset/rule model, with the ABI probed live rather than assumed and the real grant/deny split from this chapter's probes" %} **What Landlock can and cannot do.** Filesystem mediation -- open/read/write/execute/rename, scoped to a path -- has been solid since @@ -298,7 +298,7 @@ needed just to name it as a rule target), bind the rule with `no_new_privs` already be set. Rust's `landlock` crate (source in `rust/src/main.rs`) compresses the same four calls into one builder chain -- `Ruleset::default().handle_access(...).create()?.add_rule(...)?.restrict_self()?` --- and adds one thing the raw syscall path does not give you for free: it +-- and adds one thing the raw syscall path does not give you automatically: it checks the returned `RulesetStatus`, so a kernel that silently best-effort-ignores an unsupported right is caught (`bail!` if `status.ruleset != RulesetStatus::FullyEnforced`) instead of quietly @@ -339,8 +339,8 @@ worked" from "the sandbox is broken." Both Landlock and seccomp are, by default, **per-thread** kernel state -- they restrict the calling thread, not automatically every thread in the -process. That is invisible in C++ and Rust here because both builds are -deliberately single-threaded before the sandbox goes on: `main` calls +process. That is invisible in C++ and Rust here because both builds stay +single-threaded before the sandbox goes on: `main` calls `apply_landlock` and `install_seccomp` before the epoll loop starts any concurrency of its own, so "the calling thread" and "the whole process" are the same thing. @@ -484,7 +484,7 @@ start being something a container runtime installs *for* it. --- -

verified — every transcript +

verified: every transcript and number above was produced this session, reproduced on both the plain host (kernel 7.1, libseccomp 2.6.0, GCC 16.1.1, Go 1.26.5, Rust/Cargo 1.97.1) and the systems-target diff --git a/_docs/34-our-programs-in-containers.md b/_docs/34-our-programs-in-containers.md index de52198..de71563 100644 --- a/_docs/34-our-programs-in-containers.md +++ b/_docs/34-our-programs-in-containers.md @@ -151,11 +151,11 @@ implementation strategies across the three languages, because Go's and Rust's runtimes are not safe to `fork()` once other threads or goroutines exist. C++ uses a raw `fork(2)` for both the worker and each short-lived job -- no `exec`, just a different function running in the child -- because -`app`'s C++ build is deliberately single-threaded up to that point, so the +`app`'s C++ build is single-threaded by design up to that point, so the forking thread and the whole process are the same thing. Go and Rust instead **self-reexec**: `serve` resolves its own binary path (`os.Executable()` / `std::env::current_exe()`) and starts `worker`/`job` -as genuinely separate processes running the same binary with a different +as separate processes running the same binary with a different argument, then reaps each one with `Cmd.Wait()` / `Child::wait()` -- ordinary `wait4(2)` under the hood, one goroutine or thread parked per child *is* the reap loop, no `SIGCHLD` involved anywhere in those two @@ -254,7 +254,7 @@ is cooperatively stopped (`stop_token`, an atomic flag, a channel close) before `serve` waits on the worker, so no job-spawn races the shutdown sequence. -The one hardcoded assumption worth naming: `cpu_max_display()` only +One hardcoded assumption deserves a direct look: `cpu_max_display()` only handles cgroup v2's single-line `cpu.max` format (`"max"` or `" "`); a host still running cgroup v1 would need the older `cpu.cfs_quota_us`/`cpu.cfs_period_us` pair instead, and none of these @@ -288,7 +288,7 @@ inherited the *blocked* mask -- so when `serve` later forwarded `SIGTERM` to it with `kill(worker_pid, SIGTERM)`, the signal did not run its default (terminate) action. It just sat **pending**, forever: the worker was unkillable by anything short of `SIGKILL`, exactly the failure mode -`naive` demonstrates deliberately at the container's pid-1 layer, now +`naive` demonstrates at the container's pid-1 layer, now reproduced accidentally one process down. The fix is one line in the child, immediately after `fork()`: @@ -300,7 +300,7 @@ child, immediately after `fork()`: which mirrors `pmon`'s reset-before-`execvp` from chapter 12 almost exactly -- the difference is that there is no `exec` here to reset the -mask for free, so the reset has to be written explicitly. Go and Rust +mask automatically, so the reset has to be written explicitly. Go and Rust never meet this bug at all, and not by luck: both languages avoid `fork()`-without-`exec()` entirely, because forking a multithreaded runtime (goroutine scheduler threads, Rust's allocator and async @@ -339,7 +339,7 @@ the container's pid and mount namespaces on top: alt="A host band above a podman-user-namespace band. podman exec reaches the container PID 1 directly from inside the container runtime. podman unshare nsenter first joins podman's own user namespace, then nsenter --target --pid --mount reaches the same PID 1 from there -- both confirming Name: app and /proc/1/exe resolving to /usr/local/bin/app. A dashed, failing arrow shows plain nsenter without podman unshare returning EPERM: reassociate to namespaces failed. Below, real evidence boxes show the subuid mapping user:524288:65536 and a live zombie scan of /proc/[0-9]*/stat reporting zero." caption="Figure 34.3 -- podman exec and podman unshare nsenter reach the same container PID 1 from two different starting namespaces; plain nsenter cannot" %} -A live `gdb` attach is the surprise worth reporting plainly: because +A live `gdb` attach is the real surprise: because rootless podman's user namespace is *owned by the same host uid* that is running `gdb`, ordinary ptrace permission checks succeed even with no `podman unshare` step at all -- diff --git a/_docs/35-virtualization-and-kvm.md b/_docs/35-virtualization-and-kvm.md index 8e1630f..7269c56 100644 --- a/_docs/35-virtualization-and-kvm.md +++ b/_docs/35-virtualization-and-kvm.md @@ -2,7 +2,7 @@ title: "Virtualization and KVM" order: 35 part: "Containers and Virtualization" -description: "How a VM actually runs — VMX root/non-root modes, EPT, and virtio devices as the host kernel moonlighting as a hypervisor — the isolation spectrum from bare process through container and microVM to full VM, the rust-vmm/Firecracker/cloud-hypervisor landscape, and overheadbench's measured syscall/mem/io deltas across all three." +description: "How a VM actually runs: VMX root/non-root modes, EPT, and virtio devices as the host kernel moonlighting as a hypervisor, the isolation spectrum from bare process through container and microVM to full VM, the rust-vmm/Firecracker/cloud-hypervisor landscape, and overheadbench's measured syscall/mem/io deltas across all three." duration: "50 minutes" --- @@ -10,14 +10,14 @@ Every daemon in this book has, until now, run either directly on the host or inside a Podman container sharing the host's kernel. This chapter puts a third option on the table: a full KVM guest with its own kernel, and asks the one question a systems programmer actually needs answered before choosing -between them — not "which is more secure" in the abstract, but *what does +between them: not "which is more secure" in the abstract, but *what does each one cost*, measured in the same units this book has used throughout: nanoseconds per syscall, gigabytes per second, operations per second. `overheadbench` is a tri-language microbenchmark built for exactly this: the same binary, unmodified, runs on the bare host, inside the `systems-target` KVM guest, and inside a rootless Podman container, and the three runs land in -one table. The isolation spectrum this chapter walks — process, container, -microVM, full VM — is not a taxonomy exercise; it is the same spectrum every +one table. The isolation spectrum this chapter walks (process, container, +microVM, full VM) is not a taxonomy exercise; it is the same spectrum every one of this book's daemons could be deployed onto, and this is the chapter that prices each rung. @@ -27,10 +27,10 @@ builds and runs it; its `README.md` covers the CLI and the sanity ranges {% include excalidraw.html file="35-kvm-virtio-stack" - alt="Three horizontal bands. Top: host, VMX root mode, showing kvm.ko and kvm_intel.ko managing EPT tables, a QEMU/VMM userspace box running the vCPU thread and virtio-blk backend, and host NVMe/btrfs where a real fsync lands. Middle: guest, VMX non-root mode, showing the guest kernel serving getpid in-guest with no host trap, overheadbench in guest ring 3 issuing the syscall bench's raw getpid loop and the io bench's open/write/fsync calls, a virtio-blk front-end driver, and a virtqueue descriptor ring. Bottom: the trapped path for fsync — fsync in the guest causes a VM exit, the KVM exit handler hands off to the QEMU vCPU thread, the virtio-blk backend issues the real host fsync and triggers VM enter back into the guest. A fourth band gives the measured overheadbench deltas: syscall about 88ns/call on both host and guest because getpid never triggers a vmexit, mem about 14 to 15 GB/s on both because the EPT walk cost is amortized after the TLB warms up, and io about 1.3k ops/s on the host against about 0.5k ops/s on the guest, the real vmexit tax." - caption="Figure 35.1 — the VMX root/non-root boundary, the virtio-blk round trip that fsync actually pays, and where overheadbench's own measured host-vs-guest deltas land" %} + alt="Three horizontal bands. Top: host, VMX root mode, showing kvm.ko and kvm_intel.ko managing EPT tables, a QEMU/VMM userspace box running the vCPU thread and virtio-blk backend, and host NVMe/btrfs where a real fsync lands. Middle: guest, VMX non-root mode, showing the guest kernel serving getpid in-guest with no host trap, overheadbench in guest ring 3 issuing the syscall bench's raw getpid loop and the io bench's open/write/fsync calls, a virtio-blk front-end driver, and a virtqueue descriptor ring. Bottom: the trapped path for fsync: fsync in the guest causes a VM exit, the KVM exit handler hands off to the QEMU vCPU thread, the virtio-blk backend issues the real host fsync and triggers VM enter back into the guest. A fourth band gives the measured overheadbench deltas: syscall about 88ns/call on both host and guest because getpid never triggers a vmexit, mem about 14 to 15 GB/s on both because the EPT walk cost is amortized after the TLB warms up, and io about 1.3k ops/s on the host against about 0.5k ops/s on the guest, the real vmexit tax." + caption="Figure 35.1: the VMX root/non-root boundary, the virtio-blk round trip that fsync actually pays, and where overheadbench's own measured host-vs-guest deltas land" %} -> **Tools used** — `overheadbench` itself, `podman` (build and run, host), +> **Tools used**: `overheadbench` itself, `podman` (build and run, host), > `ssh`/`scp` via `scripts/lab/deploy-to-vm.sh` and the Python runner > (`scripts/test-all-examples.py`) (host), `virsh` (host, confirming the lab > is up), `lscpu` (host and `systems-target` guest), `strace -c` (host and @@ -43,29 +43,29 @@ builds and runs it; its `README.md` covers the CLI and the sanity ranges A container's isolation is namespaces and cgroups layered over *one* kernel; every syscall a containerized process makes is a syscall the host kernel itself serves, at host cost. A KVM guest is a different animal: it has its -own kernel, its own page tables, its own idea of what hardware exists — and +own kernel, its own page tables, its own idea of what hardware exists, and the host kernel is the thing making that illusion real. Since 2006, Intel and AMD CPUs have shipped hardware virtualization extensions (VMX on Intel, confirmed on this host by `lscpu`'s `Virtualization: VT-x` line) that add a second CPU mode dimension alongside ring 0/ring 3: **VMX root mode**, where the host kernel and hypervisor run, and **VMX non-root mode**, where the -*entire* guest — its kernel at ring 0, its processes at ring 3 — runs mostly +*entire* guest (its kernel at ring 0, its processes at ring 3) runs mostly unmodified. `/dev/kvm`, backed by the `kvm` and `kvm_intel` kernel modules (both loaded on this host, `lsmod | grep kvm` confirms `kvm_intel`, `kvm`, and `irqbypass`), is the interface QEMU drives: it calls `ioctl(KVM_RUN, ...)` on a vCPU file descriptor, the CPU drops into non-root mode, and guest -instructions execute directly on the real CPU — no instruction-level -emulation — until something the guest cannot handle itself forces a **VM +instructions execute directly on the real CPU (no instruction-level +emulation) until something the guest cannot handle itself forces a **VM exit** back to root mode. The key fact this chapter measures is *what actually causes an exit*. -Ordinary instructions — arithmetic, memory loads and stores, and crucially -**syscalls the guest kernel can serve itself**, like `getpid(2)` — never +Ordinary instructions (arithmetic, memory loads and stores, and crucially +**syscalls the guest kernel can serve itself**, like `getpid(2)`) never leave non-root mode at all. The guest kernel's syscall entry point runs exactly like it would on bare metal; there is no hypervisor in that path to trap into. What *does* exit are instructions that touch state the guest does -not really own: port I/O, certain MSR accesses, and — the one this chapter's -`io` bench hits on every iteration — an access to an emulated device like +not really own: port I/O, certain MSR accesses, and, the one this chapter's +`io` bench hits on every iteration, an access to an emulated device like `virtio-blk`, the paravirtualized block device virtio defines so guest I/O doesn't have to trap on every disk instruction the way full hardware emulation would. `virtio-blk` still requires a round trip: the guest driver @@ -75,7 +75,7 @@ mode again) issues the *real* `fsync(2)` against the host filesystem, and a VM entry resumes the guest. Extended Page Tables (EPT) are the third piece: they let the guest maintain its own page tables while the CPU walks a second, hardware-managed table translating guest-physical to host-physical addresses, -so ordinary memory access needs no hypervisor involvement either — only the +so ordinary memory access needs no hypervisor involvement either: only the first touch of a page, which fills the EPT-backed TLB entry, costs anything beyond native. @@ -84,56 +84,56 @@ beyond native. {% include excalidraw.html file="35-isolation-spectrum" alt="Four boxes left to right along a spectrum from shared host kernel to separate guest kernel. Bare process: no isolation boundary, syscall equals host syscall, escape equals root on host. Podman container: namespaces plus cgroup v2 plus seccomp, one host kernel and one syscall table, syscall and mem about equal host and io about equal host on a bind mount, escape equals a kernel bug in namespace or cgroup code. MicroVM, Firecracker or cloud-hypervisor built on rust-vmm: own guest kernel with a minimal virtio device set, about 125 milliseconds boot and under 5 MiB overhead for Firecracker, escape equals a bug in the roughly five devices the VMM exposes or in KVM itself. Full KVM/QEMU VM, the systems-target lab guest: own guest kernel with a full device model, syscall about equal host and io about three times slower due to virtio-blk, escape equals a bug in the full emulated device model or in KVM. A note below cites that rust-vmm crates such as kvm-ioctls, virtio-queue, and vm-memory underpin both Firecracker v1.16.1 and Cloud Hypervisor v53.0." - caption="Figure 35.2 — process, container, microVM, and full VM, each priced by escape surface and by this chapter's measured overheadbench numbers" %} + caption="Figure 35.2: process, container, microVM, and full VM, each priced by escape surface and by this chapter's measured overheadbench numbers" %} -Four points, not two. A **bare process** has no isolation boundary at all — a -syscall it makes *is* a host syscall, full stop, and an escape is simply +Four points, not two. A **bare process** has no isolation boundary at all: a +syscall it makes *is* a host syscall, and an escape is "you are already root, or you exploit a bug to become root." A **container** (Chapter 34's Podman UBI images) adds namespaces and cgroup v2 as a soft boundary layered over the *same* kernel: still one syscall table, so `overheadbench`'s `syscall` and `mem` numbers inside a container should track -the host almost exactly — and the `io` bench, run against a directory bind- +the host almost exactly, and the `io` bench, run against a directory bind- mounted from the host's real filesystem rather than the container's overlay storage, does too, because it is still the host's block layer doing the work. The escape surface for a container is a kernel bug in the -namespace/cgroup/seccomp code path itself — the code the shared kernel runs +namespace/cgroup/seccomp code path itself: the code the shared kernel runs on the container's behalf. A **full KVM/QEMU VM** (`systems-target`, this book's lab guest) is the opposite end: its own kernel, its own process table, its own view of -hardware, with a full emulated device model — network, block, USB, ACPI, -BIOS. The escape surface shrinks to "a bug in that entire device model, or in -KVM's guest/host boundary itself" — a much smaller, better-audited attack +hardware, with a full emulated device model (network, block, USB, ACPI, +BIOS). The escape surface shrinks to "a bug in that entire device model, or in +KVM's guest/host boundary itself", a much smaller, better-audited attack surface than "the whole host kernel," at the cost of a full second kernel to boot, patch, and pay overhead for on every path that has to leave the guest. -Between them sits the **microVM** — Firecracker and cloud-hypervisor's +Between them sits the **microVM**: Firecracker and cloud-hypervisor's answer to "a real second kernel, but priced like a container." A microVM still runs its own guest kernel under KVM, so the escape surface is the same -category as a full VM, but the device model is deliberately reduced: no -BIOS, no ACPI, no graphics, no USB — Firecracker (v1.16.1 as of this +category as a full VM, but the device model is reduced by design: no +BIOS, no ACPI, no graphics, no USB. Firecracker (v1.16.1 as of this writing) implements exactly five devices: virtio-net, virtio-block, virtio-vsock, a serial console, and a minimal keyboard controller. That minimalism is why Firecracker boots a guest in roughly 125 ms with under 5 MiB of per-instance memory overhead, and why AWS runs it under Lambda and Fargate at up to 150 microVMs per second per host. Both Firecracker and Cloud Hypervisor (v53.0, which added VFIO passthrough, CPU/memory hotplug, -and — recently — Windows guest support) are built from **rust-vmm**: a -community of reusable, independently audited Rust crates — +and, recently, Windows guest support) are built from **rust-vmm**: a +community of reusable, independently audited Rust crates: `kvm-ioctls`/`kvm-bindings` for the `/dev/kvm` ioctl surface, `vm-memory` for guest address space management, `virtio-queue` and per-device crates for the virtio transport. Firecracker (AWS) helped seed rust-vmm; Cloud Hypervisor (originally Intel, now Linux Foundation-governed) leans on the same crates for a broader feature set. The microVM's promise is literally this chapter's spectrum collapsed to one point: KVM's isolation -guarantee at closer to container startup cost — which is why it is the +guarantee at closer to container startup cost, which is why it is the answer teams reach for when "container" isn't isolated enough and "full VM" is too slow to spin up per request. ## How the code works -`overheadbench`'s three benches are intentionally tiny and identical across -languages, because the *comparison* is the content, not the algorithm. The +`overheadbench`'s three benches are small and identical across +languages by design, because the *comparison* is the content, not the algorithm. The one worth walking in detail is `syscall`, since it is the bench whose result this chapter's whole VMX argument hinges on: @@ -204,26 +204,26 @@ spelling makes that guarantee visible in the source, not implicit in a libc version); the clock (`std::chrono::steady_clock`, `time.Now`/`Since`, `Instant::now`) is a **monotonic** one, because Chapter 24 already established that a wall-clock read can step backward mid-measurement; and -each language guards the compiler from proving the loop is dead — +each language guards the compiler from proving the loop is dead: `black_box` in Rust wraps the return value, C++ takes no extra step here because the syscall itself is an unremovable side effect, and the Go build relies on the same property. The `mem` bench needs an explicit anti-dead-code guard because summing a buffer *is* pure, elidable computation: C++ stashes the sum in a `volatile` sink, Rust calls `black_box(sum)`, Go checks the sum -against an unreachable sentinel inside an `if` whose body writes to stderr — +against an unreachable sentinel inside an `if` whose body writes to stderr: three different languages' answer to the same compiler-optimizer problem. The `io` bench's fragile-but-deliberate choice is documented in its own comment: the scratch directory is created **relative to the current working directory**, not the system temp directory, because on this host (and many -dev hosts) `/tmp` is `tmpfs` — `fsync` there is nearly free, and the number +dev hosts) `/tmp` is `tmpfs`: `fsync` there is nearly free, and the number stops measuring storage at all. `examples/` sits on real NVMe-backed btrfs, which is what makes the `io` numbers below meaningful as a storage-overhead comparison rather than a memcpy benchmark in disguise. The other half of "how the code works" for this chapter is the multi-stage `Containerfile` that gets the same Go binary into a rootless Podman -container — `examples/35-virtualization-and-kvm/go/Containerfile`, -reproduced verbatim: +container (`examples/35-virtualization-and-kvm/go/Containerfile`, +reproduced verbatim): ``` # Multi-stage build for overheadbench (chapter 35). Build stage compiles a @@ -252,18 +252,18 @@ ENTRYPOINT ["/usr/local/bin/overheadbench"] CMD ["--bench", "all"] ``` -The build stage is `ubi10/go-toolset` — Red Hat's s2i-flavored Go 1.26.5 +The build stage is `ubi10/go-toolset`, Red Hat's s2i-flavored Go 1.26.5 image, the exact same toolchain pin this book uses on the host, so the container's binary is not just architecturally comparable to the host's, it is a build of the identical source with the identical compiler version. It compiles with `CGO_ENABLED=0`: `getpid(2)` needs no libc, so the result is a fully static binary that runs unmodified on `ubi10/ubi-micro`, a runtime image so minimal it has no shell, no package manager, nothing but the -handful of files UBI's licensing and CA-bundle policy require. The one -build-time trap worth stating plainly: setting an explicit `WORKDIR` before +handful of files UBI's licensing and CA-bundle policy require. One +build-time trap deserves a direct look: setting an explicit `WORKDIR` before the `COPY`/`RUN` steps would have the directory created root-owned at the point the instruction runs, and every later `RUN` in a rootless build -executes as the image's own non-root user (uid 1001) — the fix is to stay in +executes as the image's own non-root user (uid 1001); the fix is to stay in the s2i image's pre-existing `/opt/app-root/src`, which the base image already made writable for that uid. @@ -280,14 +280,14 @@ the first bucket in all three languages: C++'s `parse_args` returns `cfg.error()` then `kUsage` before returning 2; Go's `parseArgs` returns a plain `error` that `main` prints beside the same `usage` constant; Rust's `parse_args` returns `Result` and the `main` match arm -does the identical two-line print. The second bucket — the `io` bench's +does the identical two-line print. The second bucket (the `io` bench's `open`/`write`/`fsync` calls actually failing, which none of the runs in this chapter hit but which a full disk or a read-only bind mount would -trigger — propagates through `std::expected`, +trigger) propagates through `std::expected`, `(float64, error)`, and `Result` respectively, each wrapped one more time (`"io bench failed: " + ...`) so the message names which of the three phases failed, and exits 1 with no usage line, because the arguments -were fine — the environment was not. +were fine; the environment was not. ## Concurrency lens @@ -295,7 +295,7 @@ This chapter's isolation spectrum is built entirely from **per-thread kernel state**, and that constraint shows up on both the container and the VM side in the same shape Chapter 14 already met with capabilities. On the VM side: a vCPU file descriptor returned by `KVM_CREATE_VCPU` is only valid to drive -with `ioctl(KVM_RUN, ...)` **from the thread that created it** — call it from +with `ioctl(KVM_RUN, ...)` **from the thread that created it**: call it from any other thread and the ioctl fails. That is why QEMU pins exactly one host thread per vCPU rather than dispatching `KVM_RUN` calls from a thread pool; the vCPU's register state, its `KVM_RUN` mmap'd shared page, and its @@ -306,7 +306,7 @@ kernel's `task_struct`, not a process-wide property, exactly like the uids and capability sets Chapter 14 walked. A single-threaded C++ or Rust build that calls `setns` then immediately does its namespaced work never notices; a Go program is the case that bites, because the Go runtime is -multi-threaded and goroutines migrate between OS threads by default — a +multi-threaded and goroutines migrate between OS threads by default: a goroutine that calls `setns` and then, without more, tries to act as though it is still in the new namespace can be rescheduled onto a *different* OS thread that never made that call. The fix is the same shape as Chapter 14's @@ -316,7 +316,7 @@ current OS thread for the rest of its life (or until `UnlockOSThread`), so the namespace membership and the goroutine that believes it holds it stay attached to the same kernel thread. The general rule repeats from Chapter 14: whenever a kernel API's state lives on the thread rather than the -process — credentials, capabilities, namespace membership, a KVM vCPU — a +process (credentials, capabilities, namespace membership, a KVM vCPU), a runtime that hides its threading from you (Go's goroutines, but the same risk applies to any green-thread or fiber scheduler) needs an explicit pin, or the state and the code that thinks it owns the state quietly drift apart. @@ -366,12 +366,12 @@ Three repeats of each (host: 84.8–91.8 ns/call, 12.6–15.5 GB/s, 1105–1505 ops/s; guest: 87.7–88.1 ns/call, 14.3–15.0 GB/s, 463–521 ops/s; container: 94.0–102.8 ns/call, 13.5–15.4 GB/s, 1474–1524 ops/s) put the pattern beyond noise: **`syscall` and `mem` land in the same band across all three -environments** — a container shares the host's syscall table entirely, and +environments**: a container shares the host's syscall table entirely, and the guest's `getpid` never triggers a VM exit at all, matching Figure 35.1's claim exactly. **`io` is where the VM actually pays**: roughly 2.9–3× slower in the guest than on the host or in the container, the measured cost of every `fsync` taking the virtio-blk round trip this chapter's diagram walks -— VM exit, QEMU's backend thread, the real host `fsync`, VM entry — instead +(VM exit, QEMU's backend thread, the real host `fsync`, VM entry) instead of running straight through in-kernel like the container's bind-mounted write does. The full cross-language sweep on the host, run through this book's standard runner, confirms the three languages track each other: @@ -387,7 +387,7 @@ example cpp go rust ## Cross-check `overheadbench` prints its own timing, so an independent check has to verify -the printed numbers correspond to real syscalls actually made — not just +the printed numbers correspond to real syscalls actually made, not just trust the arithmetic. `strace -c`, run on both sides of the VMX boundary, does exactly that by counting syscalls a second way. On the host, wrapping `--bench io --iters 50`: @@ -405,7 +405,7 @@ bench=io metric=1233.15 unit=ops/s 100.00 0.006032 28 210 total ``` -Exactly 50 `fsync` calls for `--iters 50` — the count `overheadbench` claims +Exactly 50 `fsync` calls for `--iters 50`: the count `overheadbench` claims to have run is the count the kernel actually saw, independent of the program's own bookkeeping. Run identically on `systems-target`: @@ -419,36 +419,36 @@ bench=io metric=512.96 unit=ops/s Same 50 `fsync` calls, and the *unstraced* metric it printed (512.96 ops/s) falls right inside the guest's 463–521 ops/s band from the build/run/observe -section above — the trace confirms the count without disturbing the +section above; the trace confirms the count without disturbing the comparison this chapter is making. Note what `strace` cannot show, on either side: it observes the syscall the guest kernel serves, never the VM exit that syscall may or may not trigger underneath. `strace -c`'s own reported -`usecs/call` for `getpid` under tracing was 9,139 ns — two orders of -magnitude above the untraced ~85 ns this chapter reports — because `ptrace` +`usecs/call` for `getpid` under tracing was 9,139 ns, two orders of +magnitude above the untraced ~85 ns this chapter reports, because `ptrace` stops the tracee on every syscall entry and exit; that overhead is exactly why `overheadbench` times itself with its own monotonic clock instead of trusting a tracer's numbers, the same "the observation tool changes what it measures" caveat Chapter 14 hit with `strace` across a privilege drop. There -is no dashboard panel for this chapter's numbers by design — `overheadbench` +is no dashboard panel for this chapter's numbers by design: `overheadbench` is a one-shot CLI, and the table above *is* the observation surface. ## What you learned - A VM exit is not triggered by "leaving the guest," it is triggered by - the guest touching state it does not itself own — `getpid(2)` never + the guest touching state it does not itself own: `getpid(2)` never exits, but `fsync(2)` against `virtio-blk` does, which is exactly why `overheadbench`'s `syscall`/`mem` numbers track the host across all three environments while `io` is the one that is measurably ~3× slower in a full KVM guest. - The isolation spectrum has four rungs, not two: bare process, container (shared kernel, namespaces/cgroups as a soft boundary), microVM - (rust-vmm-built Firecracker v1.16.1 / Cloud Hypervisor v53.0 — a real - guest kernel behind a deliberately minimal device model), and full - KVM/QEMU VM — each trading escape-surface size against boot time and + (rust-vmm-built Firecracker v1.16.1 / Cloud Hypervisor v53.0, a real + guest kernel behind a minimal-by-design device model), and full + KVM/QEMU VM, each trading escape-surface size against boot time and per-path overhead. - Kernel state that lives per-thread, not per-process, is a recurring trap: a KVM vCPU fd must be driven from the thread that created it, and - `setns(2)` namespace membership is equally per-thread — a Go program must + `setns(2)` namespace membership is equally per-thread: a Go program must call `runtime.LockOSThread()` before either, the same lesson Chapter 14's capability drop taught for uids and capability sets. - A benchmark's own numbers need an independent check: `strace -c`'s exact @@ -456,13 +456,13 @@ is a one-shot CLI, and the table above *is* the observation surface. both host and guest, while also showing where that same tool's overhead (`ptrace`'s stop-on-every-syscall cost) makes it unfit to trust for timing. -Next, Part 10 opens with **observability** — turning `/proc`, `/sys`, and +Next, Part 10 opens with **observability**: turning `/proc`, `/sys`, and cgroup statistics into the kind of live picture this chapter had to build by hand, one `overheadbench` run at a time. --- -

verified — every transcript +

verified: every transcript and number above was produced this session. Host: Fedora 44, kernel 7.1, Intel 8-core/16-thread laptop CPU, lscpu reporting Virtualization: VT-x, lsmod showing diff --git a/_docs/36-proc-sys-and-the-agent.md b/_docs/36-proc-sys-and-the-agent.md index 46c94e7..2b7811f 100644 --- a/_docs/36-proc-sys-and-the-agent.md +++ b/_docs/36-proc-sys-and-the-agent.md @@ -2,23 +2,23 @@ title: "/proc, /sys, and the agent: a USE-method snapshot read straight from the kernel, no tools required" order: 36 part: "Observability" -description: "sysagent v0 is a USE-method metrics collector that reads /proc and /sys/fs/cgroup directly — no root, no shelled-out tools — computing CPU utilization from a two-read /proc/stat delta spanning --interval-ms, run-queue and load from /proc/loadavg, memory from /proc/meminfo, per-disk I/O from /proc/diskstats, network counters from /proc/net/dev, and cgroup PSI (falling back to system-wide /proc/pressure) gathered concurrently with the sampling sleep, exposed identically by a one-shot sample --json and a serve --port /metrics HTTP endpoint, with byte-for-byte identical field names across C++, Go, and Rust." +description: "sysagent v0 is a USE-method metrics collector that reads /proc and /sys/fs/cgroup directly (no root, no shelled-out tools), computing CPU utilization from a two-read /proc/stat delta spanning --interval-ms, run-queue and load from /proc/loadavg, memory from /proc/meminfo, per-disk I/O from /proc/diskstats, network counters from /proc/net/dev, and cgroup PSI (falling back to system-wide /proc/pressure) gathered concurrently with the sampling sleep, exposed identically by a one-shot sample --json and a serve --port /metrics HTTP endpoint, with byte-for-byte identical field names across C++, Go, and Rust." duration: "50 minutes" --- Chapter 35's `overheadbench` measured syscall, memory, and I/O overhead across -a bare process, a container, and a VM — but every one of those numbers came +a bare process, a container, and a VM, but every one of those numbers came from watching `strace -c` and hand-timed loops wrapped around each run. That's the right instrument for a one-off measurement campaign; it is not how a program watches a system continuously, in production, for the months between deploys. This chapter builds the tool that does: `sysagent`, a USE-method -metrics collector that reads `/proc` and `/sys/fs/cgroup` directly — no root, +metrics collector that reads `/proc` and `/sys/fs/cgroup` directly (no root, no subprocess, no tool you'd have to trust is still installed on the box next -year. Chapter 37's `loadmix` builds the opposite kind of tool: an `analyze` +year). Chapter 37's `loadmix` builds the opposite kind of tool: an `analyze` mode that shells out to `vmstat`, `mpstat`, `iostat`, `sar`, and `pidstat` exactly the way an on-call engineer would, and parses their plain-text tables by column name instead of reading kernel counters directly. The split between -the two chapters is deliberate — this one is the case for reading the +the two chapters is deliberate: this one is the case for reading the kernel's own accounting straight from the source, and the numbers it produces are the ones a long-running agent can afford to compute every second without forking a process to do it. @@ -29,14 +29,14 @@ drive it. {% include excalidraw.html file="36-proc-datasource-map" - alt="A pipeline diagram of sysagent's take_snapshot. Left, a single /proc/stat read at t0 landing in a small box labeled 'cumulative counters since boot, not a percentage.' Center, a band labeled 'gathered concurrently with the --interval-ms sleep' containing five parallel boxes — /proc/loadavg, /proc/meminfo, /proc/diskstats, /proc/net/dev, and cgroup PSI (cpu.pressure/memory.pressure/io.pressure, falling back to /proc/pressure) — each writing into its own field of one shared Snapshot, all joined before the sleep's deadline is allowed to pass. A second /proc/stat read at t1 sits to the right of the band; arrows from both t0 and t1 feed a delta box computing cpu_util_pct/cpu_user_pct/cpu_system_pct as 100 times (busy1-busy0)/(total1-total0). A note reads 'two cumulative reads, --interval-ms apart, become one percentage.' Below, the completed Snapshot feeds two consumers drawn side by side: sample --json printing the object once to stdout, and serve --port P's GET /metrics handler calling take_snapshot fresh on every request. A small inset box shows the PSI fallback path: this process's own cgroup v2 delegate (resolved via the '0::' line in /proc/self/cgroup) tried first, the system-wide /proc/pressure/* files tried second, and psi_available:false — not an error — if neither is readable." - caption="Figure 36.1 — sysagent's take_snapshot: a t0/t1 /proc/stat delta bracketing five concurrently-gathered /proc + cgroup PSI sources, all landing in one Snapshot that sample --json and serve /metrics both emit" %} + alt="A pipeline diagram of sysagent's take_snapshot. Left, a single /proc/stat read at t0 landing in a small box labeled 'cumulative counters since boot, not a percentage.' Center, a band labeled 'gathered concurrently with the --interval-ms sleep' containing five parallel boxes (/proc/loadavg, /proc/meminfo, /proc/diskstats, /proc/net/dev, and cgroup PSI (cpu.pressure/memory.pressure/io.pressure, falling back to /proc/pressure)), each writing into its own field of one shared Snapshot, all joined before the sleep's deadline is allowed to pass. A second /proc/stat read at t1 sits to the right of the band; arrows from both t0 and t1 feed a delta box computing cpu_util_pct/cpu_user_pct/cpu_system_pct as 100 times (busy1-busy0)/(total1-total0). A note reads 'two cumulative reads, --interval-ms apart, become one percentage.' Below, the completed Snapshot feeds two consumers drawn side by side: sample --json printing the object once to stdout, and serve --port P's GET /metrics handler calling take_snapshot fresh on every request. A small inset box shows the PSI fallback path: this process's own cgroup v2 delegate (resolved via the '0::' line in /proc/self/cgroup) tried first, the system-wide /proc/pressure/* files tried second, and psi_available:false (not an error) if neither is readable." + caption="Figure 36.1: sysagent's take_snapshot: a t0/t1 /proc/stat delta bracketing five concurrently-gathered /proc + cgroup PSI sources, all landing in one Snapshot that sample --json and serve /metrics both emit" %} -> **Tools used** — none, by design: `sysagent` reads `/proc/*` and +> **Tools used**: none, by design: `sysagent` reads `/proc/*` and > `/sys/fs/cgroup/*` directly, with no subprocess, no bcc-tools/bpftrace, and > no root. The only tool this chapter reaches for to observe it from the -> outside is `curl` (host), against the `serve --port P` `/metrics` endpoint -> — `curl` ships in Fedora's base install and isn't separately checked by +> outside is `curl` (host), against the `serve --port P` `/metrics` endpoint; +> `curl` ships in Fedora's base install and isn't separately checked by > `scripts/check-host.sh`, the same way Chapter 39 notes for `cpupower` and > `objdump`. @@ -50,25 +50,25 @@ sysagent sample [--json] [--interval-ms N] # one snapshot to stdout sysagent serve --port P [--interval-ms N] # the same snapshot over HTTP ``` -`sample` takes one reading and prints it — `key=value` lines by default, +`sample` takes one reading and prints it: `key=value` lines by default, a single-line JSON object with `--json`. `serve` exposes the identical snapshot over `GET /metrics`, taking a fresh reading on every request; any other path 404s. Both walk the same six sources: `/proc/stat` (as a delta, covered next), `/proc/loadavg`, `/proc/meminfo`, `/proc/diskstats`, `/proc/net/dev`, and cgroup PSI. What makes this example worth its own -chapter isn't any one of those parsers — each is a plain-text file, `open`, -split fields, done — it's that the field names landing in the JSON are a +chapter isn't any one of those parsers (each is a plain-text file, `open`, +split fields, done); it's that the field names landing in the JSON are a **deterministic, cross-language schema**: `cpu_util_pct`, `mem_total_kb`, `disks[].reads`, `psi_available`, and every other key are byte-identical whether the binary is the C++, Go, or Rust build. Point one Grafana dashboard, one `jq` filter, or one test suite at any of the three and it -works unmodified — the entire point of README.md's field table, which this +works unmodified: the entire point of README.md's field table, which this chapter's `verify.lua` checks key by key. ## The `/proc/stat` delta: two cumulative reads, one percentage `/proc/stat`'s `cpu` line is not a gauge. It's eight cumulative tick counters -— `user`, `nice`, `system`, `idle`, `iowait`, `irq`, `softirq`, `steal` — +(`user`, `nice`, `system`, `idle`, `iowait`, `irq`, `softirq`, `steal`) that only ever grow, counted in USER_HZ ticks since the machine booted. A single read of that line tells you nothing about *current* utilization; it tells you the total CPU-seconds this machine has ever spent in each state. @@ -155,9 +155,9 @@ again (`t1`), and only then computes the delta: ``` `irq`, `softirq`, and `steal` fold into `d_busy` alongside `user`/`nice`/ -`system` because all three represent CPU time genuinely spent — interrupt +`system` because all three represent real CPU time spent: interrupt handling, softirq processing, and (on a VM) time stolen by the hypervisor for -another guest — none of it is idle time, even though none of it is `user` or +another guest; none of it is idle time, even though none of it is `user` or `system` either. The `d_total > 0.0`/`dTotal > 0` guard exists because a sufficiently short `--interval-ms` on an otherwise-idle single-CPU host could in principle produce a zero-tick window (the kernel's tick granularity is @@ -166,21 +166,21 @@ stay at their zero-initialized default rather than dividing by zero. ## Graceful degradation: PSI is optional, not an error -Pressure Stall Information — the `some avg10=`/`avg60=`/`avg300=` lines in -`cpu.pressure`, `memory.pressure`, and `io.pressure` — tells you what +Pressure Stall Information (the `some avg10=`/`avg60=`/`avg300=` lines in +`cpu.pressure`, `memory.pressure`, and `io.pressure`) tells you what fraction of the last 10/60/300 seconds *some* task was stalled waiting on -that resource, which is a genuinely different signal from utilization: a CPU +that resource, which is a fundamentally different signal from utilization: a CPU can be 100% utilized with zero pressure (one thread pinning one core, nothing else waiting) or 40% utilized with real pressure (many threads taking turns, each one stalled part of the time). `read_psi` looks in two places, in -order: this process's own cgroup v2 delegate first — resolved by reading the +order: this process's own cgroup v2 delegate first, resolved by reading the single `0::` line out of `/proc/self/cgroup` and joining it onto -`/sys/fs/cgroup/{cpu,memory,io}.pressure` — falling back to the +`/sys/fs/cgroup/{cpu,memory,io}.pressure`, falling back to the system-wide `/proc/pressure/{cpu,memory,io}` files if the cgroup-scoped ones aren't there. Neither location is guaranteed: PSI can be compiled out of the kernel, or a container can be handed a cgroup namespace with no delegated `.pressure` files at all. When that happens, `sysagent` doesn't fail the -whole snapshot — `psi_available` comes back `false`, the three `avg10` +whole snapshot: `psi_available` comes back `false`, the three `avg10` fields default to `0.0`, and every other field in the schema is unaffected. A missing PSI source is data, not an error; the schema has a bit for exactly that case so a dashboard built against it never has to special-case "did @@ -188,8 +188,8 @@ this host even expose PSI." ## How the code works -`main` dispatches on `argv[1]` — `sample` or `serve`, anything else prints -the usage line and exits — and both subcommands share one hand-rolled +`main` dispatches on `argv[1]` (`sample` or `serve`, anything else prints +the usage line and exits), and both subcommands share one hand-rolled `Flags` scan (`--json`, `--interval-ms N`, `--port P`) kept in the same shape across all three languages precisely so the argument-parsing code never becomes the interesting part of the diff. `cmd_sample` calls @@ -199,27 +199,27 @@ the HTTP loop and calls `take_snapshot` again on every `GET /metrics`. `to_json`/`ToJSON`/`to_json` is the piece that actually delivers the cross-language guarantee the schema promises, and it earns that guarantee by -being deliberately unglamorous: none of the three languages hands the +staying unglamorous by design: none of the three languages hands the `Snapshot` to a generic serializer. C++ builds the object with an `ostringstream` and literal `"key":value,` fragments in a fixed order; Go does the same with `strings.Builder` and `fmt.Fprintf`; Rust with `String` -and `push_str`. A generic serializer — `nlohmann::json`, `encoding/json`, -`serde_json` — would have been less code, but none of the three guarantees +and `push_str`. A generic serializer (`nlohmann::json`, `encoding/json`, +`serde_json`) would have been less code, but none of the three guarantees the *same* key order or the same float formatting as the others out of the box, and a schema that's supposed to be byte-identical across languages can't leave that up to three different libraries' defaults. Hand-writing the same fixed field order three times, with the same `%.2f`-style two-decimal formatting (`fmt2`/`f2`/`{:.2}`) for every float, is what actually makes -`verify.lua`'s key-by-key checks — and a `diff` between any two languages' -output — meaningful. +`verify.lua`'s key-by-key checks, and a `diff` between any two languages' +output, meaningful. -The HTTP layer is the one place the three implementations genuinely diverge +The HTTP layer is the one place the three implementations actually diverge in mechanism rather than just syntax, and it follows the book's established chatterd pattern: Go leans on `net/http`'s stdlib server, registering `/metrics` on a `ServeMux` and letting `http.Server.Shutdown` handle a clean `SIGINT`/`SIGTERM` stop. C++ and Rust hand-roll the same tiny surface -instead — a raw listening socket, `poll(2)` waiting on both the listener and -a `signalfd`/`SignalFd` doorbell, and one inline HTTP/1.1 response writer — +instead (a raw listening socket, `poll(2)` waiting on both the listener and +a `signalfd`/`SignalFd` doorbell, and one inline HTTP/1.1 response writer) because pulling in a framework for one endpoint would hide the exact mechanism this book has been building toward since Chapter 21's `chatterd` and Chapter 24's signal-driven shutdown, not simplify it. @@ -228,12 +228,12 @@ and Chapter 24's signal-driven shutdown, not simplify it. `sysagent` has one usage-error surface and one runtime-error surface, and they map to exit codes **2** and **1** respectively across all three -languages. A **usage error** — no subcommand, an unrecognized subcommand, -`serve` with no `--port`, or a non-positive `--interval-ms`/`--port` — +languages. A **usage error** (no subcommand, an unrecognized subcommand, +`serve` with no `--port`, or a non-positive `--interval-ms`/`--port`) prints the single usage line to stderr and exits 2; every language funnels every parse failure through the same `usage()` call before returning. A -**runtime error** — `/proc/stat`, `/proc/loadavg`, `/proc/meminfo`, -`/proc/diskstats`, or `/proc/net/dev` failing to open or parse — surfaces as +**runtime error** (`/proc/stat`, `/proc/loadavg`, `/proc/meminfo`, +`/proc/diskstats`, or `/proc/net/dev` failing to open or parse) surfaces as `sysagent: error: ` on stderr and exit 1; `/proc/stat` itself is checked first, before any concurrent gathering even begins, since the whole snapshot brackets around its two readings. A successful `sample` or a clean @@ -242,8 +242,8 @@ signal-driven `serve` shutdown exits **0**. PSI sits outside that error surface entirely, and Rust's type signatures make the guarantee visible at compile time rather than just by convention: `read_loadavg`, `read_meminfo`, `read_diskstats`, and `read_netdev` all -return `Result` — they *can* fail, and `take_snapshot` propagates -that failure. `read_psi` returns `Option` — there is no `Err` variant +return `Result`: they *can* fail, and `take_snapshot` propagates +that failure. `read_psi` returns `Option`: there is no `Err` variant for it to construct, so a PSI-read failure literally cannot become a runtime error; the type system rules it out before `take_snapshot` is ever called. C++ mirrors the same asymmetry by discipline rather than by type: every @@ -251,22 +251,22 @@ other worker thread calls `note_error(...)` on failure, but `psi_thread` never does, its `if` guards only the success path. Go hardcodes the same asymmetry directly into the fan-in loop, which explicitly `continue`s past `r.source == "psi"` before counting it toward the required sources or a -possible `firstErr`. Three different enforcement mechanisms — a type the +possible `firstErr`. Three different enforcement mechanisms (a type the compiler checks, a code path that's simply never written, a loop that -explicitly skips one name — for the same one-sentence rule: a missing PSI +explicitly skips one name) for the same one-sentence rule: a missing PSI source degrades the snapshot, it never fails it. ## Concurrency lens Five of `sysagent`'s six sources are independent of each other and of the `/proc/stat` delta they're bracketed by, so `take_snapshot` overlaps all five -reads with the `--interval-ms` sleep instead of paying for each sequentially -— the wall-clock cost of one `sample` is `max(interval_ms, gather_time)`, not +reads with the `--interval-ms` sleep instead of paying for each sequentially: +the wall-clock cost of one `sample` is `max(interval_ms, gather_time)`, not `interval_ms + gather_time`. Every language spawns the same five workers (`loadavg`, `meminfo`, `diskstats`, `netdev`, `psi`), sleeps, and joins before `t1` is read, guaranteeing every field is filled in before the delta is -computed — but *how* each language gets a result out of a worker back into -the shared `Snapshot` is where the three genuinely part ways: +computed, but *how* each language gets a result out of a worker back into +the shared `Snapshot` is where the three actually part ways: {% include codetabs.html langs="C++|Go|Rust" %} @@ -476,25 +476,25 @@ pub fn take_snapshot(interval_ms: u64) -> Result { ``` C++ and Go both mutate the *same* shared `snap` directly from inside each -worker — `load_thread` writes `snap.load1`/`load5`/`load15`, `mem_thread` +worker: `load_thread` writes `snap.load1`/`load5`/`load15`, `mem_thread` writes the three `mem_*` fields, and so on. This is race-free, but only because every worker touches a *disjoint* set of fields: distinct struct fields are distinct objects for data-race purposes in both languages' memory models, so five threads writing to five non-overlapping field groups -of one struct never race — Go additionally needs the channel send/receive +of one struct never race. Go additionally needs the channel send/receive in `results <- namedErr{...}` / `r := <-results` to establish the happens-before edge that makes each goroutine's writes visible to the -receiver, which the buffered channel provides for free. Rust's `thread::scope` +receiver, which the buffered channel provides automatically. Rust's `thread::scope` takes a structurally different shape for a reason that isn't stylistic: the five spawned closures (`s.spawn(read_loadavg)`, etc.) don't capture `snap` at all, because two closures each holding `&mut snap` to write disjoint fields -would still require *two simultaneous mutable borrows of the same value* — +would still require *two simultaneous mutable borrows of the same value*, something the borrow checker refuses regardless of whether the underlying fields actually overlap, since it reasons about the whole object, not the fields a closure happens to touch. So Rust's workers return owned values through their `JoinHandle`s instead, and every assignment into `snap` happens in the *scope* thread itself, sequentially, immediately after each -`.join().unwrap()` — safe by construction, where C++ and Go are safe by the +`.join().unwrap()`: safe by construction, where C++ and Go are safe by the programmer having correctly kept every worker's field set disjoint. ## Build, run, observe @@ -520,7 +520,7 @@ being captured for this chapter: `mem_total_kb=65545860` is ~62.5 GiB, this host's actual installed RAM; `psi_available:true` because this box's cgroup v2 delegate exposes PSI, so the process's own `cpu.pressure`/`memory.pressure`/`io.pressure` files were -readable — no fallback to `/proc/pressure` was needed on this run. With no +readable; no fallback to `/proc/pressure` was needed on this run. With no arguments at all, all three builds print the usage line to stderr and exit 2: ```console @@ -542,8 +542,8 @@ sysagent: shutting down ``` The second `cpu_util_pct` (7.21) differs from the first (6.27) because -`/metrics` takes a genuinely new `--interval-ms`-apart sample on every -request rather than caching one — the same live measurement `sample` +`/metrics` takes a fresh `--interval-ms`-apart sample on every +request rather than caching one: the same live measurement `sample` prints, just reachable over the network. The unknown path 404s, and `SIGINT` prints the shutdown line and exits 0. @@ -566,8 +566,8 @@ example cpp go rust Two different cross-checks back two different kinds of claim. The first is a **static fact**: `mem_total_kb` is `MemTotal` straight out of `/proc/meminfo`, unaffected by system load or timing, so `verify.lua` reads -it two ways in the same run — once through `serve`'s `/metrics` response, -once through a direct `awk '/^MemTotal:/{print $2}' /proc/meminfo` — and +it two ways in the same run: once through `serve`'s `/metrics` response, +once through a direct `awk '/^MemTotal:/{print $2}' /proc/meminfo`, and requires the two numbers to agree **exactly**, not approximately. On this host that's `65545860` from both sources, every time; any mismatch would mean the parser is reading the wrong field or the wrong file, not measurement @@ -579,18 +579,18 @@ actually tracks real CPU load rather than returning a constant or a miscomputed delta. `verify.lua` takes an idle `sample --json` baseline, spins up `nproc` tight busy-loop children for slightly longer than a `--interval-ms 500` window, samples again, and checks that `cpu_util_pct` -rose by a real margin — retried up to three times, with a 12-percentage-point +rose by a real margin: retried up to three times, with a 12-percentage-point floor (or an outright ≥55% reading, for the case where the host is already loaded before the check even starts). The margin and the retries exist because this runs on a real, possibly-noisy development box, not an isolated -benchmark rig — the check has to distinguish "this host happened to be busy +benchmark rig; the check has to distinguish "this host happened to be busy for an unrelated reason" from "the parser is broken," and a single non-retried comparison couldn't. ## What you learned - **`/proc/stat` is a cumulative counter, not a gauge.** A percentage only - exists as the delta between two readings `--interval-ms` apart — + exists as the delta between two readings `--interval-ms` apart: `take_snapshot` reads it once before the concurrent gather, once after, and computes `cpu_util_pct` only from the difference. - **Gather independent `/proc` sources concurrently with the sampling @@ -598,8 +598,8 @@ non-retried comparison couldn't. `--interval-ms` window: the wall-clock cost of a sample is `max(interval_ms, gather_time)`, not the sum. - **A missing signal should degrade the data, not fail the call.** PSI - absent — kernel support missing, or a container with no cgroup v2 - delegate — sets `psi_available:false` rather than returning an error; + absent (kernel support missing, or a container with no cgroup v2 + delegate) sets `psi_available:false` rather than returning an error; Rust enforces this at the type level (`Option`, no `Err` variant to construct), C++ and Go enforce the same rule by never writing the failure path for that one source. @@ -610,14 +610,14 @@ non-retried comparison couldn't. formatting would drift between languages. - **Rust's borrow checker forces a different concurrency shape, not just different syntax.** C++ and Go both mutate one shared struct directly - from five worker threads — safe because the field sets are disjoint, but + from five worker threads, safe because the field sets are disjoint, but only by the programmer's discipline. Rust can't express that pattern at all without extra ceremony, so `thread::scope`'s workers return owned values and every write into `snap` happens sequentially in the parent, - after each `.join()` — safe by construction instead. + after each `.join()`: safe by construction instead. - **`mem_total_kb` served over HTTP matched a direct `/proc/meminfo` read exactly**, and a busy-loop child measurably raised `cpu_util_pct` over an - idle baseline — two cross-checks confirming the schema reports what the + idle baseline: two cross-checks confirming the schema reports what the kernel actually has, not what the code merely claims. Chapter 37 keeps the same Utilization/Saturation/Errors lens but inverts the @@ -625,12 +625,12 @@ source of truth entirely: instead of reading `/proc` directly the way this chapter does, `loadmix analyze` shells out to `uptime`, `vmstat`, `mpstat`, `iostat`, `sar`, and `pidstat` exactly as an on-call engineer would, and parses each tool's plain-text table by column name rather than reading -kernel counters itself. Read the two chapters back to back — same checklist, +kernel counters itself. Read the two chapters back to back: same checklist, same USE method, two structurally opposite ways of answering it. --- -

verified — on the Fedora 44 +

verified: on the Fedora 44 reference host (kernel 7.1, 16 CPUs, ~62 GiB RAM) this session: python3 scripts/test-all-examples.py --only 36-proc-sys-and-the-agent --mode local printed PASS PASS PASS (3 passed, 0 failed; @@ -652,6 +652,6 @@ confirmed a busy-loop child measurably raised cpu_util_pct above an idle baseline. Not exercised: the psi_available:false degraded path (this host's cgroup v2 delegate exposes PSI, so every sample this session read psi_available:true) and any run against the -lab VM — this is a local-mode example (mode: local in the +lab VM: this is a local-mode example (mode: local in the manifest) that reads the current host's own /proc, not a VM's.

diff --git a/_docs/37-use-method-60s-analysis.md b/_docs/37-use-method-60s-analysis.md index 416ee3e..0163faf 100644 --- a/_docs/37-use-method-60s-analysis.md +++ b/_docs/37-use-method-60s-analysis.md @@ -2,24 +2,24 @@ title: "The USE method in 60 seconds: a checklist that runs the real tools, not a shortcut around them" order: 37 part: "Observability" -description: "loadmix is Brendan Gregg's Utilization/Saturation/Errors checklist as a runnable program — a saturation generator that pegs one resource at a time, and an analyze mode that shells out to uptime, vmstat, mpstat, iostat, sar, and pidstat exactly as a human would, parses each tool's plain-text table by column name from its own header row, runs the five interval samplers concurrently so a 60-second checklist costs 60 seconds and not five minutes, and names the resource whose Saturation signal fired hardest." +description: "loadmix is Brendan Gregg's Utilization/Saturation/Errors checklist as a runnable program: a saturation generator that pegs one resource at a time, and an analyze mode that shells out to uptime, vmstat, mpstat, iostat, sar, and pidstat exactly as a human would, parses each tool's plain-text table by column name from its own header row, runs the five interval samplers concurrently so a 60-second checklist costs 60 seconds and not five minutes, and names the resource whose Saturation signal fired hardest." duration: "50 minutes" --- -Chapter 36's `sysagent` built a USE-method collector from first principles — +Chapter 36's `sysagent` built a USE-method collector from first principles: reading `/proc/stat`, `/proc/net`, `/proc/diskstats`, and cgroup PSI directly, no external process, no tool you'd have to trust. That is the right call for a program that runs continuously in production. But it is not how most engineers actually triage a system that's in trouble *right now*: they SSH in and reach for `uptime`, `vmstat 1`, `mpstat -P ALL 1`, `iostat -xz 1`, `pidstat 1`, and -`free`, running the checklist Brendan Gregg popularized as "the USE method" — +`free`, running the checklist Brendan Gregg popularized as "the USE method": for every resource, ask whether it's **Utilized**, **Saturated**, or -**Erroring** — because those tools already exist, are already trusted, and +**Erroring**, because those tools already exist, are already trusted, and already parse a live system correctly. `loadmix` is that checklist wired into a program: a saturation generator that pegs one resource on command, and an `analyze` mode that runs the real tools exactly as a human would, reads their -plain-text tables the way a human reads them — by column *name*, from each -tool's own header row — and tells you which resource is in trouble and why. +plain-text tables the way a human reads them (by column *name*, from each +tool's own header row) and tells you which resource is in trouble and why. The code is in `examples/37-use-method-60s-analysis/`. The run script there builds/sets up and runs it; its `README.md` covers what it does and how to @@ -27,13 +27,13 @@ drive it. {% include excalidraw.html file="37-use-method-checklist" - alt="A pipeline diagram of loadmix analyze. Left column, four sequential pre-samples run once at start: uptime, dmesg tail, top -b -n1, and ss -s, each landing in a single printed 'tool' line. Center, a band labeled 'five interval samplers, spawned concurrently' containing five parallel boxes — vmstat 1 N, mpstat -P ALL 1 N, iostat -xz 1 N, sar -n DEV 1 N, and pidstat 1 N — each writing to its own temp file, all five joined by a single wait so the band's total wall time is N seconds, not 5N; a note under the band reads 'N-second checklist costs ~N seconds'. Right, each sampler's output feeds a column-name parser (Row::get against the tool's own header row) into a 4x3 signal matrix: rows cpu/mem/io/net, columns Utilization/Saturation/Errors, each cell a fired=true-or-false verdict against a fixed threshold — cpu's Saturation threshold shown as cpus+2 to make the guest-relative scaling visible. An arrow from the matrix's Saturation column into a final verdict box reads 'resource with the highest Saturation-value/threshold ratio wins'. A small inset box below the diagram shows the saturate side: app --resource cpu|mem|io|net --seconds N spinning a worker pool that the analyze pipeline, run concurrently against it, is able to see and attribute." - caption="Figure 37.1 — the analyze pipeline: four one-shot pre-samples, five interval samplers run concurrently, column-name parsing into a 4×3 USE signal matrix, and the highest-Saturation-ratio verdict" %} + alt="A pipeline diagram of loadmix analyze. Left column, four sequential pre-samples run once at start: uptime, dmesg tail, top -b -n1, and ss -s, each landing in a single printed 'tool' line. Center, a band labeled 'five interval samplers, spawned concurrently' containing five parallel boxes (vmstat 1 N, mpstat -P ALL 1 N, iostat -xz 1 N, sar -n DEV 1 N, and pidstat 1 N) each writing to its own temp file, all five joined by a single wait so the band's total wall time is N seconds, not 5N; a note under the band reads 'N-second checklist costs ~N seconds'. Right, each sampler's output feeds a column-name parser (Row::get against the tool's own header row) into a 4x3 signal matrix: rows cpu/mem/io/net, columns Utilization/Saturation/Errors, each cell a fired=true-or-false verdict against a fixed threshold, cpu's Saturation threshold shown as cpus+2 to make the guest-relative scaling visible. An arrow from the matrix's Saturation column into a final verdict box reads 'resource with the highest Saturation-value/threshold ratio wins'. A small inset box below the diagram shows the saturate side: app --resource cpu|mem|io|net --seconds N spinning a worker pool that the analyze pipeline, run concurrently against it, is able to see and attribute." + caption="Figure 37.1: the analyze pipeline: four one-shot pre-samples, five interval samplers run concurrently, column-name parsing into a 4×3 USE signal matrix, and the highest-Saturation-ratio verdict" %} -> **Tools used** — `uptime`, `dmesg`, `top`, `ss`, `vmstat`, `free` +> **Tools used**: `uptime`, `dmesg`, `top`, `ss`, `vmstat`, `free` > (procps-ng/util-linux/iproute2, base Fedora on the `systems-target` guest) > plus `mpstat`, `iostat`, `sar`, `pidstat` (the `sysstat` package, installed -> by the guest's own cloud-init) — all on the `systems-target` VM, which +> by the guest's own cloud-init), all on the `systems-target` VM, which > `loadmix analyze` invokes directly as subprocesses. No tool here is > `loadmix`-specific; this chapter's whole premise is that the checklist is > the same nine commands you'd type by hand. @@ -51,7 +51,7 @@ app analyze [--seconds N] run the checklist and name ``` `--resource cpu` spins a pool of worker threads doing nothing but xorshift -arithmetic — no syscalls, no allocation in the hot loop, pure CPU. `--resource +arithmetic, no syscalls, no allocation in the hot loop, pure CPU. `--resource mem` touches 1.35× the guest's `MemTotal` across a worker pool, deliberately overshooting available RAM to push real pages into swap. `--resource io` opens one file per worker with `O_DSYNC` and `pwrite`s a 16 KiB buffer of @@ -61,14 +61,14 @@ net` floods UDP datagrams over loopback from a pool of senders to one receiver. Each saturator prints a `loadmix: start ...` line with the resource, requested duration, and worker count, runs for (approximately) that duration, and prints a `loadmix: done ... ops= bytes=` line. `analyze` -does none of that — it induces no load of its own, and instead spends its +does none of that; it induces no load of its own, and instead spends its `--seconds` window watching the system through the same nine tools listed above. ## Parsing by column name, not position -The single design decision that makes `analyze` durable — the reason it -survives a `sysstat` or `procps-ng` version bump that reorders columns — is +The single design decision that makes `analyze` durable (the reason it +survives a `sysstat` or `procps-ng` version bump that reorders columns) is that every parser reads a tool's header row first and looks fields up by *name* against that row, never by a hardcoded index. `Row` (C++), `row` (Go), and `Row` (Rust) all express the same idea: a table row is a name list zipped @@ -183,7 +183,7 @@ fn make_row( ``` `header_skip`/`data_skip` exist because a header row and its data rows don't -always start aligned — `mpstat -P ALL`'s `Average:` rows both begin with a +always start aligned: `mpstat -P ALL`'s `Average:` rows both begin with a label token (`Average: CPU` / `Average: all`) that has to be skipped before `%idle` in the header lines up with the actual idle percentage in the data. `parse_mpstat` puts this together end to end: it scans every line looking for @@ -295,7 +295,7 @@ parsers (`parse_vmstat`, `parse_iostat`, `parse_sar_pkts`) each carry one more quirk worth knowing before you read the source: `vmstat` has no `Average:` line at all, and its first sample is a since-boot average rather than a live interval, so `parse_vmstat` scans every data row, discards row 0, and takes -the *max* of `r` (the run queue) across the rest — the steady-state +the *max* of `r` (the run queue) across the rest, the steady-state saturation signal, not one diluted by averaging across the whole window. `iostat -xz` has the identical since-boot-first-row quirk, plus `-z` hides idle devices entirely, so `parse_iostat` tracks block boundaries with its own @@ -303,8 +303,8 @@ idle devices entirely, so `parse_iostat` tracks block boundaries with its own ## How the code works -`cmd_analyze` runs in three phases. First, four one-shot pre-samples — -`uptime`, `dmesg | tail -n 5`, `top -b -n1`, `ss -s` — each captured through +`cmd_analyze` runs in three phases. First, four one-shot pre-samples +(`uptime`, `dmesg | tail -n 5`, `top -b -n1`, `ss -s`) each captured through `run_capture`, the one function in the file with a real failure mode (the shell itself failing to start), which is why it's the one that returns `std::expected`/`error`/`Result` per this book's convention. `dmesg` on an @@ -313,8 +313,8 @@ for that string and reports `status=denied` rather than treating it as a capture failure, because an unprivileged `analyze` run is the common case, not an edge case. -Second, the five interval samplers — `vmstat`, `mpstat -P ALL`, `iostat -xz`, -`sar -n DEV`, `pidstat`, each invoked as ` 1 N` — are spawned +Second, the five interval samplers (`vmstat`, `mpstat -P ALL`, `iostat -xz`, +`sar -n DEV`, `pidstat`, each invoked as ` 1 N`) are spawned **concurrently**, each redirected to its own temp file, and only then joined. This matters more than it looks: each of those five commands blocks for `N` seconds sampling at a 1-second interval, so running them one after @@ -338,34 +338,34 @@ net Utilization: pkts/s >= 2000 Saturation: pkts/s >= 8000 cpu's Saturation threshold is the one that scales with the machine it's running on rather than being a flat number: `cpus + 2`, so a run queue longer than the guest's own core count plus a little headroom means threads are -genuinely waiting for a CPU that isn't there — the textbook definition of +waiting for a CPU that isn't there, the textbook definition of saturation, not an arbitrary constant that would be wrong on a differently sized host. Errors is printed for every resource but always `fired=false`, because this chapter's saturators induce load, not kernel-logged faults; the line exists so the U/S/E triple is complete and so a future chapter that *does* wire in `dmesg`-sourced error counts has somewhere to plug them in. The verdict is whichever resource has the highest `Saturation value / -Saturation threshold` ratio — not the highest raw Saturation value, which +Saturation threshold` ratio, not the highest raw Saturation value, which would let net's `8000`-scale threshold always lose to cpu's `4`-scale one regardless of which resource is actually worse off. ## Errors, three ways `loadmix` has one failure surface and one exit code for it: a **usage -error** — no `--resource`/`--seconds` given at all, an unrecognized +error** (no `--resource`/`--seconds` given at all, an unrecognized `--resource` value, `--seconds 0` or negative, or an unrecognized flag to -`analyze` — prints the two-line usage banner to stderr and exits **2**. C++ +`analyze`) prints the two-line usage banner to stderr and exits **2**. C++ funnels every parse path through `usage()` and returns `2` from `main`; Go calls `usage()` then `os.Exit(2)`; Rust calls `usage()` then `std::process::exit(2)`. There is no separate "runtime error" exit code here, -unlike Chapter 39's codec-bug path or Chapter 40's connection-refused path — +unlike Chapter 39's codec-bug path or Chapter 40's connection-refused path; `analyze`'s subprocess calls are designed to degrade gracefully instead of failing the whole run: a denied `dmesg`, an empty `ss -s`, or a sampler that produces no data rows all show up as a `0.00`/`denied`/empty field in the output rather than a non-zero exit, because the entire point of a diagnostic checklist is that it still reports *something* under the same restricted privileges a real on-call engineer would have. A successful run of either -mode — saturate or analyze — exits **0**. +mode (saturate or analyze) exits **0**. ## Concurrency lens @@ -374,14 +374,14 @@ worker pool that loops until `stop_requested() || now >= deadline`, then join. C++'s pool is a `std::vector`; Go's is a `sync.WaitGroup` over goroutines with `sync/atomic` counters; Rust's is `std::thread::scope`, the `jthread` analogue where worker threads are RAII-joined when the scope block -ends. All three converge on identical worker-count formulas — `max(4, cpus*3)` -for cpu, `max(4, cpus*4)` for io and net, `max(2, cpus)` for mem — and +ends. All three converge on identical worker-count formulas (`max(4, cpus*3)` +for cpu, `max(4, cpus*4)` for io and net, `max(2, cpus)` for mem) and identical atomic counters accumulated across the pool. -Where the three languages genuinely diverge is in how the parent thread stays +Where the three languages diverge is in how the parent thread stays alive long enough for that deadline to matter. C++'s `std::jthread` runs `request_stop()` *before* `join()` in its destructor, which means the instant -the enclosing scope's spawn loop finishes, every worker is told to stop — +the enclosing scope's spawn loop finishes, every worker is told to stop, before the deadline the workers are checking has had any chance to arrive. Left alone, this was a real bug in the first cut of this example: every saturator exited in single-digit milliseconds regardless of `--seconds`, @@ -420,7 +420,7 @@ close and trigger the RAII-driven stop-then-join: Go and Rust never needed this hack, because in both ports each worker checks its *own* deadline directly inside its loop condition (`time.Now().Before(deadline)`, `Instant::now() < deadline`) rather than waiting on an external stop signal -raised by the parent — the parent simply calls `wg.Wait()` / lets +raised by the parent; the parent simply calls `wg.Wait()` / lets `thread::scope` return, which blocks until every worker's own loop condition has already become false on its own. There is no analogue of `jthread`'s stop-before-join ordering to fight, because nothing external tells these @@ -428,8 +428,8 @@ workers to stop early; they stop themselves, on schedule, and the parent's join is purely a rendezvous rather than part of the termination signal. The second, unrelated concurrency story in this chapter is `analyze`'s five -interval samplers, spawned as five real OS processes — `fork`+`execvp` in -C++, `exec.Command` on five goroutines in Go, `Command::spawn` in Rust — with +interval samplers, spawned as five real OS processes (`fork`+`execvp` in +C++, `exec.Command` on five goroutines in Go, `Command::spawn` in Rust) with a single joint wait afterward. This isn't thread concurrency inside the program at all; it's five independent subprocesses racing the wall clock together so that, as the source comment puts it, "a 12s analyze window costs @@ -444,7 +444,7 @@ together so that, as the source comment puts it, "a 12s analyze window costs This is a **VM example**: the saturation signals only mean something measured against a real, otherwise-quiet kernel, so `verify.lua` stages the built binary directly to `/tmp/lsp37-/app` on `systems-target` via plain -`scp`/`ssh` and drives it there — not through `demo.sh run +`scp`/`ssh` and drives it there, not through `demo.sh run TARGET=...`, whose `deploy-to-vm.sh` wraps commands in `ssh -t` and injects CRLF that breaks the tool's plain-text output. @@ -468,11 +468,11 @@ analyze: verdict resource=cpu ratio=0.75 analyze: done ``` -`dmesg status=denied` is expected — the guest's unprivileged `fedora` user -gets `Operation not permitted` from `dmesg`, and `analyze` reports that -plainly rather than failing. This run finished in **10.417s** wall-clock for +`dmesg status=denied` is expected: the guest's unprivileged `fedora` user +gets `Operation not permitted` from `dmesg`, and `analyze` reports it +rather than failing. This run finished in **10.417s** wall-clock for a `--seconds 10` window, confirming the five samplers really do run -concurrently — five serial ` 1 10` invocations would have taken close +concurrently; five serial ` 1 10` invocations would have taken close to 50 seconds. Now the payoff: a background saturator, then a concurrent checklist run @@ -492,7 +492,7 @@ analyze: done `busy_pct` rose from the idle baseline of **1.34** to **100.00**; `run_queue` rose from **3.00** to **7.00–8.00** across three independent captures on this -2-vCPU guest — comfortably clearing both the `busy_pct >= 60` and +2-vCPU guest, comfortably clearing both the `busy_pct >= 60` and `run_queue >= cpus+2 = 4` thresholds by a wide margin, every time. Utilization fired true, and the verdict landed on `cpu`: `analyze`, run concurrently against a background `loadmix --resource cpu` saturator it knows @@ -525,51 +525,51 @@ example cpp go rust ## Cross-check: the same load, seen two ways -The load-attribution run above is itself a cross-check — `analyze`'s verdict +The load-attribution run above is itself a cross-check: `analyze`'s verdict is only trustworthy if it agrees with what you already know is true, since you launched the saturator yourself. But it's worth pairing `analyze`'s own report against `top`'s independent view, captured in the very same run: `top -b -n1`'s `%Cpu(s)` line read `97.8 us, 2.2 sy, 0.0 id` while `mpstat`'s `busy_pct` (100.0 − `%idle`) read `100.00`. The two numbers come from -different tools sampling at different instants over different windows — -`top` a single one-shot snapshot, `mpstat` an average over the whole -`analyze --seconds` window — so they are not expected to match to the decimal +different tools sampling at different instants over different windows +(`top` a single one-shot snapshot, `mpstat` an average over the whole +`analyze --seconds` window), so they are not expected to match to the decimal place, and they don't; what they agree on is the story: this guest's CPUs are saturated, not idle, not lightly loaded. A verdict that `mpstat` alone -disagreed with `top` about — one reporting near-idle while the other reported -saturated — would be the sign of a parsing bug, not measurement noise, since +disagreed with `top` about (one reporting near-idle while the other reported +saturated) would be the sign of a parsing bug, not measurement noise, since both tools are reading the same underlying `/proc/stat` counters through different sampling windows. ## What you learned -- **The USE method is a checklist, not a tool** — for every resource, ask +- **The USE method is a checklist, not a tool**: for every resource, ask whether it's Utilized, Saturated, or Erroring, against a threshold you choose up front. `loadmix analyze` is that checklist wired into a program that runs the same nine commands (`uptime`, `dmesg`, `top`, `ss`, `vmstat`, `mpstat`, `iostat`, `sar`, `pidstat`, `free`) a person would run by hand. - **Parse plain-text tool output by column name, from the tool's own header - row, never by hardcoded position** — `Row::get`/`row.getOr` walk a name + row, never by hardcoded position**: `Row::get`/`row.getOr` walk a name list to find the field a caller asked for, which is what keeps every parser correct across a `sysstat` or `procps-ng` version that reorders columns. -- **Run independent interval samplers concurrently, not serially** — five +- **Run independent interval samplers concurrently, not serially**: five ` 1 N` processes spawned together and joined once made a `--seconds 10` checklist finish in 10.417s instead of the ~50s five serial invocations would have cost. - **A saturation threshold should scale with the machine it runs on where - that's the accurate signal** — cpu's `run_queue >= cpus + 2` uses the + that's the accurate signal**: cpu's `run_queue >= cpus + 2` uses the guest's own core count, unlike net's flat `8000 pkts/s`, because "threads waiting for a CPU" only means something relative to how many CPUs exist. - **`std::jthread`'s RAII order (`request_stop()` before `join()`) can silence - a worker before its own deadline ever fires** — the fix is holding the + a worker before its own deadline ever fires**: the fix is holding the parent scope open with `sleep_until(deadline)`; Go and Rust never needed the workaround because each worker checks its own deadline directly rather than waiting on an externally raised stop signal. - **`analyze`, run concurrently against a background `--resource cpu` saturator it knows nothing about, correctly attributed the load**: `busy_pct` 1.34 → 100.00, `run_queue` 3.00 → 7.00–8.00, Utilization fired - true, verdict `cpu` — stable across three independent attempts on the lab + true, verdict `cpu`, stable across three independent attempts on the lab guest. Chapter 38 takes the numbers this checklist prints to a terminal and sends @@ -579,7 +579,7 @@ happens. --- -

verified — on the +

verified: on the systems-target lab guest (Fedora 44, kernel 6.19.10-300.fc44, 2 vCPU nested KVM) this session: python3 scripts/test-all-examples.py --only @@ -604,5 +604,5 @@ showing no OOM-kill and full memory/swap recovery after exit. Not exercised: a privileged dmesg capture (the guest's unprivileged fedora account reports status=denied, which analyze treats as an expected, non-failing path) and any run -outside the lab guest — this is a VM example whose saturation signals are +outside the lab guest; this is a VM example whose saturation signals are only meaningful against a real, otherwise-quiet kernel.

diff --git a/_docs/38-opentelemetry-to-lgtm.md b/_docs/38-opentelemetry-to-lgtm.md index aceb089..23869d2 100644 --- a/_docs/38-opentelemetry-to-lgtm.md +++ b/_docs/38-opentelemetry-to-lgtm.md @@ -2,7 +2,7 @@ title: "OpenTelemetry to LGTM: instrument a service, then prove the spans and metrics actually arrived" order: 38 part: "Observability" -description: "lsp-otel is a traced request/response server that exports OpenTelemetry traces and metrics over OTLP/HTTP to the LGTM stack from Chapter 3 — a parent request span wrapping work and respond children, a requests_total counter and a request_duration histogram — and the verify step proves it by querying Tempo and Prometheus through Grafana rather than trusting the program's own word, across C++, Go, and Rust each tagged with its own service.name." +description: "lsp-otel is a traced request/response server that exports OpenTelemetry traces and metrics over OTLP/HTTP to the LGTM stack from Chapter 3: a parent request span wrapping work and respond children, a requests_total counter and a request_duration histogram, and the verify step proves it by querying Tempo and Prometheus through Grafana rather than trusting the program's own word, across C++, Go, and Rust each tagged with its own service.name." duration: "50 minutes" --- @@ -11,14 +11,14 @@ Chapter 36's `sysagent` served one JSON snapshot per HTTP request. Both make you *present* to read the number: someone has to be running the command, or scraping the endpoint, at the moment the system is in trouble. This chapter closes that gap. `lsp-otel` is a small traced server that emits the three OpenTelemetry -signal shapes — a distributed **trace** per request, a monotonic **counter**, and -a latency **histogram** — and exports them over OTLP/HTTP into the LGTM stack +signal shapes: a distributed **trace** per request, a monotonic **counter**, and +a latency **histogram**, and exports them over OTLP/HTTP into the LGTM stack (Loki, Grafana, Tempo, Mimir plus an OpenTelemetry collector) that Chapter 3 stood up, so the next time a request is slow you can go *look* at when it happened instead of needing to have been watching. And because "I sent telemetry" is the easiest claim in observability to get quietly wrong, the whole -point of the verification here is that it queries the stack back — Tempo for the -traces, Prometheus for the metrics — and refuses to believe the export happened +point of the verification here is that it queries the stack back (Tempo for the +traces, Prometheus for the metrics) and refuses to believe the export happened until the data is actually there to read. The code is in `examples/38-opentelemetry-to-lgtm/`. It is a **local-mode @@ -27,10 +27,10 @@ the exporter posts to the collector on `localhost:4318`. {% include excalidraw.html file="38-otlp-pipeline" - alt="A left-to-right pipeline. Left, a band labeled 'serve — one traced request per input line' containing: a parent span box 'request (attr request.seq)' with two child span boxes below it, 'work' (sleep 1+seq%4 ms) and 'respond' (write ok seq), plus two metric boxes, 'requests_total (+1 per request)' and 'request_duration (ms)'. An arrow labeled OTLP leaves the band into an 'OTLP/HTTP exporter' box annotated 'batch ~2s, flush on SIGINT', which POSTs, over an amber arrow labeled POST /v1/{traces,metrics}, into an 'OTel collector :4318' box. The collector fans out along two arrows — 'spans' up to a 'Tempo — traces' box and 'metrics' across to a 'Prometheus — metrics' box — and both of those feed a 'Grafana query :3000' box on the right. A caption along the bottom reads: cpp · go · rust — each tags service.name = lsp-otel-, so the three never collide in the shared stack." - caption="Figure 38.1 — lsp-otel's telemetry path: a per-request span tree plus two metrics, batched through the OTLP/HTTP exporter to the collector on :4318, fanned to Tempo and Prometheus, and read back through Grafana — which is exactly how the export is verified" %} + alt="A left-to-right pipeline. Left, a band labeled 'serve: one traced request per input line' containing: a parent span box 'request (attr request.seq)' with two child span boxes below it, 'work' (sleep 1+seq%4 ms) and 'respond' (write ok seq), plus two metric boxes, 'requests_total (+1 per request)' and 'request_duration (ms)'. An arrow labeled OTLP leaves the band into an 'OTLP/HTTP exporter' box annotated 'batch ~2s, flush on SIGINT', which POSTs, over an amber arrow labeled POST /v1/{traces,metrics}, into an 'OTel collector :4318' box. The collector fans out along two arrows: 'spans' up to a 'Tempo: traces' box and 'metrics' across to a 'Prometheus: metrics' box, and both of those feed a 'Grafana query :3000' box on the right. A caption along the bottom reads: cpp, go, rust: each tags service.name = lsp-otel-, so the three never collide in the shared stack." + caption="Figure 38.1: lsp-otel's telemetry path: a per-request span tree plus two metrics, batched through the OTLP/HTTP exporter to the collector on :4318, fanned to Tempo and Prometheus, and read back through Grafana, which is exactly how the export is verified" %} -> **Tools used** — `podman` (host, runs the `lsp-lgtm` observability stack — +> **Tools used**: `podman` (host, runs the `lsp-lgtm` observability stack: > OTLP on `:4318`, Grafana on `:3000`) and `curl` (host, the only tool the > verification needs: it queries Prometheus and Tempo through Grafana's > datasource proxy). The stack is the same `lsp-lgtm` container the @@ -40,35 +40,35 @@ the exporter posts to the collector on `localhost:4318`. ## Three signals, one request Every OpenTelemetry story is told in three signal types, and `lsp-otel` emits -all three from a single unit of work — one line of input to `serve`, treated as +all three from a single unit of work: one line of input to `serve`, treated as one "request": - A **trace** describes the shape and timing of that request as a tree of spans. `lsp-otel` starts a parent span named `request`, and *inside* it two child spans: `work` (a deliberately tiny, variable `1 + seq%4` ms sleep - standing in for real computation) and `respond` (writing the `ok ` reply). + standing in for actual computation) and `respond` (writing the `ok ` reply). The parent/child nesting is what lets Tempo later show you that the request's time was spent in `work`, not `respond`, or vice versa. -- A **counter** — `requests_total`, a monotonic `Int64` that goes up by one per - request — answers "how many?" It only ever increases, which is what makes it +- A **counter**, `requests_total`, a monotonic `Int64` that goes up by one per + request, answers "how many?" It only ever increases, which is what makes it a counter rather than a gauge, and what lets Prometheus compute a rate from it. -- A **histogram** — `request_duration`, in milliseconds — answers "how long?" +- A **histogram**, `request_duration`, in milliseconds, answers "how long?" across the whole population, recording one observation (the parent span's wall-clock duration) per request, so the backend can later report a p50, a p99, or a bucketed distribution without the server having to keep every sample. The second subcommand, `drive --n N --addr HOST:PORT`, is a plain TCP client -that sends `N` requests and reads `N` replies. It is deliberately **untraced**: +that sends `N` requests and reads `N` replies. It stays **untraced** by design: it generates the load that makes `serve`'s telemetry interesting, but emits none of its own, because the service under observation is `serve`, not the thing poking it. ## The request pipeline, three ways -The heart of the instrumentation is one function — `handleRequest` — that starts +The heart of the instrumentation is one function, `handleRequest`, that starts the parent span, nests the two children inside it, writes the reply, and records both metrics. It reads almost identically across the three languages because the -OpenTelemetry API is deliberately the same shape in each; the differences are +OpenTelemetry API is the same shape in each by design; the differences are idiomatic, not structural: {% include codetabs.html langs="C++|Go|Rust" %} @@ -189,41 +189,41 @@ The parent/child relationship is carried differently in each, and it is the one part worth reading closely. Go threads a `context.Context`: `tracer.Start` returns a *new* context carrying the parent span, and passing that context into the two child `Start` calls is what makes them children. Rust makes the same -context explicit — `Context::current_with_span(parent_span)` builds a context +context explicit: `Context::current_with_span(parent_span)` builds a context holding the parent, and each child is started `start_with_context(..., &parent_cx)`. C++ uses a `Scope` object: constructing `trace_api::Scope parent_scope(parent_span)` installs the parent as the current span for the lifetime of that RAII object, so the two children, started while the scope is alive, attach to it implicitly. -Three mechanisms — an explicit context value, an explicit context value again, -and a scope guard — for the identical span tree, which Tempo later shows as +Three mechanisms (an explicit context value, an explicit context value again, +and a scope guard) for the identical span tree, which Tempo later shows as `request` with `work` and `respond` beneath it. ## How the code works Each language's `initTelemetry`/`init_telemetry`/`telemetry::init` wires up two -exporters against a shared **resource** — the set of attributes that identify +exporters against a shared **resource**: the set of attributes that identify *this* service in the stack. The critical one is `service.name`, and it is -deliberately different per language: `lsp-otel-cpp`, `lsp-otel-go`, +different per language by design: `lsp-otel-cpp`, `lsp-otel-go`, `lsp-otel-rust`. Because all three export into the *same* shared LGTM stack, a distinct `service.name` is what keeps their signals from colliding and lets the verification query each language's telemetry by its own name. Both exporters speak **OTLP over HTTP/protobuf** to the collector, and both are -**batched on a short (~2 second) interval** — a batch span processor for traces, +**batched on a short (~2 second) interval**: a batch span processor for traces, a periodic reader for metrics. That interval is a deliberate choice: the SDK defaults (5-second span batches, 60-second metric batches) are fine for a long-running service but far too slow for a demo that starts, drives ten requests, and stops within a couple of seconds. A short interval plus a **flush -on shutdown** is what guarantees the last batch — often *all* the batches, at -this scale — actually leaves the process before it exits. That shutdown path is +on shutdown** is what guarantees the last batch (often *all* the batches, at +this scale) actually leaves the process before it exits. That shutdown path is the subject of the Concurrency lens below, because it turned out to be where the three languages diverge most. ## Errors, three ways -`lsp-otel` has two exit codes for failure. A **usage error** — no subcommand, an -unknown subcommand, `serve` without `--port`, or `drive` with `--n 0` or missing -— prints the two-line usage banner to stderr and exits **2**: +`lsp-otel` has two exit codes for failure. A **usage error** (no subcommand, an +unknown subcommand, `serve` without `--port`, or `drive` with `--n 0` or missing) +prints the two-line usage banner to stderr and exits **2**: ``` usage: app serve --port PORT [--otel-endpoint URL] @@ -233,9 +233,9 @@ usage: app serve --port PORT [--otel-endpoint URL] A **runtime error** is exit **1** with an `app: error: …` message: the one `verify.lua` exercises is `drive` against an address nothing is listening on, which fails to connect and prints `app: error: dial …: connection refused`. This -is the one message that is *not* byte-identical across the three languages — +is the one message that is *not* byte-identical across the three languages: each surfaces its own dial/`errno` wording, the same way Chapter 40's `connect` -errors carry each platform's `strerror` text — so `verify.lua` asserts only the +errors carry each platform's `strerror` text, so `verify.lua` asserts only the shared `app: error:` prefix and the exit code, not the full string. A clean run of either subcommand exits **0**: `serve` after a signal-driven shutdown, `drive` after `N/N` successful replies. @@ -244,8 +244,8 @@ after `N/N` successful replies. `serve` has to wait on two fundamentally different things at once: a new connection arriving, and a shutdown signal arriving. All three ports solve this -the same way in shape — a signal source and an accept loop both feed a *single* -channel that the main loop consumes one event at a time — and all three write it +the same way in shape: a signal source and an accept loop both feed a *single* +channel that the main loop consumes one event at a time, and all three write it with the idiom their language reaches for: {% include codetabs.html langs="C++|Go|Rust" %} @@ -301,7 +301,7 @@ with the idiom their language reaches for: } ``` -Go's `select` over `sigCh` and `connCh` is the native version — the language has +Go's `select` over `sigCh` and `connCh` is the native version: the language has a built-in construct for exactly "wait on whichever of these channels is ready first." Rust rebuilds the same thing from an `mpsc::channel` carrying an `Event` enum (`Signal` or `Conn`), fed by a `signal-hook` thread and an accept thread. @@ -312,7 +312,7 @@ Getting there surfaced three portability lessons worth keeping, because each is the kind of bug that leaves the program looking fine while silently doing nothing: -1. **The OTLP/HTTP exporter must POST to the full per-signal URLs** — +1. **The OTLP/HTTP exporter must POST to the full per-signal URLs**: `http://localhost:4318/v1/traces` and `…/v1/metrics`, not the bare base `http://localhost:4318`. Both the Rust and C++ ports first tried the base URL; the collector answered `404`, the export silently failed, and the *only* @@ -324,15 +324,15 @@ nothing: `NoopLogHandler` at `LogLevel::None`; the Rust build had to *not* register a `tracing-subscriber` at all. Silence is part of the port. 3. **C++ only: a background job inherits `SIGINT` set to `SIG_IGN`.** A - non-interactive shell running `serve &` — exactly how `verify.lua` starts it — + non-interactive shell running `serve &` (exactly how `verify.lua` starts it) sets `SIGINT`/`SIGQUIT` to *ignore* in the child before `exec`, and that disposition is inherited. `pthread_sigmask` + `sigwait()` alone cannot consume a signal whose disposition is "ignore"; the port has to install a real (no-op) - `sigaction` first to move the disposition off `SIG_IGN` — precisely what Go's + `sigaction` first to move the disposition off `SIG_IGN`, precisely what Go's `signal.Notify` and Rust's `signal-hook` do under the hood, which is why only the C++ port had to do it by hand. (Two smaller C++-only wrinkles rounded it out: `Histogram::Record` needs the `(value, Context{})` overload under this - build's ABI v1, and the OTLP static archives have a genuine link cycle that + build's ABI v1, and the OTLP static archives have a real link cycle that needs `-Wl,--start-group`.) ## Build, run, observe @@ -341,7 +341,7 @@ nothing: [host]$ cd examples/38-opentelemetry-to-lgtm && ./demo.sh build ``` -Run the server, drive ten requests through it, and stop it — the SIGINT flush is +Run the server, drive ten requests through it, and stop it; the SIGINT flush is what pushes the final batch out: ```console @@ -353,7 +353,7 @@ app: drive sent 10/10 ok app: shutting down (interrupt) ``` -Now read the telemetry back out of the stack — not out of the program. Metrics +Now read the telemetry back out of the stack, not out of the program. Metrics land in Prometheus; querying `requests_total` through Grafana's datasource proxy shows the counter for whichever language just ran: @@ -364,7 +364,7 @@ shows the counter for whichever language just ran: ``` The runner drives all three languages against the live stack and asserts the -full contract — including the stack queries — for each: +full contract (including the stack queries) for each: ```console [host]$ python3 scripts/test-all-examples.py --only 38-opentelemetry-to-lgtm --mode local @@ -394,35 +394,35 @@ exactly **10** in Prometheus, for all three languages: And the *traces* for the same runs are in Tempo: a search by `service.name=lsp-otel-` returns traces whose `rootServiceName` is that name and whose `rootTraceName` is `request`, and fetching one back shows its -spans are exactly `work`, `respond`, and the `request` parent — the tree +spans are exactly `work`, `respond`, and the `request` parent: the tree `handle_request` built. Metrics saying "ten requests happened" and traces saying "here is the shape of each of those requests" are two different subsystems agreeing about the same work, which is a far stronger signal that the -instrumentation is correct than either backend alone — and neither number came +instrumentation is correct than either backend alone, and neither number came from the program telling us about itself. ## What you learned -- **A service emits observability as three signal types** — traces (a tree of +- **A service emits observability as three signal types**: traces (a tree of parent/child spans describing one request's shape and timing), a monotonic counter (`requests_total`, "how many"), and a histogram (`request_duration`, - "how long", one observation per request) — and `lsp-otel` produces all three + "how long", one observation per request), and `lsp-otel` produces all three from a single `handle_request`. -- **Parent/child span nesting is carried by context** — Go and Rust thread an +- **Parent/child span nesting is carried by context**: Go and Rust thread an explicit context value into each child `Start`, C++ installs the parent with a - `Scope` guard — and it is what turns three separate spans into the + `Scope` guard, and it is what turns three separate spans into the `request → {work, respond}` tree Tempo shows. - **Export is batched, and a flush on shutdown is what makes the last data - arrive** — a short (~2s) interval plus a signal-driven flush, not the slow SDK + arrive**: a short (~2s) interval plus a signal-driven flush, not the slow SDK defaults, is why a start-drive-stop demo's telemetry shows up at all. - **Prove the export landed by querying the backend, never the program.** The verify step reads `requests_total` and `request_duration_milliseconds_count` - out of Prometheus and the trace tree out of Tempo, through Grafana's proxy — an + out of Prometheus and the trace tree out of Tempo, through Grafana's proxy: an export that 404'd silently (the wrong-endpoint bug both non-Go ports hit) fails this check even though the program looked healthy. - **The same OTLP contract needs per-language care**: the full `/v1/traces` and `/v1/metrics` URLs, silencing each SDK's default logging to keep the output - contract, and — in C++ — installing a real signal handler to override the + contract, and, in C++, installing a real signal handler to override the `SIG_IGN` a background shell hands a child before `sigwait` can ever see the signal. @@ -432,7 +432,7 @@ make most microbenchmarks quietly measure the wrong thing. --- -

verified — on the Fedora 44 +

verified: on the Fedora 44 reference host (kernel 7.1) this session, with the lsp-lgtm LGTM stack running (OTLP on :4318, Grafana on :3000): python3 scripts/test-all-examples.py --only 38-opentelemetry-to-lgtm @@ -450,5 +450,5 @@ each language, querying through Grafana's datasource proxy returned drive printed drive sent 10/10 ok and exited 0; a drive to a dead address exited 1 with an app: error: prefix. Not exercised: any run with the LGTM stack down (verify.lua SKIPs that -case rather than failing) and any lab-VM run — this is a host-only, local-mode +case rather than failing) and any lab-VM run: this is a host-only, local-mode example.

diff --git a/_docs/39-benchmarking-without-lies.md b/_docs/39-benchmarking-without-lies.md index 96c5b1f..f6b5cc9 100644 --- a/_docs/39-benchmarking-without-lies.md +++ b/_docs/39-benchmarking-without-lies.md @@ -2,15 +2,15 @@ title: "Benchmarking without lies: warmup, coordinated omission, and the percentile the mean hides" order: 39 part: "Performance and Low Latency" -description: "benchlab measures the ch21 chatterd frame codec two ways in the same binary — a real run with warmup discard, per-iteration timing, and a coordinated-omission-corrected p99, and --lie's single unwarmed wall-clock sample — cross-checked with perf stat cycles/instructions and an objdump proof that an unwatched microbenchmark just gets compiled away." +description: "benchlab measures the ch21 chatterd frame codec two ways in the same binary: a real run with warmup discard, per-iteration timing, and a coordinated-omission-corrected p99, and --lie's single unwarmed wall-clock sample, cross-checked with perf stat cycles/instructions and an objdump proof that an unwatched microbenchmark just gets compiled away." duration: "50 minutes" --- Chapter 37 gave you the USE method's 60-second checklist for a system already in trouble, and Chapter 38 wired that telemetry into the LGTM stack you can query back. Both assume the numbers arriving are real. This chapter asks the -harder question: when *you* produce the number — a microbenchmark of your own -function, timed on your own laptop — what makes it trustworthy? The uncomfortable +harder question: when *you* produce the number (a microbenchmark of your own +function, timed on your own laptop), what makes it trustworthy? The uncomfortable answer is that most benchmarks people paste into commit messages and PR descriptions are not measuring what they claim to. `benchlab` is built to show exactly how, by measuring the same thing two ways in the same binary: once @@ -23,10 +23,10 @@ drive it. {% include excalidraw.html file="39-benchmark-pipeline" - alt="Two bands. Top, benchlab real path: a warmup loop of W untimed calls to run_once feeding a timed loop of N iterations each timed with the monotonic clock into a raw samples array with a running checksum, splitting into a sort-and-percentile path producing min/median/p99/max and a parallel co_correct path producing co_p99_ns, co_n, and checksum, both landing in a printed table. Bottom, benchlab --lie: one untimed call straight to a wall-clock delta straight to a printed elapsed_ns, no percentiles because there is nothing to compute one from. A note reads same codec, same fixed workload — the harness, not the code under test, is what changes the story." - caption="Figure 39.1 — the same codec measured two ways: warmup, per-iteration timing, and coordinated-omission correction versus one unwarmed wall-clock sample" %} + alt="Two bands. Top, benchlab real path: a warmup loop of W untimed calls to run_once feeding a timed loop of N iterations each timed with the monotonic clock into a raw samples array with a running checksum, splitting into a sort-and-percentile path producing min/median/p99/max and a parallel co_correct path producing co_p99_ns, co_n, and checksum, both landing in a printed table. Bottom, benchlab --lie: one untimed call straight to a wall-clock delta straight to a printed elapsed_ns, no percentiles because there is nothing to compute one from. A note reads same codec, same fixed workload: the harness, not the code under test, is what changes the story." + caption="Figure 39.1: the same codec measured two ways, warmup, per-iteration timing, and coordinated-omission correction versus one unwarmed wall-clock sample" %} -> **Tools used** — `perf stat`, `cpupower`, `objdump`, `g++` (host, ad hoc +> **Tools used**: `perf stat`, `cpupower`, `objdump`, `g++` (host, ad hoc > compile for the dead-code-elimination demonstration). `perf` and `g++`/`gcc` > are checked by `scripts/check-host.sh`; `cpupower` and `objdump` ship with > Fedora's `kernel-tools` and `binutils`, which this book already requires. @@ -40,7 +40,7 @@ the steady state a real caller experiences. The allocator has not yet carved out the arena it will reuse for the rest of the run; the branch predictor has not yet learned which way `decode_frame`'s length check usually goes; on a JIT'd or GC'd runtime the picture is worse still (Go's escape analysis and GC -pacer are both steady-state phenomena — irrelevant here since the codec is +pacer are both steady-state phenomena; irrelevant here since the codec is allocation-light, but the discipline generalizes). Skip the warmup and your "benchmark" is measuring cold-start cost, which is a real and useful number, just not the one anyone means by "how fast is `encode_frame`." @@ -50,12 +50,12 @@ runs of the same fixed workload on the same host produce ten different numbers, because the scheduler, the cache state, and (on this laptop) the CPU's own clock all move between runs. `benchlab`'s answer is to never trust one number: it collects `N` individually-timed samples and reports the -distribution — min, median, p99, max — not a single average. +distribution (min, median, p99, max), not a single average. ## Fixing the clock: the governor is part of the experiment Before any of those samples mean anything, the thing that's supposed to be -constant across the run — the CPU's clock rate — has to actually be close to +constant across the run, the CPU's clock rate, has to be close to constant. Modern Intel chips run under `intel_pstate`, which picks a frequency per core, per instant, based on load and a policy called the governor. On the reference host for this chapter: @@ -94,7 +94,7 @@ analyzing CPU 1: `powersave` with turbo **Active: yes** means the core running your benchmark loop can legitimately be executing the same instruction stream at 800 MHz in one sample and north of 4.6 GHz two samples later, purely because the -governor decided the load justified a boost — or a neighboring core's thermal +governor decided the load justified a boost, or a neighboring core's thermal budget forced a step down. That is a confound with nothing to do with your code, and it lands directly in `max_ns` and in the tail percentiles this chapter cares about. The fix, on a host you control, is to pin the governor @@ -102,7 +102,7 @@ before you benchmark: `echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor` (or the equivalent `sudo cpupower frequency-set -g performance`) locks every core to its highest non-turbo sustained frequency for the duration of the run. This chapter's own -numbers below were captured *without* making that change — deliberately, so +numbers below were captured *without* making that change, so that the variance the next two sections discuss is the real variance this host produces under its default policy, not variance edited away before you could see it. If you reproduce these numbers and they differ, check your @@ -113,13 +113,13 @@ governor first. `benchlab`'s timed loop is a **closed loop**: iteration `N+1` does not start its clock until iteration `N` has returned. This is the natural way to write a benchmark loop, and it is also, in Gil Tene's phrase, the exact shape that -produces **coordinated omission**. Suppose iteration 4 stalls — a page fault, +produces **coordinated omission**. Suppose iteration 4 stalls (a page fault, a context switch onto a different core, a neighbor process winning the -scheduler's attention — for 1000x the typical iteration cost. The closed loop +scheduler's attention) for 1000x the typical iteration cost. The closed loop records exactly *one* sample for that stall: iteration 4's latency. But a real caller is not a closed loop. A client sending one request every 47 ns (this run's median) does not pause its request rate while iteration 4's stall plays -out — five, ten, a hundred more requests queue up behind it, each waiting its +out: five, ten, a hundred more requests queue up behind it, each waiting its turn, each experiencing a delay close to the stall's own length. The closed loop's benchmark omitted every one of those queued requests from its sample set. It didn't just under-measure the tail; it *erased* the part of the @@ -128,15 +128,15 @@ distribution the tail exists to describe. {% include excalidraw.html file="39-coordinated-omission" alt="Two stacked bands. Top, benchlab's raw closed-loop timer: iterations 1 through 3 each a small box around 40 nanoseconds, iteration 4 an accent box labeled STALL page fault slash sched, 1057 nanoseconds, one sample. Bottom, what a fixed-rate caller would have seen: the same three small requests, the same stalling request 4, then two dashed ghost boxes labeled request 5 queued and request 6 queued at roughly 1018 and 979 nanoseconds, backfilled synthetic samples continuing down toward the expected interval floor. A dashed vertical line connects the two request 4 boxes labeled same stall. Captions read the raw p99 over a 1000-sample window buries the single stall sample, while the corrected co_p99 backfills the queued requests so co_p99_ns is always at least p99_ns." - caption="Figure 39.2 — one stall, recorded once by the closed loop (top) versus what a fixed-rate caller actually experiences (bottom); co_correct backfills the difference" %} + caption="Figure 39.2: one stall, recorded once by the closed loop (top) versus what a fixed-rate caller actually experiences (bottom); co_correct backfills the difference" %} -The correction — HdrHistogram calls it `recordValueWithExpectedInterval` — is +The correction (HdrHistogram calls it `recordValueWithExpectedInterval`) is mechanical once you see the picture: for every raw sample `v` that exceeds an *expected interval* `E` (the per-op time a sustained caller would target), backfill synthetic samples `v - E, v - 2E, ...` down to `E`, standing in for the requests that would have queued behind the stall. `benchlab` sets `E` to the run's own raw median, floored at 1 ns, so the correction needs no extra -flag and comes from the same run it corrects — and caps the backfill at +flag and comes from the same run it corrects, and caps the backfill at 5,000,000 synthetic samples so one pathological stall can't grow the array without bound. On a run captured this session: @@ -149,36 +149,36 @@ benchlab: co_p99_ns=16930 expected_interval_ns=42 co_n=107077 benchlab: checksum=000000000079c3e0 ``` -The raw `p99_ns` is **86 nanoseconds** — a single stall out near `max_ns` +The raw `p99_ns` is **86 nanoseconds**: a single stall out near `max_ns` (56,595 ns, almost 1350x the median) barely registers, because it is exactly one sample out of the 1000-widest slice the p99 looks at. The corrected -`co_p99_ns` is **16,930 nanoseconds** — nearly 200x the raw figure — because +`co_p99_ns` is **16,930 nanoseconds**, nearly 200x the raw figure, because that one 56 microsecond stall backfilled thousands of synthetic queued samples into the tail, and enough of them landed above the 99th-percentile cutoff to move it by two orders of magnitude. Both numbers came from the *same* 100,000-iteration run; the only difference is whether the harness accounts for the requests coordinated omission would otherwise omit. Reporting -the raw number alone — which is what most naive benchmark loops do — is not +the raw number alone (which is what most naive benchmark loops do) is not wrong arithmetic, it is measuring the wrong thing. ## HDR-style percentiles, and why the mean lies None of the numbers above are an arithmetic mean, and that is deliberate. Latency distributions are heavy-tailed: the vast majority of calls finish in -tens of nanoseconds, and a small minority — the ones a scheduler, a page -fault, or (in a GC'd runtime) a collection pause touches — finish orders of +tens of nanoseconds, and a small minority (the ones a scheduler, a page +fault, or, in a GC'd runtime, a collection pause touches) finish orders of magnitude slower. Average those together and the mean is dragged toward the tail by exactly the outliers a reader most needs to see reported accurately, while simultaneously hiding *how bad* the tail actually gets: a mean of 200 ns is consistent with "every call took 200 ns" and with "999 calls took 40 ns and one took 160,200 ns," and those are utterly different systems to operate. A -**percentile** answers a sharper question — "what latency do 99% of calls beat?" -— and doesn't blend the tail into the body. `benchlab`'s `percentile()` +**percentile** answers a sharper question: "what latency do 99% of calls beat?" +It doesn't blend the tail into the body. `benchlab`'s `percentile()` function is the simplest correct version of this idea, nearest-rank over a fully sorted sample array (`sorted[floor(p * (n-1))]`); a production HdrHistogram would instead bucket values logarithmically to report percentiles in constant memory over unbounded sample counts, which is the one simplification -worth naming here — this harness sorts a real in-memory array, which is fine +this harness makes: it sorts a real in-memory array, which is fine at these iteration counts and would need revisiting past tens of millions of samples. @@ -188,7 +188,7 @@ The final trap has nothing to do with clocks: it's that a sufficiently smart compiler will happily prove your benchmark's result is never used and delete the code that produces it, leaving you an accurate measurement of an empty loop. Every `run_once` call in `benchlab` folds its result into a running -`checksum` for exactly this reason — the accumulation is a real, observable +`checksum` for exactly this reason: the accumulation is a real, observable side effect (it's printed at the end), so nothing upstream of it can be proven dead. To see the failure mode this guards against, compile a stripped version of the same idea and read what GCC actually produced: @@ -207,15 +207,15 @@ version of the same idea and read what GCC actually produced: ``` `bench_discarded` runs a checksum function over the workload bytes `iters` -times and never uses the result — GCC 16.1.1 at `-O2` proved the whole loop +times and never uses the result: GCC 16.1.1 at `-O2` proved the whole loop has no observable effect and compiled it to a bare `ret`. Nothing ran. A benchmark harness with this shape reports a real, reproducible, extremely fast time for code that never executed even once. `bench_sunk` is the -identical loop with one change — the result feeds a `volatile` global — and +identical loop with one change (the result feeds a `volatile` global), and the disassembly shows the real loop, vectorized with SSE, doing actual work on every iteration. `benchlab`'s checksum plays exactly the role of that -`volatile` sink, and it earns its keep twice: it stops the optimizer from -proving the loop dead, and — because the workload is fixed — it doubles as a +`volatile` sink, and it does two jobs at once: it stops the optimizer from +proving the loop dead, and, because the workload is fixed, it doubles as a correctness oracle. All three languages produce the identical checksum for the identical fixed workload (`--op encode --iters 20000 --warmup 1000`, this session): @@ -230,7 +230,7 @@ benchlab: checksum=00000000001f6710 ``` A diverging checksum across languages would mean a codec bug, not -measurement noise — the ns timings are expected to vary run to run, the +measurement noise: the ns timings are expected to vary run to run, the checksum is not. Real benchmark frameworks solve the same dead-code problem with a dedicated @@ -240,9 +240,9 @@ API rather than a manual accumulator: Google Benchmark's -bench . -benchmem` and the companion `benchstat` tool for comparing two runs statistically rather than eyeballing them) all exist to answer exactly this question at scale, across many benchmarked functions, with proper -statistical comparison between runs. `benchlab` deliberately uses none of -them: it is a from-scratch harness precisely so the mechanism — warmup, -per-iteration timing, coordinated-omission correction, the dead-code trap — +statistical comparison between runs. `benchlab` uses none of +them by design: it is a from-scratch harness precisely so the mechanism (warmup, +per-iteration timing, coordinated-omission correction, the dead-code trap) is visible in ~150 lines you can read end to end, identically shaped in all three languages, rather than hidden inside a framework's own reporting format. @@ -319,29 +319,29 @@ fn co_correct(raw: &[i64], expected_interval_ns: i64, cap: usize) -> Vec { } ``` -Every sample `v` is pushed to `out` unconditionally first — a normal, +Every sample `v` is pushed to `out` unconditionally first: a normal, non-stalling iteration contributes exactly itself, which is why `co_n >= n` always and equals `n` on a run with no stalls at all. The `if` guard skips -correction entirely for samples at or below the interval — those are the +correction entirely for samples at or below the interval: those are the *fast* calls, nothing queued behind them. Once `v` exceeds the interval, the `while`/`for` loop walks backward from `v - E` in steps of `E`, pushing one synthetic sample per step: this is precisely "how many requests, each spaced `E` apart, would have piled up during a stall of length `v`?" The cap check inside the loop (not just at the top) matters because a single -pathological stall can demand millions of synthetic samples in one pass — +pathological stall can demand millions of synthetic samples in one pass: without the inner check, that one sample could blow the cap on its own before the outer loop ever gets to see it. C++ mutates a `std::vector` under manual `reserve`/`push_back`; Go uses `append` against a capacity-preallocated slice and a `for`-with-condition in place of `while`; Rust needs a labeled `'outer` loop because breaking the outer `for` from inside the inner `while` -has no unlabeled spelling — three idioms, the same three-branch algorithm, +has no unlabeled spelling. Three idioms, the same three-branch algorithm, provably matching because `verify.lua` runs all three against the identical fixed workload and checks the identical invariants (`co_p99_ns >= p99_ns`, `co_n >= n`) rather than the identical source text. `percentile()` is the other half: given a *sorted* array and a fraction `p`, nearest-rank indexing (`sorted[floor(p * (len - 1))]`) picks the sample at -that rank directly — no interpolation, no bucketing, correct as long as the +that rank directly: no interpolation, no bucketing, correct as long as the array really is sorted first, which `main` guarantees by calling `sort` (C++'s `std::sort`, Go's `sort.Slice`, Rust's `sort_unstable`) immediately before computing `median_ns`, `p99_ns`, and again after `co_correct` before @@ -356,14 +356,14 @@ next iteration. ## Errors, three ways `benchlab` has two failure surfaces, and they map to two different exit -codes across all three languages. A **usage error** — no `--op` given for a -real run, an unrecognized `--op` value, or `--warmup >= --iters` (which would -leave zero samples to time) — prints the two-line usage banner to stderr and +codes across all three languages. A **usage error** (no `--op` given for a +real run, an unrecognized `--op` value, or `--warmup >= --iters`, which would +leave zero samples to time) prints the two-line usage banner to stderr and exits **2**; C++ funnels every parse path through `usage_and_exit()`, Go through `usage()` plus `os.Exit(2)`, Rust through a `usage() -> !` that calls `std::process::exit(2)` and therefore never returns, which the type checker -enforces. A **codec bug** — `decode_frame` rejecting a frame from a workload -this harness controls and never corrupts — is exit **1**, and it is the one +enforces. A **codec bug** (`decode_frame` rejecting a frame from a workload +this harness controls and never corrupts) is exit **1**, and it is the one path `verify.lua` never exercises, because it should be unreachable on this fixed input; it exists so a future change to the workload or the codec fails loudly rather than silently producing a bogus checksum. C++ checks the @@ -374,11 +374,11 @@ loudly rather than silently producing a bogus checksum. C++ checks the still `errors.Is`-matches the original; Rust's `decode_frame` returns `Result<(u8, Vec), String>` and `run_once` propagates it with a bare `?`, letting `main`'s `match` turn any `Err` into `ExitCode::from(1)`. A successful -run of either mode — the real path or `--lie` — exits **0**. +run of either mode (the real path or `--lie`) exits **0**. ## Concurrency lens -`benchlab` itself is deliberately single-threaded — running the timed loop on +`benchlab` itself is single-threaded by design: running the timed loop on one thread is what makes "iteration N+1 starts after N returns" a clean, analyzable closed loop in the first place; multiply the callers and you'd need to reason about contention on top of coordinated omission, a strictly @@ -394,7 +394,7 @@ and corrects for. Chapter 26's `taskset -c 0,2` pinning is the mitigation: running `taskset -c 4 ./cpp/build/release/app ...` confines the benchmark to one core for the whole run, trading the ability to migrate away from a noisy neighbor for the elimination of migration-induced stalls as a source -of tail latency — worth doing whenever you want the tail to reflect the code +of tail latency, useful whenever you want the tail to reflect the code under test rather than the scheduler's decisions about it. ## Build, run, observe @@ -415,8 +415,8 @@ benchlab: lie elapsed_ns=1077 sink=76 ``` The real run and the lie run measure the identical codec against the -identical fixed workload — the same `alice\0the quick brown fox jumps over -the lazy dog, three times, for benchlab` `DELIVER` frame — and produce +identical fixed workload, the same `alice\0the quick brown fox jumps over +the lazy dog, three times, for benchlab` `DELIVER` frame, and produce answers that look nothing alike: a five-line distribution with a correction term, versus a single number with no way to tell whether it's typical or a fluke. Neither `p99_ns=` nor `median_ns=` ever appears in `--lie`'s output; @@ -458,12 +458,12 @@ from scheduling noise: ``` 95,724,894 instructions over 41,029,644 cycles is an IPC of **~2.33** for -105,000 total roundtrip calls (5,000 warmup plus 100,000 timed) — a plausible, +105,000 total roundtrip calls (5,000 warmup plus 100,000 timed), a plausible, stable figure for a branch-light encode/decode over a ~78-byte frame, and a number that doesn't move with scheduler noise the way `max_ns` does, because `cycles`/`instructions` count retired work, not wall time. If a change to the codec doubled `cycles:u` for the same `instructions:u`, that would be cache or -branch-misprediction cost growing, not timing variance — a distinction the ns +branch-misprediction cost growing, not timing variance: a distinction the ns percentiles alone can't make. Second, the corrected-versus-naive percentiles from that same invocation are @@ -476,17 +476,17 @@ explicitly: | sample count | 100,000 | 107,077 | Same run, same 100,000 timed iterations, same single 56,595 ns stall visible -in `max_ns` — one number treats that stall as a footnote, the other treats it +in `max_ns`: one number treats that stall as a footnote, the other treats it as the ~197x-worse tail a real fixed-rate caller would have actually felt. Neither number is "wrong" in isolation; reporting only the left column while -implying it describes what users experience is the lie this chapter's title -refers to. +implying it describes what users experience is the mistake this chapter +warns against. ## What you learned - A closed-loop benchmark (iteration N+1 waits for N) records a stall as one sample when a real fixed-rate caller would have queued several requests - behind it — Gil Tene's coordinated omission — and the fix is to backfill + behind it (Gil Tene's coordinated omission), and the fix is to backfill synthetic samples from the raw value down to the expected interval, which is exactly what turned this chapter's `86 ns` raw p99 into a `16,930 ns` corrected one from the identical run. @@ -495,7 +495,7 @@ refers to. simplest correct version, and it's what real HdrHistogram bucketing approximates at scale. - A benchmark whose result nothing uses is a benchmark the compiler is free - to delete — GCC turned an unwatched loop calling a real function `iters` + to delete: GCC turned an unwatched loop calling a real function `iters` times into a bare `ret`, confirmed by reading the actual `objdump` output; folding every call's result into a printed checksum (`benchlab`'s own correctness oracle, identical across all three languages) is what keeps @@ -507,14 +507,14 @@ refers to. between samples, and pinning to `performance` before benchmarking removes that confound from the tail you're trying to measure. -Chapter 40 takes the discipline this chapter earned — real percentiles, a -pinned clock, a tail you trust — and spends it: stripping a hot path down to +Chapter 40 takes the discipline this chapter earned (real percentiles, a +pinned clock, a tail you trust) and spends it: stripping a hot path down to pinned cores, zero allocation, and the low-latency techniques that make the corrected p99 small in the first place. --- -

verified — on the Fedora 44 +

verified: on the Fedora 44 reference host (kernel 7.1, Intel 8-core/16-thread laptop CPU, 16 CPUs, intel_pstate/powersave governor, turbo active) this session: python3 scripts/test-all-examples.py --only @@ -524,7 +524,7 @@ cpp/go/rust). The cited real run printed min_ns=39 median_ns=47 p99_ns=73 max_ns=5480 co_p99_ns=75; the perf stat run printed 41,029,644 cycles:u, 95,724,894 instructions:u, and (that specific run) -p99_ns=86 co_p99_ns=16930 max_ns=56595 — the side-by-side +p99_ns=86 co_p99_ns=16930 max_ns=56595: the side-by-side percentile table above is that exact run's output, not a constructed example. --lie printed elapsed_ns=1077 sink=76 with no p99_ns=/median_ns= anywhere in its output. The @@ -535,7 +535,7 @@ loop) came from a throwaway file compiled ad hoc with g++ -std=c++23 checksum 00000000001f6710 for --op encode --iters 20000 --warmup 1000 was confirmed identical across all three built binaries this session. Not exercised: pinning the governor to performance -(this host's numbers are all under the default powersave policy, -deliberately, as the CPU-frequency section explains) and any run against a -lab VM — this is a host-only, local-mode example with no VM/LGTM +(this host's numbers are all under the default powersave policy +by design, as the CPU-frequency section explains) and any run against a +lab VM: this is a host-only, local-mode example with no VM/LGTM requirement.

diff --git a/_docs/40-low-latency-fastpath.md b/_docs/40-low-latency-fastpath.md index 7576abe..e3a4156 100644 --- a/_docs/40-low-latency-fastpath.md +++ b/_docs/40-low-latency-fastpath.md @@ -2,19 +2,19 @@ title: "The low-latency fast path: pin a core, reuse one buffer, and stop asking the scheduler for permission" order: 40 part: "Performance and Low Latency" -description: "chatterd-fastpath strips the ch21 echo hot path down to a pinned core, one reused 64-byte buffer, and an optional busy-poll spin, then proves each move at the kernel level — Cpus_allowed_list and voluntary_ctxt_switches read from /proc//status, not the program's own claim. The three languages reach identical behavior by three different shutdown mechanisms, and Go's CPU count lies about its container through a taskset mask rather than a cgroup." +description: "chatterd-fastpath strips the ch21 echo hot path down to a pinned core, one reused 64-byte buffer, and an optional busy-poll spin, then proves each move at the kernel level: Cpus_allowed_list and voluntary_ctxt_switches read from /proc//status, not the program's own claim. The three languages reach identical behavior by three different shutdown mechanisms, and Go's CPU count lies about its container through a taskset mask rather than a cgroup." duration: "50 minutes" --- Chapter 39 taught you to trust a latency number: warm the code, time each iteration, correct for coordinated omission, and report the tail instead of the mean. This chapter spends that discipline. If you now believe a `p99` you -measured, the next question is how to make it *small* — and the answer, on Linux, +measured, the next question is how to make it *small*, and the answer, on Linux, is a short list of moves that each remove one source of the multi-microsecond stalls Chapter 39's `co_p99` was busy accounting for. `chatterd-fastpath` is the same echo hot path from Chapter 21's `chatterd`, stripped to exactly those moves: pin the process to one CPU, reuse a single stack buffer for every message so the -loop never allocates, and — optionally — spin on a non-blocking `recv` instead of +loop never allocates, and, optionally, spin on a non-blocking `recv` instead of blocking, trading a whole core for the elimination of the sleep/wake round trip through the scheduler. A `naive` server in the same binary does the opposite of each, so you can race them under identical placement and watch the gap. @@ -26,19 +26,19 @@ there. {% include excalidraw.html file="40-fastpath-datapath" - alt="Two bands showing one echo round trip each. Top, the naive server: a client frame arrives at a server that may run on any of the online CPUs; per message it calls make_unique to allocate a fresh 64-byte heap buffer, then a blocking read(2) that puts the thread to sleep — an arrow labeled voluntary context switch loops out through a box labeled kernel scheduler and back, waking the thread when data arrives — then write(2) echoes and the heap buffer is freed. A counter reads voluntary_ctxt_switches roughly one per message. Bottom, the fastpath server: pinned by sched_setaffinity to a single CPU shown boxed and shaded, one reused 64-byte stack buffer for the whole connection, and a busy-poll loop that spins on non-blocking recv(2) over EAGAIN, never entering the scheduler — no context-switch arrow, the thread stays on its core. A counter reads voluntary_ctxt_switches roughly zero. A note reads same 64-byte frame, same round trip — the difference is who the thread has to ask for its CPU back." - caption="Figure 40.1 — one echo round trip, naive versus fastpath: a per-message heap buffer and a blocking read that sleeps through the scheduler, against one reused buffer on a pinned core that never gives the CPU up" %} + alt="Two bands showing one echo round trip each. Top, the naive server: a client frame arrives at a server that may run on any of the online CPUs; per message it calls make_unique to allocate a fresh 64-byte heap buffer, then a blocking read(2) that puts the thread to sleep (an arrow labeled voluntary context switch loops out through a box labeled kernel scheduler and back, waking the thread when data arrives) then write(2) echoes and the heap buffer is freed. A counter reads voluntary_ctxt_switches roughly one per message. Bottom, the fastpath server: pinned by sched_setaffinity to a single CPU shown boxed and shaded, one reused 64-byte stack buffer for the whole connection, and a busy-poll loop that spins on non-blocking recv(2) over EAGAIN, never entering the scheduler, no context-switch arrow, the thread stays on its core. A counter reads voluntary_ctxt_switches roughly zero. A note reads same 64-byte frame, same round trip: the difference is who the thread has to ask for its CPU back." + caption="Figure 40.1: one echo round trip, naive versus fastpath, a per-message heap buffer and a blocking read that sleeps through the scheduler, against one reused buffer on a pinned core that never gives the CPU up" %} -> **Tools used** — `taskset` (systems-target VM), `ssh`/`scp` (host, the lab's +> **Tools used**: `taskset` (systems-target VM), `ssh`/`scp` (host, the lab's > own transport). The pinning and busy-poll proofs read `/proc//status` on -> the guest directly — procfs, no extra tool. `taskset` ships in +> the guest directly: procfs, no extra tool. `taskset` ships in > `util-linux-core` on the Fedora 44 guest; the lab transport is the same > `scripts/lab/` tooling every VM example uses. ## Three moves that shrink the tail Every technique in `fastpath` targets a specific line item in a latency budget, -and each is worthless — or actively harmful — applied blindly. Naming them up +and each is worthless, or actively harmful, applied blindly. Naming them up front: - **Pin to one CPU** (`sched_setaffinity(2)`). A thread the scheduler is free to @@ -51,19 +51,19 @@ front: - **Reuse one buffer** (zero heap allocation in the loop). `fastpath` reads into a single 64-byte stack array for the entire life of a connection; `naive` `make_unique`s a fresh heap buffer for every message. At 64 bytes the allocator - cost is small, but it is *variable* — most calls hit the thread-local cache and + cost is small, but it is *variable*: most calls hit the thread-local cache and return in nanoseconds, and the occasional one that has to grab the arena lock or fault in a fresh page does not, and that variance lands in the tail. - **Busy-poll** (spin on non-blocking `recv`). A blocking `read(2)` on an idle socket puts the thread to sleep; when the next frame arrives, the kernel has to - wake it, which is a scheduler round trip — a **voluntary context switch** — on + wake it, which is a scheduler round trip, a **voluntary context switch**, on every single message. Busy-poll never sleeps: it spins on a non-blocking `recv` that returns `EAGAIN` until data is there, burning 100% of the pinned core to keep the thread on-CPU and ready. This is the most expensive move (a whole core, gone) and the one with the largest single effect on median latency, as the numbers below show. -The rest of the program is deliberately ordinary: a fixed 64-byte wire frame +The rest of the program is ordinary by design: a fixed 64-byte wire frame (magic `CF`, a version and type byte, a big-endian `seq`, and zero padding), a `measure` subcommand that runs `N` synchronous ping-pong round trips against either server and prints the same percentile table Chapter 39 built, and a @@ -72,8 +72,8 @@ placement so the *only* variable is the in-server discipline. ## Pinning, and the count that lies about its container -`pin_to_cpu` is four lines of real work — validate the CPU index, build a -one-CPU affinity mask, install it — wrapped around one decision that turns out to +`pin_to_cpu` is four lines of real work (validate the CPU index, build a +one-CPU affinity mask, install it) wrapped around one decision that turns out to diverge across the three languages more sharply than anything else in the program: @@ -154,7 +154,7 @@ Rust both ask `sysconf(_SC_NPROCESSORS_ONLN)`, which returns exactly that (2 on the lab guest) and is blind to the caller's affinity mask. The obvious Go equivalent, `runtime.NumCPU()`, is *not* the same thing: it returns the size of the process's **CPU affinity mask**. Those two numbers agree on an idle machine -and diverge the instant something narrows the mask — which is precisely what the +and diverge the instant something narrows the mask, which is precisely what the benchmark driver does. Phase B of `run-latency-bench.sh` launches each server under `taskset -c 1`, so the fastpath process starts life with a one-CPU mask, `runtime.NumCPU()` returns **1**, and the Go server's own range check rejects the @@ -165,13 +165,13 @@ app: error: cpu 1 out of range (0..0) ``` This is the same failure mode as `hardware_concurrency()` reporting a container's -cgroup quota instead of the host's core count — the running theme of this part of -the book — reached through an unexpected door. Here the count lies not because a +cgroup quota instead of the host's core count (the running theme of this part of +the book) reached through an unexpected door. Here the count lies not because a cgroup capped it but because a `taskset` affinity mask narrowed it, and the fix is the same in spirit: ask the kernel the question you actually mean. `onlineCPUs()` in the Go port reads `/sys/devices/system/cpu/online` and counts the ranges there, matching the `sysconf` semantics the other two languages get -for free: +automatically: ```go func onlineCPUs() int { @@ -185,12 +185,12 @@ func onlineCPUs() int { ``` The Go port also carries two lines the others don't need. `sched_setaffinity(2)` -acts on a **thread**, and a goroutine is not a thread — the runtime multiplexes +acts on a **thread**, and a goroutine is not a thread: the runtime multiplexes goroutines over OS threads (Ms) and will migrate the one running your loop at will. `runtime.LockOSThread()` nails the hot loop's goroutine to one M and `GOMAXPROCS(1)` stops the runtime from parking worker threads on other cores; only then does pinning that thread mean anything. And the affinity call is made -*twice* — once with `0` (the calling thread, which is what actually restricts the +*twice*: once with `0` (the calling thread, which is what actually restricts the loop) and once with `os.Getpid()` (the thread-group leader, which is the mask `/proc//status` reports and the benchmark reads back). C++ and Rust are single-threaded here, so one `sched_setaffinity(0, …)` both pins the loop and @@ -274,7 +274,7 @@ fn read_full_busy(fd: RawFd, buf: &mut [u8]) -> bool { ``` There is nothing in the `EAGAIN` branch but a stop-flag check and `continue`. No -`sched_yield`, no `nanosleep`, no `poll` — those would all, in different ways, +`sched_yield`, no `nanosleep`, no `poll`: those would all, in different ways, hand the CPU back to the kernel, which is the exact cost busy-poll exists to avoid. The thread stays runnable and on its core, re-issuing `recv` as fast as the CPU will let it, so the instant a frame lands the very next `recv` returns it with @@ -282,7 +282,7 @@ no wakeup latency at all. The blocking path (`read_full_blocking`, not shown) is the ordinary alternative: a plain blocking `read` that the kernel parks on until data arrives. -The difference is not subtle, and it does not require a profiler to see — it is +The difference is not subtle, and it does not require a profiler to see: it is counted for you in `/proc//status`. Every time a blocking `read` sleeps and is woken, the kernel records a **voluntary context switch** against the process. Running a 20,000-message `measure` against each server, both `taskset`-pinned to @@ -297,7 +297,7 @@ before: voluntary_ctxt_switches: 2 after: voluntary_ctxt_switches: 3 ``` -The blocking server took **one voluntary context switch per message** — it slept +The blocking server took **one voluntary context switch per message**: it slept and was woken 20,498 times to serve 20,000 requests plus warmup. The busy-poll server slept **once** for the whole run. That single number is the mechanism the median latency reflects: on this run the naive `p50` was 11,426 ns and the @@ -311,7 +311,7 @@ Both servers share one accept skeleton: a `poll(2)` on the listener with a 200 m timeout so the stop flag is checked a few times a second even when idle, then `accept4`, `TCP_NODELAY`, and a per-connection echo loop that serves one client to completion before returning to `accept`. The whole program runs on **one thread** -by design — the entire point is that a latency-critical hot loop should not be +by design: the entire point is that a latency-critical hot loop should not be handing work between threads, so there is no thread pool and no per-connection goroutine or task. The fastpath loop reads into the one 64-byte buffer and writes it straight back; the naive loop `make_unique`s (C++), `make([]byte, …)`s (Go), or @@ -321,7 +321,7 @@ scope, the deliberate pessimization the benchmark is built to expose. `measure` is the client half, and its percentile math is exactly Chapter 39's: sort the samples, then nearest-rank index with `idx = ceil(p/100 · n)` clamped to `[1, n]`. It builds each frame with `seq = i`, times the round trip between two -`CLOCK_MONOTONIC` reads, and — a correctness check, not a timing one — verifies +`CLOCK_MONOTONIC` reads, and, a correctness check, not a timing one, verifies the echo came back with the same `seq` before recording the sample, so a cross-wired or corrupted connection fails loudly rather than reporting fast garbage. Warmup round trips (500 by default in the benchmark) are discarded before @@ -330,8 +330,8 @@ recording starts, for the reasons Chapter 39 spent a section on. ## Errors, three ways `fastpath` has two exit codes for two kinds of failure, and they mean different -things. A **usage error** — an unknown subcommand, a missing `--pin`, a `measure` -with no `--n`, a port outside 1–65535 — prints the three-line usage banner to +things. A **usage error** (an unknown subcommand, a missing `--pin`, a `measure` +with no `--n`, a port outside 1–65535) prints the three-line usage banner to stderr and exits **2**. Argument parsing is strict full-string unsigned, so `--n 5x` is a usage error, not a silent truncation to 5. @@ -341,18 +341,18 @@ naming what went wrong: `cpu 99 out of range (0..1)` from `pin_to_cpu`, `notanip: not an IPv4 address` when the target host isn't a dotted quad (the servers use `inet_pton`, not a DNS lookup, so a hostname is an error by design), and `measure: connection closed at iteration N` if a server dies mid-run. The -three languages carry these three ways — C++ returns `std::expected` and prints the `error()` string; Go returns `error` values with the `strerror` text unwrapped so the message reads like C's rather than Go's layered `dial tcp4 …: connect: …`; Rust returns `Result` with the message -pre-formatted at the failure site — but the bytes on stderr are identical across +pre-formatted at the failure site, but the bytes on stderr are identical across all three on the same guest, because `verify.lua` asserts exactly that. ## Concurrency lens -The interesting concurrency in this chapter is not in the hot loop — that is -single-threaded on purpose — it is in **shutdown**, and it is the one place where -identical observable behavior demanded three genuinely different mechanisms. +The interesting concurrency in this chapter is not in the hot loop; that is +single-threaded by design. It is in **shutdown**, and it is the one place where +identical observable behavior demanded three distinctly different mechanisms. {% include codetabs.html langs="C++|Go|Rust" %} @@ -408,7 +408,7 @@ C++ and Rust are showing the *same* function and the same idea: install a signal handler that does nothing but set an atomic stop flag, with `sa_flags` explicitly **not** `SA_RESTART`. That flag is the whole trick. Without `SA_RESTART`, a blocking `read(2)` or `poll(2)` already in progress when `SIGINT` arrives does not -resume — it returns `EINTR`, the loop unwinds, sees the stop flag, and exits. The +resume: it returns `EINTR`, the loop unwinds, sees the stop flag, and exits. The signal handler and the blocked syscall cooperate through the kernel's default `EINTR` behavior, and no second thread is involved. Rust can copy the C++ design line for line because Rust's standard library makes real blocking syscalls; the @@ -418,19 +418,19 @@ descriptors via `nix` where every `EINTR` is handled explicitly. Go cannot use this technique at all, which is why its tab shows a completely different function. The Go runtime installs its own signal handlers with -`SA_RESTART`, and — more fundamentally — a "blocking" read on a `net.Conn` is not +`SA_RESTART`, and, more fundamentally, a "blocking" read on a `net.Conn` is not sitting in a blocking syscall to begin with. The runtime's network poller parks the goroutine on an `epoll` registration and lets the OS thread go do other work; no signal will make that parked read return `EINTR`, because there is no syscall -in flight to interrupt. So Go reaches identical behavior — `SIGINT`, then a clean -`app: … shutting down` and exit 0 — by a different route entirely: the +in flight to interrupt. So Go reaches identical behavior (`SIGINT`, then a clean +`app: … shutting down` and exit 0) by a different route entirely: the `signal.Notify` goroutine sets the stop flag and then **closes the listener and the live connection**. Closing an fd is what wakes a poller-parked read; it is the `shutdown(2)`-driven wakeup Chapter 21's `chatterd` used, and it is the idiomatic Go answer to "interrupt a blocked I/O wait." The busy-poll path needs one extra -touch — its socket is a dup'd raw fd outside the poller — so `trigger` also calls -`shutdown(SHUT_RDWR)` on it to break the spin. Three mechanisms — non-`SA_RESTART` -`EINTR`, non-`SA_RESTART` `EINTR` again, and close-the-fd — for one observable +touch: its socket is a dup'd raw fd outside the poller, so `trigger` also calls +`shutdown(SHUT_RDWR)` on it to break the spin. Three mechanisms (non-`SA_RESTART` +`EINTR`, non-`SA_RESTART` `EINTR` again, and close-the-fd) for one observable outcome, which `verify.lua` pins down with a bounded 10-second wait so a server that *fails* to shut down surfaces as a clear failure rather than a hung run. @@ -442,7 +442,7 @@ that *fails* to shut down surfaces as a clear failure rather than a hung run. The `run-latency-bench.sh` driver does the real work. Phase A brings up a naive and a fastpath server with no external pinning and reads each one's -`Cpus_allowed_list` from `/proc//status` — this is the pinning proof, taken +`Cpus_allowed_list` from `/proc//status`: this is the pinning proof, taken from the kernel rather than from anything the program prints about itself. Phase B then races the two under identical `taskset` placement (both servers pinned to the same CPU, the client to the other), running three interleaved trials of each so a @@ -460,10 +460,10 @@ app: percentiles_ns tag=naive-3 p50=11320 ... n=20000 app: percentiles_ns tag=fastpath-3 p50=8184 ... n=20000 ``` -`naive-cpus-allowed` is `0-1`, the full online set — the naive server never +`naive-cpus-allowed` is `0-1`, the full online set; the naive server never touches its affinity mask. `fastpath-cpus-allowed` is `1`, a single CPU, because `sched_setaffinity` narrowed it. Median of the three naive `p50`s is 11,320 ns; -median of the three fastpath `p50`s is 8,184 ns — the fast path is about 1.38× +median of the three fastpath `p50`s is 8,184 ns: the fast path is about 1.38× faster on this run, and the margin held across every language and every capture this session (1.38×–1.62×), never inverted, never marginal. The runner drives all three languages against the same `verify.lua`: @@ -483,7 +483,7 @@ example cpp go rust ## Cross-check: two kernel-level signals, neither the program's own word The chapter's two central claims are checked by two independent numbers, both read -from `/proc//status` on the guest — not from anything the server prints about +from `/proc//status` on the guest, not from anything the server prints about itself, which is the discipline that makes them proofs rather than assertions: | claim | procfs field | naive | fastpath | @@ -497,7 +497,7 @@ like the naive server, and the benchmark would be measuring two unpinned process fighting over both cores. The second row proves the busy-poll move: a server that claimed to busy-poll but was secretly blocking would show ~20,000 voluntary context switches like the naive one. Both numbers come from the kernel's own -accounting, and both are exactly what the mechanism predicts — which is why +accounting, and both are exactly what the mechanism predicts, which is why `verify.lua` asserts the `Cpus_allowed_list` shape directly and why the context-switch contrast, though not gated, is the single clearest picture of what busy-poll buys and what it costs. @@ -505,8 +505,8 @@ busy-poll buys and what it costs. ## What you learned - **Pinning, zero allocation, and busy-poll each remove one specific source of - tail latency** — migration jitter, allocator variance, and the scheduler - sleep/wake round trip respectively — and each has a real cost (a core you can't + tail latency** (migration jitter, allocator variance, and the scheduler + sleep/wake round trip respectively), and each has a real cost (a core you can't reclaim, most of all for busy-poll), so they are tools for a dedicated latency-critical path, not defaults. - **Busy-poll's whole effect is visible as a context-switch count**: a blocking @@ -519,23 +519,23 @@ busy-poll buys and what it costs. line is a claim, and the two are allowed to disagree when something is broken. - **A CPU count lies about its container through more than one door.** Go's `runtime.NumCPU()` reports the affinity-mask size, so under `taskset -c 1` it - returns 1 where `sysconf(_SC_NPROCESSORS_ONLN)` returns the host's 2 — the same + returns 1 where `sysconf(_SC_NPROCESSORS_ONLN)` returns the host's 2, the same class of bug as `hardware_concurrency()` reading a cgroup quota, reached via an affinity mask instead. For an absolute-CPU-index range check, the system-wide online count is the one you mean. - **Identical observable behavior can require different mechanisms per runtime.** C++ and Rust unblock a shutdown with a non-`SA_RESTART` handler and `EINTR`; Go's network poller means no syscall is in flight to interrupt, so it closes the fd - instead — same `SIGINT`, same exit 0, three different routes there. + instead: same `SIGINT`, same exit 0, three different routes there. -Chapter 41 takes everything from this part and the last — the USE checklist, the +Chapter 41 takes everything from this part and the last (the USE checklist, the LGTM pipeline, benchmarks you can trust, and this chapter's pinned, allocation-free -hot path — and assembles it into the capstone: a supervised, sandboxed, fully +hot path) and assembles it into the final project: a supervised, sandboxed, fully observed two-VM fleet. --- -

verified — on the +

verified: on the systems-target lab guest (Fedora 44, kernel 6.19.10-300.fc44, 2 vCPU nested KVM) this session: python3 scripts/test-all-examples.py --only 40-low-latency-fastpath --mode @@ -555,5 +555,5 @@ and 1.51× (rust), all clearing the verify.lua runtime.NumCPU()-under-taskset range-check failure (cpu 1 out of range (0..0)) was reproduced directly before the onlineCPUs() fix and is gone after it. Not exercised: any run -outside the lab guest — this is a VM example whose numbers are only meaningful on +outside the lab guest: this is a VM example whose numbers are only meaningful on a quiet, pinned machine.

diff --git a/_docs/41-capstone-fleet.md b/_docs/41-capstone-fleet.md index 885ba49..9528175 100644 --- a/_docs/41-capstone-fleet.md +++ b/_docs/41-capstone-fleet.md @@ -1,8 +1,8 @@ --- -title: "The capstone fleet: pmon supervises chatterd, sysagent, and fwatch across two capability-dropped, Landlock-sandboxed, OTLP-observed hosts" +title: "The final project fleet: pmon supervises chatterd, sysagent, and fwatch across two capability-dropped, Landlock-sandboxed, OTLP-observed hosts" order: 41 part: "Performance and Low Latency" -description: "41-capstone-fleet is one binary with four subcommands that assembles four of the book's six recurring programs into a single supervised system: pmon drops the capability bounding set and sets no_new_privs, then self-re-execs chatterd, sysagent, and fwatch as restartable children, forwarding SIGTERM/SIGINT and printing a pmon: health line every tick; chatterd (ch21-27) is bridged across two hosts so a message sent on one node is delivered on the other tagged from=NICK@node; sysagent (ch36) exports cpu%/mem%/load1 as OTel gauges over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; and fwatch (ch07-09, 33) watches its sandbox directory under an optional Landlock ruleset whose enforced ABI — 7 on the lab guest's 6.19.10 kernel, 9 on the Fedora 44 host — is itself a live cross-check of the kernel-version probing discipline Chapter 33 established. Verified on the systems-target/systems-peer two-VM lab, with OTLP reaching the host's Grafana LGTM stack over the libvirt gateway." +description: "41-capstone-fleet is one binary with four subcommands that assembles four of the book's six recurring programs into a single supervised system: pmon drops the capability bounding set and sets no_new_privs, then self-re-execs chatterd, sysagent, and fwatch as restartable children, forwarding SIGTERM/SIGINT and printing a pmon: health line every tick; chatterd (ch21-27) is bridged across two hosts so a message sent on one node is delivered on the other tagged from=NICK@node; sysagent (ch36) exports cpu%/mem%/load1 as OTel gauges over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; and fwatch (ch07-09, 33) watches its sandbox directory under an optional Landlock ruleset whose enforced ABI (7 on the lab guest's 6.19.10 kernel, 9 on the Fedora 44 host) is itself a live cross-check of the kernel-version probing discipline Chapter 33 established. Verified on the systems-target/systems-peer two-VM lab, with OTLP reaching the host's Grafana LGTM stack over the libvirt gateway." duration: "60 minutes" --- @@ -16,33 +16,33 @@ path (21–27, 34, 38, 40); `sysagent` learned to read `/proc` and `fwatch` learned to watch a directory with polling, then inotify, then a Landlock sandbox around itself (07–10, 33). This chapter does not teach a new syscall. It assembles: one binary, four subcommands, where `pmon` is the -fleet's own init — it drops the capability bounding set for itself and every +fleet's own init: it drops the capability bounding set for itself and every descendant, then self-re-execs `chatterd`, `sysagent`, and `fwatch` as children it restarts on unexpected exit, forwards its own signals to, and reports on with a `pmon: health` line every tick. -Two copies of that fleet run in this chapter — one on loopback, for -`verify.lua`, and one for real, on the two-VM lab Chapter 2 built. +Two copies of that fleet run in this chapter: one on loopback, for +`verify.lua`, and one on the actual two-VM lab Chapter 2 built. `chatterd` is bridged between `systems-target` and `systems-peer` so a message sent on one host is delivered on the other; each guest's `sysagent` exports gauges over OTLP, through the libvirt gateway, to the LGTM stack Chapter 3 stood up on the host; and `fwatch` runs sandboxed under Landlock inside each fleet, its enforced ABI itself differing between the host kernel -and the guest kernel — a real, verified number, not a claim. Nearly every +and the guest kernel: a real, verified number, not a claim. Nearly every part of this book from Setting Up through Performance and Low Latency shows up somewhere in the run below. {% include excalidraw.html file="41-fleet-topology" alt="Two side-by-side VM boxes labeled systems-target (192.168.124.7) and systems-peer (192.168.124.95), each drawn as a large rounded rectangle representing one fleet. Inside each box, a top band labeled 'pmon: capability bounding set dropped, no_new_privs=1' encloses three child boxes side by side: chatterd, sysagent, and fwatch, each connected upward to a small pmon box with a supervise/restart arrow labeled 'fork+exec, restart on unexpected exit, forward SIGTERM'. The fwatch box additionally sits inside its own smaller nested box labeled 'Landlock ruleset: reads restricted to sandbox dir' with its enforced ABI number shown as a variable (7 on this guest's 6.19.10 kernel). A bidirectional arrow runs directly between the two chatterd boxes, labeled 'bridge: bridge@target / bridge@peer, DELIVER re-broadcast with from=NICK@node'. Below both VM boxes, a horizontal band represents the libvirt virbr0 bridge and its gateway address 192.168.124.1; two upward arrows from each guest's sysagent box pass through the gateway band labeled 'OTLP/HTTP :4318' and converge on a host-side box labeled 'LGTM stack (Grafana + Prometheus + Tempo), Chapter 3' sitting outside and above both VM boxes. A small caption near the gateway notes 'same route deploy-to-vm.sh forwards OTEL_EXPORTER_OTLP_ENDPOINT over.'" - caption="Figure 41.1 — the two-VM fleet: pmon supervises chatterd/sysagent/fwatch on each of systems-target and systems-peer inside a capability-bounding-set-drop boundary, fwatch additionally sandboxed under Landlock, chatterd bridged directly between the two guests, and both guests' sysagent exporting OTLP over the libvirt gateway to the host's LGTM stack" %} + caption="Figure 41.1: the two-VM fleet: pmon supervises chatterd/sysagent/fwatch on each of systems-target and systems-peer inside a capability-bounding-set-drop boundary, fwatch additionally sandboxed under Landlock, chatterd bridged directly between the two guests, and both guests' sysagent exporting OTLP over the libvirt gateway to the host's LGTM stack" %} -> **Tools used** — `ssh`/`scp` (host, to stage the built binary onto both -> `systems-target` and `systems-peer` and drive `pmon` on each — the same +> **Tools used**: `ssh`/`scp` (host, to stage the built binary onto both +> `systems-target` and `systems-peer` and drive `pmon` on each, the same > `scripts/lab/deploy-to-vm.sh` transport every VM example uses), `curl` > (host, querying Grafana's Prometheus proxy for the `sysagent_cpu_pct` > gauge). The libvirt bridge's gateway address (`192.168.124.1`) is the -> route guest-side OTLP traffic takes to reach the host's LGTM stack — a +> route guest-side OTLP traffic takes to reach the host's LGTM stack: a > network path, not a tool. Capability-bounding-set drop and Landlock are > syscalls the program makes itself (`prctl(2)`, `landlock_create_ruleset(2)` > and friends); seccomp was already proven alongside Landlock in Chapter 33 @@ -66,30 +66,30 @@ fwatch snapshot DIR fwatch watch DIR [--sandbox] [--timeout-ms T] ``` -`pmon` never execs a *different* binary for chatterd, sysagent, or fwatch — +`pmon` never execs a *different* binary for chatterd, sysagent, or fwatch: it re-execs **itself**, with a different `argv[0]` command word, the same self-re-exec technique Chapter 34's container entrypoint used. Concretely: `mkdir` the sandbox directory, drop the capability bounding set once, resolve the running binary's own path (`os.Executable()` in Go, `readlink("/proc/self/exe")` in C++ and Rust), then fork+exec that same path -three times with three different argument lists — `chatterd serve …`, +three times with three different argument lists: `chatterd serve …`, `sysagent --node … --interval-ms 2000`, `fwatch watch DIR --sandbox`. {% include excalidraw.html file="41-supervise-lifecycle" alt="A state-machine diagram for one supervised child. A box labeled starting has an arrow to a box labeled up, annotated 'fork+exec succeeded, pid recorded, pmon: started service=NAME pid=PID printed'. From up, a dashed arrow labeled 'blocking wait (waitpid/cmd.Wait/nix::waitpid)' loops back into a decision diamond: 'shutting_down flag set?'. The no-branch leads to a box labeled restarting, annotated 'pmon: restart service=NAME attempt=N reason=... printed, restarts counter incremented, 300ms backoff sleep', with an arrow looping back up to starting. The yes-branch leads to a box labeled down. Off to the side, a separate small loop shows a clock icon labeled 'health tick every --health-interval-ms' with an arrow into a box printing 'pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0'. A third flow at the bottom shows SIGTERM/SIGINT arriving at a box labeled 'signal thread (sigwait / signal.Notify channel / SigSet::wait)', with an arrow to 'shutting_down.store(true)', then a fan-out arrow to 'kill(child_pid, SIGTERM) for every recorded pid', ending in a box labeled 'pmon: shutdown, exit 0' once every per-child supervisor has returned." - caption="Figure 41.2 — pmon's per-child supervise/restart lifecycle (starting → up → wait → restarting → up again on an unexpected exit), the health tick alongside it, and the SIGTERM → forward-to-children → shutdown path" %} + caption="Figure 41.2: pmon's per-child supervise/restart lifecycle (starting → up → wait → restarting → up again on an unexpected exit), the health tick alongside it, and the SIGTERM → forward-to-children → shutdown path" %} Before any of that forking happens, `pmon` drops every capability it might hold from its own bounding set and sets `no_new_privs`. `PR_CAPBSET_DROP` -removes a capability from the *bounding set*, not the current effective set — +removes a capability from the *bounding set*, not the current effective set: the guarantee it gives is that no future `execve` of a setuid-root binary, by this process or any descendant, can ever regain that capability, because the bounding set only shrinks across `fork`/`exec`. That guarantee holds even when the calling process is already unprivileged, which is exactly `pmon`'s situation on an unprivileged lab-guest account: a loopback run prints `pmon: capabilities bounding_set_dropped=0 no_new_privs=1`, and `0` is -the *correct* number here — there was nothing in the bounding set to drop, +the *correct* number here: there was nothing in the bounding set to drop, and the line still proves the `prctl` calls ran and `no_new_privs` is set. {% include codetabs.html langs="C++|Go|Rust" %} @@ -152,14 +152,14 @@ pub fn drop_bounding_set() { All three loop `prctl(PR_CAPBSET_DROP, c, …)` from `0` through `CAP_CHECKPOINT_RESTORE` (40, current as of a 6.x kernel) and ignore `EINVAL` -from probing past whatever the running kernel's actual last capability is — +from probing past whatever the running kernel's actual last capability is, the loop is future-proof by construction rather than by a version check. Once the bounding set is shrunk, every child `pmon` forks inherits the narrower set automatically; nothing in `chatterd`, `sysagent`, or `fwatch` has to know this happened. With the bounding set dropped, `pmon` spawns and supervises the three -children — one goroutine per service in Go, one `std::thread` per service in +children: one goroutine per service in Go, one `std::thread` per service in C++, one `std::thread` per service in Rust, each running the same restart-on-unexpected-exit loop Figure 41.2 diagrams: @@ -315,21 +315,21 @@ fn run_child_supervisor( ``` Go delegates the fork+exec mechanics to `os/exec.Command`, which is why its -version is shorter — `cmd.Start()`/`cmd.Wait()` hide the raw syscalls C++ and +version is shorter: `cmd.Start()`/`cmd.Wait()` hide the raw syscalls C++ and Rust make directly (`fork()`, then `execv()` in the child, then `waitpid()` in the parent). But the *shape* is identical across all three: start, record the pid, print `pmon: started`, block until the child exits, check whether this exit was requested (`shutting_down`) or not, and if not, count a restart, print `pmon: restart … reason=…`, back off 300 ms, and loop. A child that fails to even start (`cmd.Start()` erroring, `fork()` returning -`< 0`) is not fatal to `pmon` — it logs, waits 500 ms, and retries, because a +`< 0`) is not fatal to `pmon`; it logs, waits 500 ms, and retries, because a supervisor's job is to keep trying, not to give up on the first bad attempt. ## chatterd, bridged across two hosts -`chatterd` (Chapters 21–27) speaks the same wire frame it always has — +`chatterd` (Chapters 21–27) speaks the same wire frame it always has: magic `"CH"`, version 1, a type byte, a big-endian `u16` length, then payload -(`proto.go`/`.hpp`/`.rs`, unchanged) — and this chapter adds exactly one new +(`proto.go`/`.hpp`/`.rs`, unchanged), and this chapter adds exactly one new idea on top: with `--peer HOST:PORT` set on both sides, a message sent on one node is delivered to a listener on the other. The mechanism needs no protocol change at all. A **bridge worker** is just an ordinary client of @@ -340,14 +340,14 @@ target dials peer, joins as "bridge@target" (lives in peer's client registry) peer dials target, joins as "bridge@peer" (lives in target's client registry) ``` -A local `MSG` is broadcast as `DELIVER` to every registered connection — -including a bridge connection dialed in from the other side — exactly like +A local `MSG` is broadcast as `DELIVER` to every registered connection +(including a bridge connection dialed in from the other side) exactly like any other client. When a bridge *worker* itself receives a `DELIVER` over the connection it dialed out, it re-broadcasts locally with `includeBridges=false`, so the message never goes back out over any bridge: no ping-pong between two nodes forwarding the same line back and forth forever. The one piece of new logic lives in the bridge worker's receive -loop — tag the nick with the *sending* node's name, if it isn't tagged +loop: tag the nick with the *sending* node's name, if it isn't tagged already, before re-broadcasting locally: {% include codetabs.html langs="C++|Go|Rust" %} @@ -500,14 +500,14 @@ fn bridge_worker( } ``` -`peer_node` here is the *name of the node this worker dialed into* — so when +`peer_node` here is the *name of the node this worker dialed into*: so when `dave` sends a message locally on `target`, `target`'s hub broadcasts `DELIVER` to every local client including the `bridge@peer` connection that `peer`'s own bridge worker dialed in with; that worker reads the `DELIVER` back over the connection it opened, sees `dave` has no `@` in it yet, tags it `dave@target`, and re-broadcasts locally on `peer` with `includeBridges=false`. A listener on `peer` sees -`chatterd: received from=dave@target text=…` — the `@target` suffix is the +`chatterd: received from=dave@target text=…`: the `@target` suffix is the proof the message actually crossed the wire, not that it was echoed somewhere local. The reconnect loop (dial, backoff, retry) means either side can start first, or the other side can restart under `pmon`, without the @@ -526,41 +526,41 @@ registered as OTel **observable gauges** (`sysagent.cpu.pct`, sample under a mutex and reports it tagged with a `node` attribute, exported over OTLP/HTTP on a periodic reader. When the endpoint is unset, `sysagent` prints `sysagent: otel disabled (OTEL_EXPORTER_OTLP_ENDPOINT not set)` and -runs exactly the same otherwise — telemetry here is additive, never a +runs exactly the same otherwise; telemetry here is additive, never a precondition for the service coming up. `sysagent saturate --resource cpu|mem --seconds N` is the chaos half: it spins worker threads doing nothing but arithmetic, or holds a touched, resident memory allocation, so a -concurrently-running `sysagent`'s own `cpu_pct`/`mem_pct` visibly moves — the +concurrently-running `sysagent`'s own `cpu_pct`/`mem_pct` visibly moves: the USE-method callback this fleet can point at itself. -`fwatch` is Chapter 07–10's file watcher — `inotify_init1`, one -non-recursive watch on a directory, `create`/`modify`/`delete` lines — with +`fwatch` is Chapter 07–10's file watcher (`inotify_init1`, one +non-recursive watch on a directory, `create`/`modify`/`delete` lines) with Chapter 33's Landlock ruleset available as `--sandbox`. The ruleset restricts filesystem *reads* to the watched directory: `pmon` passes fwatch `watch SANDBOX_DIR --sandbox`, and before the inotify loop starts, `fwatch` probes the running kernel's Landlock ABI with `landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, builds a ruleset scoped to that one directory, and calls `landlock_restrict_self` -after setting `no_new_privs` — the identical sequence Chapter 33 proved, +after setting `no_new_privs`, the identical sequence Chapter 33 proved, reused rather than re-derived. `fwatch: landlock ABI=N enforced dir=…` is -printed only once the kernel has actually accepted the restriction, and — as -the cross-check below shows — `N` is not a constant baked into the program; +printed only once the kernel has actually accepted the restriction, and, as +the cross-check below shows, `N` is not a constant baked into the program; it is whatever the kernel underneath happens to support. Capability-bounding- set drop (applied once, by `pmon`, before any child exists) and Landlock -(applied per-process, by `fwatch` itself) are this capstone's two sandbox +(applied per-process, by `fwatch` itself) are this final project's two sandbox layers; seccomp, Chapter 33's third layer, was proven there and is not reapplied here. ## How the code works -`main` in all three languages dispatches on `argv[1]` — `pmon`, `chatterd`, -`sysagent`, or `fwatch` — through the same hand-rolled `argFlag`/`hasFlag`/ -`firstPositional` scan the rest of the book's argument parsing uses, funnel- -ing every malformed invocation through one `usage()` call. `chatterd serve` +`main` in all three languages dispatches on `argv[1]` (`pmon`, `chatterd`, +`sysagent`, or `fwatch`) through the same hand-rolled `argFlag`/`hasFlag`/ +`firstPositional` scan this book's argument parsing has used throughout, +funneling every malformed invocation through one `usage()` call. `chatterd serve` and `sysagent` (without `saturate`) each call `initTelemetry`, which is additive by design: it reads `OTEL_EXPORTER_OTLP_ENDPOINT`, and if it's unset, returns a no-op `telemetry` struct backed by `otel.Tracer`/ -`otel.Meter` no-op implementations rather than failing the subcommand — a +`otel.Meter` no-op implementations rather than failing the subcommand: a service the fleet depends on for chatting and sampling `/proc` should never refuse to start because an observability backend is unreachable or unconfigured. When the endpoint *is* set, `chatterd` wraps every delivered @@ -568,28 +568,28 @@ message in a `chatterd.deliver` span tagged with the sender's nick, text length, node, and whether it arrived over a bridge, and prints `chatterd: trace_id=… node=… from=…` for each one; `sysagent` registers its three gauges and lets a periodic OTLP/HTTP exporter push them on its own -schedule. Self-re-exec is the one place the three languages genuinely +schedule. Self-re-exec is the one place the three languages truly diverge in mechanism: Go's `os.Executable()` plus `filepath.Abs` resolves the running binary through `/proc/self/exe` internally, while C++ and Rust -call `readlink("/proc/self/exe", …)` directly — same symlink, three ways to +call `readlink("/proc/self/exe", …)` directly: same symlink, three ways to read it, all landing on the same absolute path that every forked child then `execv`s. ## Errors, three ways `app` has two usage-error surfaces and one runtime-error surface. The **top- -level usage error** — no command, an unrecognized subcommand, `chatterd`/ -`fwatch` with no sub-subcommand, `chatterd send` missing `--nick`/`--text` — +level usage error** (no command, an unrecognized subcommand, `chatterd`/ +`fwatch` with no sub-subcommand, `chatterd send` missing `--nick`/`--text`) prints the shared `usage: app ` banner and exits **2**. `sysagent saturate` has its **own**, narrower usage error: a missing `--resource` prints `usage: sysagent saturate --resource cpu|mem --seconds N -[--workers K|--mb M]` — a second banner, not the top-level one, because +[--workers K|--mb M]`: a second banner, not the top-level one, because `saturate` is itself a small program with its own contract. **Runtime errors** are exit **1** with a `service: error message` line: `chatterd: listen H:P: …` on a bind failure, `fwatch: Landlock not supported by this kernel` when the probe returns ABI 0, `pmon: mkdir DIR: …` if the sandbox directory can't be created. What `pmon` does *not* treat as a runtime error -is a single failed fork or exec attempt for one of its children — that's +is a single failed fork or exec attempt for one of its children; that's logged (`pmon: start service=…: …`) and retried after a backoff, never surfaced as a fatal error, because a process supervisor exists precisely to absorb transient start failures rather than propagate them. A clean `pmon` @@ -598,8 +598,8 @@ runs to its `--timeout-ms` deadline or a delivered signal all exit **0**. ## Concurrency lens -Every piece of this book's other running thread — which concurrency model -each language reaches for — shows up in `pmon`'s two jobs: consuming exactly +Every piece of this book's other running thread (which concurrency model +each language reaches for) shows up in `pmon`'s two jobs: consuming exactly one `SIGTERM`/`SIGINT` no matter how many children are running, and supervising each child without one child's restart blocking another's. @@ -616,19 +616,19 @@ has actually stopped. C++ and Rust reach for the same alternative to a signal *handler*: a dedicated **`sigwait`-consuming thread**. Both block `SIGTERM` and `SIGINT` in the main thread with `pthread_sigmask`/`SigSet::thread_block()` *before* -spawning any other thread — a blocked mask is inherited by every thread +spawning any other thread: a blocked mask is inherited by every thread created afterward, so only the one thread that later calls `sigwait`/`SigSet::wait()` (nix's wrapper over the same syscall) ever consumes the signal; no async-signal-unsafe work happens in a handler at all, because there is no handler. Each of the three services then gets its -own `std::thread`, and — unlike Go's `exec.Command`-based children — that +own `std::thread`, and, unlike Go's `exec.Command`-based children, that thread performs the `fork()`/`execv()` **itself**, directly: `pmon`'s C++ and Rust ports are "fork+threads" in the sense the task names them, one OS thread owning one child's entire fork/wait/restart lifecycle. Every forked child clears its own inherited copy of the blocked signal mask (`pthread_sigmask(SIG_SETMASK, &empty, …)` in C++, `SigSet::empty(). -thread_set_mask()` in Rust) immediately after `fork()` and before `execv()` -— without this, the child would inherit a *blocked* `SIGTERM`/`SIGINT` and +thread_set_mask()` in Rust) immediately after `fork()` and before `execv()`: +without this, the child would inherit a *blocked* `SIGTERM`/`SIGINT` and its own shutdown-signal handling (the same `install_signal_flag` mechanism `chatterd`/`sysagent`/`fwatch` use standalone) could never fire, the exact bug Chapter 34's container entrypoint first ran into and @@ -636,7 +636,7 @@ documented. Where C++ and Rust part ways is the health-tick wait: C++'s main thread holds a `std::condition_variable` and calls `wait_for(lock, interval, predicate)`, waking early the moment every worker thread's `active` counter reaches zero and otherwise printing health once per timeout; Rust's main -thread has no condvar at all — it just `thread::sleep`s the health interval +thread has no condvar at all; it just `thread::sleep`s the health interval and polls `JoinHandle::is_finished()` on every worker, a coarser but simpler wake pattern that trades a small amount of shutdown latency (up to one health interval) for not needing a shared mutex/condvar pair at all. @@ -648,9 +648,9 @@ health interval) for not needing a shared mutex/condvar pair at all. ``` `verify.lua` follows the convention Chapter 23 established for `vm-peer` -examples: the assertion itself runs entirely on loopback — the fleet, the +examples: the assertion itself runs entirely on loopback (the fleet, the cross-node bridge across two `127.0.0.1` ports, and telemetry to a local -LGTM stack — while the real two-host run is the chapter's demo. A loopback +LGTM stack) while the real two-host run is the chapter's demo. A loopback fleet, driven by hand the same way, shows every piece in one place: ```console @@ -667,7 +667,7 @@ pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fw pmon: shutdown ``` -(`ABI=9` here is the reference host's own Landlock version — see the +(`ABI=9` here is the reference host's own Landlock version; see the cross-check below for the guest's different number.) A local listener receiving a message sent by another client on the same node, and the cross-node bridge with two nodes on loopback, are exactly the shapes @@ -728,14 +728,14 @@ Two independent proofs back this chapter's two biggest claims, and neither comes from a line the program prints about its own intentions. The first is Landlock's enforced ABI. `fwatch --sandbox` never hardcodes a -version — it asks the running kernel with +version: it asks the running kernel with `landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, the exact probe Chapter 33 established, and prints whatever comes back. On the Fedora 44 reference host (kernel `7.1`), where this chapter's loopback `verify.lua` runs, that probe returns **ABI 9**. On the `systems-target` lab guest (kernel `6.19.10-300.fc44`) this session, the identical code, the identical syscall, returned **ABI 7**. That is not a -bug and not a discrepancy to reconcile — it is the accurate, verified +bug and not a discrepancy to reconcile; it is the accurate, verified consequence of two different kernel versions supporting two different sets of Landlock features, and it is the same lesson Chapter 33 taught about attach targets and struct offsets, reached through a sandbox ABI instead: a @@ -745,8 +745,8 @@ own probe is what keeps it right on both. The second is the OTLP path itself. `sysagent: otel enabled endpoint=…` proves the exporter was configured; it does not prove a single byte reached anywhere. The real proof is on the other end of the gateway: after the -two-host bridge run above, `sysagent_cpu_pct{node="peer"}` — queried through -Grafana's Prometheus proxy — was present with value **1.75**, a gauge that +two-host bridge run above, `sysagent_cpu_pct{node="peer"}` (queried through +Grafana's Prometheus proxy) was present with value **1.75**, a gauge that started life inside an unprivileged process on `systems-peer`, was exported over OTLP/HTTP to `http://192.168.124.1:4318`, crossed the libvirt bridge's gateway, and landed in the host's own Prometheus, labeled by the @@ -762,7 +762,7 @@ LGTM stack exists to receive, closed end to end. recorded child pid on shutdown. - **`PR_CAPBSET_DROP` is meaningful even on an already-unprivileged process.** `bounding_set_dropped=0` on the lab guest's unprivileged - account is the *correct* number — the guarantee (no future `execve` of a + account is the *correct* number: the guarantee (no future `execve` of a setuid binary can regain a dropped capability) holds regardless of what was there to begin with. - **A bridge is just a client with a special nick.** `chatterd`'s cross-host @@ -772,33 +772,33 @@ LGTM stack exists to receive, closed end to end. a delivered nick is the observable proof a message actually crossed. - **A sandbox's enforced feature set is a kernel fact, not a program's assumption.** The identical Landlock probe returned ABI 9 on the Fedora - 44 host and ABI 7 on the lab guest's 6.19.10 kernel this session — the + 44 host and ABI 7 on the lab guest's 6.19.10 kernel this session, the same class of version drift Chapter 33 taught, reached through a capability-negotiation syscall instead of an attach point. - **Telemetry is additive, and the guest→host OTLP path is real, not assumed.** Every service ran identically with `OTEL_EXPORTER_OTLP_ENDPOINT` - unset; with it set, `sysagent_cpu_pct{node="peer"}` — 1.75 on this - session's run — reached the host's Prometheus over the libvirt gateway, + unset; with it set, `sysagent_cpu_pct{node="peer"}` (1.75 on this + session's run) reached the host's Prometheus over the libvirt gateway, the same route `deploy-to-vm.sh` forwards for every OTLP-emitting VM demo. - **Identical supervision, three concurrency shapes.** Go leans on `os/exec.Command`, goroutines, a `sync.WaitGroup`, and a `signal.Notify` channel; C++ and Rust both fork and `execv` directly inside a dedicated `std::thread` per child and consume signals on a separate `sigwait`/ - `SigSet::wait` thread rather than a handler — and even between those two, + `SigSet::wait` thread rather than a handler, and even between those two, C++'s health tick wakes early off a condition variable while Rust's just - polls `is_finished()` on a sleep — the same restart/health/shutdown - contract, four genuinely different mechanisms underneath. + polls `is_finished()` on a sleep: the same restart/health/shutdown + contract, four distinctly different mechanisms underneath. -This closes the six-program arc that began in Chapter 7: four of the six — -`pmon`, `chatterd`, `sysagent`, `fwatch` — now run together, supervised, +This closes the six-program arc that began in Chapter 7: four of the six +(`pmon`, `chatterd`, `sysagent`, `fwatch`) now run together, supervised, sandboxed, and observed, across two real hosts. The book continues into -three deep dives — embedding Lua, Rust macros for systems code, the Go -runtime — but every syscall, sandbox layer, and signal-handling discipline -those chapters lean on was proven here, running for real. +three deep dives (embedding Lua, Rust macros for systems code, the Go +runtime), but every syscall, sandbox layer, and signal-handling discipline +those chapters lean on was proven here, on real hardware. --- -

verified — on the Fedora 44 +

verified: on the Fedora 44 reference host (kernel 7.1) and the two lab guests systems-target (192.168.124.7) / systems-peer (192.168.124.95, kernel 6.19.10-300.fc44) this session: @@ -814,13 +814,13 @@ was delivered to a listener on peer as chatterd: received from=dave@target text=hello across hosts, proving the cross-host bridge; fwatch --sandbox on the guest printed fwatch: landlock ABI=7 enforced, against ABI=9 on the host's -loopback run — the same probe, two different kernels, a real and verified +loopback run: the same probe, two different kernels, a real and verified difference; and with OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1: 4318 set on both guests, sysagent_cpu_pct{node="peer"} reached the host's Prometheus (queried through Grafana's proxy) with value 1.75, confirming the guest→host OTLP path over the libvirt gateway. Not exercised: seccomp filtering of any fleet child (Chapter 33 -proved seccomp alongside Landlock; this capstone reuses only the +proved seccomp alongside Landlock; this final project reuses only the capability-bounding-set-drop and Landlock layers, not seccomp) and a run where the LGTM stack was unreachable (that path reports SKIP for the telemetry assertions, not a failing example, per the manifest's diff --git a/_docs/42-embedded-lua-scripting.md b/_docs/42-embedded-lua-scripting.md index d985296..17f58f4 100644 --- a/_docs/42-embedded-lua-scripting.md +++ b/_docs/42-embedded-lua-scripting.md @@ -2,37 +2,37 @@ title: "Embedding Lua as a policy engine: one policy.lua, three host runtimes" order: 42 part: "Deep Dives" -description: "lua-policy moves pmon's restart/backoff decision out of the compiled binary and into a single sandboxed policy.lua, embedded three structurally different ways -- C++23 via sol2, Go via gopher-lua (Lua 5.1), Rust via mlua (Lua 5.4) -- and reloaded live on a real SIGHUP with no process restart. Verified on the Fedora 44 host: byte-identical decisions across all three languages (diff-confirmed), PASS 19/FAIL 0 per language, and three genuinely different sandbox-rejection diagnostics for the same malicious policy." +description: "lua-policy moves pmon's restart/backoff decision out of the compiled binary and into a single sandboxed policy.lua, embedded three structurally different ways -- C++23 via sol2, Go via gopher-lua (Lua 5.1), Rust via mlua (Lua 5.4) -- and reloaded live on a real SIGHUP with no process restart. Verified on the Fedora 44 host: byte-identical decisions across all three languages (diff-confirmed), PASS 19/FAIL 0 per language, and three distinct sandbox-rejection diagnostics for the same malicious policy." duration: "45 minutes" --- Chapter 12 taught `pmon` to treat a signal as data instead of an interruption: block `SIGCHLD` process-wide, then read it off a `signalfd` on -the next turn of an ordinary `poll` loop. Chapter 41's capstone fleet leaned +the next turn of an ordinary `poll` loop. Chapter 41's final-project fleet leaned on the same discipline for `SIGTERM`/`SIGINT`. This chapter reaches for a -different signal — `SIGHUP`, "please re-read your configuration" — to +different signal (`SIGHUP`, "please re-read your configuration") to reload something `pmon` has never had before: a restart policy that is not compiled in. Every earlier `pmon` shipped its restart/backoff logic as C++, Go, or Rust source, rebuilt whenever the policy changed. This chapter's `pmon` keeps the -supervisor's *mechanism* — track a child, replay its exit — in the host +supervisor's *mechanism* (track a child, replay its exit) in the host language, and moves the restart *decision* into `policy.lua`, one file with one contract: `on_exit(info) -> decision`. Embedding Lua is not new. What is worth a chapter is embedding the *same* file into three sandboxed -interpreters built three structurally different ways — sol2's C++ DSL over +interpreters built three structurally different ways: sol2's C++ DSL over PUC-Rio 5.4, gopher-lua's pure-Go 5.1 VM, mlua's Rust bindings over a -vendored PUC-Rio 5.4 — and proving all three reach byte-identical restart +vendored PUC-Rio 5.4, and proving all three reach byte-identical restart decisions from it, live reload included. {% include excalidraw.html file="42-host-script-boundary" alt="Three stacked host-runtime boxes on the left -- C++23/sol2 against PUC-Rio Lua 5.4, Go/gopher-lua against Lua 5.1, Rust/mlua against a vendored PUC-Rio Lua 5.4 -- each with a bidirectional arrow into a center box labeled policy.lua, one file, marked on_exit(info) going in and decision{action,delay_ms,reason} coming back. The policy.lua box sits inside a shaded sandbox-boundary band with a ghost box reading blocked: os, io, package, debug (never opened); nilled: load/loadfile/dofile/require/collectgarbage/loadstring. A host table box on the right connects to policy.lua with a bidirectional arrow labeled host.log / host.now_ms. A caption note reads Lua 5.1 (gopher-lua) vs Lua 5.4 (sol2, mlua) forces the shared subset -- one policy.lua drives all three." - caption="Figure 42.1 — the host↔script boundary: three differently-built sandboxed Lua states, one shared policy.lua, and the host table it calls back into" %} + caption="Figure 42.1: the host↔script boundary: three differently-built sandboxed Lua states, one shared policy.lua, and the host table it calls back into" %} -> **Tools used** — `lua` (host, drives `verify.lua` and is the PUC-Rio 5.4 +> **Tools used**: `lua` (host, drives `verify.lua` and is the PUC-Rio 5.4 > reference interpreter `policy.lua` was checked against), `kill` (host, -> sends the real `SIGHUP` that triggers the live reload — a shell builtin / +> sends the real `SIGHUP` that triggers the live reload, a shell builtin / > coreutils command), and the three build toolchains `cmake`+`conan` / `go` > / `cargo` (host). `lua`, `cmake`, `conan`, `go`, and `cargo` all appear in > `scripts/check-host.sh`; `kill` ships in Fedora's base install. @@ -40,7 +40,7 @@ decisions from it, live reload included. ## The host↔script boundary: `on_exit(info) -> decision` `pmon`'s contract with `policy.lua` is four callbacks wide (Figure 42.1), -and every one crosses the boundary as plain data — numbers, strings, and +and every one crosses the boundary as plain data: numbers, strings, and tables, never a function pointer or closure the host has to reason about: `on_exit(info)` in, a `{action, delay_ms, reason}` decision back, plus two callbacks the policy can call into the host, `host.log(msg)` and @@ -48,8 +48,8 @@ callbacks the policy can call into the host, `host.log(msg)` and Every simulated child exit becomes one call across that boundary. The host builds an `info` table from a hardcoded `Event`/`event`/`ExitEvent` struct, -calls `on_exit` under a **protected** call — so a bug in the policy becomes -a returned error, never a crash — and reads back three fields from the +calls `on_exit` under a **protected** call, so a bug in the policy becomes +a returned error, never a crash, and reads back three fields from the table Lua handed back. That marshaling looks remarkably similar across all three despite three unrelated Lua C APIs underneath: @@ -157,7 +157,7 @@ fn decide(lua: &Lua, ev: &ExitEvent) -> mlua::Result<()> { a Lua-side error inside `on_exit` comes back to the host as data, not a `longjmp` or a process-ending panic. `signal` is the one field every binding treats as *conditionally present*: this scenario never kills a child with a -signal, so C++'s `info["signal"]` is simply never set, and Go's +signal, so C++'s `info["signal"]` is never set, and Go's `event.signal` being a `*string` makes that "absent, not nil" contract explicit in the host's own type. @@ -230,7 +230,7 @@ end Four rules, checked in order, and every one is a decision the *host* never makes: a clean exit always `stop`s; a tight streak of fast failures (three inside `fast_crash_ms`, 2000 ms) `escalate`s regardless of lifetime restart -count — the case a flat counter misses, since 8 restarts over an hour is +count, the case a flat counter misses, since 8 restarts over an hour is normal churn and 3 in 3 seconds is a broken binary; burning through the lifetime `max_restarts` budget also `escalate`s; anything else `restart`s with a backoff doubled from whatever delay the host last used, floored at @@ -240,31 +240,31 @@ is under 130 lines of Lua, none of it duplicated in any of the three hosts. ## One `policy.lua`, two Lua versions `policy.lua` is a single 128-line file, and it is not just shared by -convention — it is written to run correctly under **two different Lua +convention: it is written to run correctly under **two different Lua language versions** at once. sol2 and mlua both build against PUC-Rio **Lua 5.4.6**; gopher-lua is a pure-Go reimplementation of **Lua 5.1**. Lua 5.2 through 5.4 added integer division (`//`), bitwise operators, `goto`, and to-be-closed variables (``); 5.1 has none of them. `policy.lua`'s own -header comment states the constraint plainly: it "uses only features +header comment states the constraint directly: it "uses only features present in BOTH Lua 5.1 and 5.4 ... so the exact same file drives all three hosts." Concretely, the backoff doubling reads `delay = last_backoff_ms * 2`, not a bitwise left-shift, and every formatted string goes through `string.format`, identical in both versions. This is the same lesson the -book has repeated elsewhere — a `hardware_concurrency()` call lying about a -container, a `runtime.NumCPU()` lying about an affinity mask — reached here +book has repeated elsewhere: a `hardware_concurrency()` call lying about a +container, a `runtime.NumCPU()` lying about an affinity mask, reached here through language-version drift instead of kernel drift. ## The sandbox: what `policy.lua` cannot touch -`policy.lua` never sees `os`, `io`, `package`, or `debug` — those libraries -are simply never opened, in any binding — and `load`, `loadfile`, `dofile`, +`policy.lua` never sees `os`, `io`, `package`, or `debug`; those libraries +are never opened, in any binding, and `load`, `loadfile`, `dofile`, `require`, `collectgarbage`, and `loadstring` are explicitly nilled out afterward, even though their *owning* libraries were never opened either. That belt-and-suspenders removal matters because those six globals are reachable straight from `base`, the one library every binding does open; a global with the same name could otherwise be reintroduced by a careless policy. The three bindings build that sandbox with three different APIs, -but the shape is identical — open four libraries, remove six globals, +but the shape is identical: open four libraries, remove six globals, register one `host` table: {% include codetabs.html langs="C++|Go|Rust" %} @@ -385,42 +385,42 @@ fn build_sandbox() -> mlua::Result { The Go tab is longest for a real reason: `gopher-lua`'s `SkipOpenLibs` leaves `os`/`io`/`package`/`debug`/`coroutine` as ordinary `nil` globals, and its -own nil-index error is generic — it never names *which* global was nil. -sol2 and mlua get PUC-Rio's actual error message for free; gopher-lua's host +own nil-index error is generic: it never names *which* global was nil. +sol2 and mlua inherit PUC-Rio's actual error message automatically; gopher-lua's host installs a **guard table** per blocked library instead, with a metamethod -that names the field explicitly — a strictly more informative error for +that names the field explicitly, a strictly more informative error for whoever runs `check`. This split shows up again, unprompted, in "Errors, three ways" below. ## How the code works -Both subcommands — `run` and `check` — funnel through the same three +Both subcommands (`run` and `check`) funnel through the same three functions: `make_sandbox`/`newSandbox`/`build_sandbox` builds the interpreter, `load_policy`/`loadPolicy` runs the chunk under a protected load and calls the policy's optional `on_load()` hook (only for the `policy loaded: version=...` banner), and `decide`/`callOnExit`/`decide` -drives one `on_exit` call. `run` replays a fixed two-phase scenario — three +drives one `on_exit` call. `run` replays a fixed two-phase scenario (three simulated exits against v1, a real `SIGHUP`, three more against whatever the -policy file now contains — hardcoded because the point here is the policy +policy file now contains) hardcoded because the point here is the policy engine, not a process reaper. `check` loads one policy path, probes `on_exit` once, and reports `pmon: policy ok version=...` or `pmon: policy -error: ...`, exiting 0 or 1 — the "will this file behave" gate `demo.sh` -calls before ever running a policy for real. Every print in the C++ port +error: ...`, exiting 0 or 1, the "will this file behave" gate `demo.sh` +calls before the supervisor ever runs that policy. Every print in the C++ port routes through a `say()` helper that calls `std::fflush(stdout)`: `demo.sh` polls the redirected stdout file for the `awaiting SIGHUP` sentinel, and an unflushed write would leave it staring at a stale, empty file. ## Errors, three ways -Each host's own source comments name its own "errors, three ways" split — +Each host's own source comments name its own "errors, three ways" split: three unrelated idioms for the identical job, turning an interpreter failure into data instead of a crash. **C++**: fallible operations return `std::expected`, converting a failed `sol::protected_function_result` into its `sol::error` message; `die()` is reserved for CLI-usage/OS-resource failures with no data-driven recovery. **Go**: `die()` prints the fully `%w`-wrapped error chain and exits 1; -`luaErr()` unwraps gopher-lua's `*lua.ApiError` via `errors.As` — never a -bare type assertion — so a caller sees `attempt to index a nil value +`luaErr()` unwraps gopher-lua's `*lua.ApiError` via `errors.As` (never a +bare type assertion), so a caller sees `attempt to index a nil value (global 'os')` rather than a Go-shaped wrapper string. **Rust**: CLI parse failures `bail!` out of `parse_args`; host-side operations return `anyhow::Result` composed with `.context()`; every boundary crossing into @@ -448,16 +448,16 @@ stack traceback: ... ``` -sol2 and mlua report the *identical* diagnostic shape — PUC-Rio's own -"attempt to index a nil value (global 'os')" plus a native stack traceback — +sol2 and mlua report the *identical* diagnostic shape: PUC-Rio's own +"attempt to index a nil value (global 'os')" plus a native stack traceback, because both are the real reference interpreter underneath, reached through different Rust/C++ FFI. Go's line differs by design: it's `newSandbox`'s guard-table metamethod firing, and it names something PUC-Rio's message -doesn't — *which field* (`execute`) the policy tried to reach. Same -rejection, same line number, three genuinely different diagnostics. +doesn't: *which field* (`execute`) the policy tried to reach. Same +rejection, same line number, three distinct diagnostics. -`policy-broken.lua`'s missing `end` is a **load**-time failure — caught -before `on_exit` is ever called — and it shows the same split one syntax +`policy-broken.lua`'s missing `end` is a **load**-time failure (caught +before `on_exit` is ever called), and it shows the same split one syntax error later: ```console @@ -469,17 +469,17 @@ pmon: policy error: loading ../policy-broken.lua: ../policy-broken.lua line:16(c pmon: policy error: syntax error: [string "../policy-broken.lua"]:16: 'end' expected (to close 'if' at line 13) near 'return' ``` -Once again cpp and rust agree almost to the byte — PUC-Rio's own parser -message — and gopher-lua's parser reports the same line and near-token in +Once again cpp and rust agree almost to the byte, PUC-Rio's own parser +message, and gopher-lua's parser reports the same line and near-token in its own format. In every case the running supervisor is unaffected: `check` loads the bad file in isolation, reports the rejection, and exits 1; a `run`-time `SIGHUP` reload that hits a bad policy would report the same -error and simply keep serving whatever policy was already loaded — a +error and keep serving whatever policy was already loaded; a `policy-*.lua` typo is a reported problem, never a crash of `pmon` itself. ## Concurrency lens -Every host consumes the *same* `SIGHUP` through three different primitives — +Every host consumes the *same* `SIGHUP` through three different primitives: the same divergence Chapter 41 mapped for `SIGTERM`, reached here through a reload path with one new wrinkle: after the signal lands, **every** binding discards the whole interpreter and builds a brand-new one. @@ -558,26 +558,26 @@ void wait_for_sighup(int sigfd) { C++ reuses ch12/ch13's pattern outright: `SIGHUP` is blocked process-wide with `sigprocmask` up front (in `make_sighup_fd`, not shown), then consumed -as ordinary readability on a `signalfd` inside a `poll` loop — no signal +as ordinary readability on a `signalfd` inside a `poll` loop; no signal handler runs at all. Go reaches for the idiomatic equivalent: `signal.Notify` -hands a buffered channel to the Go runtime's own signal-handling machinery — -delivery happens on a dedicated runtime thread, never in user code — and -`cmdRun` simply blocks on `<-sigCh`. Rust's `signal-hook` sits between those +hands a buffered channel to the Go runtime's own signal-handling machinery: +delivery happens on a dedicated runtime thread, never in user code, and +`cmdRun` blocks on `<-sigCh`. Rust's `signal-hook` sits between those two: it installs a real, async-signal-safe `sigaction` handler, but that -handler is only permitted to flip an `AtomicBool` — no allocation, no Lua -call, no I/O — and `cmd_run`'s own thread polls that flag every 20 ms. The +handler is only permitted to flip an `AtomicBool` (no allocation, no Lua +call, no I/O), and `cmd_run`'s own thread polls that flag every 20 ms. The flag is registered **before** the "awaiting SIGHUP" line prints, closing a race a naïve implementation would have: a `SIGHUP` landing the instant after the sentinel appears is never missed. All three converge on the same rebuild discipline once the signal lands: -throw the whole interpreter away and construct a fresh one — the +throw the whole interpreter away and construct a fresh one: the correctness argument for reload, not a code-simplicity shortcut. A policy can define arbitrary local state, upvalues, and closures; mutating a live state in place would leave open the chance the *old* policy's state survives into the *new* one's behavior. Building fresh guarantees the -opposite. One asymmetry worth naming even though this chapter never -exercises it: Go's `*lua.LState` is **not** goroutine-safe — a real `pmon` +opposite. One asymmetry applies even though this chapter never +exercises it: Go's `*lua.LState` is **not** goroutine-safe; a real `pmon` wanting concurrent `on_exit` calls against a `gopher-lua` policy would need its own serialization, where sol2's and mlua's states carry the identical single-threaded constraint under a different name. @@ -590,7 +590,7 @@ single-threaded constraint under a different name. `./demo.sh run` copies `policy.lua` into a scratch file, starts `pmon run --policy `, waits for the `awaiting SIGHUP` sentinel, copies -`policy-v2.lua` over the same path, and sends a real `kill -HUP` — no fixed +`policy-v2.lua` over the same path, and sends a real `kill -HUP`, with no fixed sleep, no guesswork. The output is identical, byte for byte, across all three languages (diff-confirmed this session): @@ -607,13 +607,13 @@ pmon: decision child=api action=restart delay_ms=100 reason="api: restart 1 afte pmon: decision child=worker action=escalate delay_ms=0 reason="worker: giving up after 2 restarts (last exit code 1)" ``` -Read as a story: v1 loads (500 ms base, give up after 8). `web` exits 0 — -`stop`. `worker` crashes twice — `restart` at 500 ms, then the doubled +Read as a story: v1 loads (500 ms base, give up after 8). `web` exits 0, +`stop`. `worker` crashes twice: `restart` at 500 ms, then the doubled 1000 ms backoff. `pmon` parks on `SIGHUP`; `demo.sh` swaps in `policy-v2.lua` and signals the same, never-restarted PID. Reload picks up v1.1.0 (100 ms base, give up after only 2). `api`'s fresh crash backs off 100 ms, v2's tighter base; `worker`, already carrying 2 lifetime restarts, -immediately hits v2's `max_restarts=2` and `escalate`s — the same event +immediately hits v2's `max_restarts=2` and `escalate`s, the same event that would have been a normal `restart` under v1's `max_restarts=8`. The policy changed behavior with zero rebuild and zero process restart. @@ -644,7 +644,7 @@ Each language's `verify.lua` reported `PASS 19 / FAIL 0`. {% include excalidraw.html file="42-sighup-reload-lifecycle" alt="A horizontal sequence of six boxes inside a band labeled one pmon process -- PID never changes: load v1 (base=500ms max_restarts=8), phase-1 decisions (web: stop; worker: restart 500ms to 1000ms), awaiting SIGHUP (a dashed ghost box), reload (fresh Lua state rebuilt, on_load logs v1.1.0), phase-2 decisions (api: restart 100ms; worker: escalate after 2 restarts), and exit 0. Above the awaiting-SIGHUP box, a ghost box shows the operator action -- edit the policy file, then kill -HUP $(pgrep app) -- with an arrow into the flow. Below it, three sub-boxes show the three per-language consumption mechanisms: C++ signalfd + poll, Go signal.Notify(ch), Rust signal_hook flag (AtomicBool + 20ms poll). A bottom note reads: the process is NOT restarted -- only a fresh, sandboxed Lua state is built in place and the decision logic changes underneath it." - caption="Figure 42.2 — the SIGHUP reload lifecycle: v1's decisions, a real SIGHUP consumed three different ways, a fresh Lua state built in place, and v1.1.0's tighter decisions — same process throughout" %} + caption="Figure 42.2: the SIGHUP reload lifecycle: v1's decisions, a real SIGHUP consumed three different ways, a fresh Lua state built in place, and v1.1.0's tighter decisions, same process throughout" %} ## Cross-check: one script, three interpreters, byte-identical decisions @@ -653,9 +653,9 @@ against: `policy.lua`, unmodified, drives sol2/PUC-Lua 5.4, gopher-lua's pure-Go 5.1 VM, and mlua/PUC-Lua 5.4 to the *exact same sequence of restart decisions*, including the exact backoff numbers and the exact `reason` strings Lua's `string.format` produced. That is not "the same shape of -output" — it is a byte-for-byte `diff` of three processes built from three -different compilers, three different runtimes, and (for Go) a genuinely -different Lua language version, running the identical two-phase scenario +output"; it is a byte-for-byte `diff` of three processes built from three +different compilers, three different runtimes, and (for Go) a different +Lua language version, running the identical two-phase scenario with the identical `SIGHUP` reload in between. This is possible because every value crossing the boundary is host-primitive data, never a Lua value the host interprets loosely; it is *correct* because of the shared-subset @@ -670,7 +670,7 @@ not three green exit codes agreeing by coincidence. - **The host↔script boundary is four callbacks, all plain data**: `on_exit(info) -> decision` plus `host.log`/`host.now_ms`, crossed as - tables of numbers and strings, never a closure — what makes a + tables of numbers and strings, never a closure: what makes a diff-identical run across three unrelated Lua bindings possible. - **A shared-subset policy file survives a language-version gap, not just a binding-API gap.** `policy.lua` runs identically on PUC-Rio Lua 5.4 (sol2, @@ -683,7 +683,7 @@ not three green exit codes agreeing by coincidence. - **The sandbox-rejection diagnostic differs by binding, and that is informative, not a bug.** sol2 and mlua share PUC-Rio's native `attempt to index a nil value (global 'os')`; gopher-lua's guard table reports - `sandbox: global 'os' is not available (blocked field 'execute')` — + `sandbox: global 'os' is not available (blocked field 'execute')`, strictly more specific, since gopher-lua's own nil-index error never names the global at all. - **`SIGHUP` reload rebuilds the interpreter, never restarts the process.** @@ -697,27 +697,27 @@ not three green exit codes agreeing by coincidence. separate thread polls every 20 ms, armed before the "awaiting" line prints to close the missed-signal race. -Two deep dives remain in this part — Rust macros for systems code, and the -Go runtime from a systems programmer's seat — the same instinct turned on a +Two deep dives remain in this part: Rust macros for systems code, and the +Go runtime from a systems programmer's seat, the same instinct turned on a single language's own machinery. --- -

verified — on the Fedora 44 +

verified: on the Fedora 44 reference host this session: python3 scripts/test-all-examples.py --only 42-embedded-lua-scripting printed 3 passed, 0 failed (build cpp/go/rust: ok; verify cpp/go/rust: PASS), with each language's own verify.lua reporting PASS 19 / FAIL 0. A full local regression (--mode local) gave 96 passed, 0 failed, 3 skipped (the 3 skips are the unrelated 38-otel -example, LGTM unreachable) — no regressions from this chapter's example. +example, LGTM unreachable), with no regressions from this chapter's example. ./demo.sh <lang> run produced byte-identical output across cpp/go/rust (diff-confirmed): v1.0.0 (base 500ms, max_restarts=8) driving a `stop` for a clean exit and a `restart` backoff that doubled 500ms → 1000ms, then a real kill -HUP reload to v1.1.0 (base 100ms, max_restarts=2) driving a 100ms `restart` and an `escalate` once the worker's lifetime restart count (2, carried over from phase one) hit v2's -lower ceiling — the same process throughout, PID unchanged. The sandbox +lower ceiling, the same process throughout, PID unchanged. The sandbox rejection diagnostics for policy-evil.lua (os.execute) and policy-broken.lua (missing `end`) were captured verbatim per language via ./demo.sh <lang> check ../policy-evil.lua @@ -730,6 +730,6 @@ guard table reports "sandbox: global 'os' is not available (blocked field vendored features plus signal-hook 0.3 and anyhow 1 (Rust); the host's own lua is 5.4.8 (PUC-Rio), driving verify.lua itself. Not exercised: this -example is mode: local per examples/manifest.yaml — +example is mode: local per examples/manifest.yaml; there is no VM or LGTM path for it, so no lab-guest or telemetry run applies here.

diff --git a/_docs/43-rust-macros-for-systems.md b/_docs/43-rust-macros-for-systems.md index 2768725..7219073 100644 --- a/_docs/43-rust-macros-for-systems.md +++ b/_docs/43-rust-macros-for-systems.md @@ -2,14 +2,14 @@ title: "Rust macros for systems code: proc-macros, syn/quote, and what miri proves about unsafe" order: 43 part: "Deep Dives" -description: "sysmacros is chapter 43's proc-macro crate, powering two Rust-only techniques in the app binary -- #[derive(SysError)], a miniature thiserror built with syn 2.0's AST and quote's token emission, and #[instrument], a compile-time tracing attribute that times a function with Instant -- plus a sound unsafe unaligned read that cargo +nightly miri validates clean and, under the deliberately-named --oob-read path, catches as undefined behaviour. The book's first single-language deep dive, verified on the Fedora 44 host with verify.lua PASS 8/FAIL 0, the exact six-line stable stdout, and miri's own out-of-bounds diagnostic reproduced byte for byte." +description: "sysmacros is chapter 43's proc-macro crate, powering two Rust-only techniques in the app binary -- #[derive(SysError)], a miniature thiserror built with syn 2.0's AST and quote's token emission, and #[instrument], a compile-time tracing attribute that times a function with Instant -- plus a sound unsafe unaligned read that cargo +nightly miri validates clean and, under the --oob-read path, catches as undefined behaviour. The book's first single-language deep dive, verified on the Fedora 44 host with verify.lua PASS 8/FAIL 0, the exact six-line stable stdout, and miri's own out-of-bounds diagnostic reproduced byte for byte." duration: "35 minutes" --- Every chapter so far has mirrored the same behavior across C++23, Go, and -Rust, because the point of those chapters was a Linux mechanism — a -syscall, a signal, a namespace — that all three languages sit on top of -equally. This one breaks that pattern on purpose. Proc-macros are a Rust +Rust, because the point of those chapters was a Linux mechanism (a +syscall, a signal, a namespace) that all three languages sit on top of +equally. This one breaks that pattern by design. Proc-macros are a Rust compiler feature with no C++ or Go equivalent worth building a codetab around, and `miri`, the interpreter that catches undefined behaviour a normal build would run right past, only exists because Rust's `unsafe` @@ -22,33 +22,33 @@ still doing exactly what its comment claims. `sysmacros` is a small cargo workspace: a `proc-macro = true` crate holding two macros, and an `app` binary that uses them. `#[derive(SysError)]` turns an errno-style enum into a `Display` + `std::error::Error` impl from -one `#[error("...")]` attribute per variant — a miniature `thiserror`, +one `#[error("...")]` attribute per variant: a miniature `thiserror`, built from scratch with `syn 2.0` and `quote`. `#[instrument]` wraps a function to print its entry, its exit, and its wall-clock duration, without a tracing runtime. And `read_record` is one `unsafe` unaligned -read of a `repr(C)` struct out of a byte buffer — sound on the path that +read of a `repr(C)` struct out of a byte buffer: sound on the path that checks its bounds, and undefined behaviour on the `--oob-read` path that skips the check, which is exactly the distinction `miri` exists to draw. {% include excalidraw.html file="43-proc-macro-pipeline" alt="Two horizontal bands. Top band: the app crate, main.rs's annotated source (#[derive(SysError)] enum PmonError, #[instrument] fn hash_chunk) feeding rustc, which packages a TokenStream and hands it off; on the right, rustc splices tokens back into the AST, typechecks, and compiles the result into the app binary (generated Display + Error impls, the instrumented hash_chunk). Bottom band: sysmacros, a separate proc-macro crate compiled FIRST, containing two nodes -- syn::parse_macro_input! turning the TokenStream into a real DeriveInput/ItemFn AST, and quote! emitting a new TokenStream2 (the Display impl or the wrapped function). Arrows cross the band boundary labeled TokenStream (macro input) going down and TokenStream (macro output) coming back up. A note at the bottom reads: sysmacros never runs at app runtime -- it is pure tokens-in/tokens-out, executed only while rustc is compiling app." - caption="Figure 43.1 — the compile-time-only pipeline: annotated source becomes a TokenStream, the separately-compiled sysmacros crate parses it with syn and re-emits new tokens with quote, and rustc splices the result into app" %} + caption="Figure 43.1: the compile-time-only pipeline: annotated source becomes a TokenStream, the separately-compiled sysmacros crate parses it with syn and re-emits new tokens with quote, and rustc splices the result into app" %} -> **Tools used** — `cargo` (host, build/run the pinned stable toolchain — +> **Tools used**: `cargo` (host, build/run the pinned stable toolchain, > `rustc 1.97.1` per `rust-toolchain.toml`, in `scripts/check-host.sh`), -> `cargo +nightly miri` / the `miri` component (host — a nightly toolchain +> `cargo +nightly miri` / the `miri` component (host, a nightly toolchain > plus `miri` installed via `rustup`, exercised throughout this chapter but -> **not** part of the stable CI gate, so it is intentionally not in -> `scripts/check-host.sh`), `lua` (host, drives `verify.lua` — in +> **not** part of the stable CI gate, so it does not appear in +> `scripts/check-host.sh`), `lua` (host, drives `verify.lua`, in > `scripts/check-host.sh`). ## Two macros, one crate: `sysmacros` `sysmacros/Cargo.toml` sets `proc-macro = true` in its `[lib]` section, which is what makes it a fundamentally different kind of dependency: -rustc compiles it as a *compiler plugin* — a program that runs during -`app`'s own compilation, not inside `app` itself — before it ever touches +rustc compiles it as a *compiler plugin*, a program that runs during +`app`'s own compilation, not inside `app` itself, before it ever touches `main.rs`. That is why this is a two-crate cargo workspace rather than one package: a proc-macro can only live in its own crate, compiled first, and `app`'s `Cargo.toml` depends on it the ordinary way (`sysmacros = { path = @@ -57,7 +57,7 @@ execute the finished `app` binary; every line of it runs once, at build time, and produces tokens that rustc splices into the program in its place. -The macro `PmonError` actually uses is the plainer of the two ideas — +The macro `PmonError` actually uses is the plainer of the two ideas: each variant just needs a message: ```rust @@ -79,9 +79,9 @@ their own fields by name; the tuple variant (`PolicyRejected`) formats its one field positionally as `{0}`. `#[instrument]` sits on `hash_chunk` below it the same way `#[error(...)]` sits on an enum variant: an annotation the macro reads, not code that runs before the function does. -Both macros end at the same place — a `syn::Result` that is +Both macros end at the same place: a `syn::Result` that is either the new code or a `syn::Error` describing exactly what was wrong -with the input — which "Errors, three ways" returns to below. +with the input, which "Errors, three ways" returns to below. ## A sound unsafe read, and what miri proves @@ -107,13 +107,13 @@ fn read_record(buf: &[u8], oob: bool) -> Record { An `unsafe` block is a promise to the compiler: "I checked something the type system can't, and this operation is valid anyway." `read_unaligned` -needs three things to be true — the pointer is readable for +needs three things to be true: the pointer is readable for `size_of::()` bytes, those bytes form a valid `Record` (satisfied trivially here, since `Record` is `repr(C)` and made only of integers), and the read doesn't race another write. The `assert!` right above the `unsafe` block is what makes the sound path's promise true: it checks the buffer really does have 8 bytes before offset 0 is read from. `--oob-read` -sets `off = 4` and, by design, skips that `assert!` — the read now reaches +sets `off = 4` and, by design, skips that `assert!`; the read now reaches for offset `4..12` against an 8-byte `buf`, and nothing in a normal `cargo build`/`cargo run` will tell you that happened. That silence is the whole reason `miri` exists: it is not a linter looking at your source, it @@ -123,7 +123,7 @@ the instant a read or write steps outside them. {% include excalidraw.html file="43-miri-bounds-check" alt="An 8-byte buffer BUF: [u8; 8] drawn as eight numbered byte boxes 0 through 7, labeled above as magic: u32 (bytes 0..4) and slots: u32 (bytes 4..8), followed by four dashed ghost boxes numbered 8 through 11 marked past the end (never allocated). Below the real bytes, a blue SOUND window box spans bytes 0 through 7, labeled: read_record(buf, oob=false), read_unaligned::(ptr.add(0)), window [0..8) fully inside the 8-byte alloc, cargo +nightly miri run: exit 0, no UB. Below and offset right, an amber window box spans bytes 4 through 11 (crossing from the real buffer into the ghost bytes), labeled: --oob-read: read_record(buf, oob=true), read_unaligned::(ptr.add(4)), window [4..12) -- 4 bytes past the 8-byte alloc, miri: only 4 bytes from the end of the allocation (main.rs:57). Dashed amber boundary markers point from the OOB window up into the byte row at the offset-4 and offset-12 boundaries." - caption="Figure 43.2 — the same 8-byte Record read from two offsets: the SOUND window [0..8) miri accepts, and the --oob-read window [4..12) that runs 4 bytes past the allocation, exactly what miri's diagnostic names" %} + caption="Figure 43.2: the same 8-byte Record read from two offsets: the SOUND window [0..8) miri accepts, and the --oob-read window [4..12) that runs 4 bytes past the allocation, exactly what miri's diagnostic names" %} ## How the code works @@ -211,26 +211,26 @@ fn expand_sys_error(input: &DeriveInput) -> syn::Result { ``` `attr.path().is_ident("error")` is syn 2.x's API for "is this attribute -named `error`" — `path()` is a method, not a field, one of the surface +named `error`"; `path()` is a method, not a field, one of the surface changes from syn 1.x. `#[proc_macro_derive(SysError, attributes(error))]` (not shown above) declares `error` as an *inert helper attribute*: it tells rustc that `#[error("...")]` on a variant is this derive's business, not an unknown-attribute error. `error_attr.parse_args::()` (spelled `parse_args()?` above, with the type driven by the `LitStr` annotation on `message`) parses whatever is inside the parentheses as a -single string literal — exactly the `"open {path} failed: {errno}"` from +single string literal: exactly the `"open {path} failed: {errno}"` from `PmonError::Open`. The three `Fields` arms are why `PolicyRejected`'s tuple variant formats differently from the two named-field variants: for `Fields::Named`, the destructured pattern `Self::#vident { #(#idents),* }` binds `path` and `errno` as local names, so `write!(f, #message)` alone is -enough — edition 2024's inline format-arg capture resolves `{path}` and +enough; edition 2024's inline format-arg capture resolves `{path}` and `{errno}` in the literal directly against those bindings, with no positional arguments at all. For `Fields::Unnamed`, there is no name to capture, so the arm binds synthetic identifiers (`f0`, `f1`, ...) and passes them to `write!` explicitly as `#(#idents),*` after the message. The whole function returns one `quote!` block containing both the -`Display` impl and the `Error` impl, and that block — not `PmonError`'s -own source — is what rustc actually type-checks and compiles into `app`. +`Display` impl and the `Error` impl, and that block (not `PmonError`'s +own source) is what rustc actually type-checks and compiles into `app`. `expand_instrument` takes the same shape and applies it to a function instead of an enum: @@ -292,35 +292,35 @@ fn expand_instrument(func: ItemFn) -> syn::Result { ``` `parse_macro_input!(item as ItemFn)` (in the caller, not shown) gives -`expand_instrument` a full parsed function — signature, visibility, -attributes, and body — as one `syn::ItemFn`. The loop over `sig.inputs` +`expand_instrument` a full parsed function (signature, visibility, +attributes, and body) as one `syn::ItemFn`. The loop over `sig.inputs` rejects anything that isn't a plain identifier parameter (`self` receivers, destructuring patterns) with a `syn::Error` rather than silently mangling it. The generated function keeps the original `#(#attrs)*`, `#vis`, and -`#sig` untouched — same name, same signature, same visibility — and +`#sig` untouched (same name, same signature, same visibility), and replaces only `#block`, the body, with one that prints an entry line, runs the original body inside a closure so `#block`'s own `return`s and `?` still work as written, times it with `Instant`, prints the exit line with the return value and the elapsed microseconds, and returns the result. `hash_chunk(len: usize) -> u64` goes in as ordinary Rust and comes out as a function with the identical signature that also happens to log -itself — the caller never sees a difference except the printed lines. +itself; the caller never sees a difference except the printed lines. ## Errors, three ways Every earlier chapter's "errors, three ways" contrasted three languages; this one has only Rust, so the three ways are the three moments an error in this program can actually surface. **Compile time**: a malformed macro -input — `#[derive(SysError)]` on a struct, or a variant missing -`#[error("...")]` — becomes a `syn::Error`, and `derive_sys_error` turns +input (`#[derive(SysError)]` on a struct, or a variant missing +`#[error("...")]`) becomes a `syn::Error`, and `derive_sys_error` turns that into real tokens with `err.to_compile_error().into()`, so a bad macro invocation is a compiler error at the call site, never a panic inside `sysmacros` itself. **Run time**: once compiled, `PmonError`'s -three variants are ordinary values — `println!("error: {open_err}")` +three variants are ordinary values: `println!("error: {open_err}")` drives the generated `Display` impl the same way any hand-written error type would, producing `error: open /etc/shadow failed: 13` and the two lines after it. **A checked run**: `read_record`'s `--oob-read` path -produces no error at all under a normal `cargo run` — the read succeeds, +produces no error at all under a normal `cargo run`: the read succeeds, reads whatever bytes happen to sit past the allocation, and the program keeps going, which is precisely what makes undefined behaviour dangerous rather than merely broken. `cargo +nightly miri run` is what turns that @@ -331,12 +331,12 @@ allocation. ## Concurrency lens -There is a genuine concurrency angle here, and it is not the hot loop — +There is a real concurrency angle here, and it is not the hot loop; there isn't one. `expand_sys_error` and `expand_instrument` run inside rustc's own build process, once per macro invocation, as ordinary synchronous function calls; nothing in `sysmacros` spawns a thread or shares mutable state across invocations, because a proc-macro never needs -to — it is handed one `TokenStream`, returns one `TokenStream`, and is +to; it is handed one `TokenStream`, returns one `TokenStream`, and is done. `#[instrument]`'s `Instant::now()`/`.elapsed()` pair is just as thread-agnostic at the *other* end of the pipeline: the generated wrapper times whatever thread calls the instrumented function, on that thread's @@ -346,7 +346,7 @@ stdout by nothing more than `println!`'s own internal lock. Contrast a real tracing framework like `tracing`: `#[tracing::instrument]` has to enter a span on a thread- or task-local subscriber stack, so concurrent spans compose into a tree a `Registry` maintains. This chapter's -`#[instrument]` composes with concurrency by having nothing to compose — +`#[instrument]` composes with concurrency by having nothing to compose: it wraps whatever the function does and gets out of the way, at the cost of never answering "which call is this exit line for" once two threads print at once. @@ -358,7 +358,7 @@ print at once. ``` `./demo.sh rust run` builds (if needed) and runs the release binary on the -pinned stable toolchain — the exact path `verify.lua` exercises: +pinned stable toolchain: the exact path `verify.lua` exercises: ```console [host]$ ./demo.sh rust run @@ -372,29 +372,29 @@ record: magic=0x4b565053 slots=256 Six lines, three techniques: lines 1–3 are `#[derive(SysError)]`'s generated `Display`, one line per variant shape (named-field, multi-named-field, -tuple); lines 4–5 are `#[instrument]` around `hash_chunk(len=4096)` — the +tuple); lines 4–5 are `#[instrument]` around `hash_chunk(len=4096)`: the hash itself (`10911624183873311525`) is a deterministic FNV-1a over `0..4096`, so it is identical on every run, and only the microsecond figure (`4us` here) varies; line 6 is the sound `read_record` call printing the `Record` it read. `./demo.sh rust miri` runs the identical scenario under -`cargo +nightly miri run --quiet` — the interpreter, not the compiled -binary. It prints the same six lines and exits 0 (no UB reported), with +`cargo +nightly miri run --quiet` (the interpreter, not the compiled +binary). It prints the same six lines and exits 0 (no UB reported), with one visible difference: the instrument line reads `<- hash_chunk = 10911624183873311525 (491785us)` instead of `(4us)`. The -hash value does not change — determinism holds whether the code runs -natively or under an interpreter — but the wall-clock duration jumps from +hash value does not change: determinism holds whether the code runs +natively or under an interpreter, but the wall-clock duration jumps from 4 microseconds to roughly 491,785 microseconds, a slowdown of about five orders of magnitude. That is not a regression to worry about: `miri` is a UB checker, not a profiler, and it re-executes every memory operation through its own bookkeeping instead of the CPU directly. The lesson is in -the contrast, not the absolute number — a wall-clock measurement taken +the contrast, not the absolute number: a wall-clock measurement taken under `miri` tells you nothing about the compiled program's real performance, only about whether its memory operations are sound. `./demo.sh rust miri -- --oob-read` runs the same binary with the -deliberately out-of-bounds path enabled, and this time `miri` stops +out-of-bounds path enabled, and this time `miri` stops before printing the sixth line at all: ```console @@ -413,13 +413,13 @@ error: Undefined Behavior: memory access failed: attempting to access 8 bytes, b `alloc25` is `miri`'s own name for the specific allocation `BUF` occupies in its interpreted heap; `+0x4` is the offset the failing read started at (`off = 4` on the `--oob-read` path); "only 4 bytes from the end of the -allocation" is `miri` doing arithmetic a native run never does — `BUF` is +allocation" is `miri` doing arithmetic a native run never does: `BUF` is 8 bytes, the read needs 8 more bytes starting at offset 4, and offset 4 plus 8 is 12, four bytes past where the allocation actually ends. The line and column (`src/main.rs:57:14`) point at the exact `unsafe` block quoted above. On stable Rust this same `--oob-read` invocation runs to completion, silently reading whatever garbage bytes happen to sit past -`BUF` in memory — no error, no crash, no signal that anything was wrong. +`BUF` in memory, no error, no crash, no signal that anything was wrong. The gate the CI-equivalent runner checks is the stable path only: @@ -439,20 +439,20 @@ PASS 8 / FAIL 0 Two independent claims, two independent proofs. The first is that `#[derive(SysError)]`'s generated `Display` impl is not an approximation -of a hand-written systems error type — it *is* one: `verify.lua` asserts +of a hand-written systems error type; it *is* one: `verify.lua` asserts the exact three lines `PmonError`'s variants must produce (`open /etc/shadow failed: 13`, `child 4242 exited with status 1`, `policy rejected: sandbox blocked os.execute`), plus the `#[instrument]` -entry/exit shape and the parsed `Record` fields, for `PASS 8 / FAIL 0` — +entry/exit shape and the parsed `Record` fields, for `PASS 8 / FAIL 0`: eight assertions against behaviour, not "the binary exited 0." The second is that the sound `read_record` path really is sound and the `--oob-read` path really is a bug, and that claim is checked by an interpreter rather than taken on faith: the identical scenario run under `cargo +nightly miri run` exits 0 with no UB reported on the default path, -and reports the precise out-of-bounds access — 4 bytes past an 8-byte -allocation, at `src/main.rs:57` — the moment `--oob-read` is passed. The +and reports the precise out-of-bounds access (4 bytes past an 8-byte +allocation, at `src/main.rs:57`) the moment `--oob-read` is passed. The 491785us-versus-4us timing gap is the same proof from a different angle: -it shows `miri` is genuinely re-executing every memory access through its +it shows `miri` re-executes every memory access through its own interpreter rather than fast-pathing to the native instruction, which is exactly the mechanism that lets it catch a bounds violation a native `cargo run` sails straight through. @@ -461,14 +461,14 @@ is exactly the mechanism that lets it catch a bounds violation a native - **A proc-macro crate is a second, separate compilation**: `sysmacros` sets `proc-macro = true` and is compiled by rustc as a compiler plugin - before `main.rs` is ever touched — every proc-macro is `TokenStream` in, + before `main.rs` is ever touched; every proc-macro is `TokenStream` in, `TokenStream` out, and none of that code exists at `app`'s runtime. - **`#[derive(SysError)]` earns the name "miniature `thiserror`"**: one `syn::Result` function parses each variant's `#[error("...")]` string with `syn 2.0`, builds a `Display` match arm per `Fields` shape (named, tuple, unit) with `quote!`, and edition 2024's inline format-arg capture resolves `{path}`/`{errno}` straight against - the destructured match bindings — no positional arguments needed. + the destructured match bindings, no positional arguments needed. - **Malformed macro input is a `compile_error!`, never a panic**: both macros convert a `syn::Error` into real tokens via `err.to_compile_error()`, so a missing `#[error(...)]` attribute or a @@ -477,26 +477,26 @@ is exactly the mechanism that lets it catch a bounds violation a native - **`unsafe` is a promise the type system stopped checking, and `miri` checks it anyway**: the sound `read_record` path's `assert!` makes its `SAFETY` comment true; the `--oob-read` path skips that assert and reads - 4 bytes past an 8-byte allocation — silent on stable, a named Undefined + 4 bytes past an 8-byte allocation: silent on stable, a named Undefined Behavior report under `cargo +nightly miri run`. - **`miri` is a correctness checker, not a profiler**: the identical scenario took 4 microseconds natively and roughly 491,785 microseconds - under `miri` — a real, expected, five-order-of-magnitude interpreter + under `miri`, a real, expected, five-order-of-magnitude interpreter slowdown that says nothing about `app`'s real performance and everything about whether its memory operations are sound. - **`#[instrument]` is thread-agnostic because it holds no state at all**: it wraps whatever function it's given in a `println!`/`Instant`/`println!` sandwich with no shared subscriber or span stack, unlike a runtime - tracing crate that has to maintain one — composable with any threading + tracing crate that has to maintain one; composable with any threading model precisely because it has nothing to synchronize. Two deep dives remain in this part: this chapter's Rust-only look at -compile-time metaprogramming, and the next — the Go runtime itself, from +compile-time metaprogramming, and the next: the Go runtime itself, from a systems programmer's seat. --- -

verified — on the Fedora +

verified: on the Fedora 44 reference host this session: python3 scripts/test-all-examples.py --only 43-rust-macros-for-systems reported 1 passed, 0 failed (rust column only; this is the book's first single-language @@ -511,7 +511,7 @@ this run), and the parsed record: magic=0x4b565053 slots=256 line. cargo +nightly miri run --quiet (the sound path) exited 0 with no Undefined Behavior reported, printing the same six lines with the instrument line's duration at 491785us instead of -4us — a real ~5-order-of-magnitude interpreter slowdown, not a +4us, a real ~5-order-of-magnitude interpreter slowdown, not a regression. cargo +nightly miri run --quiet -- --oob-read reproduced the exact Undefined Behavior diagnostic quoted above: memory access failed at alloc25+0x4, "only 4 bytes from the end of @@ -522,7 +522,7 @@ pins: syn 2.0 (features full), quote 1.97.1 (rust-toolchain.toml) with a separately installed nightly toolchain plus the miri component via rustup. Not exercised: this example is mode: local -per examples/manifest.yaml — there is no VM or LGTM path for +per examples/manifest.yaml; there is no VM or LGTM path for it; and the --oob-read path is only ever run under miri in this book, never asserted on stable Rust, where the same call is undefined behaviour rather than a checkable outcome.

diff --git a/_docs/44-go-runtime-for-systems.md b/_docs/44-go-runtime-for-systems.md index 58a7917..fb10ed8 100644 --- a/_docs/44-go-runtime-for-systems.md +++ b/_docs/44-go-runtime-for-systems.md @@ -24,43 +24,43 @@ excursions the tri-language chapters made room for. {% include excalidraw.html file="44-gmp-scheduler" alt="A global run queue band at top holds overflow G's. Below, two P columns (P0.M0 running a spinner G, P1.M1 running a ticker G) each show a local run queue (256 slots + runnext), plus a box for P2 through P15 on this 16-CPU host. A dashed amber steal half arrow runs P1 into P0. To the right, a dashed sysmon box (no P, its own M) lists retake, forcegc, netpoll backstop, and the Go 1.25+ GOMAXPROCS recheck, with a solid amber SIGURG to asyncPreempt arrow into P0's running M. Caption: M:N -- many G onto few M via P; GOMAXPROCS=1 still preempts since sysmon's signal needs no spare P." - caption="Figure 44.1 — the GMP model: G's queued locally (256 slots + runnext) and globally, GOMAXPROCS P's bound to M's, work-stealing between them, and sysmon's SIGURG reaching in from outside the P pool" %} + caption="Figure 44.1: the GMP model: G's queued locally (256 slots + runnext) and globally, GOMAXPROCS P's bound to M's, work-stealing between them, and sysmon's SIGURG reaching in from outside the P pool" %} -> **Tools used** — `go build` / `go run` (host, the `go 1.26.x` check in +> **Tools used**: `go build` / `go run` (host, the `go 1.26.x` check in > `scripts/check-host.sh`). `GODEBUG` is a runtime environment variable the Go > binary itself reads at startup, not a separate tool this chapter installs > or shells out to; `/proc/self/status` is opened and parsed by the program's > own `readThreadCount` helper (below), not scraped by a host command. No VM, -> no root, no LGTM — `examples/44-go-runtime-for-systems` is `mode: local` in +> no root, no LGTM: `examples/44-go-runtime-for-systems` is `mode: local` in > `examples/manifest.yaml`. ## The GMP model: many goroutines, few threads Go's scheduler has three kinds of object, and the names are the whole mental model. A **G** is a goroutine: a small growable stack, an -instruction pointer, and a status — cheap enough that `sched` spawns +instruction pointer, and a status, cheap enough that `sched` spawns thousands without a second thought. An **M** is a machine: an actual OS thread, the thing `clone(2)` creates. A **P** is a processor: a scheduling context, exactly `GOMAXPROCS` of them, and the resource token -that makes this an M:N scheduler — an M cannot run Go code without +that makes this an M:N scheduler: an M cannot run Go code without holding a P. Each P owns a local run queue (a 256-slot ring buffer plus a single-slot `runnext` fast path); a global run queue is the overflow and injection target every P eventually checks. When a P runs dry, it -steals — `findRunnable`/`stealWork` picks a random other P and takes +steals: `findRunnable`/`stealWork` picks a random other P and takes **half** its local queue, not one goroutine at a time. None of that explains how a tight loop with no yield point ever gets interrupted. Before Go 1.14 it didn't: a bare `for {}` could wedge the runtime indefinitely (golang/go#10958). The fix is `sigPreempt = -_SIGURG` — chosen because no real program uses `SIGURG` for anything +_SIGURG`, chosen because no real program uses `SIGURG` for anything else. `preemptone()` sets a cooperative flag first, then, if async preemption is enabled, calls `preemptM`, sending `SIGURG` to the specific thread running the goroutine; the signal handler redirects into `asyncPreempt`, which saves registers, parks the goroutine at a safe point, and lets the scheduler run something else. `sched` proves this exists: with `runtime.GOMAXPROCS(1)` forced, one goroutine spins -2,000,000,000 iterations of pure integer arithmetic — comfortably past -the runtime's 10ms forced-preemption window — while a second does +2,000,000,000 iterations of pure integer arithmetic (comfortably past +the runtime's 10ms forced-preemption window) while a second does nothing but increment a counter in its own tight loop. On a purely cooperative scheduler the second goroutine would never run at all under one P; this run's `ticker_incr=388757095` says it ran the whole spin. @@ -111,22 +111,22 @@ one P; this run's `ticker_incr=388757095` says it ran the whole spin. ``` The goroutine that spins is launched *second*, deliberately: the most -recently spawned goroutine claims `runnext`, so it — not the ticker — is +recently spawned goroutine claims `runnext`, so it (not the ticker) is guaranteed to run first when `main` yields below. Behind this sits **sysmon**, a dedicated M with no P of its own, looping with a 20µs-to-10ms doubling backoff. Each tick it can retake a P stuck in a syscall or a goroutine past `forcePreemptNS` (10ms), force a GC if none has run in two minutes, trigger the SIGURG preemption above, poll the -network as a backstop if nobody has in over 10ms, and — new since Go -1.25 — re-evaluate `GOMAXPROCS` at most once a second. That last item is +network as a backstop if nobody has in over 10ms, and (new since Go +1.25) re-evaluate `GOMAXPROCS` at most once a second. That last item is the other headline scheduler change here: Go 1.25 made the default `GOMAXPROCS` **container-aware**, reading cgroup v2 `cpu.max` (or v1's quota/period pair) and taking the minimum of logical CPU count, affinity -mask, and cgroup throughput limit — floor of 2, fractional limits round -up. `GODEBUG=containermaxprocs=0` reverts to the old, `NumCPU()`-only +mask, and cgroup throughput limit (floor of 2, fractional limits round +up). `GODEBUG=containermaxprocs=0` reverts to the old, `NumCPU()`-only behavior; Go 1.26 adds only observability metrics on top, no algorithm change. On this 16-CPU host with nothing limiting this process, `knobs: -GOMAXPROCS=16 NumCPU=16` — nothing narrows either. +GOMAXPROCS=16 NumCPU=16`; nothing narrows either. ## The pacer's bargain: GOGC, GOMEMLIMIT, and Green Tea @@ -137,7 +137,7 @@ sub-millisecond), with a hybrid write barrier keeping the tri-color invariant intact. The **pacer** decides when to start the next cycle, and `GOGC` is the dial: target heap size is `Live + (Live + roots) * GOGC / 100`, so `GOGC=100` (the default) roughly doubles the live heap before -the next collection. Loosen `GOGC` and the pacer waits longer — double +the next collection. Loosen `GOGC` and the pacer waits longer: double the memory overhead, roughly halve the GC CPU cost. `GOGC=off` (`debug.SetGCPercent(-1)`) removes the percentage trigger entirely, exactly what `gc` does for its whole run, so every `NumGC` delta below is @@ -173,26 +173,26 @@ func measureCyclesMemLimit(limit int64) int { `runtime.GC()`, set a percent, run a fixed 64 MiB churn workload, read the `NumGC` delta. Run at 100 and at 800 against identical churn, and this session shows the textbook shape: `gogc100_cycles=21` against -`gogc800_cycles=2` — the looser percent needs a tenth as many collections +`gogc800_cycles=2`: the looser percent needs a tenth as many collections for the same garbage. `measureCyclesMemLimit` isolates a second knob, `GOMEMLIMIT` (`debug.SetMemoryLimit`, since Go 1.19): a **soft** ceiling -on runtime-managed memory, respected *even with `GOGC=off`* — why "GOGC +on runtime-managed memory, respected *even with `GOGC=off`*: why "GOGC off, GOMEMLIMIT set" is coherent rather than contradictory: don't collect on a heap-growth ratio at all, only when about to breach an absolute -ceiling. The docs call it deliberately soft ("no guarantees... only some +ceiling. The docs call it soft by design ("no guarantees... only some reasonable amount of effort"), since a hard limit risks the GC thrashing -to stay under it. Holding `GOGC=800` fixed and varying only the limit — -unset (`math.MaxInt64`) against a tight 8 MiB — this session measured +to stay under it. Holding `GOGC=800` fixed and varying only the limit, +unset (`math.MaxInt64`) against a tight 8 MiB, this session measured `memlimit_unset_cycles=2` against `memlimit_tight_cycles=122`: the tight limit forces sixty times as many collections as `GOGC` alone would ever trigger. Go 1.26 changes what's doing the marking, without touching any of the above. **Green Tea**, opt-in in Go 1.25 (`GOEXPERIMENT=greenteagc`), is -enabled by default in 1.26 — the release notes: "now enabled by default +enabled by default in 1.26; the release notes: "now enabled by default after incorporating feedback." Instead of chasing individual object pointers, it organizes marking around 8 KiB pages, scanning in memory -order with per-page bitmaps — cache-friendly, and on Ice-Lake/Zen-4-class +order with per-page bitmaps, cache-friendly, and on Ice-Lake/Zen-4-class CPUs it also uses vector instructions to scan whole pages in a handful of cycles. The measured effect, per Google's benchmarks, is 10-40% less GC CPU overhead on GC-heavy programs (if GC was 10% of total CPU, a 1-4% @@ -205,36 +205,36 @@ build-time signal only, visible via `go version -m ` or ## The netpoller: why 2,000 goroutines are not 2,000 threads A classic blocking syscall and a network read look identical from inside -a goroutine — `conn.Read(buf)` either way — but they take different +a goroutine (`conn.Read(buf)` either way), but they take different paths through the runtime, which is why Go can run thousands of concurrently-blocked goroutines on a double-digit thread count. A real blocking read calls `entersyscall`, transitioning the goroutine to `_Gsyscall` and optimistically **keeping its P**, betting the syscall -returns fast; if it doesn't, sysmon's `retake` notices — within ~20µs if +returns fast; if it doesn't, sysmon's `retake` notices (within ~20µs if other work is waiting, otherwise within `forcePreemptNS` (10ms) -regardless — steals the P away, and `handoffp` wakes a parked M or, if +regardless), steals the P away, and `handoffp` wakes a parked M or, if none is idle, `clone(2)`s a new one. Block N goroutines this way and the thread count tracks N. A network or pipe read on a non-blocking fd exits differently. `internal/poll`'s read loop calls the raw syscall once; on `EAGAIN` it calls `pollDesc.waitRead` → `netpollblock` → `gopark`, dropping the -goroutine into `_Gwaiting`, detached **with no P or M held at all** — its +goroutine into `_Gwaiting`, detached **with no P or M held at all**: its identity is stored as a bare pointer inside the fd's `pollDesc`, and the M that was running it is immediately free to find other work. The wakeup side is one process-wide `epoll` instance (`epoll_create1` plus an `eventfd` for the runtime's own wake-itself-up calls), each fd armed edge-triggered. The wrapper is named `EpollWait` but the syscall it -issues is `epoll_pwait` — what an `strace -f` on any Go binary shows, +issues is `epoll_pwait`, what an `strace -f` on any Go binary shows, never `epoll_wait`. `findRunnable` polls this netpoller opportunistically before resorting to work-stealing, and blockingly when there's nothing -else to do — an M with no work drops its P and calls `netpoll(delay)`, +else to do; an M with no work drops its P and calls `netpoll(delay)`, temporarily *becoming* the netpoller thread. {% include excalidraw.html file="44-goroutines-not-threads" alt="Top band: 2000 goroutines blocked on os.Pipe reads feed one shared netpoller (epoll_pwait, per-fd pollDesc) labeled parked, no M; the OS-thread box reads threads=12, bound=80. Bottom band, the contrast: N goroutines calling syscall.Read on a raw, never-epoll-registered fd each block one M in read(2); the thread box shows the count climbing toward N via sysmon retake, handoffp, startm/newm. Caption: goroutines != threads -- 2000 parked, 12 threads, bound 80." - caption="Figure 44.2 — the netpoller contrast: 2,000 goroutines parked in one shared epoll with no M held, against a raw blocking read that pins one M per blocked goroutine" %} + caption="Figure 44.2: the netpoller contrast: 2,000 goroutines parked in one shared epoll with no M held, against a raw blocking read that pins one M per blocked goroutine" %} The `netpoll` theme measures exactly this: 2,000 `os.Pipe()` pairs, one goroutine per pipe blocked on a `Read` of the unwritten end. @@ -265,7 +265,7 @@ goroutine per pipe blocked on a `Read` of the unwritten end. } ``` -Thread count is read from `/proc/self/status`'s own `Threads:` field — an +Thread count is read from `/proc/self/status`'s own `Threads:` field: an OS-level cross-check independent of anything the Go runtime claims about itself: @@ -302,7 +302,7 @@ func readThreadCount() int { This session: `netpoll: baseline goroutines=1 threads=12` before the fan-out, then `netpoll: parked goroutines=2000 baseline=1 total=2001` and -`netpoll: parked threads=12 bound=80 threads<=bound ok` after it — 2,000 +`netpoll: parked threads=12 bound=80 threads<=bound ok` after it: 2,000 more live goroutines, *zero* additional threads, well inside a bound (`4*GOMAXPROCS+16 = 80`) set orders of magnitude below 2,000 so it only fails if something regresses to one-thread-per-blocked-goroutine. Go 1.26 @@ -311,7 +311,7 @@ status. A P whose M was in a syscall used to carry that status directly; as of 1.26, `pp.status` only ever takes `_Pidle`/`_Prunning`/`_Pgcstop`/`_Pdead`, and whether a P's M is really in a syscall is read off the **goroutine's** status instead -(`setBlockOnExitSyscall`) — the release notes attribute a ~30% cut in +(`setBlockOnExitSyscall`); the release notes attribute a ~30% cut in baseline cgo-call overhead to this, a narrower change than the still-open proposal for a second dedicated syscall thread per M (golang/go#58492). @@ -319,7 +319,7 @@ proposal for a second dedicated syscall thread per M (golang/go#58492). Every theme follows the same shape: compute something, print a fixed-text comparison label with the measured numbers substituted in, and -end the line with a verdict word the program itself decides — +end the line with a verdict word the program itself decides: `fmt.Printf("... %s\n", verdict(cond))`, where `verdict` renders `"ok"` or `"FAIL"` from a boolean. `verify.lua` matches the label text plus the literal `ok`; if the condition flips, `FAIL` prints instead and the @@ -339,31 +339,31 @@ clock. With only one language, "three ways" means three moments an error surfaces, not three languages producing the same one. **A computed -assertion**: every `