diff --git a/README.md b/README.md index 36f5076..abacfbf 100644 --- a/README.md +++ b/README.md @@ -11,8 +11,8 @@ example doubles as a standalone demo. `systems-peer`) for anything privileged, kernel-observing, or two-host; Podman LGTM stack (Grafana/Loki/Tempo/Mimir) + Performance Co-Pilot for observability chapters. -- **eBPF scope**: tooling only (bcc-tools, bpftrace, bpftool) — we observe our - userspace programs; we do not write kernel eBPF. +- **eBPF scope**: tooling only (bcc-tools, bpftrace, bpftool); we observe our + userspace programs and do not write kernel eBPF. ## Local development @@ -24,7 +24,7 @@ bundle exec jekyll serve --baseurl "" # http://127.0.0.1:4000/ ## Verification model CI validates site integrity and that examples *build*. It cannot run VM, root, -or observability demos — the authoritative gate is a host run of +or observability demos: the authoritative gate is a host run of `scripts/test-all-examples.py --mode all` with the lab up. A demo is marked **verified** only when it produced its claimed observable effect, never merely because it compiled. @@ -33,12 +33,12 @@ because it compiled. This project is dual-licensed to separate the code from the writing: -- **Code** — everything under `examples/`, plus build and helper scripts and - inline code snippets — is licensed under the **Apache License 2.0** +- **Code**: everything under `examples/`, plus build and helper scripts and + inline code snippets, is licensed under the **Apache License 2.0** ([`LICENSE`](LICENSE)). -- **Content** — the tutorial and book prose (chapters under `_docs/`), +- **Content**: the tutorial and book prose (chapters under `_docs/`), diagrams and images under `assets/`, and the slide decks under - `presentation/` — is licensed under **Creative Commons Attribution 4.0 + `presentation/`, is licensed under **Creative Commons Attribution 4.0 International (CC BY 4.0)** ([`LICENSE-docs.md`](LICENSE-docs.md)). Copyright 2026 PatternCatalyst (Robert Sedor). diff --git a/_docs/00-outline.md b/_docs/00-outline.md index a819fb3..48450df 100644 --- a/_docs/00-outline.md +++ b/_docs/00-outline.md @@ -13,7 +13,7 @@ switchable code tabs, and every example doubles as a standalone demo. {% include excalidraw.html file="00-course-map" - alt="Course map: four journey stages - Foundations, Core systems, Operating context, and Mastery - shown as bands containing the fourteen parts with their chapter ranges, and the six artifacts fwatch, shmkv, chatterd, pmon, bugfarm, and sysagent along the bottom, each with an arrow to the part where it grows." + alt="Course map: four stages, Foundations, Core systems, Operating context, and Mastery, shown as bands containing the fourteen parts with their chapter ranges, and the six artifacts fwatch, shmkv, chatterd, pmon, bugfarm, and sysagent along the bottom, each with an arrow to the part where it grows." caption="Figure 0.1: the course at a glance." %} Two threads run through the whole book and never stop: @@ -63,8 +63,8 @@ artifacts that grow chapter by chapter: Examples run **locally** on your Fedora 44 machine unless they need root, kernel observation, or a second host; those run in the disposable KVM lab -(`systems-target`, and `systems-peer` for two-host networking) that Chapter 2 -provisions. Observability chapters use a local Podman LGTM stack (Grafana, +(`systems-target`, and `systems-peer` for two-host networking) that the lab +setup provisions. Observability chapters use a local Podman LGTM stack (Grafana, Loki, Tempo, Mimir) plus Performance Co-Pilot from its container image. Nothing experimental ever runs un-snapshotted on your workstation. diff --git a/_docs/01-prerequisites.md b/_docs/01-prerequisites.md index e89897a..b1e85f6 100644 --- a/_docs/01-prerequisites.md +++ b/_docs/01-prerequisites.md @@ -13,8 +13,8 @@ book was written against, which is why this chapter is less "install these packages" and more "here is *why* each pin exists, and here is one script that tells you whether your host matches." By the end you will have run the book's first program, `examples/01-hello-syscalls`, in all three languages, and -cross-checked with `strace` that each of them really made the syscalls the -source code claims. +cross-checked with `strace` that each of them made the syscalls the source +code claims. The code is in `examples/01-hello-syscalls/`. `./demo.sh` there builds and runs all three implementations; its `README.md` covers the run-script contract @@ -81,7 +81,7 @@ per run; this one accumulates every result into a table and exits non-zero only at the end, so one pass gives you the full shopping list. Second, it distinguishes **hard requirements** (`fail`, which gates the exit code) from **soft** ones (`warn`, informational). That split encodes what the book -actually needs versus what is merely convenient. +needs versus what is merely convenient. The hard requirements, and the reason each is hard: @@ -299,7 +299,7 @@ are `CStr`-like views, and `to_string_lossy()` handles the (theoretical) non-UTF-8 case instead of panicking. Three languages, three defaults: C++ makes you construct the error discipline yourself, Go hands you an error value and conventions, and Rust moves part of the manpage into the type system. -Chapter 5 turns this comparison into policy. +This comparison is the seed of the book's errors-three-ways policy. ## Build, run, observe @@ -383,7 +383,7 @@ step ever requires the IDE. - Every example follows one `demo.sh` contract: `build`, `run`, and `TARGET=` for deploy-to-guest. It is *verified* only by its observable output, which `verify.lua` pattern-matches. -- `strace` confirms what a binary actually asks the kernel for, and already +- `strace` confirms what a binary asks the kernel for, and already reveals runtime differences: 64 syscalls for the C++ hello, 217 for Go's. Next, the **KVM lab**: provisioning the `systems-target` and `systems-peer` diff --git a/_docs/02-lab-setup.md b/_docs/02-lab-setup.md index ccf60e0..e8ebf1a 100644 --- a/_docs/02-lab-setup.md +++ b/_docs/02-lab-setup.md @@ -6,8 +6,9 @@ description: "Provision two disposable Fedora KVM guests with cloud-init: an obs duration: 45 minutes --- -Chapter 1 proved that three toolchains build and run the same program on your -workstation. This chapter builds the other half of the book's infrastructure: +The hello-syscalls example already proved that three toolchains build and run +the same program on your workstation. This chapter builds the other half of +the book's infrastructure: two disposable Fedora VMs (`systems-target` and `systems-peer`) that absorb everything you should never do to the machine you work on. A systems-programming book keeps asking you to run code as root, watch the kernel from the inside, @@ -36,7 +37,7 @@ a path on your workstation is a bad afternoon. **Kernel observation** (bpftrace one-liners, perf profiles, the BCC tools) attaches instrumentation to a live kernel, and you want that kernel to be one you can reboot without losing your editor. **Destructive demos** (OOM kills, disk-full behavior, broken network -configs) only teach if you actually break something, which +configs) only teach if you break something, which means you need something safe to break. A VM answers all three at once, and adds the property no container gives you: @@ -44,7 +45,7 @@ A VM answers all three at once, and adds the property no container gives you: you kernel-version drift, can't be probed as an isolated kernel, and can't be snapshot-reverted with its kernel state. The lab guests run their own `6.19.10-300.fc44` kernel while the host runs `7.1`, a visible -difference this chapter turns into its cross-check. +difference used below as a cross-check. That said, the VM is not the default. The book's decision rule, applied at the top of every example: @@ -148,8 +149,8 @@ base keeps provisioning fast and the snapshot meaningful. *and* disk, because the guest is running, so a revert restores it *running*, tools warm, in seconds. The discipline has two halves: -- **Snapshot immediately after provisioning** (and after any deliberate - addition you want to keep). This is your known-good tooled baseline. +- **Snapshot immediately after provisioning** (and after any addition you + want to keep). This is your known-good tooled baseline. - **Revert after every destructive demo**: `scripts/lab/revert-vm.sh systems-target` puts the guest back to `lab-ready`, no forensics, no "did that experiment leave state behind?" doubt. @@ -291,8 +292,8 @@ lab: built once, reverted forever. ## Cross-check: prove the code runs *there* -Chapter 1's example prints the kernel it runs on, which makes it a perfect -independent check that `deploy-to-vm.sh` really executes on the guest rather +The hello-syscalls example prints the kernel it runs on, which makes it a +perfect independent check that `deploy-to-vm.sh` executes on the guest rather than, say, quietly running the local binary. Run it both ways: ```bash diff --git a/_docs/03-observability-stack.md b/_docs/03-observability-stack.md index ce5d091..d49198d 100644 --- a/_docs/03-observability-stack.md +++ b/_docs/03-observability-stack.md @@ -46,8 +46,8 @@ same tools (`curl`, `podman ps`) you would use against the real thing, is a skill this book leans on again when the lab VMs and later the page cache force the same kind of judgment call. -Chapter 1 gave you three toolchains; Chapter 2 gave you two disposable VMs. -This chapter adds the last piece of the workbench: somewhere for telemetry to +The prerequisites gave you three toolchains; lab setup gave you two disposable +VMs. This chapter adds the last piece of the workbench: somewhere for telemetry to *go*. Many chapters in this book emit metrics, traces, or logs (a syscall counter here, a request span there), and printing them to a terminal only gets you so far. What you want is a real backend you can query, graph, and @@ -57,8 +57,8 @@ image, run rootless under Podman, listening on exactly two ports you will memorize: **4318** for sending and **3000** for looking. The infrastructure is in `examples/_infra/`: one compose file and a README. -There is no per-chapter example here; every telemetry-emitting example from -Chapter 38 onward depends on what you stand up now. +There is no per-chapter example here; every telemetry-emitting example, once +OpenTelemetry instrumentation begins, depends on what you stand up now. {% include excalidraw.html file="03-telemetry-pipeline" @@ -146,7 +146,7 @@ polls Grafana's `/api/health` from inside the container with a generous `start_period`, because first boot unpacks and starts all five services and takes ~30 seconds; the health state is what scripts wait on instead of sleeping. Finally the network gets the fixed name `lsp-observability` so any -future container (the PCP suite in Chapter 45, for instance) can join it by +future container (the PCP suite, for instance) can join it by name rather than by guessing a generated one. ## The prerequisite: the Podman user socket @@ -333,12 +333,12 @@ endpoint, two network paths, both answering. `.122.1`), and `deploy-to-vm.sh` forwards `OTEL_ENDPOINT` through `sudo env` because sudo strips the environment. -The stack sits idle until Chapter 38 instruments real programs in all three -languages against it, and Chapter 45 adds the PCP, Cockpit, and SystemTap -suites alongside. With the toolchains, the lab, and the telemetry pipeline in +The stack sits idle until OpenTelemetry instrumentation puts real programs in +all three languages against it, and the Linux analysis suites add PCP, +Cockpit, and SystemTap alongside. With the toolchains, the lab, and the telemetry pipeline in place, the workbench is complete; next, Foundations begins where everything in this book bottoms out: the syscall boundary and the ABI beneath it. --- -

verified: on the Fedora 44 host: podman.socket enabled, podman compose -f examples/_infra/compose-lgtm.yaml up -d brought lsp-lgtm (otel-lgtm:0.8.1) to healthy, /api/health returned ok, the OTLP HTTP POST to :4318/v1/traces returned 200, and the Tempo query by trace ID through Grafana's datasource proxy returned the span r02-gate-smoke (service lsp-smoke). Not yet exercised: exporting telemetry from inside a lab VM via the 192.168.124.1 gateway; the endpoint-forwarding path is verified only as far as deploy-to-vm.sh running host-built binaries on the guest; a VM-side OTLP export lands with Chapter 38.

+

verified: on the Fedora 44 host: podman.socket enabled, podman compose -f examples/_infra/compose-lgtm.yaml up -d brought lsp-lgtm (otel-lgtm:0.8.1) to healthy, /api/health returned ok, the OTLP HTTP POST to :4318/v1/traces returned 200, and the Tempo query by trace ID through Grafana's datasource proxy returned the span r02-gate-smoke (service lsp-smoke). Not yet exercised: exporting telemetry from inside a lab VM via the 192.168.124.1 gateway; the endpoint-forwarding path is verified only as far as deploy-to-vm.sh running host-built binaries on the guest; a VM-side OTLP export lands with OpenTelemetry instrumentation.

diff --git a/_docs/04-syscalls-and-the-abi.md b/_docs/04-syscalls-and-the-abi.md index abc6b2e..6c758fd 100644 --- a/_docs/04-syscalls-and-the-abi.md +++ b/_docs/04-syscalls-and-the-abi.md @@ -47,10 +47,10 @@ means later chapters on concurrency, I/O multiplexing, and signals can point at a runtime behavior and explain it in terms you already have, instead of treating each language's quirks as unrelated trivia. -Chapter 1 ended on a number it did not explain: the C++ hello made 64 syscalls, -the Go hello made 217, and we waved at "the runtime" and moved on. This chapter +The prerequisites left a number unexplained: the C++ hello made 64 syscalls, +the Go hello made 217, and "the runtime" took the blame without proof. This chapter is where that debt gets paid. With the lab and observability stack built, Part -1 starts at the only interface that actually matters in this book, the +1 starts at the only interface that matters in this book, the user/kernel boundary, and answers three questions precisely: what *is* a system call at the instruction level, why do some libc calls never reach the kernel at all (the vDSO), and what path does each of our three languages take @@ -314,8 +314,8 @@ unix.EOPNOTSUPP)` still matches through the added context. rustix never materializes a global at all: the negative `rax` *is* the `Err(Errno)`, and the fallback triage becomes a match arm: `Err(Errno::OPNOTSUPP | Errno::ISDIR | Errno::INVAL)`. Same kernel fact, -three idioms; Chapter 5 turns this three-way comparison into the book's -error-handling policy. +three idioms; this three-way comparison becomes the book's error-handling +policy. ### Concurrency lens @@ -330,7 +330,7 @@ C++ and Rust stay a single thread from `execve` to exit, which is exactly why their traces need no `-f`. The portable lesson is the `EINTR` loops: once any process has signal handlers (as Go's runtime just demonstrated), *every* blocking syscall can return early, so retry-on-`EINTR` is not -defensive decoration but the baseline discipline; Chapter 12 (signals) +defensive decoration but the baseline discipline; safe signal handling makes it precise. ## Build, run, observe @@ -382,7 +382,7 @@ distinct boundary signatures: none of them wrong, all of them visible. ## Cross-check: counting every trap -Chapter 1 counted 64 syscalls for the C++ hello and 217 for Go's. `strace +The prerequisites counted 64 syscalls for the C++ hello and 217 for Go's. `strace -c` on the three `sysprobe` binaries (`-f` for Go) deepens that finding with a full accounting; trimmed to the key rows from this session's runs: @@ -395,8 +395,8 @@ rust : 73 total — 12 mmap, 5 openat, 6 write, 2 getrandom, 1 nanosleep ``` The program's own work is four steps in every case; the rest is runtime -overhead, and the ratio matches Chapter 1's almost exactly (roughly 4× for -Go, near-parity for C++ and Rust). C++'s 21 `mmap`s are the dynamic loader +overhead, and the ratio almost exactly matches what the prerequisites +counted (roughly 4× for Go, near-parity for C++ and Rust). C++'s 21 `mmap`s are the dynamic loader mapping four libraries; Rust's 73 is C++-shaped because its *startup* still goes through glibc even though its syscalls do not. Go's 283 is dominated by runtime self-management: signals, threads, and its sleeping monitor. As an @@ -405,7 +405,7 @@ instructions) confirms *where* each binary keeps its boundary crossings: nowhere (glibc's), everywhere (a static runtime's), and inline at 32 rustix call sites. Two tools, one story. -

unverified: On the lab VM: the kernel-side view of this chapter (watching sys_enter_* tracepoints fire with bpftrace as sysprobe runs, counting traps from inside the kernel instead of via ptrace) needs root and a disposable kernel, so it is not run here; chapter 30 (Debugging part) exercises exactly this on systems-target.

+

unverified: On the lab VM: the kernel-side view of this chapter (watching sys_enter_* tracepoints fire with bpftrace as sysprobe runs, counting traps from inside the kernel instead of via ptrace) needs root and a disposable kernel, so it is not run here; the eBPF observation toolkit exercises exactly this on systems-target.

## What you learned @@ -421,7 +421,7 @@ call sites. Two tools, one story. leaves a distinct fingerprint in `strace`. - `strace -c` puts numbers on runtime overhead: 70 (C++) vs 283 (Go, all threads) vs 73 (Rust) traps for four steps of real work, the same shape - as Chapter 1's 64-vs-217, now fully attributed. + as the prerequisites' 64-vs-217, now fully attributed. Next, **errors, three ways**: the `-errno` fact each wrapper just handed us becomes a policy (`std::expected`, wrapped Go errors, and `thiserror`-style diff --git a/_docs/05-errors-three-ways.md b/_docs/05-errors-three-ways.md index d0bf399..e3b2634 100644 --- a/_docs/05-errors-three-ways.md +++ b/_docs/05-errors-three-ways.md @@ -39,15 +39,16 @@ type systems arguing about what to do with it, and one shared contract irrelevant from the outside. That contract is not local to this chapter. Every program in the rest of the -book, the concurrent hasher in Chapter 6, the file watcher built across Part -2, the benchmark in Chapter 8, inherits the EINTR-retry and short-write-resume +book, the concurrent hasher built three ways, the file +watcher built across Part 2, the page-cache durability benchmark, +inherits the EINTR-retry and short-write-resume rules established here, and every one of them reaches for the same values-versus-exceptions line when a new failure mode shows up. Getting the taxonomy right once, with a program small enough to read start to finish, is what lets later chapters spend their words on new material instead of re-litigating what a correct error policy looks like. -Chapter 4 walked a syscall down to the boundary and back, and ended on the +Syscalls and the ABI walked a syscall down to the boundary and back, and ended on the detail this chapter is built from: when a syscall fails, the kernel does not throw, log, or explain: it puts a small negative integer in a register and returns. Everything your language calls an "error" is constructed, in user @@ -357,8 +358,8 @@ return means EOF, `write_all` per chunk, then a checked close) but each language spends its effort differently. **C++** builds the discipline out of `std::expected`: every fallible helper returns `std::expected`, `last_error()` snapshots `errno` into -`std::generic_category()` (the category for POSIX errno values; Chapter 1's -`uname` wrapper used `system_category`; for Linux errno they render the +`std::generic_category()` (the category for POSIX errno values; the +prerequisites' `uname` wrapper used `system_category`; for Linux errno they render the same, and the book standardizes on `generic_category` from here on), and the move-only RAII `Fd` guarantees no descriptor outlives its owner. The asymmetry between `dst->close()` (explicit, `[[nodiscard]]`, result mapped @@ -401,14 +402,14 @@ typed enum earns its boilerplate. ### Concurrency lens -`copyx` is deliberately single-threaded (one loop, one buffer, no shared +`copyx` is single-threaded (one loop, one buffer, no shared state), but the error machinery already matters for concurrency. C's `errno` is thread-local precisely so two threads failing simultaneously do not corrupt each other's error state, and `last_error()` copies it into a value *immediately* because the very next libc call on the same thread may overwrite it. Go and Rust sidestep the global entirely: the errno arrives as a return value, so there is nothing thread-local to protect. And the EINTR -policy is really a concurrency policy in disguise: signals are asynchronous +policy is a concurrency policy in disguise: signals are asynchronous interruptions, and a program whose I/O loops retry EINTR is one whose correctness does not depend on when a signal lands. That property gets load-tested in the Processes, Signals, Privilege part, when `pmon` starts @@ -485,7 +486,7 @@ syscall, one identical message, one identical `+++ exited with 3 +++`. > the aggregate view of this chapter (*which* errnos your whole system's > syscalls are failing with, live) belongs to eBPF tooling in the class of > bpftrace one-liners counting negative syscall returns by errno. That is not -> runnable on this host without privileges; Chapter 30 (Debugging part) +> runnable on this host without privileges; the eBPF observation toolkit > exercises it in the `systems-target` VM against these same binaries. ## What you learned @@ -513,4 +514,4 @@ and scoped threads under `Send`/`Sync`. --- -

verified: all evidence in this chapter was produced on the Fedora 44 host this session: each of the three binaries printed copied 200000 bytes (exit 0) with cmp exiting 0, copyx: nope.bin: No such file or directory (exit 2), and copyx: write /dev/full: No space left on device (exit 3); the strace excerpt is real output showing write(4, …, 65536) = -1 ENOSPC before the exit-3; the Go trace showed the failing write on descriptor 5 with cpu.max held on 3, and the Rust trace showed the fragmented stderr writes; ls -li /dev/full showed the 1, 7 character device; and scripts/test-all-examples.py --only 05-errors-three-ways reported PASS for cpp, go, and rust (3 passed, 0 failed, 0 skipped). The lab-VM eBPF callout above is the one unverified item, deferred to Chapter 30.

+

verified: all evidence in this chapter was produced on the Fedora 44 host this session: each of the three binaries printed copied 200000 bytes (exit 0) with cmp exiting 0, copyx: nope.bin: No such file or directory (exit 2), and copyx: write /dev/full: No space left on device (exit 3); the strace excerpt is real output showing write(4, …, 65536) = -1 ENOSPC before the exit-3; the Go trace showed the failing write on descriptor 5 with cpu.max held on 3, and the Rust trace showed the fragmented stderr writes; ls -li /dev/full showed the 1, 7 character device; and scripts/test-all-examples.py --only 05-errors-three-ways reported PASS for cpp, go, and rust (3 passed, 0 failed, 0 skipped). The lab-VM eBPF callout above is the one unverified item, deferred to the eBPF observation toolkit.

diff --git a/_docs/06-concurrency-three-ways.md b/_docs/06-concurrency-three-ways.md index e279355..00324e7 100644 --- a/_docs/06-concurrency-three-ways.md +++ b/_docs/06-concurrency-three-ways.md @@ -42,28 +42,27 @@ instead of rhetorical: the same four-worker pool, three times, with `/proc//task` proving that underneath the vocabulary it is always `clone(2)` threads parked on `futex(2)`. -This chapter is also where the book's two permanent threads meet: Chapter 5's -error policy has to survive concurrent execution, and the cancellation drain +This chapter is also where the book's two permanent threads meet: the +error-handling policy has to survive concurrent execution, and the cancellation drain built here, stop accepting, finish what you hold, report what completed, is the pattern every later concurrent example reuses, from the signal handling -in Part 7 to the event loops in Chapter 27. Understanding *why* each language +in Part 7 to async runtimes and coroutines. Understanding *why* each language reaches for its own idiom here, rather than just copying the syntax, is what makes those later chapters readable at a glance instead of a fresh puzzle each time. -Chapter 5 pinned down the first of this book's two permanent threads: how each +The error-handling policy pinned down the first of this book's two permanent threads: how each language turns an `errno` into something a caller can act on. This chapter pins down the second, how each language *shares work*, and it does it the only way that is fair: one program, `parhash`, written three times with each language's native concurrency toolkit, required to produce **byte-identical stdout**. Underneath, all three are the same thing: kernel threads created -with `clone(2)`, parked and woken with `futex(2)`, so what you are really +with `clone(2)`, parked and woken with `futex(2)`, so what you are comparing is vocabulary: what C++ hands you as parts, what Go's runtime takes away from you, and what Rust's compiler refuses to let you get wrong. One scope note up front: everything here is *thread* concurrency. Async (C++ -coroutines, tokio, and what the Go scheduler looks like from below) is -deferred to Chapter 27, after you have seen the event loops that -motivate it. +coroutines, tokio, and what the Go scheduler looks like from below) comes +later, after you have seen the event loops that motivate it. The code is in `examples/06-concurrency-three-ways/`. `./demo.sh` builds and runs all three implementations; its `README.md` covers the shared contract and @@ -108,7 +107,8 @@ are multiplexed by the runtime onto a small set of kernel threads (M), each of which needs a *processor* slot (P, capped by `GOMAXPROCS`, 16 on this host) to run Go code; when a goroutine blocks in a syscall its M can hand the P to another thread, and idle Ps steal runnable Gs from busy ones. That is the -entire mental model this book needs until Chapter 27. What you program against +entire mental model this book needs until it turns to async runtimes and +coroutines. What you program against is not the scheduler but the idiom on top: goroutines communicate over channels, `select` races multiple channel operations, and cancellation is a `context.Context` whose `Done()` channel closes: one convention so universal @@ -291,11 +291,11 @@ iterator ends precisely when the last worker (holding the last sender) exits. Cancellation is the bluntest of the three: `signal_hook::flag::register` points SIGINT at a shared `AtomicBool`, and walker and workers poll it with `Ordering::Relaxed`: a plain flag needs no ordering guarantees, a point -Chapter 26 returns to properly. +taken up again under atomics and lock-free programming. ### Errors, three ways -The per-file hash function carries this book's Chapter 5 policy into +The per-file hash function carries this book's error-handling policy into concurrent code. C++'s `hash_file` returns `std::expected` and is the only one of the three that must handle `EINTR` by hand: its raw `::read` loop retries on @@ -318,7 +318,7 @@ between you and the kernel. The Rust model costs you a negotiation with the compiler up front and repays it by making the forgotten-mutex bug unrepresentable. From here on, each example states which model it leans on and why; when the answer becomes "none of these: the problem wants an event -loop," that is Chapter 27's cue. +loop," that is the cue to reach for async runtimes and coroutines. ## Build, run, observe @@ -415,8 +415,8 @@ program sorts at the end instead of trusting the walk. > bcc-tools would let you watch the drain from the kernel's side: > `offcputime` showing the idle workers parked in futex waits, `exitsnoop` > catching the thread exits as they cascade. Those tools are not runnable on -> this host without root; Chapter 30 (Debugging part) builds that toolkit on -> the `systems-target` VM and points it at these exact binaries. +> this host without root; the eBPF observation toolkit, built on +> the `systems-target` VM, points it at these exact binaries. One asymmetry here: Go ran every check under its race detector, but the C++ pool ran with no equivalent referee (this example's CMake presets @@ -424,8 +424,8 @@ stop at `asan`, and no ThreadSanitizer build was exercised here). unverified: the host carries the `libtsan` runtime (`scripts/check-host.sh` checks for it) and a TSan probe links and runs, but running *this* worker pool under TSan, and what its -reports look like with the mutex removed, is Chapter 29's -job. +reports look like with the mutex removed, is left to valgrind, sanitizers, +and miri. ## What you learned diff --git a/_docs/07-fds-raii-and-the-vfs.md b/_docs/07-fds-raii-and-the-vfs.md index adbb2d4..cd1d863 100644 --- a/_docs/07-fds-raii-and-the-vfs.md +++ b/_docs/07-fds-raii-and-the-vfs.md @@ -28,7 +28,7 @@ By the end of this chapter you can: ## Why this matters Most bugs involving file descriptors come from treating "the fd" as a single -thing when it is really three: a slot in your process, a shared kernel +thing when it is three: a slot in your process, a shared kernel object, and the file itself. Conflating them is how a `dup`'d descriptor surprises you by sharing an offset, how a privileged fd leaks across `exec`, or how a deleted file keeps its storage alive because something still holds @@ -145,7 +145,7 @@ you already hold open. Once `fwatch` holds fd 3 on the root of the tree, renaming that tree out from under it changes nothing: the fd pins the inode, not the name. `AT_SYMLINK_NOFOLLOW` completes the discipline for the stat side: classify the entry itself, never what a symlink points at. (`AT_FDCWD`, -which `scan` passes for the very first open, is the pseudo-dirfd meaning +which `scan` passes for the first open, is the pseudo-dirfd meaning "resolve relative to the current working directory", the only path-based resolution in the whole program.) @@ -506,7 +506,7 @@ first arguments: only the root open uses `AT_FDCWD`; every later `openat` and `newfstatat` anchors on fd 3, 4, or 5: the dirfd chain, exactly as Figure 7.1 draws it, with every open carrying `O_CLOEXEC|O_NOFOLLOW|O_DIRECTORY`. The kernel also reports unsorted entry order (`gone.txt` before `a.txt`); -the sorted output really does come from the map. The Rust binary shows one +the sorted output comes from the map. The Rust binary shows one extra `openat(3, ".", …) = 4` per level: `Dir::read_from` acquiring its own cursor handle, just as its comment says. @@ -546,7 +546,7 @@ different tool, all agreeing. > **On the lab VM** unverified: > the eBPF view of this chapter is watching `openat` flags fleet-wide with > `opensnoop`/`bpftrace` instead of per-process `strace`; bcc-tools are not -> runnable on this host without privileges, and chapter 30 (Debugging part) +> runnable on this host without privileges, and the eBPF observation toolkit > exercises exactly that on the `systems-target` VM. ## What you learned @@ -568,7 +568,7 @@ different tool, all agreeing. one printer that formats identical text. Next, **page cache and durability**: what the kernel does with the bytes you -write, why `fwatch`'s one-second mtimes can lie, and what `fsync` actually +write, why `fwatch`'s one-second mtimes can lie, and what `fsync` promises. --- diff --git a/_docs/08-page-cache-and-durability.md b/_docs/08-page-cache-and-durability.md index 6f9ecde..2b4efc4 100644 --- a/_docs/08-page-cache-and-durability.md +++ b/_docs/08-page-cache-and-durability.md @@ -34,13 +34,13 @@ systems programming. A successful `write(2)` return code says only that your bytes reached kernel memory; whether they survive a power loss a second later depends on a sync call you have to ask for explicitly, and getting that choice wrong is how real databases lose real data after passing every test -that never pulled the power cord. This chapter turns that gap from folklore -into a measured quantity: 21 ms to "write" 64 MiB, 138 ms to write it +that never pulled the power cord. That gap is a measured quantity here: +21 ms to "write" 64 MiB, 138 ms to write it durably, watched directly in `/proc/meminfo` and `/proc//io` rather than taken on faith. -It also closes the loop Chapter 7 opened. That chapter gave you a precise -model of what a file descriptor is; this one gives you a precise model of +It also closes a loop: the fd model gave you a precise picture of what a +file descriptor is; this chapter gives you a precise model of what happens to the bytes you push through it, which is the other half of "I wrote a file" that most introductions skip. The durability tools introduced here, `fsync`/`fdatasync` semantics, the atomic-rename pattern, @@ -48,8 +48,8 @@ introduced here, `fsync`/`fdatasync` semantics, the atomic-rename pattern, chapters (event loops, and eventually `fwatch`'s own durability story) build on next. -Chapter 7 ended with a file descriptor safely wrapped in RAII and a byte -successfully written through the VFS. This chapter is about the uncomfortable +You already have a file descriptor safely wrapped in RAII and a byte +successfully written through the VFS. The uncomfortable question that follows: written *where*? A successful `write(2)` puts your bytes in kernel memory (the **page cache**) and nothing more; if the power fails a second later, they are gone, and no error will ever tell you so. The @@ -81,7 +81,7 @@ intuition suggests: the kernel has copied your bytes into page-cache pages belonging to the file, marked those pages **dirty**, and made the data visible to every subsequent `read(2)` from any process. That is all. Durability is explicitly not included (the manpage says so), and the return -value is really just confirming a memcpy into kernel memory. That is why +value is just confirming a memcpy into kernel memory. That is why `iobench --mode buffered` reports 3,013 MiB/s on this host's NVMe drive: the timed loop never touches the device. You are benchmarking `memcpy` plus bookkeeping. @@ -138,7 +138,7 @@ makes each `write(2)` DMA straight from your user buffer to the device: no copy into the page cache, no dirty pages, no writeback. The price is a contract: buffer address, file offset, and transfer length must all be aligned to the device's logical block size (4,096 bytes covers current -hardware), and a filesystem may simply refuse the flag at `open(2)` with +hardware), and a filesystem may refuse the flag at `open(2)` with `EINVAL`, which `iobench` treats as a documented outcome (exit 4), because it varies by filesystem. @@ -157,7 +157,7 @@ torn half, you use **atomic rename**: write the new content to a temporary file in the *same directory*, `fsync` the temp file, `rename(2)` it over the target (rename within a filesystem is atomic), then `fsync` the *directory* fd so the rename itself (a directory entry, i.e. metadata living in the -directory, as chapter 7's inode walk showed) is on disk too. Skip step 2 and +directory, as the inode walk earlier showed) is on disk too. Skip step 2 and a crash can leave you a correctly named empty file; skip step 4 and the old name can come back after reboot. `iobench` does not do this: a benchmark should overwrite in place, but every "save settings" feature you @@ -359,7 +359,7 @@ throughput division: on tmpfs a tiny run can round to 0 ms. ### Errors, three ways -Chapter 5's policy gets its first workout against a syscall that fails in +The error-handling policy gets its first workout against a syscall that fails in interesting ways. The key move in all three languages is *classify before you generalize*: the open error is inspected for the one value with a documented meaning (`EINVAL` in direct mode → message + exit 4) before being @@ -382,8 +382,8 @@ and begin writing them back *concurrently with the loop*; in the cross-check below, a 2 GiB write peaks at only ~1.1 GiB dirty because writeback was draining the pool while the program filled it. The final `fdatasync` is therefore a *rendezvous*, not the whole transfer: it waits for whatever -writeback has not already covered. Language runtimes add one more wrinkle -from chapter 6: a blocking `fdatasync` parks the calling OS thread, so in Go +writeback has not already covered. Language runtimes add one more wrinkle, +already familiar from concurrency three ways: a blocking `fdatasync` parks the calling OS thread, so in Go the runtime hands the P to another thread and other goroutines keep running, one reason "just fsync in a goroutine" feels free until you count threads. And timing uses `steady_clock` / `time.Now` monotonic readings / @@ -414,11 +414,11 @@ Read the buffered pair together: 21 ms to "write" 64 MiB, then 17 ms of MiB/s headline was never a device number. Syncing every 8 blocks (512 KiB) drops throughput 6.5× to 462.8 MiB/s because the device's flush latency now sits inside the loop 128 times. `O_DIRECT` lands between them at 785.2 MiB/s: -every byte really moved to the device inside the timed region, with no +every byte moved to the device inside the timed region, with no copy and no cache. Go and Rust agree with the C++ numbers (buffered 3,283.9 and 3,483.6 MiB/s with `fsync_ms=17` and `14` on the same runs); the page cache does not care which language dirtied it. After any run, -`stat -c %s bench.bin` reports exactly 67108864: same bytes, very different +`stat -c %s bench.bin` reports exactly 67108864: same bytes, different guarantees. ## Cross-check: watching the bytes move @@ -479,7 +479,7 @@ hit/miss counts while iobench runs) and biolatency (the block I/O latency histogram that separates fsync-every's many small flushes from direct mode's steady stream) would show the kernel side of these runs. They need root and a kernel with BPF tooling, so they are not run on this -host; chapter 30 (Debugging part) exercises them on the lab VM.

+host; the eBPF observation toolkit exercises them on the lab VM.

## What you learned diff --git a/_docs/09-event-loops-epoll-inotify.md b/_docs/09-event-loops-epoll-inotify.md index 6716bf1..0625047 100644 --- a/_docs/09-event-loops-epoll-inotify.md +++ b/_docs/09-event-loops-epoll-inotify.md @@ -21,14 +21,14 @@ By the end of this chapter you can: Every scalable network service, file watcher, and terminal multiplexer rests on the same decision: decouple waiting from the work the wait is for. The readiness model is how a single thread tracks thousands of file descriptors without a thread per source, and it is the direct ancestor of the event loops behind editors, sync clients, and the servers Part 6 builds. -This chapter also establishes the comparison that drives the rest of this part: an explicit, visible-in-source epoll loop versus a runtime that hides the same machinery behind channels and goroutines. Chapter 10 builds on this foundation by introducing the completion model, io_uring, as the second half of the readiness-versus-completion pair. +This chapter also establishes the comparison that drives the rest of this part: an explicit, visible-in-source epoll loop versus a runtime that hides the same machinery behind channels and goroutines. io_uring builds on this foundation by introducing the completion model, as the second half of the readiness-versus-completion pair. -Chapter 8 closed on a question durability cannot answer: "which of ten +Page-cache durability closed on a question it cannot answer: "which of ten thousand fds is ready?" This chapter answers it with the **readiness model**: the kernel will tell you *when an fd has something for you*, and a program built on that promise never blocks anywhere except the one call that asks. -`fwatch` v1 keeps the polling-era `snapshot` and `diff` subcommands from -Chapters 7 and 8 and adds `watch`: a real-time watcher that reports +`fwatch` v1 keeps the polling-era `snapshot` and `diff` subcommands built on +RAII-wrapped fds and page-cache durability and adds `watch`: a real-time watcher that reports `created`/`modified`/`deleted` per file, debounced 100 ms per path. The C++ and Rust versions build it the explicit way: one `epoll` loop over three kernel fds (inotify, timerfd, signalfd), a single thread, every step visible @@ -73,7 +73,7 @@ mask (`EPOLLIN`, readable) and a caller-chosen 64-bit cookie interest list lives **in the kernel**, which is the whole advantage over the older `poll(2)`/`select(2)`: those pass the entire fd set on every call, so each wait is O(fds); epoll pays the setup cost once and each wakeup returns -only the fds that are actually ready. `fwatch` has three fds so any of them +only the fds that are ready. `fwatch` has three fds so any of them would work; the point of learning epoll on three is that the loop is *identical* at thirty thousand, which is where Part 6's network servers take it. @@ -85,7 +85,7 @@ performance: **when** does a ready fd count as ready again? - **Level-triggered** (the default): `epoll_wait` reports the fd as long as data *remains*, a fact about the fd's state. Forget to read everything and - the next wait simply reports it again. + the next wait reports it again. - **Edge-triggered** (`EPOLLET`): the fd is reported only on the *transition* to readable, a fact about a state change. Miss data and no further wakeup comes until *new* data arrives; the only correct consumer @@ -97,7 +97,7 @@ The discipline it demands instead is smaller but real: **consume what made the fd readable**. That is why the loop below reads the timerfd's 8-byte expiration counter even though it only wants "the timer fired"; leave those 8 bytes and the timerfd stays readable, `epoll_wait` returns instantly -forever, and your loop spins at 100% CPU. Note that `fwatch` drains its +forever, and your loop spins at 100% CPU. `fwatch` drains its inotify fd to `EAGAIN` anyway: with level triggering that is an optimization (fewer wakeups per burst, and each burst is timestamped coherently); under edge it would be the difference between working and silently stalling. @@ -109,8 +109,8 @@ edge it would be the difference between working and silently stalling. > is mid-read. `EPOLLONESHOT` disarms the registration after each report; > the owning thread must re-arm with `EPOLL_CTL_MOD` when done. It is the > standard fix for multi-threaded epoll, and the reason this book's -> single-threaded loops do not need it. One loop, one thread, no re-arm -> protocol: that simplicity is a feature, not a limitation. +> single-threaded loops do not need it: one loop, one thread, no re-arm +> protocol. Interestingly, the readiness style is observable per implementation: the strace later in this chapter shows Go's netpoller registering our inotify fd @@ -141,9 +141,9 @@ when it fills, the kernel does not block the writer and does not drop silently: it appends one `IN_Q_OVERFLOW` event and discards until you catch up. A correct watcher treats `IN_Q_OVERFLOW` as "my view is stale, re-scan," which is exactly why `fwatch` kept its `snapshot`/`diff` -subcommands: the polling machinery from Chapters 7 and 8 is the recovery -path for the event machinery of Chapter 9. Also note what a directory watch does -*not* give you: recursion. Subdirectories need their own watches, the cost +subcommands: the polling machinery built on RAII-wrapped fds and page-cache +durability is the recovery path for the event machinery here. A directory watch does +*not* give you recursion. Subdirectories need their own watches, the cost model behind editors and sync clients burning thousands of `wd`s. ## Everything is an fd: timerfd and signalfd @@ -187,7 +187,7 @@ reads as `modified`, the same visible truth `diff` would have reported. Setup is four fallible constructions (`inotify_init1` + `inotify_add_watch`, `timerfd_create`, `sigprocmask` + `signalfd`, `epoll_create1` + three -`epoll_ctl` ADDs); every fd owned by Chapter 7's RAII wrapper (`Fd` in C++, +`epoll_ctl` ADDs); every fd owned by the RAII wrapper (`Fd` in C++, `OwnedFd`-backed nix types in Rust, `os.File` in Go). Then the loop; here it is in all three languages, verbatim from `examples/09-event-loops-epoll-inotify/`: @@ -390,7 +390,7 @@ channels and the kernel machinery is rented from the runtime. ### Errors, three ways Setup is where `watch` can fail, and the missing-directory case exercises -each language's Chapter 5 policy end to end. C++ threads four +each language's error-handling policy end to end. C++ threads four `std::expected` constructions together, chaining the three `epoll_ctl` adds with `.and_then(…)` so the first failure short-circuits with its `std::error_code`; the observed result is @@ -413,13 +413,13 @@ three verified below. Count the threads doing *your* work: one in C++, one in Rust, and one goroutine plus `main` in Go, multiplexed by the runtime onto its thread -pool. The single-threaded epoll design is not an accident, it is the -concurrency *strategy*: since only one thread touches `pending`, there are +pool. The single-threaded epoll design is the concurrency *strategy*: since +only one thread touches `pending`, there are no locks, no atomics, and no `EPOLLONESHOT` re-arm protocol; the event loop serializes everything by construction. Go reaches the same freedom-from-locks differently: `pending` is confined to the `select` goroutine, and other -concerns communicate by channel, which is Chapter 6's "share memory by -communicating" applied to I/O. The costs surface in `strace`: idle for 400 ms, +concerns communicate by channel, which is the concurrency-three-ways "share +memory by communicating" idiom applied to I/O. The costs surface in `strace`: idle for 400 ms, the C++ watcher makes exactly **1** `epoll_wait` call (armed once, sleeps, fires at the deadline), while the Go binary makes **10** `epoll_pwait` calls across its threads: the netpoller is consulted by the scheduler, not just @@ -551,8 +551,8 @@ lab VM: the kernel-side view of this loop, watching wakeup sources fire with bpftrace on tracepoint:syscalls:sys_enter_epoll_wait and kprobes on the eventpoll internals, and forcing a real IN_Q_OVERFLOW by shrinking max_queued_events, -needs root and a disposable kernel, so it is not run here; chapter 30 -(Debugging part) exercises exactly this on systems-target.

+needs root and a disposable kernel, so it is not run here; the eBPF +observation toolkit exercises exactly this on systems-target.

## What you learned diff --git a/_docs/10-io-uring.md b/_docs/10-io-uring.md index 8ef757d..d270386 100644 --- a/_docs/10-io-uring.md +++ b/_docs/10-io-uring.md @@ -14,15 +14,15 @@ By the end of this chapter you can: - Construct an io_uring instance with `io_uring_setup`, map the SQ ring, CQ ring, and SQE array via the canonical `mmap`s, and submit batched operations with `io_uring_enter`. - Write and link submission queue entries (`IORING_OP_READ`/`IORING_OP_WRITE`) with `IOSQE_IO_LINK` to express intra-pair ordering without a userspace round trip. - Measure when completion-based I/O helps and when it costs more than a synchronous read/write loop, using `strace -c` syscall counts and wall-clock timings. -- Account for io_uring's security exposure, including why container seccomp profiles deny it by default, and explain Go's deliberate refusal to expose a uring engine. +- Account for io_uring's security exposure, including why container seccomp profiles deny it by default, and explain why Go refuses to expose a uring engine. ## Why this matters Readiness and completion are the two mental models underlying every I/O-heavy system on Linux, and knowing which one a workload needs is a design decision with measurable consequences: this chapter's own numbers show a completion-based engine running roughly ten times slower than a synchronous loop on warm, small files, while the same engine would win decisively against deep queues on real storage or busy sockets. Fewer syscalls is not automatically faster, and this chapter proves it rather than asserting it. -The chapter also surfaces a security tradeoff worth carrying forward: io_uring's opcode-dispatched interface can bypass syscall filters written against `read`/`write`/`connect`, which is why container runtimes deny it by default. That tension between a powerful kernel interface and a shrinking attack surface returns when Part 7 moves these programs into containers. +The chapter also surfaces a security tradeoff worth carrying forward: io_uring's opcode-dispatched interface can bypass syscall filters written against `read`/`write`/`connect`, which is why container runtimes deny it by default. That tension between a broad kernel interface and a shrinking attack surface returns when these programs move into containers. -Chapter 9 ended with `fwatch` blocked in an event loop, and with a promise: +Epoll left `fwatch` blocked in an event loop, and with a promise: epoll would not be the last word on Linux I/O. Epoll answers one question: *which of these fds may I touch without blocking?* Then it leaves every actual `read(2)` and `write(2)` for you to make, one syscall per operation. @@ -33,9 +33,8 @@ Instead it *describes* batches of operations in memory the kernel shares with the process, and submits sixteen of them per syscall. That is the **completion model**, and readiness-versus-completion is the mental-model pair this whole Part has been building toward. The chapter also delivers two -deliberate surprises: our uring engine is measurably *slower* on this -workload, and one of our three languages refuses to play at all. Both are -intended, and both are worth understanding. +surprises: our uring engine is measurably *slower* on this +workload, and one of our three languages refuses to play at all. The code is in `examples/10-io-uring/`. `./demo.sh` there builds and runs all three implementations; its `README.md` covers the `scan`/`watch`/`sync` @@ -53,8 +52,8 @@ subcommands and the two-engine contract. ## Readiness asks "may I?", completion says "it is done" Every I/O interface you have met so far is synchronous at the moment of data -movement: `read(2)` in chapter 7, `write(2)` through the page cache in -chapter 8, and epoll in chapter 9, which, for all its scalability, only +movement: `read(2)` on RAII-wrapped fds, `write(2)` through the page cache, +and epoll, which, for all its scalability, only *schedules* those same calls. Readiness is a hint about the future ("fd 7 will not block right now"), and it can even be wrong: a readable socket can still return `EAGAIN` by the time you get there. @@ -104,13 +103,13 @@ runtimes responded directly: on this host, podman's default seccomp profile and its allowlist contains **zero** mentions of `io_uring_setup`, `io_uring_enter`, or `io_uring_register`: deny by default. Run the uring engine in an unmodified container and `io_uring_setup` fails with `EPERM`. -Chapter 34, when our programs move into containers, deals with that head-on. +Containerizing these programs deals with that head-on. ## Go's deliberate absence `fwatch sync --engine uring` in Go prints -`engine=uring: unsupported in Go (see chapter 10)` and exits 64. That is not -laziness: it is the chapter's design lesson. Go already solved "many I/O +`engine=uring: unsupported in Go (see chapter 10)` and exits 64: the chapter's +design lesson. Go already solved "many I/O operations, few OS threads" once, inside the runtime: every goroutine writes ordinary blocking code, and the **netpoller** multiplexes those goroutines over epoll, parking them until readiness arrives. The concurrency model *is* @@ -414,7 +413,7 @@ contract cannot express the engine at all. ### Concurrency lens Strip away the I/O and Figure 10.1 is a pair of single-producer, -single-consumer lock-free ring buffers: the kernel is simply *the other +single-consumer lock-free ring buffers: the kernel is *the other thread*. That is why `try_get_sqe`/`submit_and_wait` are written in acquire/release vocabulary: the store-release of the SQ tail publishes the filled SQEs, the kernel's load-acquire observes them, and the mirror-image @@ -426,7 +425,7 @@ not get to declare `std::atomic`. Meanwhile the kernel side is concurrent too: `IORING_FEAT_NATIVE_WORKERS` in our trace is the io-wq worker-thread pool, which executes any submitted operation that cannot complete immediately, our buffered file writes among them, a fact that returns in -the timings. Chapter 26 rebuilds this exact structure in user space, barrier +the timings. Lock-free atomics rebuild this exact structure in user space, barrier by barrier. ## Build, run, observe @@ -448,8 +447,8 @@ cache, `ms=` as printed by the tool: | `uring` | 129, 137, 130 ms | 105, 107, 107 ms | exit 64 (by design) | On this workload, our uring engine is roughly **ten times -slower**. Nothing is broken: the copies are byte-identical (checked below). -The model just costs more than it saves here. Every file is page-cache +slower**. The copies are byte-identical (checked below); the model just costs +more than it saves here. Every file is page-cache hot, so each `read(2)` in the rw loop is a short memcpy; the syscalls we so cleverly eliminated were never the bottleneck. The uring path, meanwhile, pays for what buffered file I/O cannot do asynchronously inline: the @@ -457,7 +456,7 @@ kernel punts those operations to io-wq worker threads, buying context switches and wakeups per batch, and our engine waits for each batch to fully retire before building the next, with unregistered buffers reallocated per file. This is the batch-size lesson the chapter owes you: completion -wins when queues are deep, operations are actually asynchronous, and fixed +wins when queues are deep, operations are asynchronous, and fixed costs get amortized across real waiting, not when the "I/O" is a memcpy that was already cheaper than the bookkeeping. On the small verify fixture (5 files, 1,348,634 bytes) the gap shrinks to noise: `rw` printed `ms=0`, @@ -508,8 +507,8 @@ claim above is a one-liner you can run yourself: `grep -c io_uring > the natural next probe is watching io-wq from the inside, > `bpftrace -e 'tracepoint:io_uring:io_uring_submit_req { @[comm] = count(); }'` > while the two engines run. bcc/bpftrace are not part of this host's -> toolchain by design; chapter 30 (Debugging part) builds that eBPF -> observation kit on the lab VM and exercises exactly this. +> toolchain by design; the eBPF observation toolkit covers that setup +> on the lab VM and exercises exactly this. ## What you learned @@ -526,7 +525,8 @@ claim above is a one-liner you can run yourself: `grep -c io_uring small and fully synchronous. Completion pays off at depth, on real waiting. - io_uring's power is why container seccomp profiles deny it by default - (podman's allowlist has no io_uring entries on this host; chapter 34), + (podman's allowlist has no io_uring entries on this host, confirmed when + these programs move into containers), and why Go declines to expose it at all: the netpoller already owns the event loop (golang/go#31908 stays on hold). diff --git a/_docs/11-process-lifecycle.md b/_docs/11-process-lifecycle.md index 6beae7a..755241a 100644 --- a/_docs/11-process-lifecycle.md +++ b/_docs/11-process-lifecycle.md @@ -10,7 +10,7 @@ duration: "45 minutes" By the end of this chapter you can: -- Trace a process from `fork(2)` through `execve(2)` to `wait4(2)`/`waitid(2)`, and explain what copy-on-write actually copies. +- Trace a process from `fork(2)` through `execve(2)` to `wait4(2)`/`waitid(2)`, and explain what copy-on-write copies. - Build a `O_CLOEXEC` self-pipe that lets a parent distinguish "exec failed" from "the program ran and exited." - Observe a zombie process live in `ps` and `/proc//stat`, and explain why a parent must reap it before its pid can be reused. - Distinguish exit codes from termination signals (`WIFEXITED`/`WEXITSTATUS` versus `WIFSIGNALED`/`WTERMSIG`) and apply the shell's `128+n` convention. @@ -21,7 +21,7 @@ By the end of this chapter you can: Fork, exec, and wait are the three syscalls underneath every process a Linux system runs, from a shell pipeline to a container runtime launching an entry point. Getting the gap between fork and exec right, and reaping promptly, is the difference between a supervisor that works and one that silently leaks zombies or orphans until `pid_max` runs out. -This chapter opens `pmon`, the process monitor that carries through the rest of this part: signal handling in Chapter 12, pidfd-based supervision in Chapter 13, and privilege dropping in Chapter 14 all build directly on the spawn-and-reap discipline established here. +This chapter opens `pmon`, the process monitor that carries through the rest of this part: safe signal handling, pidfd-based supervision, and privilege dropping all build directly on the spawn-and-reap discipline established here. This chapter opens the book's third part with a new long-running artifact: `pmon`, a process monitor that will grow signal forwarding, pidfd-based @@ -33,9 +33,9 @@ it**. Between a child's last instruction and the parent's `wait`, the kernel holds the corpse (exit status, resource usage, a pid that cannot be reused) as a *zombie*, and in this chapter you will create one, watch it in `ps`, read its `/proc` entry field by field, and then reap it. Along the -way we take apart what `fork(2)` really copies (almost nothing, thanks to +way we take apart what `fork(2)` copies (almost nothing, thanks to copy-on-write), why exec failure needs a back channel, and what Go's -`os/exec` and Rust's `Command` are actually doing under their tidy APIs: the +`os/exec` and Rust's `Command` are doing under their tidy APIs: the answer, straight from `strace`, involves three different flavors of `clone`. The code is in `examples/11-process-lifecycle/`. `./demo.sh` there builds all @@ -87,8 +87,8 @@ parent until the exec, which is fork with the copy amortized to zero. One hole remains in the two-step model: if `execvp` itself fails (the command does not exist), the failure happens *in the child*, after fork, and the parent has no direct way to distinguish "exec failed" from "the program -ran and exited". pmon closes the hole with a **self-pipe**, and chapter 7's -`O_CLOEXEC` discipline is what makes it work. Both pipe ends are opened +ran and exited". pmon closes the hole with a **self-pipe**, and the +`O_CLOEXEC` discipline from RAII-wrapped fds is what makes it work. Both pipe ends are opened close-on-exec, from `examples/11-process-lifecycle/cpp/src/main.cpp`: ```cpp @@ -111,13 +111,13 @@ struct ExecPipe { If exec succeeds, the kernel closes the child's write end atomically at exec time and the parent's `read` returns 0 (EOF, no child code required). If exec fails, the child writes the raw `errno` into the pipe and `_exit(127)`s. The -parent reads either nothing or four bytes that name the exact failure. This -trick is not pedagogy: the cross-check shows Go and Rust doing precisely the +parent reads either nothing or four bytes that name the exact failure. The +cross-check shows Go and Rust doing precisely the same dance inside their standard libraries. ## Zombies, orphans, and the duty to reap -When a process exits, the kernel cannot simply delete it: the parent may not +When a process exits, the kernel cannot delete it: the parent may not have asked yet how it died, and the pid must not be recycled while an answer is owed. So the address space, fds, and threads are freed immediately, but the task record (pid, exit status, rusage) stays as a **zombie** (`Z` in @@ -325,7 +325,7 @@ Walk the C++ tab first, because everything else is a compressed version of it. `steady_clock::now()` is taken *before* the fork so wall time covers the child's whole life, and steady (not system) time so an NTP step cannot make a child appear to run backwards. `fork` splits the world; the child branch -resets its copy of the read end (chapter 7's move-only `Fd` makes "close +resets its copy of the read end (the move-only `Fd` makes "close exactly once" a type property, and `ExecPipe` is just two of them), calls `execvp` (the `p` variant walks `PATH` like a shell) and, only if exec *returned*, ships `errno` up the pipe and dies with `_exit(127)`. `_exit`, @@ -386,8 +386,8 @@ The two `SAFETY` comments are the whole cost of dropping below `std::process`: a zeroed `rusage` is a valid out-parameter, and the two `&mut`s are exclusive for the duration of the call. `nix` decodes the raw status into a typed `WaitStatus`, giving the `match` in `monitor` the same -two-armed shape as `WIFEXITED`/`WIFSIGNALED`. Note the deliberate asymmetry -with C++: `RUSAGE_CHILDREN` aggregates every reaped child, `wait4` meters +two-armed shape as `WIFEXITED`/`WIFSIGNALED`. The asymmetry +with C++ is deliberate: `RUSAGE_CHILDREN` aggregates every reaped child, `wait4` meters one; identical numbers here, different interfaces, and the distinction will matter the moment pmon supervises more than one process. @@ -448,7 +448,7 @@ so `os/exec` confines the child to a tiny fixed path of raw syscalls, and the strace shows the runtime spawning its scheduler threads (`CLONE_THREAD` clones) before the one process-creating `clone(CLONE_VM|CLONE_PIDFD|CLONE_VFORK|SIGCHLD)`. That `CLONE_PIDFD` flag -is this part's chapter 13 arriving early: Go receives a *pidfd* for the +is pidfd-based process management arriving early: Go receives a *pidfd* for the child and waits via `waitid(P_PIDFD, ...)`, immune to pid-reuse races: it identifies the child by fd, not by number. Rust's `CLONE_CLEAR_SIGHAND` (via `posix_spawn`) solves a different race: the child starts with default diff --git a/_docs/12-signals-and-safe-handling.md b/_docs/12-signals-and-safe-handling.md index 412ec34..469c572 100644 --- a/_docs/12-signals-and-safe-handling.md +++ b/_docs/12-signals-and-safe-handling.md @@ -19,15 +19,15 @@ By the end of this chapter you can: ## Why this matters -A signal handler runs at an arbitrary instant inside arbitrary code, which makes it one of the easiest places in a Linux program to introduce a deadlock or a race that only shows up under load. The discipline this chapter teaches, converting signals into data and handling them in a plain loop, is what separates a supervisor that restarts cleanly from one that occasionally hangs on SIGTERM. +A signal handler runs at an arbitrary instant inside arbitrary code, which makes it one of the easiest places in a Linux program to introduce a deadlock or a race that only shows up under load. The discipline here, converting signals into data and handling them in a plain loop, is what separates a supervisor that restarts cleanly from one that occasionally hangs on SIGTERM. -This chapter turns pmon from a one-shot runner (Chapter 11) into an actual supervisor: restart on crash, clean shutdown, and configuration reload all ride the same signal-to-data conversion. Chapter 13 closes the one race this design still carries, a recycled pid receiving a forwarded signal meant for a dead child. +Turning pmon from a one-shot runner into an actual supervisor needs restart on crash, clean shutdown, and configuration reload, all riding the same signal-to-data conversion. Pinning process identity to a file descriptor instead of a reused pid closes the one race this design still carries: a recycled pid receiving a forwarded signal meant for a dead child. -Chapter 11 taught `pmon` to run one child and report how it died. Version 1 +`pmon` previously ran one child and reported how it died. Version 1 makes it a real supervisor: restart a crashing child with a doubling backoff, shut down cleanly on SIGTERM or SIGINT, and replace the child on SIGHUP. That -sounds like three `signal()` calls, and that is exactly the trap. The one new -idea of this chapter is that a signal handler is close to lawless territory +sounds like three `signal()` calls, and that is exactly the trap. The new +idea here is that a signal handler is close to lawless territory where almost no code is safe to run, so well-built programs run *no* meaningful code there at all: they convert signals into ordinary data (a file descriptor read, a channel receive, a byte in a pipe) and consume that data in a plain @@ -85,13 +85,13 @@ The **pending set** is where a generated-but-undelivered signal sits, and for standard signals it is literally one bit each: a second SIGCHLD arriving while one is already pending *coalesces* into the same bit. This is the core of SIGCHLD semantics: the signal means "one **or more** children changed -state", never "exactly one". The robust consumer treats SIGCHLD as a doorbell -and asks `waitpid(…, WNOHANG)` what actually happened; `pmon` has exactly one +state", never "exactly one". A correct consumer treats SIGCHLD as a doorbell +and asks `waitpid(…, WNOHANG)` what happened; `pmon` has exactly one child at a time, so a targeted `waitpid(child, …, WNOHANG)` is the whole answer; the general reap-until-empty loop arrives with pmon's later growth. Delivery itself happens only when a signal is pending *and* unblocked in some -thread, at an arbitrary instant between two instructions, which is the -problem the next section is about. +thread, at an arbitrary instant between two instructions: the source of +every handler hazard. ## Why handlers are a minefield @@ -123,7 +123,7 @@ the handler does exactly one safe thing (`write` one byte to it), and the main loop, already blocked in `poll`, wakes up, drains the pipe, and handles the event as ordinary code. The pipe is the bridge from signal context to loop context, and `write(2)` is async-signal-safe. The Rust implementation -is deliberately the historical exhibit, with the unsafe parts delegated to +is the historical exhibit, with the unsafe parts delegated to `signal-hook`: each signal is registered twice, a flag and a doorbell, from `examples/12-signals-and-safe-handling/rust/src/main.rs`: @@ -469,22 +469,22 @@ processes; supervision semantics belong to `supervise` only. ## Errors, three ways -The contract is unchanged from chapter 11: `pmon: error: …` on stderr and +The contract is unchanged from pmon v0: `pmon: error: …` on stderr and exit 1 for runtime failures, usage on exit 2, but signals sharpen the -`EINTR` policy from chapter 4 into three distinct spellings. C++ retries +`EINTR` policy from the syscall ABI into three distinct spellings. C++ retries `poll` on `EINTR` explicitly, and `reap_blocking` loops `waitpid` the same way; under `supervise` the mask makes those interruptions nearly impossible, but `run` reaps with the default mask, so the loop still matters there. Rust matches `Errno::EINTR` as a value in both `poll` and `waitpid`; its -handlers really do interrupt syscalls, as the cross-check will show, and +handlers do interrupt syscalls, as the cross-check shows, and additionally tolerates `Errno::ECHILD` ("no child to reap") as a stale doorbell rather than an error. Go never sees `EINTR` at all: the runtime installs every handler with `SA_RESTART` and retries interrupted waits inside `cmd.Wait`, so user code only ever receives the final verdict. -One deliberate acceptance shows up in all three `stop_child`s: `ESRCH` from -`kill` means the child is already gone mid-shutdown: a race you expect, -handled at the call site, while everything else carries context up to one -printer, exactly the chapter-5 policy. +One accepted race shows up in all three `stop_child`s: `ESRCH` from +`kill` means the child is already gone mid-shutdown, handled at the call +site, while everything else carries context up to one printer, exactly +the shared error-reporting policy. ## Concurrency lens @@ -515,7 +515,7 @@ per-thread, and signalfd only sees signals *blocked* in every thread that might otherwise take delivery. The C++ code blocks the mask before any thread could exist, so the invariant holds trivially; in a threaded program you must block-then-spawn so children inherit the mask. Rust's mechanism is -the only one where user-adjacent code truly runs in signal context, and its +the only one where user-adjacent code runs in signal context, and its concurrency budget is exactly one atomic store plus one non-blocking `write`: a full doorbell drops the byte harmlessly, because bytes, like pending bits, are allowed to coalesce. @@ -601,7 +601,7 @@ session's supervisor pid 2617196 issued exactly **114 `rt_sigaction` calls**: 58 probes reading old dispositions plus 56 installs of the runtime handler (`sa_handler=0x484d40`, `sa_mask=~[]`, `SA_ONSTACK|SA_RESTART|SA_SIGINFO`) covering SIGHUP through `SIGRT_32`, the -identical 114-call storm chapter 4 counted for a program that never touched +identical 114-call storm already counted for a program that never touched `os/signal`, because it is runtime initialization, not `signal.Notify`. Zero `signalfd4`, zero `poll`/`ppoll` (the runtime waits in `futex` and its netpoller). Rust sits between: 13 `rt_sigaction`, probe+install pairs for @@ -661,11 +661,11 @@ stack-overflow guard the strace already showed. Three processes, one hex format, and each design legible straight from the kernel's ledger. > **On the lab VM** unverified: -> the kernel-side view of this chapter is `bpftrace` on +> the kernel-side view here is `bpftrace` on > `tracepoint:signal:signal_generate` and `signal_deliver`, watching the > pending-set handoff fleet-wide instead of per-process: which pid raised > each SIGCHLD, where each SIGURG landed. That needs root and a disposable -> kernel, so it is not run here; chapter 30 (Debugging part) exercises +> kernel, so it is not run here; the eBPF observation toolkit exercises > exactly this on `systems-target`. ## What you learned @@ -710,4 +710,4 @@ and ERESTART_RESTARTBLOCK on poll; and the SigCgt 0, SigCgt fffffffd7fc1feff, SigCgt 14443) were read from live supervisors and decoded with the Python shown. The "On the lab VM" bpftrace callout is unverified as -marked and deferred to chapter 30.

+marked and deferred to the eBPF observation toolkit.

diff --git a/_docs/13-pidfd-modern-process-mgmt.md b/_docs/13-pidfd-modern-process-mgmt.md index 8b9c68e..ef59039 100644 --- a/_docs/13-pidfd-modern-process-mgmt.md +++ b/_docs/13-pidfd-modern-process-mgmt.md @@ -21,7 +21,7 @@ By the end of this chapter you can: A supervisor that names its children by pid carries a race it usually never notices: any reap it does not control, a runtime, a library, a signal handler, can free that pid for reuse before the supervisor acts on it again. On a busy build server or a container with a small `pid_max`, that window is not theoretical. -Pidfds close the race by making a process a referenceable object instead of a reused number, and they compose directly with the event-loop model Chapter 9 built: a process exit becomes ordinary fd readiness. This chapter makes pidfds pmon's default engine, completing the supervision story Chapters 11 and 12 started before Chapter 14 turns to privilege. +Pidfds close the race by making a process a referenceable object instead of a reused number, and they compose directly with the epoll-based event-loop model: a process exit becomes ordinary fd readiness. Making pidfds pmon's default engine completes the supervision story that restart-on-crash and signal handling started, before identity and privilege take over. `pmon` v1 ended the last chapter supervising a crashing child correctly: no async handlers, SIGCHLD arriving as data through a signalfd, a `WNOHANG` reap @@ -32,7 +32,7 @@ come to mean a completely different process, and `kill(2)` will cheerfully signal the stranger. The one new idea in this chapter is that a process can be a *file descriptor*: `pidfd_open(2)` hands you an fd that refers to the process itself, not to its number, pollable in exactly the event-loop style -Chapter 9 built, reapable with `waitid(2)`, signalable with +built on epoll, reapable with `waitid(2)`, signalable with `pidfd_send_signal(2)`, and immune to reuse by construction. `pmon` v2 makes that its default engine and keeps the v1 sigchld engine beside it, byte-identical in output, so you can strace the difference. @@ -61,8 +61,8 @@ table, nothing more; on this host the table wraps at zombie: the pid stays reserved *only* until someone collects the exit status. (3) Someone does, and here is the trap: in any program bigger than a toy, "someone" is not always you. A library thread that spawns helpers, a -language runtime that reaps everything (Chapter 11 showed Go's runtime -holding exactly this power), a signal handler someone else installed: any +language runtime that reaps everything (Go's runtime holds exactly this +power), a signal handler someone else installed: any `wait()` in the process can free the pid. (4) The kernel recycles 4242 for an unrelated process. Under default settings that needs the counter to wrap, but a busy build server wraps in hours, and a container with a small @@ -85,12 +85,12 @@ The fix arrived in a burst across Linux 5.1–5.4, and this host's manpages (kernel `7.1`) document each piece. `pidfd_open(2)`'s manpage (HISTORY: "Linux 5.3") says it "creates a file descriptor that refers to the process whose PID is specified in pid"; the close-on-exec flag is set automatically, so -the Chapter 7 discipline costs nothing here. The fd's superpower is stated +the RAII file-descriptor discipline costs nothing here. The fd's superpower is stated under NOTES: it "can be monitored using poll(2), select(2), and epoll(7). When the process that it refers to terminates, these interfaces indicate the file descriptor as readable": a process exit becomes an ordinary readiness -event, composable with every other fd your Chapter 9 event loop owns. There -is nothing to actually read: `read(2)` on a pidfd fails with `EINVAL`. The +event, composable with every other fd your epoll event loop owns. There +is nothing to read: `read(2)` on a pidfd fails with `EINVAL`. The manpage also warns that an fd obtained by opening `/proc/` (the older trick) "is not pollable and can't be waited on with waitid(2)". @@ -98,7 +98,7 @@ For your *own* children there is a spelling with no window at all: `clone(2)`/`clone3(2)` with `CLONE_PIDFD` (since Linux 5.2) delivers the pidfd atomically at spawn time, via `parent_tid` for `clone` or `cl_args.pidfd` for `clone3`, the same "set the property at creation, not -after" move as `O_CLOEXEC` in Chapter 7. Acting through the fd: +after" move as `O_CLOEXEC`. Acting through the fd: `pidfd_send_signal(2)` (Linux 5.1) "sends the signal sig to the target process referred to by pidfd", and `waitid(2)` grew an ID type `P_PIDFD` (Linux 5.4) to "wait for the child referred to by the PID file @@ -135,7 +135,7 @@ probe against it on this stable toolchain fails with `E0658: use of unstable library feature 'linux_pidfd'`, pointing at tracking issue #82971. That is the verified line: Go's runtime rides pidfds invisibly; stable Rust makes you call `pidfd_open` yourself (our example uses `rustix`); where either -lands next is a research note, not something this chapter will claim. +lands next is a research note, not a claim made here. ## How the code works @@ -148,7 +148,7 @@ the two things a wait produced: a reaped status, or a stop request (`signal` or `timeout`). C++ spells that as `std::variant`, Rust as an `enum Outcome`, and Go, with `select` as its native "whichever happens first", doesn't need the type at -all. The C++ `Fd` owner is the same class Chapter 7 built; Rust gets +all. The C++ `Fd` owner is the same RAII fd wrapper used elsewhere; Rust gets `OwnedFd` straight from `rustix::process::pidfd_open`, and Go holds the raw integer from `unix.PidfdOpen` with an explicit `unix.Close`. @@ -295,7 +295,7 @@ Only then does readability on `fds[0]` mean "the child terminated", and naming the child *by the fd*, so even in a process where some library reaps promiscuously, this call can never collect a stranger. `status_from` classifies `si_code == CLD_EXITED` versus killed/dumped, the same siginfo -plumbing as Chapter 11. +plumbing as pmon v0's exit reporting. Go cannot put a channel in a `poll` set, so it inverts the bridge: one goroutine parks in `unix.Poll` on the pidfd alone, infinite timeout, and @@ -303,10 +303,10 @@ goroutine parks in `unix.Poll` on the pidfd alone, infinite timeout, and then merges three event sources (exit, `signal.Notify` delivery, deadline) exactly as the C++/Rust `poll` merged two fds and a timeout. On a stop, the `<-exited` after `PidfdSendSignal` does double duty: it waits for -the child to actually die (the pidfd goes readable), and it guarantees the +the child to die (the pidfd goes readable), and it guarantees the goroutine has finished touching `pidfd` before `unix.Close(pidfd)` runs: close an fd a goroutine is still polling and the slot number could be -recycled mid-poll, which is Chapter 7's double-close lesson wearing a +recycled mid-poll, which is the double-close lesson wearing a concurrency hat. The stop path is the syscall the diagram promised: @@ -330,15 +330,15 @@ get the fd at clone time the way Go's runtime does. ## Errors, three ways -The retry policy from Chapter 5 lands on three calls here: `poll` and +The EINTR retry policy lands on three calls here: `poll` and `waitid` retry `EINTR` in a loop in C++ (`errno != EINTR` breaks), Go (`errno == unix.EINTR { continue }` around `Syscall6`, and the poller goroutine's `err == unix.EINTR`), and Rust (`Err(rustix::io::Errno::INTR) => continue`). Expected failures are policy at the call site: `pidfd_send_signal` on the stop path ignores its result (zombie or `ESRCH`, both fine), and the sigchld engine treats a `WNOHANG` -reap that returns nobody as "coalesced SIGCHLD, keep waiting": Chapter 12's -lesson encoded in all three languages. Everything unexpected carries context +reap that returns nobody as "coalesced SIGCHLD, keep waiting": the +coalesced-signal lesson encoded in all three languages. Everything unexpected carries context up to one printer with the raw error preserved: C++ wraps `errno` in `std::error_code` inside `std::expected`, Go wraps with `%w` so `errors.Is`/`errors.As` still see the errno underneath, Rust chains @@ -362,7 +362,7 @@ child-side mask reset (`POSIX_SPAWN_SETSIGMASK`, `pre_exec`) is non-negotiable: any thread may be the one that spawns. Note also what the strace shows about fd-table behavior under concurrency: every spawn prints `pidfd=4` because the previous pidfd was closed before the next -`pidfd_open`, and Chapter 7's lowest-free-slot rule hands the number back, +`pidfd_open`, and the kernel's lowest-free-slot fd rule hands the number back, while in Go the runtime's *own* pidfd for the same child sits one slot over. Two fds, one process, no interference: identity multiplied is still identity. @@ -404,7 +404,8 @@ pgrep exit=1 The `pgrep` exit 1 is the observable that matters: no supervised `sleep` outlived its supervisor. (The `[.]` keeps `pgrep -f` from matching its own -command line: a pid-namespace-sized footgun in miniature.) The Rust binary +command line, since pattern-matching the process list can catch the +matching command itself.) The Rust binary produces the same crash-loop shape (children 2622026–2622028 on this host), and the runner (`python3 scripts/test-all-examples.py --only 13-pidfd-modern-process-mgmt`) reports `PASS PASS PASS` with `verify.lua`'s @@ -447,7 +448,7 @@ followed by our `pidfd_open(2623708, 0) = 4` and the identical **The fd itself, in `/proc`.** Pause a live supervision (`supervise --timeout-ms 8000 -- sleep 6.66`, supervisor pid 2624388) and -read the fd table the way Chapter 7 taught: +read the fd table directly: ```bash [host]$ ls -l /proc/2624388/fd | tail -2 @@ -470,11 +471,11 @@ code on our side. (`NSpid` previews Part 10: inside a pid namespace the same process has a different number, but the same pidfd.) > **On the lab VM** unverified: -> the fleet-wide view of this chapter is bcc-tools: `exitsnoop` timestamping +> the fleet-wide view here is bcc-tools: `exitsnoop` timestamping > every process exit and `killsnoop` logging who signalled whom, which would > show `pmon`'s SIGTERM (and would *not* show a pid-reuse mishit, because > there is none). bcc-tools need root and kernel headers, so they are not run -> on this host; Chapter 30 in the Debugging part exercises them on the +> on this host; the eBPF observation toolkit exercises them on the > `systems-target` VM. ## What you learned @@ -483,7 +484,7 @@ process has a different number, but the same pidfd.) steps, and the dangerous reap in step 3 is often performed by a runtime or library you don't control. - `pidfd_open` (or `CLONE_PIDFD` at clone time) pins process identity to an - fd: exit becomes `poll` readability (Chapter 9's event-loop model now + fd: exit becomes `poll` readability (the epoll event-loop model now covers processes), `waitid(P_PIDFD)` reaps exactly that process, and `pidfd_send_signal` yields `ESRCH` rather than signalling a stranger; `/proc//fdinfo/`'s `Pid:` line names the pinned process. diff --git a/_docs/14-identity-and-privilege.md b/_docs/14-identity-and-privilege.md index 496de85..31a6037 100644 --- a/_docs/14-identity-and-privilege.md +++ b/_docs/14-identity-and-privilege.md @@ -18,7 +18,7 @@ By the end of this chapter you can: ## Why this matters -Most daemons start as root because they need a privileged resource (a port below 1024, a protected path) and must stop being root the moment they no longer do. Getting that transition wrong in either direction, leaving root behind by accident or stripping the one capability the service needed, is a recurring real-world vulnerability class, not a theoretical one. +Most daemons start as root because they need a privileged resource (a port below 1024, a protected path) and must stop being root the moment they no longer do. Getting that transition wrong in either direction, leaving root behind by accident or stripping the one capability the service needed, is a recurring vulnerability class in deployed services. This chapter gives pmon the final piece of its supervision story: dropping privilege correctly while carrying exactly one capability across an identity change and an `execve`, proven with a paired positive and negative run rather than a single claim. It closes out Part 3's process-and-identity arc before Part 4 turns to virtual memory. @@ -43,7 +43,7 @@ positive/negative verification. {% include excalidraw.html file="14-credential-sets-caps" - alt="Two bands. The top band shows the three uids: real, effective, saved, each moving 0 to 65534, with a note that setresuid moves all three at once and the saved uid is the way back that the drop destroys for good. The bottom band shows pmon's kept-cap path as six boxes: pmon as root with full permitted and effective sets, prctl PR_SET_KEEPCAPS, setresuid to 65534 with permitted kept, capset pinning 0x400 into permitted inheritable and effective, prctl AMBIENT_RAISE putting 0x400 into ambient, and execve into bindprobe bound to port 80 at uid 65534. A dashed negative-control row underneath skips the prctl and capset steps: setresuid scrubs permitted to zero and execve reaches bindprobe with every set zero and bind denied, exit 3. Notes carry the real /proc status values: the kept-cap child reads 0000000000000400 in CapInh CapPrm CapEff CapAmb with CapBnd 000001ffffffffff." + alt="Two bands. The top band shows the three uids: real, effective, saved, each moving 0 to 65534: setresuid moves all three at once and the saved uid is the way back that the drop destroys for good. The bottom band shows pmon's kept-cap path as six boxes: pmon as root with full permitted and effective sets, prctl PR_SET_KEEPCAPS, setresuid to 65534 with permitted kept, capset pinning 0x400 into permitted inheritable and effective, prctl AMBIENT_RAISE putting 0x400 into ambient, and execve into bindprobe bound to port 80 at uid 65534. A dashed negative-control row underneath skips the prctl and capset steps: setresuid scrubs permitted to zero and execve reaches bindprobe with every set zero and bind denied, exit 3. Notes carry the real /proc status values: the kept-cap child reads 0000000000000400 in CapInh CapPrm CapEff CapAmb with CapBnd 000001ffffffffff." caption="Figure 14.1: the three-uid state machine and the five capability sets across pmon's drop; every hex value comes from /proc/<pid>/status on systems-target, read during the cross-check below" %} > **Tools used**: `ssh`/`scp` and the Python runner @@ -128,7 +128,7 @@ confirmed below). Each *process* carries five sets, visible as the `Cap*` lines of `/proc//status`: - **Permitted**, the ceiling: capabilities the process may use or raise. -- **Effective**, the subset the kernel actually consults in checks. +- **Effective**, the subset the kernel consults in checks. - **Inheritable**, offered across `execve`, but only into binaries whose *file* inheritable set accepts them. - **Bounding**, a mask over what `execve` can ever grant. On the guest it @@ -433,8 +433,8 @@ A few fragile points: `--keep-cap` accepts only and generalizing to caps ≥ 32 means touching word 1; the staged binary must live somewhere the dropped user can read (`/tmp`, not `/home/fedora`); and `pmon` does not call `setsid` or close inherited fds; real daemons -layer those on, and chapter 7's `O_CLOEXEC` discipline is half of why they -can skip the fd audit. +layer those on, and `O_CLOEXEC` discipline on every open descriptor is half +of why they can skip the fd audit. ## Errors, three ways @@ -466,8 +466,7 @@ lockstep. But `pmon`'s C++ and Rust builds bypass glibc for `capset` because a privilege drop must cover every task, and `execve` replaces the whole process anyway. Go cannot make that promise: the runtime spawns threads before `main`, and a raw in-process dance would leave sibling -threads still root, a real vulnerability class, not a theoretical one. -That is exactly why the Go build pushes the entire sequence into the +threads still root. That is exactly why the Go build pushes the entire sequence into the fork/exec child. The general rule: change identity either before threads exist or in a fresh child, never "around" a running thread pool. @@ -597,8 +596,8 @@ drop starts from. > stream with symbolic names, but it failed to compile against kernel > 6.19 on this guest; Part 8 works through that toolchain properly.) -One deliberate non-observation: you cannot follow the drop with `strace` -from outside, because ptrace attachment is severed at a privilege +One thing you cannot observe this way: you cannot follow the drop with +`strace` from outside, because ptrace attachment is severed at a privilege transition. The port bind *is* the observation surface here, and there is no dashboard panel by design; this example emits no telemetry. diff --git a/_docs/15-virtual-memory.md b/_docs/15-virtual-memory.md index 029b4ca..2063de2 100644 --- a/_docs/15-virtual-memory.md +++ b/_docs/15-virtual-memory.md @@ -41,7 +41,7 @@ This chapter builds the vocabulary, VMAs, faults, VSZ/RSS/PSS, overcommit, that every later memory and IPC chapter assumes, including `shmkv`, the key-value store the rest of this part builds directly on mapped pages. -Part 3 closed with `pmon` dropping privileges in chapter 14; this part turns +Part 3 closed with `pmon` dropping privileges; this part turns inward, to the memory every one of those processes lives in. The one new idea is that **an address is a promise, not a page**: when `malloc`, `make`, or `Vec` hands you 64 MiB, the kernel has recorded a range in a bookkeeping @@ -105,7 +105,7 @@ ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] (Whitespace compressed; the columns are start–end, permissions, file offset, device, inode, and path.) The fourth permission character is the one people miss: `p` means *private*, writes trigger copy-on-write and never reach the -file; `s` would mean *shared*, the mode chapter 16 is built on. Read +file; `s` would mean *shared*, the mode shared mappings are built on. Read the file top to bottom and the classic layout falls out. The binary itself appears four times: `r-xp` text, `r--p` read-only data, a second `r--p` that is the relro region (writable during relocation, sealed read-only before @@ -116,7 +116,7 @@ one region the legacy `brk(2)` syscall grows. Then a jump to the high top-down: the locale archive `cat` mapped read-only, `libc` in the same four-segment pattern as the binary, anonymous `rw-p` slabs with inode 0 (allocator arenas and thread machinery), and the kernel-provided `[vvar]` + -`[vdso]` pages that let chapter 4's `clock_gettime` skip the syscall +`[vdso]` pages that let `clock_gettime` skip the syscall entirely. Near the top, `[stack]` grows downward toward the mmap region; the `[vsyscall]` page at the very top is a frozen compatibility relic. The gaps between regions are the point, not an accident: unmapped gigabytes that turn @@ -219,7 +219,7 @@ otherwise assume most promises are never fully touched. The proof is in the second read: this 64 GiB host has *already* promised 48.5 GiB against a nominal `CommitLimit` of 41.2 GiB (swap + 50% of RAM, per `overcommit_ratio`), and nothing is wrong: in mode 0 the limit is -advisory. Mode `2` enforces it (`malloc` actually starts returning `NULL`); +advisory. Mode `2` enforces it (`malloc` starts returning `NULL`); mode `1` promises anything. The price of overcommit is paid at touch time: if a fault needs a frame and there is nothing left to reclaim, the **OOM killer** picks the process with the highest badness score: roughly its @@ -228,8 +228,8 @@ volunteers). This shell runs at `100` (`cat /proc/self/oom_score_adj`), a nudge from systemd making user sessions die before system services. The lesson for a systems programmer: on mode-0 Linux, allocation failure mostly does not arrive as a clean error at the call site; it arrives as `SIGKILL` -at the worst touch. Chapter 22's cgroup work sharpens this with per-service -memory limits. +at the worst touch. cgroup memory limits sharpen this further, with +per-service caps instead of a single host-wide policy. The last piece of the VMA toolkit is `madvise(2)`: hints on a range you already own. `MADV_WILLNEED` starts readahead; `MADV_SEQUENTIAL` and @@ -241,9 +241,9 @@ out of transparent huge pages, which matters here because this host's THP mode is `madvise` (`cat /sys/kernel/mm/transparent_hugepage/enabled` prints `always [madvise] never`), meaning 2 MiB pages only where explicitly requested. This is not exotic: the Go runtime in the cross-check below calls -`madvise(…, MADV_NOHUGEPAGE)` on its own metadata as a matter of course, and -chapter 17 uses `MADV_DONTNEED` semantics to explain what "freeing" memory -to an allocator actually returns to the kernel. +`madvise(…, MADV_NOHUGEPAGE)` on its own metadata, and `MADV_DONTNEED` +semantics are why "freeing" memory to an allocator does not always return +it to the kernel. ## How the code works @@ -251,7 +251,7 @@ Three small structures carry `memmap`. A `Baseline` (`{rss_kb, minor, major}`) is captured *before* each allocation because `/proc/self/status`'s `VmRSS` and `getrusage`'s `ru_minflt`/`ru_majflt` are cumulative since process start; only deltas isolate the window we care about. A `Mapping` -owns an mmap'd range exactly the way chapter 7's `Fd` owns a descriptor: +owns an mmap'd range exactly the way the RAII `Fd` owns a descriptor: move-only in C++, `Drop` calls `munmap` in Rust, `defer unix.Munmap(m)` in Go, so no error path leaks a VMA. A `TempFile` unlinks its path on destruction, which is why `verify.lua` can assert the walk file is gone even @@ -344,11 +344,11 @@ fn run_stack(bytes: usize, page: usize) -> Result<()> { } ``` -The shared discipline: the worker is `noinline` so the 4 MiB frame really -exists as a frame, and the baseline is taken by the *caller*, because the -compiler (stack probing in C++/Rust, stack growth in Go) touches frame pages -at function entry; measure from inside and you miss the very faults you came -for. The divergence: C++ and Rust run the worker on the OS thread, so their +The shared discipline: the worker is `noinline` so the 4 MiB frame exists as +a frame instead of being optimized into registers, and the baseline is taken +by the *caller*, because the compiler (stack probing in C++/Rust, stack +growth in Go) touches frame pages at function entry; measure from inside and +you miss the faults you came to measure. The divergence: C++ and Rust run the worker on the OS thread, so their excerpt shows the `[stack]` VMA growing; Go runs it on a fresh goroutine whose stack is runtime-managed heap spans, so its excerpt shows an anonymous region: same contract, different mechanism, and the accurate place to see @@ -382,7 +382,7 @@ memmap: error: /no/such/file: No such file or directory (os error 2) (Rust, exi Three renderings of the same `ENOENT`, because each rides its language's native machinery: `std::error_code::message()` inside `std::expected` plumbing, a `%w`-wrapped `*os.PathError`, `anyhow`'s `{:#}` chain over -`io::Error`. Unlike chapter 5's `fwatch`, this contract pins the prefix and +`io::Error`. Unlike `fwatch`'s contract, this one pins the prefix and the exit code but leaves message text native, and `verify.lua` asserts exactly that split, while errors the *program* generates, like mapping an empty file (`mmap` of length 0 would fail anyway, so `open_for_map` checks @@ -401,7 +401,7 @@ zero-fills eagerly during construction, so the faults land in the `memset` `16598`: the same 16,384 plus a few hundred faults from runtime machinery (GC metadata, spans, a second OS thread growing something) because `getrusage(RUSAGE_SELF)` sums *every thread in the process*, and a Go -program is never really alone. The same reasoning explains why `perf stat` +program is never alone. The same reasoning explains why `perf stat` whole-process counts (below) always exceed the in-window deltas, and why `verify.lua` asserts `minor > 0` and RSS floors instead of equalities. One more thread-shaped trap from the stack tabs: Go's worker goroutine could @@ -516,7 +516,8 @@ story. > logging every OOM-killer decision with the victim's badness score, and > `bpftrace` histograms over the `page_fault_user` tracepoint replacing > per-process `getrusage`. Those need root on a kernel you can afford to -> pressure; chapter 30, on the `systems-target` VM, does exactly that. +> pressure; the eBPF observation toolkit, on the `systems-target` VM, does +> exactly that. ## What you learned @@ -535,7 +536,7 @@ story. - Overcommit mode 0 means `Committed_AS` may exceed `CommitLimit` (it does, on this host); the bill arrives at touch time via reclaim or the OOM killer's `oom_score_adj`-weighted choice, and `madvise` is your lever for - telling the kernel what a range is really for. + telling the kernel what a range is for. Next, **mmap and shared mappings**: `MAP_SHARED`, `msync`, and two processes seeing one page, the machinery `shmkv` v0 is built from. diff --git a/_docs/16-mmap-and-shared-mappings.md b/_docs/16-mmap-and-shared-mappings.md index 3bd3096..197cf12 100644 --- a/_docs/16-mmap-and-shared-mappings.md +++ b/_docs/16-mmap-and-shared-mappings.md @@ -71,12 +71,12 @@ Every `mmap` call picks one cell of a 2×2. The first axis is **what backs the pages**: a *file-backed* mapping names an fd and an offset, so its pages are literally page-cache pages of that file; an *anonymous* mapping (`MAP_ANONYMOUS`) is backed by nothing but zero-fill: this is what your -allocator asks for when it wants a fresh arena, which is chapter 17's whole -subject. The second axis is **who sees your writes**. `MAP_SHARED` means +allocator asks for when it wants a fresh arena, squarely allocator and +GC-runtime territory. The second axis is **who sees your writes**. `MAP_SHARED` means writes go to the backing object itself: for a file, straight into the shared page cache, visible immediately to every other process mapping the same range. `MAP_PRIVATE` means copy-on-write: your mapping starts out referencing the -very same page-cache pages (reads are free and shared), but the page-table +same page-cache pages (reads are free and shared), but the page-table entries are write-protected, and the first store to a page faults, at which point the kernel copies that page and points only *your* mapping at the copy. Your writes never reach the file, and after the copy, other processes' writes @@ -179,8 +179,8 @@ processes. Interop works because the format is defined in bytes, not in types: magic `"SHKV1\0"` at offset 0, a **u32 little-endian** slot count at offset 6, then -256-byte slots: key NUL-padded in `[0..64)`, value in `[64..256)`. Note that -offset 6 is *misaligned* for a 4-byte integer, and none of the +256-byte slots: key NUL-padded in `[0..64)`, value in `[64..256)`. Offset 6 +is *misaligned* for a 4-byte integer, and none of the three implementations care, because none of them ever casts a pointer into the mapping: C++ assembles the u32 from four bytes with shifts, Go uses `binary.LittleEndian.PutUint32`, Rust `u32::to_le_bytes`: byte moves have no @@ -383,9 +383,9 @@ barrier, and the kernel orders nothing between mappers. shmkv v0's defense is its usage model: commands are whole processes run sequentially, so every write is ordered by process exit, and the Rust unsafe-boundary comment is where that assumption is written down rather than left implicit. This is the -cliff-hanger by design: chapter 20 puts atomics and futexes *inside* a shared -mapping to make concurrent access actually safe. Two language-specific hazards -round it out. Go's `[]byte` view means the race detector and GC offer no help +gap left open by design: shared-memory coordination puts atomics and +futexes *inside* a shared mapping to make concurrent access safe. Two +language-specific hazards round it out. Go's `[]byte` view means the race detector and GC offer no help across processes, and the slice must never be touched after `Munmap`, which `close` enforces by nil-ing it. And any mapper can be killed by a peer that truncates the file (in-bounds loads become `SIGBUS`), which is exactly the @@ -506,7 +506,7 @@ language, `PASS 48 / FAIL 0` for cpp, go, and rust in this session's run. memory look like any other pointer, and Go disguises the mapping as a `[]byte` you must never touch after `Munmap`. -Next, **allocators and GC runtimes**: what actually happens between your +Next, **allocators and GC runtimes**: what happens between your `new`/`make`/`Box` and the anonymous mappings this chapter taught you to see. --- diff --git a/_docs/17-allocators-and-gc-runtimes.md b/_docs/17-allocators-and-gc-runtimes.md index ffc2850..837b930 100644 --- a/_docs/17-allocators-and-gc-runtimes.md +++ b/_docs/17-allocators-and-gc-runtimes.md @@ -30,14 +30,14 @@ By the end of this chapter you can: Every allocation-heavy service eventually asks the same question: where did the memory go? Answering it requires knowing what your language's allocator -actually does with a `free`, a dropped `Box`, or a Go object the collector +does with a `free`, a dropped `Box`, or a Go object the collector has not traced yet, because a dashboard only reports the outcome, not the mechanism, and "my program freed everything but RSS didn't drop" is a question that costs an on-call engineer real time if the mechanism is not already familiar. This chapter opens the allocator and the GC pacer directly: glibc's arena -and tcache, what `pmr`/`bumpalo` arenas really buy you, and the live +and tcache, what `pmr`/`bumpalo` arenas buy you, and the live `GOGC`/`GOMEMLIMIT` knobs that turn Go's memory footprint into a negotiated policy rather than a mystery. @@ -107,7 +107,7 @@ pointer through a slab, never free individuals, release the whole slab in one move when the batch of work is done. C++23 spells it `std::pmr::monotonic_buffer_resource` (a memory resource whose `deallocate` is a **no-op**); everything comes back when the resource is -destroyed. What pmr actually changes is therefore two things and only two +destroyed. What pmr changes is therefore two things and only two things: **allocation locality** (the batch's strings sit contiguously in one slab instead of scattered across malloc chunks) and **free-at-once** (batch teardown is one release, not a thousand `free` calls). It does not lower @@ -292,8 +292,9 @@ slab*. Same reason `make_kv` builds numbers with `std::to_chars` via `append_num`: `std::to_string` would hand back an ordinary heap `std::string`, silently routing churn around the arena. The survivors are copied out explicitly (`std::string{key}`) because the index must outlive the -resource; forgetting that copy is the classic pmr lifetime bug, and it is the -exact discipline in one line: *bulk-free the churn, copy out what survives.* +resource; skipping that copy leaves a dangling reference into freed arena +memory, and the discipline is one line: *bulk-free the churn, copy out what +survives.* The Rust version encodes the same rules in types instead of comments. `BumpString::new_in(&bump)` borrows the arena, so the inner block must end diff --git a/_docs/18-pipes-fifos-splice.md b/_docs/18-pipes-fifos-splice.md index e0455b7..ea391f8 100644 --- a/_docs/18-pipes-fifos-splice.md +++ b/_docs/18-pipes-fifos-splice.md @@ -99,7 +99,7 @@ asked for 1, got: 4096 Sixteen 4096-byte pages, 65536 bytes; ask for a 1-byte pipe and the kernel rounds up to one page; unprivileged growth stops at `/proc/sys/fs/pipe-max-size` (1048576 here). `pmon` uses `pipe2` rather than -`pipe` for the same reason chapter 7 put `O_CLOEXEC` in every open: the flag +`pipe` for the same reason every RAII-wrapped file descriptor puts `O_CLOEXEC` in every open: the flag lands atomically, with no window for a concurrently spawned child to inherit the fd. `strace -f -e trace=pipe2` on the C++ supervisor shows exactly two calls per attempt: `pipe2([5, 6], O_CLOEXEC) = 0` and @@ -177,7 +177,7 @@ and go back to waiting for the next reader. {% include excalidraw.html file="18-log-topology" - alt="Three horizontal bands. Top, pmon supervise pid 2633754: the sh child pid 2633756 writes fd 1 and fd 2 into two kernel pipe nodes with inodes 16462630 and 16462631, which feed a poll loop node listing signalfd 4, pidfd 6, and pipe read ends 5 and 7, appending framed out and err lines into a live.log node opened append-only as fd 3. Middle, pmon tail pid 2634759: a log fd 3 node with fdinfo pos 10 connects by an amber splice arrow labelled with the real splice call returning 22 to a FIFO writer fd 4 node whose flags 02100001 show O_NONBLOCK cleared. Bottom, kernel FIFO plus reader: the live.fifo inode of type S_IFIFO flows into a cat reader holding fd 3, with a note that a detached reader causes EPIPE and the relay re-splices undelivered bytes to the next reader." + alt="Three horizontal bands. Top, pmon supervise pid 2633754: the sh child pid 2633756 writes fd 1 and fd 2 into two kernel pipe nodes with inodes 16462630 and 16462631, which feed a poll loop node listing signalfd 4, pidfd 6, and pipe read ends 5 and 7, appending framed out and err lines into a live.log node opened append-only as fd 3. Middle, pmon tail pid 2634759: a log fd 3 node with fdinfo pos 10 connects by an amber splice arrow labelled with the real splice call returning 22 to a FIFO writer fd 4 node whose flags 02100001 show O_NONBLOCK cleared. Bottom, kernel FIFO plus reader: the live.fifo inode of type S_IFIFO flows into a cat reader holding fd 3; a detached reader causes EPIPE, and the relay re-splices undelivered bytes to the next reader." caption="Figure 18.2: pmon v4's log pipeline end to end; every pid, fd number, pipe inode, flag value, and splice return comes from the live runs traced in this chapter" %} ## How the code works @@ -228,7 +228,7 @@ supervisor would be holding its own pipe open, waiting on itself. Closing the parent's copies immediately after spawn makes child-exit imply pipe-EOF, which is what lets `drain_to_eof` terminate. -From there the pipes simply join the multiplexing each engine already had. +From there the pipes join the multiplexing each engine already had. C++ polls four fds: signalfd, both pipe read ends, and (pidfd engine) the pidfd, marking a drained pipe's slot as `fd = -1`, which `poll(2)` ignores. Rust builds the same four-slot `rustix::event::poll` set beside @@ -243,7 +243,7 @@ log is complete before `pmon: child=… exited status=…` appears, an ordering `verify.lua` asserts. `tail` is the second half: create the FIFO (`EEXIST` is tolerated only if -the existing inode really is a FIFO; anything else is error, exit 1), open +the existing inode is a FIFO; anything else is error, exit 1), open the log read-only at offset 0 so a late reader gets the backlog, then relay forever. The C++ and Rust relays move bytes with `splice(2)`; Go uses `io.Copy` per reader session. The heart of each: @@ -345,13 +345,14 @@ consuming them, which is how you would split this log stream to two FIFOs. The fragile bits: the 50 ms `sleep`-poll for both "log has -no new bytes" and "no reader yet" is deliberate simplicity; chapter 9's -inotify machinery could make both event-driven, and a real system would use -it; the FIFO relay serves one reader at a time (a second simultaneous +no new bytes" and "no reader yet" is deliberate simplicity; an epoll-based +event loop with inotify could make both event-driven, and a real system would +use it; the FIFO relay serves one reader at a time (a second simultaneous opener would interleave records with the first, since FIFOs have one ring, not one per reader); and log rotation is unhandled: `tail` follows an fd, not -a name, so a renamed-away log keeps feeding the old inode. Chapter 19 -replaces "one anonymous byte stream" with a real control plane. +a name, so a renamed-away log keeps feeding the old inode. A Unix domain +socket with fd passing replaces "one anonymous byte stream" with a real +control plane. ## Errors, three ways @@ -389,8 +390,9 @@ explicitly; Go's `signal.Ignore(syscall.SIGPIPE)` does the same job with a runtime subtlety: Go would otherwise re-raise a fatal SIGPIPE only for writes to fds 1 and 2, and the FIFO writer is fd 3, but the explicit ignore documents intent and covers the stdout prints too. And the -supervisor still resets the child's signal mask at spawn (chapter 12's -lesson): a captured child is still a child. +supervisor still resets the child's signal mask at spawn, the same safe-signal-handling +discipline every spawn site in this book follows: a captured child is still a +child. ## Build, run, observe diff --git a/_docs/19-unix-sockets-fd-passing.md b/_docs/19-unix-sockets-fd-passing.md index ba37037..370418d 100644 --- a/_docs/19-unix-sockets-fd-passing.md +++ b/_docs/19-unix-sockets-fd-passing.md @@ -28,20 +28,20 @@ By the end of this chapter you can: ## Why this matters A process that wants to be found and trusted by a stranger needs more than -a shared pipe: it needs an address, a way to check who is really calling, +a shared pipe: it needs an address, a way to verify who is calling, and sometimes a way to hand over an already-open resource instead of re-deriving it. UNIX domain sockets are the one primitive on the system built for exactly that, and `SCM_RIGHTS` gets used constantly without -anyone being clear on what actually crosses the wire. +anyone being clear on what crosses the wire. This chapter builds `pmon` v5's control plane on `SO_PEERCRED` for kernel-vouched identity and `SCM_RIGHTS` for descriptor passing, then proves with `/proc` on both processes at once that an open file description, not a number, is what moved. -Chapter 18 gave `pmon` a log pipeline built from pipes and FIFOs (fast, -kernel-buffered, and strictly byte-shaped): a pipe moves data, and only -between processes that inherited its ends. Version 5 needs something pipes +The pipes-and-FIFOs log pipeline gave `pmon` fast, kernel-buffered, +strictly byte-shaped delivery: a pipe moves data, and only between +processes that inherited its ends. Version 5 needs something pipes cannot do: a stranger (a `pmctl` process started minutes later, from another terminal) must find the running supervisor, prove who it is, ask questions, and get answers. That is a *control plane*, and the Linux @@ -51,9 +51,9 @@ two abilities no network socket has. It can tell the server exactly which uid and pid is calling (`SO_PEERCRED`), and it can carry *file descriptors* in its messages (`SCM_RIGHTS`). The second one is the headline: `pmctl logfd` never opens the log file; the supervisor hands it an already-open -descriptor, and the chapter closes the loop on chapter 7's three-layer model -by proving, with `/proc` on both processes at once, exactly which layer -crossed the socket. +descriptor, closing the loop on the three-layer fd model (fd table, open +file description, inode) by proving, with `/proc` on both processes at +once, exactly which layer crossed the socket. The code is in `examples/19-unix-sockets-fd-passing/`. `./demo.sh` there builds all three implementations; its `README.md` specifies the CLI, the @@ -110,8 +110,8 @@ u_str LISTEN 0 8 /tmp/p19.t9j0/ctl.sock 16411867 * 0 u_str LISTEN 0 1 @pmon-demo 16406037 * 0 ``` -The `@` prefix is `ss`'s spelling of that leading NUL. pmon deliberately -uses the filesystem kind: the path *is* the discovery mechanism and the +The `@` prefix is `ss`'s spelling of that leading NUL. pmon uses the +filesystem kind: the path *is* the discovery mechanism and the permission check, and `stop` unlinking it is a clean "no longer accepting" signal. The classic trap is capacity, not permissions: `sun_path` is 108 bytes, so a control socket under a deep CI workspace path fails to bind at @@ -146,16 +146,18 @@ languages. pmon only logs the identity; the one-line change to *enforce* (compare `cred.uid` against `getuid()` and hang up) is the pattern every socket-activated daemon on your machine uses. Two sharp edges: the pid is the connector's pid at connect time; the peer may have exited or the pid -been reused by the time you act on it (chapter 13's pidfd argument, again), +been reused by the time you act on it (the same argument for pidfd over a +raw pid, again), and stronger per-message guarantees exist via `SCM_CREDENTIALS` if you need them. -## What actually crosses the socket +## What crosses the socket `SCM_RIGHTS` is routinely described as "sending a file descriptor", and that phrase hides the mechanism. The number does not travel: fd 3 in the sender arrived as fd 4 in the receiver on the traced run below. What the kernel -does is precise in chapter 7's terms: it takes the sender's fd-table slot, +does is precise in terms of the fd-table/open-file-description/inode +layering: it takes the sender's fd-table slot, follows it to the **open file description**, allocates a *new slot in the receiver's fd table* (lowest free, as always), and points it at that same description. The result is exactly what `dup(2)` produces (shared offset, @@ -166,7 +168,7 @@ system-wide. {% include excalidraw.html file="19-scm-rights-transfer" alt="Two fd-table bands over a shared kernel band: sender slot 3 with close-on-exec set at open, and receiver slot 4 labeled new lowest free with close-on-exec set by MSG_CMSG_CLOEXEC, both pointing at one dark open-file-description box holding pos 47 and flags 02102002, which points down to inode 95934 for pmon.log; sendmsg with cmsg_data 3 names the sender slot, an amber dashed arrow shows the kernel installing the new receiver slot, and recvmsg reports cmsg_data 4." - caption="Figure 19.2: SCM_RIGHTS in chapter 7's three layers: two per-process slots, one open file description, one inode; pos, flags, inode, and both fd numbers are from the held-open run below" %} + caption="Figure 19.2: SCM_RIGHTS across the three fd layers: two per-process slots, one open file description, one inode; pos, flags, inode, and both fd numbers are from the held-open run below" %} That layer choice has practical consequences pmon is built around. The log is opened `O_RDWR | O_APPEND`, not `O_WRONLY`: flags live on the @@ -179,7 +181,8 @@ but the discipline costs nothing and the point generalizes.) ## How the code works -The supervision half is chapter 11–13 machinery matured: `spawn_child` +The supervision half is process-lifecycle, signal, and pidfd machinery +matured: `spawn_child` forks/execs the command with `setpgid(0,0)` so `stop` can `SIGTERM` the whole group, `dup2`s the log fd over stdout and stderr, and a reaper (`std::jthread`, goroutine, or `std::thread`) `wait`s, backs off 300 ms, and @@ -188,7 +191,7 @@ a mutex (C++/Rust condvar, Go channels). What is new is the socket plumbing. `listen_ctl` is the canonical four-step: `unlink` the stale path, `socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0)`, `bind` a `sockaddr_un` (after the 108-byte check), `listen`. The accept loop takes each connection -with `accept4(…, SOCK_CLOEXEC)` (chapter 7's rule that the cloexec bit must +with `accept4(…, SOCK_CLOEXEC)` (the rule that the cloexec bit must be set atomically applies to *every* fd source, and `accept` is one), logs `SO_PEERCRED`, reads one line, and dispatches: `status` formats the shared state, `stop` tears down, `logfd` calls the interesting function. @@ -308,7 +311,7 @@ fn recv_log_fd(conn: &UnixStream) -> Result { Three details deserve emphasis. First, `MSG_CMSG_CLOEXEC`: the received fd is installed by the kernel, not by any `open` you control, so this flag is the only atomic way to give it the cloexec bit; the recvmsg-then-`fcntl` -window is chapter 7's fork/exec race all over again. Note the layering in +window is the same fork/exec cloexec race all over again. Note the layering in Figure 19.2: both slots end up cloexec, but each got the bit *its own way*, because close-on-exec is per-slot, never shared. Second, ownership: the C++ code moves the raw int into the RAII `Fd` immediately, and Rust wraps @@ -342,7 +345,7 @@ next caller. `EINTR` is retried at every blocking call in the C++ (`read_command`, `write_all`, `waitpid`), which the runtimes of Go and Rust handle beneath their standard libraries. And the address-length failure is caught before the syscall in all three, because a 108-byte truncation by -`strncpy` (the traditional C bug) would *bind to the wrong path* rather +`strncpy` would *bind to the wrong path* rather than fail. ## Concurrency lens @@ -354,7 +357,7 @@ past its backoff and inside `spawn_child`. All three implementations re-check `stopping` *after* the spawn returns and, if the flag flipped, undo (SIGTERM and reap the child they just created) before signaling `child_done`. Without that re-check, `stop` leaks exactly one freshly -respawned process, a bug that only bites when the child dies within 300 ms +respawned process when the child dies within 300 ms of a stop. The languages phrase the wait differently: C++ and Rust block on a condvar predicate, Go closes a `stopCh` the reaper `select`s against the backoff timer, but the state machine is identical, which is what makes @@ -367,7 +370,8 @@ description, which the kernel makes an atomic append. One Go footnote: `listener.SetUnlinkOnClose(false)` keeps Go's helpfulness from unlinking the socket before the stop path does it explicitly, and `int(logFile.Fd())` in the `logfd` case is again a loan against the -`*os.File` staying live, exactly as dissected in chapter 7. +`*os.File` staying live, exactly as the fd-table/open-file-description +layering dissected above predicts. ## Build, run, observe @@ -441,7 +445,7 @@ pread64(4, "", 4096, 47) = 0 (Fields trimmed; the full lines are in the trace files.) The sender named fd **3**; the receiver got fd **4**: the number is per-process, so it did not travel. `cmsg_len=20` and `msg_controllen=24` are `CMSG_LEN(4)` and -`CMSG_SPACE(4)` from `send_fd`, live. And the very next syscalls show +`CMSG_SPACE(4)` from `send_fd`, live. And the next syscalls show `pmctl` reading all 47 bytes of log *through fd 4* at explicit offsets, no `openat` anywhere in the trace. @@ -468,7 +472,8 @@ ino: 95934 95934 47 /tmp/p19.t9j0/pmon.log ``` -Field for field equal, and each field pins a layer of the chapter 7 model. +Field for field equal, and each field pins a layer of the fd-table/ +open-file-description/inode model. `ino: 95934` on both fds matches `stat` on the path: one inode. `pos: 47` on both: one *offset*, which only makes sense if there is one open file description; 47 is the file's size because every `O_APPEND` write advanced diff --git a/_docs/20-shared-memory-coordination.md b/_docs/20-shared-memory-coordination.md index 2330f69..132c4a7 100644 --- a/_docs/20-shared-memory-coordination.md +++ b/_docs/20-shared-memory-coordination.md @@ -42,11 +42,11 @@ never block, readers that retry on a torn copy instead of taking a lock, and a measured latency ladder that ranks every coordination channel this book has built so far. -Chapter 16 (mmap and shared mappings) left `shmkv` in an awkward state: two +The mmap and shared-mappings work left `shmkv` in an awkward state: two processes share a page of key-value slots, but a reader who wants to *notice* a change has no option except reading the page again, and again, and again. This chapter fixes that, and the one new idea is smaller than you expect: for -coordination between processes, the kernel provides essentially **one +coordination between processes, the kernel provides **one primitive**: `futex(2)`, "sleep until this 32-bit word changes; wake the sleepers queued on it." Everything you have ever called a mutex, condition variable, semaphore, or `WaitGroup` is userspace protocol wrapped around that @@ -162,8 +162,8 @@ from a crashed writer) produces `seqlock read livelocked` instead of a hang. ## Process-shared atomics, three ways -All of this only works if atomic operations on *mapped* memory are really -atomic, and each language has its own rules for saying so. In C++, a +All of this only works if atomic operations on *mapped* memory are atomic +in practice, and each language has its own rules for saying so. In C++, a `std::atomic` must be constructed, but our words live in a file another process formatted, so `Shm` binds `std::atomic_ref` onto the raw bytes: @@ -194,12 +194,12 @@ from racing the same bytes non-atomically. The comment block at the top of `examples/20-shared-memory-coordination/go/main.go` spells the caveats out. > **Sidebar: where is eventfd?** `eventfd(2)` is a lighter kernel counter -> that chapter 9's event loops already waited on, and it measures well (2 µs +> that epoll-based event loops already wait on, and it measures well (2 µs > p50 on this host, same harness). But an eventfd is named by *file > descriptor*, not by anything in the shared page; to coordinate two > unrelated processes with one you must inherit it across `fork` or pass it -> through a UNIX socket with `SCM_RIGHTS`, exactly the machinery chapter 19 -> (UNIX sockets and fd passing) built for pmon's `logfd`. The futex needs +> through a UNIX socket with `SCM_RIGHTS`, exactly the fd-passing machinery +> built for pmon's `logfd`. The futex needs > none of that: its rendezvous *is* the mapped file. That is the deep > difference: fd-based channels need an fd-passing bootstrap; a shared > mapping is its own bootstrap. @@ -315,16 +315,15 @@ Same protocol, three idioms: acquire-load the word, refuse odd, copy the counter and all 512 slot bytes with a *plain* copy that is allowed to race, fence, re-check. Go leans on its sequentially-consistent atomics instead of explicit fences and yields to the goroutine scheduler rather than the OS. -The copy being torn sometimes is not a bug, it is the design; the re-check +The copy tears by design; the re-check is what makes it safe to have raced. `cmd_bench` is the measurement harness for the ladder. A consumer thread (`std::jthread`, a goroutine, a `thread::scope` spawn) blocks on one of the -three channels; the producer publishes and, this is the part that keeps the -numbers meaningful, **spins until the consumer acknowledges round `i-1` -before sending round `i`**, through an `ack` atomic. Without that -serialization, a fast producer queues ahead and you measure queue depth, not -wakeup latency. The mq channel carries the id through a real POSIX queue +three channels; the producer publishes and spins until the consumer +acknowledges round `i-1` before sending round `i`, through an `ack` atomic: +that serialization is what keeps the numbers meaningful. Without it, a fast +producer queues ahead and you measure queue depth, not wakeup latency. The mq channel carries the id through a real POSIX queue (created `O_EXCL`, unlinked by RAII/defer on every path; `verify.lua` checks `/dev/mqueue` is clean afterwards); Go builds `mq_open`, `mq_timedsend`, `mq_timedreceive`, and `mq_unlink` from raw `Syscall6` @@ -334,7 +333,7 @@ below confirms. A few fragile points: one writer per file is an unchecked assumption (two servers would interleave seqlock bumps and corrupt each other; a -`flock` would fix it and chapter 25's futex-built locks are the deeper +`flock` would fix it, and locks built from futexes are the deeper answer); the futex word is 32 bits, so a watcher blind for exactly 2³² updates would miss a wake until the 2 s timeout catches it; the ring back-fills at most 8 ids, and a watcher asleep for more sees only the latest @@ -371,8 +370,9 @@ fine; a thousand goroutines each parked in `FUTEX_WAIT` would be a thousand OS threads, which is precisely the degenerate case channels and the netpoller exist to avoid. The comment at the top of `go/main.go` documents this trade so the pattern is not copied blindly. The second thing worth -naming: the memory-ordering vocabulary of chapter 6 (acquire, release, -fences) crossed a process boundary in this chapter without changing at all. +naming: the memory-ordering vocabulary for concurrency primitives (acquire, +release, fences) crossed a process boundary in this chapter without +changing at all. Cache coherence does not know what a process is; orderings pair between any two threads that touch the same physical page, whichever address spaces they live in. That is why `seqlock_write`'s release fence in a Go server orders @@ -428,18 +428,18 @@ bench: channel=poll rounds=2000 p50_us=1056 p99_us=1083 ``` The Go and Rust benches bracket the same values (Go: 0/4, 3/6, 1003/1021; -Rust: 0/0, 2/5, 1061/1153 µs). Folding in the fd-based channels from -chapters 18 and 19 (pipe, UNIX stream socketpair, and the eventfd from the -sidebar), re-measured this session with the identical serialized-round -harness rebuilt around `read(2)`, gives the matrix: +Rust: 0/0, 2/5, 1061/1153 µs). Folding in the fd-based channels built +earlier (pipe, UNIX stream socketpair, and the eventfd from the sidebar), +re-measured this session with the identical serialized-round harness +rebuilt around `read(2)`, gives the matrix: -| Channel | Kernel object | p50 (µs) | p99 (µs) | Waiter blocks in | Chapter | +| Channel | Kernel object | p50 (µs) | p99 (µs) | Waiter blocks in | Built from | |---|---|---|---|---|---| -| futex wake | wait queue on (inode, offset) | <1 | 4 | `futex(FUTEX_WAIT)` | this one | -| pipe | pipe buffer | 2 | 5 | `read(2)` | 18 | -| eventfd | 64-bit counter | 2 | 6 | `read(2)` | 9, 19 | -| POSIX mq | message queue | 2 | 14 | `mq_timedreceive(2)` | this one | -| UNIX socket | socket buffer | 3 | 12 | `read(2)` | 19 | +| futex wake | wait queue on (inode, offset) | <1 | 4 | `futex(FUTEX_WAIT)` | this chapter | +| pipe | pipe buffer | 2 | 5 | `read(2)` | pipes/FIFOs | +| eventfd | 64-bit counter | 2 | 6 | `read(2)` | event loops; UNIX sockets/fd passing | +| POSIX mq | message queue | 2 | 14 | `mq_timedreceive(2)` | this chapter | +| UNIX socket | socket buffer | 3 | 12 | `read(2)` | UNIX sockets/fd passing | | 1 ms poll loop | none | 1,056 | 1,083 | `nanosleep(2)` | rung 0 | {% include excalidraw.html @@ -476,7 +476,7 @@ the layout puts the futex word), the op is plain `FUTEX_WAIT` (shared, no `_PRIVATE`), and `expected` climbs 0, 1, 2 as each observed update re-arms the next sleep. The serve side, traced in a separate run, is the mirror image: `futex(0x7f79491d300c, FUTEX_WAKE, 2147483647) = 1` three times: wake -`INT_MAX`, one waiter actually woken. And the mq bench under +`INT_MAX`, one waiter woken. And the mq bench under `-e trace=mq_open,mq_timedsend,mq_timedreceive,mq_unlink`: ```bash @@ -487,7 +487,7 @@ image: `futex(0x7f79491d300c, FUTEX_WAKE, 2147483647) = 1` three times: wake 2623348 mq_unlink("shmkv-bench-2623348") = 0 ``` -The kernel-level name really has no leading slash, confirming the Go +The kernel-level name has no leading slash, confirming the Go implementation's raw-syscall comment against the C++ binary going through glibc. Producer tid 2623348 sends, consumer tid 2623349 receives the same 8 bytes, and the unlink runs on exit. (That traced 5-round run reported @@ -517,14 +517,14 @@ both runs but on this host counted zero: stock Fedora ships `kernel.perf_event_paranoid = 2`, which restricts unprivileged perf to user-space counting, and context switches happen in the kernel; the elapsed-time line still tells the story (0.004 s futex vs 2.125 s poll for -the same 2,000 rounds), and the chapter 28 debugging toolbelt revisits perf +the same 2,000 rounds), and the remote-debugging toolbelt revisits perf with `CAP_PERFMON`. > **On the lab VM** unverified: > the eBPF view of this chapter is `offcputime` proving the watcher's stack > is parked in `futex_wait_queue` (not spinning) and `futexctn` summarizing > contention fleet-wide; bcc-tools need privileges this host's lane does not -> use, and chapter 30 (eBPF observation tools) exercises exactly that on the +> use, and the eBPF observation toolkit exercises exactly that on the > `systems-target` VM. ## What you learned @@ -563,6 +563,6 @@ runs performed now; the strace excerpts are real trimmed output (shared FUTEX_WAKE INT_MAX returning 1, slashless kernel mq names, mq_unlink on exit); the context-switch evidence (watch 2→10, poll 285→4,059 over 4 s; rusage 120 vs 2,002) was sampled live; and the -perf_event_paranoid = 2 limitation was observed, not assumed. +perf_event_paranoid = 2 restriction was observed, not assumed. The bcc-tools callout is unverified as marked, and all latency absolutes are bounded by this host.

diff --git a/_docs/21-tcp-sockets.md b/_docs/21-tcp-sockets.md index 3a9ad32..e4f9fde 100644 --- a/_docs/21-tcp-sockets.md +++ b/_docs/21-tcp-sockets.md @@ -22,8 +22,8 @@ By the end of this chapter you can: - Build `chatterd` v0, a thread-per-connection TCP chat room, in C++, Go, and Rust with an RAII/move-only socket that owns every fd - Identify the two failure modes of thread-per-connection service - (per-connection memory cost, lock-held-across-I/O blocking) that chapter - 22 addresses + (per-connection memory cost, lock-held-across-I/O blocking) that an + epoll event loop with a thread pool fixes - Verify a live server with `ss -tlnp`/`-tnp` and read the wire bytes at the `read`/`write` syscall boundary with `strace` @@ -33,7 +33,7 @@ A TCP server looks like a pipe with an address until the first production incident proves otherwise: a restart that cannot rebind its own port, a message that arrives split across two `read` calls, or a slow client that freezes every other connection behind a shared lock. Each of those is a -direct consequence of what TCP actually promises, ordered bytes with no +direct consequence of what TCP promises: ordered bytes with no message boundaries, not what a quick read-write test suggested. This chapter builds `chatterd` v0 from the four raw syscalls that make a @@ -56,7 +56,8 @@ on it: that a byte stream has no message boundaries, so *you* must add them. The code is in `examples/21-tcp-sockets/`. `./demo.sh` there builds all three implementations; its `README.md` fixes the CLI, the exit codes, and the wire -frame that chapters 22–24 and 27 grow but never replace. +frame that later event-loop, peer-discovery, deadline, and async-runtime work +grows but never replaces. {% include excalidraw.html file="21-server-lifecycle" @@ -102,8 +103,8 @@ Result listen_tcp(const std::string& host, std::uint16_t port) { `socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)` asks for an IPv4 (`AF_INET`) reliable byte stream (`SOCK_STREAM`, TCP), and folds close-on-exec -into the creating call the same way chapter 7 insisted every fd source must: -an `accept4` a moment later has the same discipline, because a listener that +into the creating call, the discipline every fd-creating call must follow: +an `accept4` a moment later carries the same flag, because a listener that leaks into a forked child is a port you can never rebind. The returned fd is not yet an address: `bind` gives it one. `sockaddr_in` is the IPv4 address struct, and its two fields carry one trap: `sin_port` and @@ -116,8 +117,8 @@ raw. Skip `htons` on a little-endian x86 box and you bind port 9000 as real IPv4 literal, so a hostname or garbage gets the `not an IPv4 address` error instead of a mysterious `bind` failure. Turning a *name* like `chat.example.com` into a `sockaddr` is a different job: `getaddrinfo(3)`, -which also picks IPv4-vs-IPv6 for you. This example stays numeric -and leaves resolution to chapter 23. +which also picks IPv4-vs-IPv6 for you. This example stays numeric; +name resolution is a peer-discovery concern. `bind` attaches the address; `listen(fd, 128)` flips the socket from active to **passive** (it will never `connect` out, only accept in) and sizes the @@ -155,8 +156,9 @@ about how they are grouped into `read` calls. A protocol that assumed "one real network the first time a segment split. The fix is **length-prefix framing**, wrapped in a small fixed header, and it -is fixed for the whole book: chapters 22, 24, and 27 add message *types* to -this exact 6-byte header, never a new one. Every message, both directions, +is fixed for the whole book: the event-loop server, deadline handling, and +async-runtime work ahead all add message *types* to this exact 6-byte header, +never a new one. Every message, both directions, opens with a 2-byte magic `0x43 0x48` (ASCII `"CH"`), a 1-byte version (`0x01`), a 1-byte `type`, then a 2-byte **big-endian** payload length (`0..65535`), and finally exactly that many payload bytes. @@ -172,7 +174,8 @@ chat `text` alone, nothing else; the server relays every MSG as a **DELIVER** frame whose payload is `name` `0x00` `text`, the *only* frame that still needs a NUL, because it is the only one carrying two fields to a reader who never sent them itself. `WELCOME` and `PING` are declared and reserved for -chapters 22 and 24 but never sent or read here. The length prefix means the +the event-loop server and deadline handling ahead but never sent or read +here. The length prefix means the reader never scans for a delimiter: it reads exactly the 6 header bytes, learns the type and size from them, reads exactly that many payload bytes, and is done. There is no separate "reject an oversized frame" check the way @@ -262,8 +265,8 @@ keeps calling `read` until `n` bytes have accumulated, retrying `EINTR` and returning false on EOF; Go and Rust get it from the standard library (`io.ReadFull`, `read_exact`). This loop is the whole point: a single `read` returning fewer bytes than asked is not an error, it is Tuesday, and -the reader that treats a short read as a complete message is the bug the -length prefix exists to prevent. The 6-byte header is read whole, then +treating a short read as a complete message is exactly what the length +prefix prevents. The 6-byte header is read whole, then checked before anything in it is trusted: `hdr[0]`/`hdr[1]` must be `'C'`/`'H'` and `hdr[2]` must be the version this build speaks (`0x01`), or the frame is rejected outright: a stray non-chatterd byte stream, or a future version this @@ -277,7 +280,7 @@ remember. None of the three distinguishes a bad magic/version from a short read with a diagnostic in this chapter: a truncated header, a wrong magic, and a truncated payload all end the connection the same quiet way a clean EOF does: `read_frame` returns `std::nullopt`/an `error`/(mostly) `Ok(None)`, and -the caller simply stops reading that connection. Rust's return type is the +the caller stops reading that connection. Rust's return type is the one place the distinction survives structurally (`Ok(None)` is EOF, `Err(...)` is a bad magic/version via `bail!`), but `serve_client`'s `_ => break` arm folds both back into the same outcome anyway. @@ -300,7 +303,7 @@ its fd in the destructor and nowhere else; there is not a single naked parameter dies and the connection closes; when `cmd_serve` returns, the listener `Socket` dies last and the port frees. Ownership *is* lifetime, so "who closes this fd, and when" is answered by scope rather than by vigilance, -the same lesson chapter 7 drew for files, now true for a server +the same RAII discipline that governs files, now true for a server that opens and closes a socket per client. ## Errors, three ways @@ -339,7 +342,7 @@ next caller. in `read_frame` per client, plus the listener thread in `accept`. It is the easiest server model to reason about: each connection is a straight-line blocking program, and the only shared state is the `Hub` behind a mutex, and -it is exactly the model that stops scaling, which is why chapter 22 exists. +it is exactly the model that stops scaling once connection counts climb. Two cracks are already visible. First, **cost per connection**: a thread is a megabyte or two of stack plus a scheduler entry, so ten thousand idle chat clients is gigabytes of RAM and a scheduler run queue that thrashes: the @@ -351,8 +354,8 @@ lock. Go sidesteps it structurally: each client owns a 64-deep buffered channel and a writer goroutine, and `broadcast` does a non-blocking `select { case c.out <- frame: case <-c.done: }`, so a wedged client is skipped rather than allowed to block the room. That contrast, a mutex held -across I/O versus a bounded queue per consumer, is the seam chapter 22 opens -up when it moves the server onto one epoll loop and a thread pool. For now, the +across I/O versus a bounded queue per consumer, is the seam that opens up +when the server moves onto one epoll loop and a thread pool. For now, the three shutdown strategies all reach the same observable end: C++ interrupts `poll` with SIGINT and lets each `std::jthread`'s `stop_callback` shut its socket down; Go closes the listener out from under `Accept` via a cancelled @@ -486,11 +489,11 @@ the same 6-byte magic/version/type/length header that would show up in a that two-phase read and the `43 48 01 03 00 0f` (DELIVER, length 15) header on the wire. - The RAII `Socket` owns every fd by scope, so there is no naked `close`; and - thread-per-connection is the simple model that chapter 22 must leave behind: - a thread and a stack per client, plus a broadcast that holds the roster - lock across I/O and stalls behind one slow reader. + thread-per-connection is the simple model an event-loop server leaves + behind: a thread and a stack per client, plus a broadcast that holds the + roster lock across I/O and stalls behind one slow reader. -Next, **scaling the server**: chapter 22 takes the same `chatterd` frame off +Next, **scaling the server** takes the same `chatterd` frame off threads and onto a single epoll loop with nonblocking sockets, so ten thousand idle clients cost fds instead of stacks. diff --git a/_docs/22-scaling-the-server.md b/_docs/22-scaling-the-server.md index c686e20..cf414fd 100644 --- a/_docs/22-scaling-the-server.md +++ b/_docs/22-scaling-the-server.md @@ -2,7 +2,7 @@ title: "Scaling the server" order: 22 part: "Networking" -description: "chatterd v1 holds many connections from one thread: nonblocking accept and per-connection read/write state machines, an outbound queue with EPOLLOUT backpressure reusing chapter 9's level-triggered epoll, then the chapter's central comparison, Go's goroutine-per-connection model with the runtime netpoller running epoll_pwait underneath, proven with per-process Threads counts, ss -tp, and strace." +description: "chatterd v1 holds many connections from one thread: nonblocking accept and per-connection read/write state machines, an outbound queue with EPOLLOUT backpressure reusing the level-triggered epoll model, then the chapter's central comparison, Go's goroutine-per-connection model with the runtime netpoller running epoll_pwait underneath, proven with per-process Threads counts, ss -tp, and strace." duration: "55 minutes" --- @@ -22,7 +22,7 @@ By the end of this chapter you can: - Guard a send to a dead peer with `MSG_NOSIGNAL`, and defer reaping dead connections to the end of an event batch instead of mid-iteration. - Read `/proc//status` thread counts and `ss -tp` to confirm how many OS - threads actually back a given connection-handling design. + threads back a given connection-handling design. - Measure the C10K cost directly with `strace -c`, comparing `clone3` counts between the epoll and threaded engines under load. @@ -43,15 +43,16 @@ lets you read a Go service's concurrency as threads parked on an epoll instance, not as magic, and lets you choose the right language for a given connection-handling budget instead of guessing. -Chapter 21 gave `chatterd` a length-prefixed frame protocol and the obvious -server: one OS thread per connection, each blocked in `recv`. That design is -correct and it is easy to reason about, right up to the point a decade of -network programmers named the **C10K problem**: ten thousand idle clients means -ten thousand kernel threads, ten thousand stacks, and a scheduler run queue that -spends more time context-switching than working. This chapter keeps the threaded -engine (it still passes every test) and adds a second one that holds the same N -connections from a *single* thread. The one new idea is the **readiness model -from chapter 9, aimed at sockets**: put every fd in nonblocking mode, let one +The TCP chat room design gave `chatterd` a length-prefixed frame protocol and +the obvious server: one OS thread per connection, each blocked in `recv`. That +design is correct and straightforward to reason about, right up to the point a +decade of network programmers named the **C10K problem**: ten thousand idle +clients means ten thousand kernel threads, ten thousand stacks, and a +scheduler run queue that spends more time context-switching than working. +This chapter keeps the threaded engine (it still passes every test) and adds +a second one that holds the same N connections from a *single* thread. The +one new idea is the **epoll-based readiness model, aimed at sockets**: put +every fd in nonblocking mode, let one `epoll_wait` tell you which are ready, and make each connection a little state machine with an inbound buffer and an outbound queue so a partial read or a blocked write is just *state*, never a stalled thread. Then comes the payoff: @@ -60,8 +61,8 @@ with it, because its runtime hides exactly this machinery underneath. The code is in `examples/22-scaling-the-server/`. `./demo.sh` there builds all three implementations and runs an end-to-end demo; its `README.md` specifies the -CLI, the wire format (unchanged since chapter 21), and the exit codes all three -languages share. +CLI, the wire format (unchanged from the original frame protocol), and the +exit codes all three languages share. {% include excalidraw.html file="22-connection-state-machine" @@ -101,7 +102,7 @@ struct EConn { ``` `inbuf` accumulates whatever `recv` returns until `take_frame` (the same -6-byte-header parser from chapter 21) can lift a whole frame off the front; +6-byte-header parser from the original TCP server) can lift a whole frame off the front; leftover bytes stay for next time. `outbuf` is the mirror image and the heart of the chapter: bytes we *want* to send but the kernel would not take. `want_write` records whether we currently have `EPOLLOUT` armed for this connection (so we @@ -110,7 +111,7 @@ reap after the current event batch rather than deleting it mid-iteration. The connections live in `std::unordered_map` keyed by an epoll **token**, `next_tok_` starting at 2 because tokens 0 and 1 are reserved for the listener and the `signalfd`, the same "let the kernel echo back a small integer -so dispatch is a `switch`, not an fd lookup" trick chapter 9 used. +so dispatch is a `switch`, not an fd lookup" trick epoll event loops use. ## How the code works @@ -156,7 +157,7 @@ happen; deferring the actual erase to `reap_closed` at the end of the batch is how a single-threaded loop safely mutates the container it is iterating events from. `accept_ready` loops `accept4(…, SOCK_CLOEXEC | SOCK_NONBLOCK)` until `EAGAIN`: one `EPOLLIN` on the listener can mean *many* pending connections, and -draining to `EAGAIN` is the level-triggered discipline from chapter 9 applied to +draining to `EAGAIN` is the level-triggered epoll discipline applied to the backlog. Each new connection is registered `EPOLLIN`-only and immediately handed a `WELCOME` frame via `queue`, which is the client's proof it is in the fan-out set. `on_readable` drains `recv` into `inbuf` to `EAGAIN`, then pulls @@ -166,7 +167,7 @@ connection. `queue` and `flush` are where backpressure lives, and where the three languages are worth reading side by side. `queue` appends to `outbuf` and calls `flush`; -`flush` sends as much as the kernel will take and, crucially, *decides whether to +`flush` sends as much as the kernel will take and *decides whether to arm `EPOLLOUT`* based on what is left over: {% include codetabs.html langs="C++|Go|Rust" %} @@ -287,8 +288,9 @@ always drains to `EAGAIN` and parks on the transition. ## Errors, three ways -The contract is chapter 21's: exit 0 on success, 1 on a runtime failure with one -diagnostic line, 2 on a usage error, and `verify.lua` asserts all three per +The contract carries over unchanged: exit 0 on success, 1 on a runtime failure +with one diagnostic line, 2 on a usage error, and `verify.lua` asserts all +three per language (24 checks each, 72 total). The interesting split is which side owns which error. A client against a dead port fails `connect` and exits 1 with `chatctl: cannot connect to 127.0.0.1:1: Connection refused`: `std::error_code` @@ -314,12 +316,12 @@ mutex so two broadcasters can't interleave bytes on one wire, and `std::atomic` counters; simple per connection, but every message wakes N reader threads. Go is a bounded handful of runtime threads regardless of connection count, with `pending`-style state confined to the hub goroutine and everything else -communicating by channel: chapter 6's "share memory by communicating" applied to -I/O. Measured on this host with 50 connections held open: the C++ epoll daemon -reports `Threads: 1`, the C++ threaded daemon `Threads: 51`, and the Go daemon -`Threads: 7` (near, but not pinned to, this box's 16 CPUs; the runtime grows -threads on demand up to `GOMAXPROCS` runnable). One process, one design decision, -three very different rows in `/proc//status`. +communicating by channel: the "share memory by communicating" concurrency +idiom applied to I/O. Measured on this host with 50 connections held open: the +C++ epoll daemon reports `Threads: 1`, the C++ threaded daemon `Threads: 51`, +and the Go daemon `Threads: 7` (near, but not pinned to, this box's 16 CPUs; +the runtime grows threads on demand up to `GOMAXPROCS` runnable). One process, +one design decision, three different rows in `/proc//status`. ## Build, run, observe @@ -436,8 +438,8 @@ counter, `clone3: 500` versus `clone3: 0`, *is* the C10K problem in one line. survives across wakeups rather than a thread blocked in `recv`. - Backpressure is the outbound queue plus `EPOLLOUT`: a slow reader fills its own `outbuf`, we arm write-interest, and the next `epoll_wait` drains it: no thread - blocks, no frame drops, no other client is affected. Level-triggered epoll from - chapter 9 makes this safe with only "consume what woke you" discipline. + blocks, no frame drops, no other client is affected. Level-triggered epoll + makes this safe with only "consume what woke you" discipline. - Go's goroutine-per-connection code is the same epoll engine with the loop hidden: the netpoller runs `epoll_pwait` (edge-triggered) underneath and parks goroutines, not threads: 1 vs 51 vs 7 threads in `/proc`, all serving 50 diff --git a/_docs/23-udp-and-peer-discovery.md b/_docs/23-udp-and-peer-discovery.md index 84bf8f4..5e14b3b 100644 --- a/_docs/23-udp-and-peer-discovery.md +++ b/_docs/23-udp-and-peer-discovery.md @@ -2,7 +2,7 @@ title: "UDP and peer discovery" order: 23 part: "Networking" -description: "chatterd learns to find peers it was never told about: UDP multicast beacons on 239.23.7.1, the IP_ADD_MEMBERSHIP / IP_MULTICAST_TTL setsockopt dance, a small gossip protocol deduped by name, and the bridge from a discovered datagram to the canonical chatterd TCP chat frame (introduced ch21); proven across two libvirt guests with tcpdump, ss -uap, and both transcripts." +description: "chatterd learns to find peers it was never told about: UDP multicast beacons on 239.23.7.1, the IP_ADD_MEMBERSHIP / IP_MULTICAST_TTL setsockopt dance, a small gossip protocol deduped by name, and the bridge from a discovered datagram to the canonical chatterd TCP chat frame; proven across two libvirt guests with tcpdump, ss -uap, and both transcripts." duration: "55 minutes" --- @@ -24,7 +24,7 @@ By the end of this chapter you can: - Verify multicast group membership and socket state with `ip maddr` and `ss -uap`, and capture beacons on the wire with `tcpdump -X`. - Run a two-host discovery exchange across a libvirt bridge and explain why a - TTL-1 multicast beacon stays link-local without any NAT workaround. + TTL-1 multicast beacon stays link-local without any NAT configuration. ## Why this matters @@ -42,9 +42,9 @@ ordering and reliability, so it belongs on TCP. Picking the wrong protocol for either half either drops messages you needed or pays connection overhead you did not. -Chapters 21 and 22 built `chatterd` on TCP: the canonical chatterd chat +`chatterd` already runs on TCP, carrying the canonical chat frame (magic `"CH"`, version, type, a big-endian length, and a payload) -carried between two endpoints you already knew, since you typed the peer's +between two endpoints you already knew, since you typed the peer's address on the command line. This chapter removes that assumption. `chatterd discover` finds peers it was never told about, and the primitive that makes that possible is the one TCP cannot offer: **UDP multicast**, a single @@ -52,7 +52,7 @@ that possible is the one TCP cannot offer: **UDP multicast**, a single who (or how many) they are. The shape of the chapter is a bridge: a connectionless UDP beacon announces "I exist, dial me here," a receiver hears strangers and dedupes them, and the first time it meets a new name it opens -an ordinary TCP connection and speaks the *same canonical chat frame ch21 +an ordinary TCP connection and speaks the *same canonical chat frame already introduced*: the header never changes; each version only adds message *types*. Discovery is UDP; the conversation stays TCP. The star of the run is real: two libvirt guests, `systems-target` and `systems-peer`, each @@ -64,7 +64,7 @@ beacon and frame formats, and the exit codes all three languages share. {% include excalidraw.html file="23-two-vm-topology" - alt="A libvirt virbr0 bridge band on 192.168.124.0/24 holds a central amber multicast-group box for 239.23.7.1 port 51824 TTL 1, with systems-target at 192.168.124.7 on the left and systems-peer at 192.168.124.95 on the right, each joined to the group on enp1s0 and holding udp fd 4 and tcp 9241; two dashed amber bidirectional beacon arrows (35 bytes from target, 34 bytes from peer) reach the group, and a solid bidirectional TCP arrow labelled one ch21 chat frame each way joins the two guests directly." + alt="A libvirt virbr0 bridge band on 192.168.124.0/24 holds a central amber multicast-group box for 239.23.7.1 port 51824 TTL 1, with systems-target at 192.168.124.7 on the left and systems-peer at 192.168.124.95 on the right, each joined to the group on enp1s0 and holding udp fd 4 and tcp 9241; two dashed amber bidirectional beacon arrows (35 bytes from target, 34 bytes from peer) reach the group, and a solid bidirectional TCP arrow labelled one chat frame each way joins the two guests directly." caption="Figure 23.1: the two-guest lab topology: both VMs join 239.23.7.1, beacon to the group, and bridge each discovery to a direct TCP :9241 chat frame; every address, port, and byte-length here is from the two-host run later in the chapter" %} > **Tools used**: `ss`, `tcpdump`, `ip`, `strace` (host); `tcpdump`, `ss`, @@ -112,7 +112,7 @@ two machines, where the packet is on the wire regardless. {% include excalidraw.html file="23-discovery-sequence" alt="Two horizontal bands. The upper blue band, UDP multicast discovery, holds three boxes left to right: announce loop sendto to group emitting CHATTERD1 target 9241; receive loop recv one datagram splitting four fields and checking magic; and an amber parse-and-dedup box that skips its own name and calls seen.first before dialing. An amber arrow labelled new peer bridge to TCP drops into the lower grey band, TCP bridge, whose boxes are connect with retry, write_frame slash read_frame with a 6-byte header (magic, version, type, 2-byte big-endian length) plus payload, and a dark stdout box printing discovered peer and peer says. A note reads loss and reorder tolerated." - caption="Figure 23.2: one peer's path from a connectionless beacon to a connection-oriented frame: recv, dedup by name, then dial and speak the canonical chat frame ch21 introduced (JOIN out, DELIVER back)" %} + caption="Figure 23.2: one peer's path from a connectionless beacon to a connection-oriented frame: recv, dedup by name, then dial and speak the canonical chat frame already introduced (JOIN out, DELIVER back)" %} ## How the code works @@ -235,7 +235,7 @@ each dials once and each prints exactly one `peer … says:` line. The `` field is the one fragile convention to flag: a peer advertises the address it wants to be dialed at (its `--iface`), so behind NAT or on a different subnet it would advertise -an unreachable address, a limitation a real gossip protocol solves with +an unreachable address, a case a real gossip protocol handles with observed source addresses, out of scope here. ## Errors, three ways @@ -266,7 +266,7 @@ in a `Seen` struct with a `std::mutex` (defensive, since a future second reader would need it), while Go confines its `map[string]bool` to the `recvLoop` goroutine and Rust confines its `HashSet` to the `recv_loop` thread, so neither needs a lock at all: the state is owned by one flow, which -is Chapter 6's "share memory by communicating" reflex applied to sockets. +is the "share memory by communicating" reflex, applied to sockets. Shutdown is the interesting coordination. There is no natural end to a `recv` loop, so each waits on a stop signal *and* a 200 ms socket read timeout: the timeout wakes the blocked `recv` often enough to observe the @@ -310,7 +310,7 @@ UNCONN 0 0 0.0.0.0:51833 0.0.0.0:* users:(("app",pid=2818975,fd=4)) inet 239.23.7.1 ``` -Loopback multicast is the one quirk to know: `lo` is not flagged `MULTICAST`, +Loopback multicast behaves differently from a normal link: `lo` is not flagged `MULTICAST`, but the IPv4 stack still loops group traffic when the multicast interface is set to `127.0.0.1`, which is why the local demo passes `--iface 127.0.0.1`. Output equality alone could be faked by a program that quietly skipped the @@ -350,7 +350,7 @@ UNCONN 0 0 0.0.0.0:51824 0.0.0.0:* users:(("chatterd",pid=2105,fd=4)) inet 239.23.7.1 ``` -`ip maddr` proves the kernel actually joined the group on `enp1s0` (the +`ip maddr` proves the kernel joined the group on `enp1s0` (the observable effect of `IP_ADD_MEMBERSHIP`), and `ss -uap` shows the same `UNCONN` fd-4 UDP socket as on the host. Third, the payoff: both guests' stdout, quoted verbatim from the two-host run: @@ -368,7 +368,7 @@ peer target says: hello from target ``` Each guest discovered the other exactly once and exchanged one TCP frame: the -message crossed the network between two separate hosts. No libvirt workaround was needed: both +message crossed the network between two separate hosts. No special libvirt configuration was needed: both guests sit on the same `virbr0` L2 segment, so a TTL-1 multicast beacon is delivered by the bridge as ordinary link-local traffic; libvirt's NAT only governs traffic *leaving* the host toward the internet, which discovery never @@ -395,7 +395,7 @@ does. un-fragmented packet, and a receiver dedupes by name so one peer is announced once regardless of beacon count. - The bridge from discovery to conversation is a plain `connect` to the - advertised endpoint, reusing the canonical chat frame ch21 introduced (a + advertised endpoint, reusing the canonical chat frame already introduced (a JOIN out, a DELIVER back): proven across two guests, each printing one `discovered peer` and one `peer … says:` line. @@ -422,6 +422,6 @@ IP-header TTL byte 01); ss -uanp showed the joined chatterd fd-4 UDP socket and ip maddr show enp1s0 the 239.23.7.1 membership; and both guests' stdout printed one discovered peer and one peer … says: line as quoted. -No libvirt NAT workaround was required: the guests share one virbr0 +No special libvirt NAT configuration was required: the guests share one virbr0 L2 segment. The bcc-tools tcpconnect/tcplife/offcputime callout is unverified as marked and is exercised in Part 8.

diff --git a/_docs/24-time-timers-deadlines.md b/_docs/24-time-timers-deadlines.md index d849600..95521b0 100644 --- a/_docs/24-time-timers-deadlines.md +++ b/_docs/24-time-timers-deadlines.md @@ -42,7 +42,7 @@ jittered exponential backoff is the standard discipline for reconnecting to a recovering service without a fleet of clients hammering it back down the moment it comes up. -Chapters 21–23 built `chatterd` into a peer-to-peer chat daemon that links two +`chatterd` is a peer-to-peer chat daemon that links two hosts over TCP and passes framed messages. It has one fatal naïveté: it assumes the peer stays. Pull the cable, kill the process, or let a switch drop the flow, and a `chatterd` peer sits forever in a `read` that will never return: @@ -55,13 +55,13 @@ turns a heartbeat into a liability the first time NTP corrects your wall clock. The code is in `examples/24-time-timers-deadlines/`. `./demo.sh` there builds all three implementations; its `README.md` specifies the CLI, the wire format -(the same canonical frame chapters 21–23 used, now carrying a `PING` TYPE this +(the same canonical frame the chat protocol already used, now carrying a `PING` TYPE this chapter adds), and the exit codes all three languages share. {% include excalidraw.html file="24-clock-taxonomy" alt="Top band: the four clocks clock_gettime can read as boxes: CLOCK_REALTIME (dashed, wall time, settable, can step backward), CLOCK_MONOTONIC (amber, since a boot epoch, never steps, steady_clock/Instant), CLOCK_BOOTTIME (monotonic plus suspend time), and PROCESS_CPUTIME_ID (this process's on-CPU time). Bottom band: while a peer is silent and ntpd corrects the wall clock, a dashed REALTIME box shows a wall-clock deadline rewinding so a dead peer looks alive, beside an amber MONOTONIC box where elapsed is always non-negative and the timeout fires on real silence; an amber arrow from CLOCK_MONOTONIC points to it labeled chatterd reads this." - caption="Figure 24.1: the clock taxonomy and the monotonic-vs-wall bug: a wall-clock deadline slips when the clock is stepped; a CLOCK_MONOTONIC deadline does not" %} + caption="Figure 24.1: the clock taxonomy and the monotonic-vs-wall-clock hazard: a wall-clock deadline slips when the clock is stepped; a CLOCK_MONOTONIC deadline does not" %} > **Tools used**: `strace`, `python3` (host); `cyclictest` (systems-target > VM, exercised in Part 8). Everything host-side is checked by @@ -85,7 +85,7 @@ useful for profiling, useless for wall deadlines. The rule that follows is absolute: **any duration or deadline must be measured on a monotonic clock.** A heartbeat timeout is a duration ("no traffic for 3 seconds"), so it reads `CLOCK_MONOTONIC`, never `CLOCK_REALTIME`. Figure 24.1 -is the bug you avoid by obeying this. Suppose you computed a deadline as +is the failure mode you avoid by obeying this. Suppose you computed a deadline as `wall_now() + 3s` and stored it. Halfway through, `ntpd` decides the clock is 0.5 s fast and steps it *back*. Now `wall_now()` reads earlier than when you started; your deadline recedes into the future; a peer that died stays "alive" @@ -100,7 +100,7 @@ so `time.Since` cannot be fooled by a wall-clock step. Reading the clock happens constantly (on every loop iteration `chatterd` recomputes how long until the deadline), so it had better be cheap. It is, and -the reason is chapter 4's vDSO. `CLOCK_MONOTONIC` and `CLOCK_REALTIME` reads do +the reason is the vDSO. `CLOCK_MONOTONIC` and `CLOCK_REALTIME` reads do not trap into the kernel: the kernel maps a shared page (the vDSO) into every process, updates a timekeeping structure there on each tick, and the libc `clock_gettime` wrapper reads it in userspace with no `syscall` instruction. @@ -227,8 +227,8 @@ timer wrapped in a file descriptor: `timerfd_create(CLOCK_MONOTONIC, …)` makes one, `timerfd_settime` arms it (an `it_value` with no `it_interval` is a one-shot; both set makes it periodic), and it becomes *readable* when it expires: a `read` returns a `uint64` count of expirations. That last property -is why it belongs in this chapter: a timer that is an fd drops straight into the -`poll`/`epoll` loop chapter 9 built, no signal handler required. +is why it belongs in this chapter: a timer that is an fd drops straight into an existing +`poll`/`epoll` event loop, no signal handler required. The session loop is where the deadline lives. In the C++ `run_session`, every iteration computes how long to block from the monotonic deadline, and hands @@ -252,7 +252,7 @@ MSG, or PING, all count as traffic. The `poll` array holds three fds: the socket. When `poll` returns 0, the timeout elapsed with nothing to read (the peer went silent for `timeout_ms`) and the session returns `TimedOut`. When the heartbeat fd fires, the loop reads its 8-byte tick and sends a `PING`. This -is the same one-loop-three-sources shape as chapter 9's `fwatch`, now serving +is the same one-loop-three-sources shape as the `epoll`-based `fwatch` event loop, now serving liveness instead of file events. The three languages express the deadline differently: C++ subtracts two `time_point`s, Rust does the identical arithmetic on `Instant`, and Go resets a `time.Timer` and `select`s on its @@ -311,7 +311,7 @@ error returns in Go, `Result` in Rust, all printing the same `usage:` block. The C++ and Rust sessions are single-threaded: one `poll` loop multiplexes the socket, the heartbeat timerfd, and the shutdown signalfd, so there is no shared state to lock: the timer and the deadline are just two more fds and one -`time_point`. Go inverts it, exactly as chapter 9 did: a per-connection reader +`time_point`. Go inverts it, exactly as its event-loop examples did: a per-connection reader goroutine turns the socket into a `chan frame`, and the session is a `select` over four cases: `ticker.C` (heartbeat), `deadline.C` (liveness), the frame channel, and `ctx.Done()` (shutdown from `signal.NotifyContext`). The subtlety @@ -344,7 +344,7 @@ All three languages agree to within noise: Go printed `actual=1066us` / `1005us`, Rust `1055us` / `1002us`. The number to sit with is the overshoot: a 1 ms sleep took ~1056 µs, ~56 µs long. That gap is not the clock's resolution (1 ns); it is the *scheduler's wake-up latency*, the time between the timer -expiring and your thread actually getting a CPU. On a general-purpose kernel +expiring and your thread getting a CPU. On a general-purpose kernel this is tens of microseconds and variable; the timerfd path is a touch tighter (~1013 µs) because it wakes straight out of `read` rather than through `nanosleep`'s return. Now the heartbeat scenario, driven by hand: a listener @@ -431,7 +431,7 @@ Every claim in the chapter is on those lines. `timerfd_settime` arms the heartbeat with a 300 ms interval. The `sendto(4, "CH\1\1\0\1B", 7, …)` is the 7-byte JOIN frame, `43 48 01 01 00 01 42`, the canonical header (`CH`, version 1, type 1 = JOIN) plus the name `B`, matching the wire format byte -for byte, the same header chapters 21–23 used; only the TYPE byte and the +for byte, the same header the chat protocol already used; only the TYPE byte and the engine around it differ by chatterd version. When fd 5 (the timerfd) fires, the loop reads its 8-byte tick (`\1\0\0\0\0\0\0\0`, one expiration) and sends the 6-byte PING `CH\1\5\0\0` (`43 48 01 05 00 00`). And diff --git a/_docs/28-gdb-and-remote-debugging.md b/_docs/28-gdb-and-remote-debugging.md index 717ef24..19b033a 100644 --- a/_docs/28-gdb-and-remote-debugging.md +++ b/_docs/28-gdb-and-remote-debugging.md @@ -37,7 +37,7 @@ occasionally or only in an environment you cannot freely recompile in. The three languages also make the same bug look completely different at the tooling layer: a core file is the right artifact for C++, useless for Rust's panic-and-unwind, and only accidentally adequate for Go once goroutines enter -the picture. Knowing which tool actually sees your language's failure mode, +the picture. Knowing which tool sees your language's failure mode, and which one just looks like it does, is what keeps a debugging session from turning into a search for frames that were never going to be there. @@ -92,7 +92,7 @@ gdb's job starts where that photograph does. Opened against a binary that still carries debug info; this chapter's C++ build uses the `release` CMake preset, which is `RelWithDebInfo` (`-O2 -g`, optimized *and* debuggable), gdb reconstructs source-level frames, local variables, and -even C++ types from the DWARF the compiler embedded, not from guesswork. +even C++ types straight from the DWARF the compiler embedded. That last part is what "pretty-printers" buy you: without them, a `std::string_view` is a two-word struct (`{ _M_str, _M_len }`) and you'd read its text by hand out of memory; with libstdc++'s printers loaded (they ship @@ -162,7 +162,7 @@ Thread 1 "app" received signal SIGSEGV, Segmentation fault. That result is flattering and it hides the real problem. gdb's thread list maps to **OS threads** (the `M`s in Go's `G`/`M`/`P` scheduler), not to **goroutines** (`G`s): a goroutine that is parked on a channel, blocked in a -`select`, or simply not currently scheduled onto any `M` has no OS thread for +`select`, or not currently scheduled onto any `M` has no OS thread for gdb to show you, and gdb has no notion of "goroutine 41" as a selectable frame context the way it has "thread 3". The moment your program has more than a handful of concurrent goroutines, `info threads` and `bt` stop mapping @@ -241,7 +241,7 @@ Remote debugging using systems-target:2345 afterward; the right default for a scripted reproduction, since a stray `gdbserver` left listening on a lab VM is a debug backdoor with no authentication. `target remote` is the one gdb command that changes -everything downstream of it: every `break`, `run` (really `continue`, since +everything downstream of it: every `break`, `run` (which behaves as `continue`, since the process is already started), `print`, and `bt` you already know behaves identically, dispatched over the wire instead of a local `ptrace(2)`. The binary path you pass to gdb on the host must be the *unstripped*, debug-info @@ -397,20 +397,20 @@ languages part ways, and the table is the chapter in miniature: | Go | nil deref → runtime catches `SIGSEGV`, converts to panic | 2 | no | the goroutine trace in the panic output | | Rust | `Option::unwrap()` on `None` → panic | 101 | no | `RUST_BACKTRACE=1` | -One more thing worth reading off the C++ trace directly: the `ShutdownGuard` -RAII type in `run_supervisor`'s clean path prints "supervisor exiting -cleanly" from its destructor, and that line never appears on the crash path, -confirmed in every transcript above. RAII buys you cleanup on every -*C++-level* exit (`return`, a thrown exception, `std::exit` via a registered -handler); it buys you nothing against a signal, because a signal does not -unwind the stack; the kernel simply stops the process where it stands. +The C++ trace also shows the `ShutdownGuard` RAII type in `run_supervisor`'s +clean path printing "supervisor exiting cleanly" from its destructor; that +line never appears on the crash path, confirmed in every transcript above. +RAII buys you cleanup on every *C++-level* exit (`return`, a thrown +exception, `std::exit` via a registered handler); it buys you nothing +against a signal, because a signal does not unwind the stack: the kernel +stops the process where it stands. ## What each language's tooling gives you -This chapter is tooling-heavy rather than concurrency-heavy, so the lens -here is what each debugger sees, not a data race: gdb sees raw memory, -registers, and DWARF: it is precise but blind to any concept a language -runtime invented for itself. That is exactly why it is the *only* tool for +gdb, `delve`, and `rust-gdb` differ in what they see: gdb sees raw memory, +registers, and DWARF, precise but blind to any concept a language runtime +invented for itself. +That is exactly why it is the *only* tool for the C++ crash (there is no higher-level runtime to ask) and only an *accidentally* adequate one for Go (it happened to catch this crash while the program was still single-goroutine); `delve` exists specifically to @@ -484,7 +484,7 @@ without it. Every address gdb reported above is derived from the same `.debug_line` section `readelf` and `objdump` can decode independently, with no gdb involved at all; if they agree, gdb isn't inventing anything. `readelf -S` -first confirms the binary actually carries full DWARF (it is optimized, not +first confirms the binary carries full DWARF (it is optimized, not stripped): ```console @@ -552,7 +552,7 @@ and gdb all read the same way. on `.debug_line` are the independent cross-check that confirms gdb's own addresses rather than trusting them blind. -Next, **valgrind, sanitizers, and Miri**: three very different ways of +Next, **valgrind, sanitizers, and Miri**: three different ways of catching the memory bugs a debugger only shows you *after* they've already happened. diff --git a/_docs/29-valgrind-sanitizers-miri.md b/_docs/29-valgrind-sanitizers-miri.md index 8670ff1..5df53ef 100644 --- a/_docs/29-valgrind-sanitizers-miri.md +++ b/_docs/29-valgrind-sanitizers-miri.md @@ -48,7 +48,7 @@ need to worry about (directly from this chapter's own catch/miss matrix) is what turns "run every tool on everything" into picking the right one for the binary in front of you. -Chapter 28 read a crash after it happened. This chapter catches five bugs +A debugger reads a crash after it happened. This chapter catches five bugs *before* they get the chance to crash anything; or, in one case, watches a tool fail to catch one at all. `bugfarm` is a single CLI, `app `, seeding one specific memory or @@ -70,8 +70,8 @@ directly, since instrumented builds are outside every chapter's normal {% include excalidraw.html file="29-tool-coverage-matrix" - alt="A grid with defect down the rows -- leak, uninit, uaf, overflow, race -- and C++, Go, Rust across the columns. C++/leak: valgrind memcheck caught it, definitely lost. C++/uninit: valgrind memcheck caught it, uninitialised value(s); ASan alone missed it, needs MemorySanitizer. C++/uaf: valgrind caught it (invalid write, freed) and ASan caught it (heap-use-after-free). C++/overflow: valgrind caught it (0 bytes after a block) and ASan caught it (heap-buffer-overflow). C++/race: memcheck is not built for this and was not the tool tried; helgrind caught it (possible data race) and ThreadSanitizer caught it. Go/leak: self-detected via goroutines before and after; valgrind was confused, 630 errors and 0 real signal. Go/uninit, uaf, overflow: prevented by the runtime, exit 64, no tool needed. Go/race: go -race caught it, WARNING DATA RACE, exit 66. Rust/leak: valgrind caught it, 65536 bytes lost, and miri on nightly additionally reported memory leaked. Rust/uninit, uaf, overflow, race: prevented at compile time by the borrow checker, exit 64; a note says unsafe Rust plus miri catches the use-after-free class separately, shown elsewhere in the chapter. A footer legend says amber cells mean a tool run this session caught the defect, dark cells mean the language prevents it outright, and plain cells mean a real blind spot this session measured; every cell traces to a command in Build run observe or Cross-check." - caption="Figure 29.1: the five bugfarm defects against three languages, every cell backed by a command this chapter actually ran" %} + alt="A grid with defect down the rows (leak, uninit, uaf, overflow, race) and C++, Go, Rust across the columns. C++/leak: valgrind memcheck caught it, definitely lost. C++/uninit: valgrind memcheck caught it, uninitialised value(s); ASan alone missed it, needs MemorySanitizer. C++/uaf: valgrind caught it (invalid write, freed) and ASan caught it (heap-use-after-free). C++/overflow: valgrind caught it (0 bytes after a block) and ASan caught it (heap-buffer-overflow). C++/race: memcheck is not built for this and was not the tool tried; helgrind caught it (possible data race) and ThreadSanitizer caught it. Go/leak: self-detected via goroutines before and after; valgrind was confused, 630 errors and 0 real signal. Go/uninit, uaf, overflow: prevented by the runtime, exit 64, no tool needed. Go/race: go -race caught it, WARNING DATA RACE, exit 66. Rust/leak: valgrind caught it, 65536 bytes lost, and miri on nightly additionally reported memory leaked. Rust/uninit, uaf, overflow, race: prevented at compile time by the borrow checker, exit 64; a note says unsafe Rust plus miri catches the use-after-free class separately, shown elsewhere in the chapter. A footer legend says amber cells mean a tool run this session caught the defect, dark cells mean the language prevents it outright, and plain cells mean a real blind spot this session measured; every cell traces to a command in Build run observe or Cross-check." + caption="Figure 29.1: the five bugfarm defects against three languages, every cell backed by a command this chapter ran" %} > **Tools used**: `valgrind` 3.27.1 (`--tool=memcheck`, `--tool=helgrind`, > `--tool=massif`), `cmake`/`g++` 16.1.1 with the `asan`/`tsan` CMake presets @@ -130,14 +130,14 @@ whose every load and store already carries its own inline check: no separate CPU, no re-translation, just a few extra machine instructions the compiler emitted. ASan's mechanism is **shadow memory**: one shadow byte covers eight bytes of application memory (`shadow_addr = (app_addr >> 3) + -offset`), and specific byte values mean specific things — `00` fully +offset`), and specific byte values mean specific things: `00` fully addressable, `fa` a heap redzone the allocator placed around every block to catch overflow, `fd` a freed block's bytes repainted so any further access trips instantly. {% include excalidraw.html file="29-shadow-memory" - alt="Two stacked bands. The upper application-memory band shows three boxes: a left redzone at addresses ending ff80 through ffff, a 4-byte int* p at address ending 0010 marked as new'd then delete'd with the note that writing through it is undefined behavior, and a right redzone at addresses ending 0018 through 007f. The lower shadow-memory band shows the corresponding three shadow bytes: fa labeled heap left redzone, fd labeled freed heap region and marked as the exact byte the write trips, and fa labeled heap left redzone again for the next block's guard. An arrow labeled shift right by 3 then add offset connects the freed block down to its fd shadow byte. A footer formula reads shadow address equals app address shifted right 3 plus offset, one comparison before every load or store, not a full CPU emulation, and a note gives the real legend from this session's ASan run -- 00 addressable, 01 through 07 partially addressable, fa heap redzone, fd freed heap region, f1 through f8 stack states, f9 global redzone -- plus the measured gap: 0.002s native, 0.046s under ThreadSanitizer, 1.0s under valgrind memcheck, for the same tiny program." + alt="Two stacked bands. The upper application-memory band shows three boxes: a left redzone at addresses ending ff80 through ffff, a 4-byte int* p at address ending 0010 marked as new'd then delete'd (writing through it is undefined behavior), and a right redzone at addresses ending 0018 through 007f. The lower shadow-memory band shows the corresponding three shadow bytes: fa labeled heap left redzone, fd labeled freed heap region and marked as the exact byte the write trips, and fa labeled heap left redzone again for the next block's guard. An arrow labeled shift right by 3 then add offset connects the freed block down to its fd shadow byte. A footer formula reads shadow address equals app address shifted right 3 plus offset, one comparison before every load or store, not a full CPU emulation, and a footer gives the real legend from this session's ASan run (00 addressable, 01 through 07 partially addressable, fa heap redzone, fd freed heap region, f1 through f8 stack states, f9 global redzone), plus the measured gap: 0.002s native, 0.046s under ThreadSanitizer, 1.0s under valgrind memcheck, for the same tiny program." caption="Figure 29.2: ASan's shadow-memory model, with the real freed-block shadow bytes this chapter's `uaf` run produced" %} Running the `uaf` binary under ASan pulls exactly the `fd` byte at @@ -155,8 +155,8 @@ stays silent here: its absence from these transcripts is expected, not a missed catch. **TSan** needs its own separate build (ASan and TSan cannot be linked into the same binary) and uses yet another shadow model (a vector clock per thread recording happens-before relationships) to catch a race -*while it actually executes*, meaning it can only report interleavings the -run actually hits. The overhead lands close to native either way: `race` +*while it executes*, meaning it can only report interleavings the +run hits. The overhead lands close to native either way: `race` measured **0.03-0.046s** under TSan versus **0.002s** plain, roughly the constant-factor cost of a few inlined instructions per access, not valgrind's order-of-magnitude software CPU. @@ -213,7 +213,7 @@ and `unsafe` is where Rust admits it. Each language's `main.cpp` / `main.go` / `main.rs` shares one shape: a `Bug` enum with exactly five variants (`enum class Bug` in C++, a plain string switch in Go, `enum Bug` in Rust), a `parse_bug` function returning -that enum or an error through each language's Chapter 5 error-handling +that enum or an error through each language's own error-handling idiom (`std::expected`, `(string, error)`, `Result`), and one `do_*` function per variant holding the seeded defect. The enum, not a raw string comparison scattered through `main`, is what makes "five bugs, three diff --git a/_docs/30-ebpf-observation-toolkit.md b/_docs/30-ebpf-observation-toolkit.md index f5b2333..67146b3 100644 --- a/_docs/30-ebpf-observation-toolkit.md +++ b/_docs/30-ebpf-observation-toolkit.md @@ -20,14 +20,14 @@ By the end of this chapter you can: - Attach a `uprobe`/`uretprobe` pair to time a userspace function's entry-to-return latency and read the resulting histogram. - Use `bpftool prog show`/`map show` to independently confirm what the kernel - actually has loaded, cross-checking what `bpftrace` itself reports. + has loaded, cross-checking what `bpftrace` itself reports. - Demangle Itanium (C++) and Rust v0 mangled symbol names with `c++filt` to find the exact string a uprobe must attach to. - Explain why Go's register-based ABIInternal breaks the System-V-assuming `arg0`/`argN` shorthand, and verify the mismatch by reading `RDI` against `RAX` on a live uprobe. - Compare this eBPF-based observation path against SystemTap's kernel-module - and dyninst backends for the same kind of probe. + and dyninst backends for the same probe. ## Why this matters @@ -44,14 +44,14 @@ It also surfaces a real trap worth knowing before you hit it live: tools that read function arguments by position assume the System V calling convention, and Go's compiler does not follow it. A uprobe can report a plausible-looking wrong number instead of an error, and the only defense is knowing which -register a given language and function actually use, confirmed here by +register a given language and function use, confirmed here by reading `RDI` next to `RAX` on the same call. -Chapter 28 read a crash gdb already knew about; Chapter 29 caught bugs a -compile-time instrumentation pass or a binary-translating emulator watched -for. Both chapters needed something extra glued onto the target: debug -info gdb could walk, or a sanitizer/valgrind rebuild. This chapter needs -neither. `app work --seconds N` is an ordinary, unmodified binary (no +A debugger session reads a crash it already knows about; sanitizers and +valgrind catch bugs a compile-time instrumentation pass or a +binary-translating emulator watched for. Both approaches need something +extra glued onto the target: debug info gdb can walk, or a +sanitizer/valgrind rebuild. This chapter needs neither. `app work --seconds N` is an ordinary, unmodified binary (no special build, no `-fsanitize`, not even a debugger attached), and the standard eBPF observation stack (bcc-tools, bpftrace, bpftool) watches it entirely from outside, live, on the `systems-target` lab VM this book has @@ -74,7 +74,7 @@ the full contract and a real run's tail. {% include excalidraw.html file="30-ebpf-tooling-stack" - alt="Three stacked bands. The top kernel band holds five small boxes -- kprobe/kretprobe on kernel functions, uprobe/uretprobe on userspace functions, tracepoint as the stable kernel ABI, perf_event as timer/PMU sampling, and BTF/CO-RE for struct layout without headers. Amber arrows descend from three of these into the middle eBPF-tooling band, which holds bcc-tools (execsnoop/opensnoop/biolatency/offcputime/funccount/profile), bpftrace (one-liners and scripts, probe:target /filter/ action), bpftool (prog show/map show), and c++filt (demangle the exact symbol you attach to), with a note that libbpf loads a small verified BPF program per probe that the kernel runs in-kernel on each hit. Below that, a third band holds SystemTap as the sibling technology: its classic kernel-module backend needing matching kernel-devel, beside stap --runtime=dyninst, which patches the target binary directly in userspace with no kernel module. A fourth band at the bottom shows the three target binaries -- C++'s busy_hash under the SysV ABI with a correct arg0, Rust's cmd_work also correct under the SysV ABI once c++filt demangles its v0-mangled name, and Go's cmdWork in a dark box reading arg0/RDI wrong at 1, RAX right at 9, with a dashed amber arrow from it up to the eBPF-tooling band labeled arg0 mismatch. A footer note says every amber box ran live against systems-target this session; SystemTap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, and the dyninst backend attached without erroring." + alt="Three stacked bands. The top kernel band holds five small boxes: kprobe/kretprobe on kernel functions, uprobe/uretprobe on userspace functions, tracepoint as the stable kernel ABI, perf_event as timer/PMU sampling, and BTF/CO-RE for struct layout without headers. Amber arrows descend from three of these into the middle eBPF-tooling band, which holds bcc-tools (execsnoop/opensnoop/biolatency/offcputime/funccount/profile), bpftrace (one-liners and scripts, probe:target /filter/ action), bpftool (prog show/map show), and c++filt (demangle the exact symbol you attach to); libbpf loads a small verified BPF program per probe that the kernel runs in-kernel on each hit. Below that, a third band holds SystemTap as the sibling technology: its classic kernel-module backend needing matching kernel-devel, beside stap --runtime=dyninst, which patches the target binary directly in userspace with no kernel module. A fourth band at the bottom shows the three target binaries: C++'s busy_hash under the SysV ABI with a correct arg0, Rust's cmd_work also correct under the SysV ABI once c++filt demangles its v0-mangled name, and Go's cmdWork in a dark box reading arg0/RDI wrong at 1, RAX right at 9, with a dashed amber arrow from it up to the eBPF-tooling band labeled arg0 mismatch. A footer note says every amber box ran live against systems-target this session; SystemTap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, and the dyninst backend attached without erroring." caption="Figure 30.1: the eBPF observation stack this chapter exercises live: kernel probe primitives at the bottom, bcc-tools/bpftrace/bpftool/c++filt above them, SystemTap as the sibling technology beside them, and the one real register-ABI mismatch that only shows up on the Go binary" %} > **Tools used**: bcc-tools `execsnoop`, `opensnoop`, `funccount`, @@ -88,9 +88,10 @@ the full contract and a real run's tail. ## Watching from outside: the third way to observe a program -Chapter 29 drew a line between two ways of catching a bug: **translate the -binary** (valgrind, no rebuild, a software-emulated CPU) or **instrument the -source** (sanitizers, a required rebuild, near-native overhead). eBPF-based +Binary translation and compile-time instrumentation are two ways of catching +a bug: **translate the binary** (valgrind, no rebuild, a software-emulated +CPU) or **instrument the source** (sanitizers, a required rebuild, +near-native overhead). eBPF-based observation is a third way, and it needs neither: point a probe at a running kernel function, a running userspace function, or a stable tracepoint, and the **kernel itself** runs a small, verified BPF program on every hit, @@ -183,9 +184,9 @@ Tracing block device I/O... Hit Ctrl-C to end. ``` That histogram came from five `dd ... conv=fsync` writes to `/var/tmp` -completing in the low hundreds of microseconds, a real, measured -distribution, not a synthetic example. `profile`, the sixth named tool, is -where this tour hit a real wall worth reporting rather than hiding: +completing in the low hundreds of microseconds: a measured distribution +from the lab VM's root filesystem. `profile`, the sixth named tool, hit a +real wall: ```console [vm]$ sudo /usr/share/bcc/tools/profile -F 49 -p 15912 3 @@ -204,8 +205,8 @@ snippet references a `struct ns_common` field the installed headers no longer have under that name. bpftrace's own `profile:hz:N` probe type sidesteps the header dependency and attached cleanly, but on this exact workload it reported **zero samples** at both 49Hz and 99Hz over several -seconds, not a bug, but the real result of an on-CPU sampler pointed at -a process that spends over 99% of its time asleep in `nanosleep`. +seconds: the expected result of an on-CPU sampler pointed at a process +that spends over 99% of its time asleep in `nanosleep`. `offcputime` and `profile` are answering two different questions (where did the *waiting* time go, versus where did the *running* time go), and a workload this sleep-dominated has an answer for exactly one of them. @@ -280,7 +281,7 @@ this host, a real, measured latency distribution for a function this book's example code keeps simple, produced with five lines of bpftrace and no rebuild of `app` at all. -## bpftool: naming what is actually loaded +## bpftool: naming what is loaded **bpftool** is the inspection half of the story: it reads back what the kernel currently has loaded, independent of whichever tool put it there. @@ -298,7 +299,7 @@ Two things are worth reading closely here. First, `bpftrace` compiled the attach point's name straight into the BPF program's own name (`uprobe__tmp_lsp30_cpp_app_busy_hash_1`); `bpftool` is reading that name back from the kernel, not from bpftrace's own bookkeeping, so this is an -independent confirmation the probe really loaded. Second, the kernel +independent confirmation the probe loaded. Second, the kernel classifies this program's `type` as plain **`kprobe`** even though we asked bpftrace for a `uprobe`: uprobes and kprobes share the same underlying BPF program type at the kernel ABI level; the distinction between "kernel @@ -319,7 +320,7 @@ process: `pids bpftrace(16154)` is the cross-check: this map is held open by the exact bpftrace process running our script, not a leftover from some earlier session; `bpftool` and `bpftrace` are two independent programs -agreeing on what is actually resident in the kernel right now. +agreeing on what is resident in the kernel right now. ## SystemTap: the sibling technology @@ -337,7 +338,7 @@ Incorrect version or missing kernel-devel package, use: dnf install kernel-devel ``` The installed `kernel-devel` on this image is built for `7.1.3-201.fc44`, -one version off the actually-running `6.19.10-300.fc44`, exactly the +one version off the `6.19.10-300.fc44` kernel that is running, exactly the class of drift bpftrace and bcc's BTF-backed tools were built to avoid. SystemTap also ships a userspace-only **dyninst** backend (`--runtime=dyninst`), which patches the target process's own code directly @@ -360,8 +361,9 @@ path used everywhere else in this chapter (which needed no matching for a userspace-patching approach this session did not get a positive result from in the time available. SystemTap has real strengths bpftrace does not (a proper scripting language and years of production hardening -among them), and Chapter 45 covers it in depth; this callout is only the -comparison this chapter's own live evidence supports. +among them); the broader catalog of Linux analysis suites covers it in depth. +This callout is only the comparison this chapter's own live evidence +supports. ## How the code works @@ -392,8 +394,8 @@ Go's equivalent `result` struct has no such requirement (a single goroutine owns it end to end, handed to `main` once, complete, over a channel) and Rust's `Counters` is a plain `Default`-derived struct for the same reason: the concurrency shape, not the language, decides whether the counters need -to be atomic. The comment on `busy_hash_val` documents this chapter's own -`README.md`-recorded gotcha: the first version of this workload printed +to be atomic. The comment on `busy_hash_val` documents a real compiler behavior recorded +in this example's `README.md`: the first version of this workload printed `busy_calls` but never the accumulated hash, and GCC's interprocedural analysis proved the entire `busy_hash()` call chain had no observable effect and deleted it: `funccount` and the bpftrace uprobe both reported @@ -518,7 +520,7 @@ ERROR: Unable to attach probe: uprobe:.../app:no_such_function. If this is expec `funccount`, by contrast, treats an unmatched pattern as an **empty result set**, not an error, asking it for the exact unqualified name `busy_hash` -against the *Go* binary, which actually exports `main.busy_hash`: +against the *Go* binary, which exports `main.busy_hash`: ```console [vm]$ sudo /usr/share/bcc/tools/funccount -p 1 -d 1 '/tmp/lsp30-go/app:busy_hash' @@ -540,8 +542,8 @@ three without a per-language branch anywhere in `observe.sh`. Attaching to any *other* function in the same binaries is where each language's real symbol story, and one real ABI trap, shows up. `cmd_work`, the function that dispatches `--seconds` into the workload loop, carries no such -protection in any language, so `nm` shows what each compiler really does -with a name by default: +protection in any language, so `nm` shows what each compiler does with a +name by default: ```console [host]$ nm cpp/build/release/app | grep cmd_work @@ -581,7 +583,7 @@ next section shows what it costs. {% include excalidraw.html file="30-uprobe-attachment" - alt="Three parallel columns, one per language, each four boxes tall. C++ column: nm reports the mangled symbol t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0; c++filt (auto Itanium ABI) demangles it to (anonymous namespace)::cmd_work(int); a uprobe attaches using the MANGLED string as the attach target; the probe fires with seconds_arg=5 for --seconds 5, and arg0==RDI is correct because C++ follows the plain SysV x86-64 ABI. Rust column: nm reports t _RNvCs6FzKXswKZCc_3app8cmd_work; c++filt auto-detects Rust v0 mangling and demangles it to app[hash]::cmd_work; a uprobe attaches using that same mangled string; the probe fires with seconds_arg=7 for --seconds 7, and arg0==RDI is again correct under the SysV ABI. Go column: nm reports T main.cmdWork, never mangled; no demangling step is needed since the package-qualified name is already the symbol; a uprobe attaches to the plain unmangled name main.cmdWork; the probe fires for --seconds 9 but RDI(arg0) reads 1, which is wrong, while RAX correctly reads 9 -- a dashed amber arrow connects this box back up to the tooling band above, labeled arg0 mismatch. A footer band spans all three columns explaining that Go's ABIInternal (1.17+) passes integer arguments in the order RAX, RBX, RCX, RDI, RSI, R8, R9, ... rather than the SysV order RDI, RSI, RDX, ... that bpftrace's arg0/arg1 assume, and that goroutine stacks also grow by copying, so a captured stack address can move out from under a probe that read it from the stack instead of a register." + alt="Three parallel columns, one per language, each four boxes tall. C++ column: nm reports the mangled symbol t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0; c++filt (auto Itanium ABI) demangles it to (anonymous namespace)::cmd_work(int); a uprobe attaches using the MANGLED string as the attach target; the probe fires with seconds_arg=5 for --seconds 5, and arg0==RDI is correct because C++ follows the plain SysV x86-64 ABI. Rust column: nm reports t _RNvCs6FzKXswKZCc_3app8cmd_work; c++filt auto-detects Rust v0 mangling and demangles it to app[hash]::cmd_work; a uprobe attaches using that same mangled string; the probe fires with seconds_arg=7 for --seconds 7, and arg0==RDI is again correct under the SysV ABI. Go column: nm reports T main.cmdWork, never mangled; no demangling step is needed since the package-qualified name is already the symbol; a uprobe attaches to the plain unmangled name main.cmdWork; the probe fires for --seconds 9 but RDI(arg0) reads 1, which is wrong, while RAX correctly reads 9; a dashed amber arrow connects this box back up to the tooling band above, labeled arg0 mismatch. A footer band spans all three columns explaining that Go's ABIInternal (1.17+) passes integer arguments in the order RAX, RBX, RCX, RDI, RSI, R8, R9, ... rather than the SysV order RDI, RSI, RDX, ... that bpftrace's arg0/arg1 assume, and that goroutine stacks also grow by copying, so a captured stack address can move out from under a probe that read it from the stack instead of a register." caption="Figure 30.2: the same uprobe-attach recipe (find the symbol, demangle it to read it, attach to the mangled string) across three binaries, ending in the one real, measured argument-register mismatch that shows up only on Go" %} Attaching directly to the mangled string on the C++ and Rust binaries, with @@ -665,7 +667,7 @@ drives the same contract for all three languages against the live VM: No single tool in this chapter is trusted alone: every tool's count is checked against either another tool watching the same process, or against -`bpftool` reading back what the kernel actually loaded. From the same +`bpftool` reading back what the kernel loaded. From the same `observe.sh` run quoted above, `execsnoop` (fork/exec) and `bpftrace`'s uprobe (function calls) are two completely different probe types (a tracepoint-adjacent syscall tracer versus a userspace-symbol uprobe) @@ -691,7 +693,7 @@ bpftrace itself reported: two separate programs reading the same kernel state and agreeing. The third is the mangled-symbol pair: `nm` and `c++filt` independently identify and translate `cmd_work`'s real linker name off the binary, and a live uprobe attached to that exact string is -what actually fires and reports the right `--seconds` value back, proof +what fires and reports the right `--seconds` value back, proof the demangled name and the attach target are the same symbol, not a coincidence of two unrelated strings. @@ -705,7 +707,7 @@ coincidence of two unrelated strings. bpftrace's one-liners and scripts all ran this way against an unmodified binary on `systems-target`. - `bpftool prog show`/`map show` is the independent check on what the - kernel actually has loaded: it named bpftrace's own uprobe program by + kernel has loaded: it named bpftrace's own uprobe program by the exact attach-point string bpftrace compiled into it, confirmed from the kernel's bookkeeping, not bpftrace's. - A uprobe attaches to the ELF symbol table's literal string, mangled or @@ -721,8 +723,7 @@ coincidence of two unrelated strings. Next, **per-language toolbelts**: `perf`, `pprof`, `delve`, and `cargo-flamegraph` each answer "where does the time go" their own native -way, on the same kind of unmodified binary this chapter just watched from -outside. +way, on the same unmodified binary this chapter just watched from outside. --- diff --git a/_docs/31-per-language-toolbelts.md b/_docs/31-per-language-toolbelts.md index 7846ada..e6d94d5 100644 --- a/_docs/31-per-language-toolbelts.md +++ b/_docs/31-per-language-toolbelts.md @@ -24,9 +24,9 @@ A profiler only answers "where does the time go" if you can read what it hands b The payoff is the cross-check this chapter runs at the end: three profiling tools, built by three unrelated projects, reading three different kinds of instrumentation, all converge on the same function name at the same overwhelming share of samples. That agreement, not any single tool's output, is what makes a profiling result trustworthy enough to act on. -Chapter 28 opened this part by reading a crash gdb already knew about; -Chapter 29 broke code by design and watched a sanitizer catch it. Both -chapters answer "what is wrong". This one answers a different question that +gdb already read a crash it knew about, opening this part; sanitizers broke +code by design and watched the break get caught. Both answer "what is +wrong". This one answers a different question that comes up on every program that runs correctly and still feels slow: "where does the time go, and which line is spending it?" That question has no generic answer: each language ships its own native way of pointing a @@ -47,7 +47,7 @@ this chapter draws from. {% include excalidraw.html file="31-flamegraph-reading" alt="A schematic flamegraph with two labeled axes: y is stack depth, bottom the root frame runtime.main and top the leaf running at the instant a sample was taken; x is share of samples, explicitly not time order or call order. A stack of boxes narrows from a full-width root frame up through main/main.run, into a wide accent box for spin_hot/hotSpin covering about 99% of the row's width, into an equally wide inlined divisor-loop frame directly above it, topped by a leaf sliver. A callout box explains that color here just marks which language tab a frame came from, since real profilers usually color by module or at random. A footer note says the way to find the hot path is to scan for the widest unbroken column reaching toward the top." - caption="Figure 31.1: reading a flamegraph: width is the share of samples, height is stack depth, and the widest column reaching toward the top is the function actually spending the cycles" %} + caption="Figure 31.1: reading a flamegraph: width is the share of samples, height is stack depth, and the widest column reaching toward the top is the function spending the cycles" %} > **Tools used**: `perf` record/report/stat, `clang-tidy`, GCC `-fanalyzer` > (C++); `go tool pprof`, `go tool trace`, `dlv` (Go); `perf` record/report, @@ -73,7 +73,7 @@ libraries the program calls into, so writing a CPU profile, a heap profile, or an execution trace is part of the program's own behavior, driven by flags this chapter's Go build accepts and C++/Rust do not (`--cpuprofile`, `--memprofile`, `--trace`). Rust sits in the middle: it is compiled to the -same kind of native ELF binary as C++, so `perf` attaches to it the same way, +same native ELF binary as C++, so `perf` attaches to it the same way, but the Rust ecosystem also ships `cargo flamegraph`, a Rust program that *wraps* `perf record` and then renders the result with `inferno`, a Rust reimplementation of Brendan Gregg's original Perl `stackcollapse`/`flamegraph` @@ -84,21 +84,21 @@ flamegraph draws directly and what `--top` tabulates as text. A flamegraph earns its name from the shape, not from heat: each horizontal row is one level of stack depth (the root frame along the bottom, the -function actually executing at the top), and the *width* of a box is the +function executing at the top), and the *width* of a box is the fraction of samples in which that function was somewhere on the stack at that depth. Frames are usually **sorted alphabetically within their parent**, not by call order or wall-clock time, which is the detail that trips up first-time readers expecting the x-axis to mean "when". Reading one is a single scan: follow the tallest, widest column of boxes from the bottom to the top, and the name at the top of that column is where the cycles are -actually going. Figure 31.1 draws that shape directly from this chapter's +going. Figure 31.1 draws that shape directly from this chapter's own data: `spin_hot`/`hotSpin` is that column in every one of the three profiles below. {% include excalidraw.html file="31-toolbelt-map" - alt="One shared target box, app hot or alloc with an identical result across languages, feeding three columns. The C++ column lists perf record with call-graph fp as its accent entry naming spin_hot at 99.9%, then perf report/perf stat, clang-tidy, GCC -fanalyzer, and a dashed gdb box marked Chapter 28. The Go column lists runtime/pprof as its accent entry, cpuprofile into go tool pprof top naming main.hotSpin, then memprofile with alloc_space naming main.allocChurn, runtime/trace with go tool trace, delve, and a dashed go vet box. The Rust column lists perf record with call-graph fp as its accent entry naming spin_hot inlined near 100%, then cargo flamegraph producing a real SVG, cargo clippy and cargo fmt check, a dashed tokio-console box marked forward reference to Chapter 27 not applicable here since there is no async runtime, and a dashed gdb/rust-gdb box marked Chapter 28. A note explains the amber boxes are the CPU-profiling step this chapter cross-checks against each other, and the dashed boxes are noted but out of scope." - caption="Figure 31.2: one shared target, three native toolbelts: the amber boxes are what this chapter's cross-check compares; the dashed boxes point to where gdb (Ch. 28) and tokio-console (Ch. 27) belong instead" %} + alt="One shared target box, app hot or alloc with an identical result across languages, feeding three columns. The C++ column lists perf record with call-graph fp as its accent entry naming spin_hot at 99.9%, then perf report/perf stat, clang-tidy, GCC -fanalyzer, and a dashed gdb box marked as the debugging digression. The Go column lists runtime/pprof as its accent entry, cpuprofile into go tool pprof top naming main.hotSpin, then memprofile with alloc_space naming main.allocChurn, runtime/trace with go tool trace, delve, and a dashed go vet box. The Rust column lists perf record with call-graph fp as its accent entry naming spin_hot inlined near 100%, then cargo flamegraph producing a real SVG, cargo clippy and cargo fmt check, a dashed tokio-console box marked as a forward reference to async-runtime debugging, not applicable here since there is no async runtime, and a dashed gdb/rust-gdb box marked as the debugging digression. A note explains the amber boxes are the CPU-profiling step this chapter cross-checks against each other, and the dashed boxes are noted but out of scope." + caption="Figure 31.2: one shared target, three native toolbelts: the amber boxes are what this chapter's cross-check compares; the dashed boxes point to where gdb and tokio-console belong instead" %} ## How the code works @@ -186,10 +186,10 @@ fn spin_hot(n: u64) -> u64 { All three are the same trial-division loop with the same invariant: an `i` in `[2, n)` is prime only if no `d` in `[2, sqrt(i)]` divides it evenly, found by walking `d` up from 2 and stopping the moment `d*d > i`. The loop is -intentionally the slow textbook version (no sieve, no memoized small primes, -no early-exit on evens) because a fast implementation would finish before a -sampling profiler collected enough hits to draw a clean picture, and the goal -here is a profile, not a fast prime counter. The `noinline` annotation on +the slow textbook version (no sieve, no memoized small primes, no +early-exit on evens): a fast implementation would finish before a sampling +profiler collected enough hits to draw a clean picture, and the goal here is +a profile, not a fast prime counter. The `noinline` annotation on each (`[[gnu::noinline]]`, `//go:noinline`, `#[inline(never)]`) is not decoration: at `-O2`/release optimization, all three compilers are entitled to fold a small, single-call-site function directly into its caller, which @@ -201,7 +201,7 @@ every transcript below does. The allocation-heavy sibling (`alloc_churn`/`allocChurn`/`alloc_churn`, not shown in the tabs above but present in every `main.cpp`/`main.go`/`main.rs`) -does the opposite kind of work by design: `n` iterations round-robin +does the opposite work by design: `n` iterations round-robin overwrite a 1,000-key string map, and every iteration allocates a fresh key string and a fresh value string before inserting. Where `spin_hot` never touches the heap, `alloc_churn` never does meaningful arithmetic; its job is @@ -219,8 +219,8 @@ before the timed call, `pprof.WriteHeapProfile` after it), each guarded by a `anyhow::Result` through `parse_args`/`run` and returns `ExitCode` from `main` rather than calling `std::process::exit`, so destructors still run on every exit path. None of that wiring changes the algorithm; it only exists -so the same binary can be pointed at by three very different profilers -without recompiling it differently for each one. +so the same binary can be pointed at by three different profilers without +recompiling it differently for each one. ## Errors, three ways @@ -242,13 +242,13 @@ profiler. ## Concurrency lens: what each language's tooling gives you This is a pure-tooling chapter: `app` is single-threaded in every language, -so there is no race to hunt here the way Chapters 25 and 26 hunted one. What -still differs by language is how much of the *concurrency picture* each -toolbelt can show once a program does have more than one thread of control. -`perf record --call-graph` samples every thread in a process by default, so -it is already concurrency-aware for C++ and Rust: point it at `workq` from -Chapter 25 and the flamegraph would show each worker thread's stack -separately. Go's profilers go further because they are runtime-aware: a +so there is no race to hunt here the way futex-protected shared state and +lock-free atomics hunted one. What still differs by language is how much of +the *concurrency picture* each toolbelt can show once a program does have +more than one thread of control. `perf record --call-graph` samples every +thread in a process by default, so it is already concurrency-aware for C++ +and Rust: point it at `workq`, the shared-state-and-futex example, and the +flamegraph would show each worker thread's stack separately. Go's profilers go further because they are runtime-aware: a `runtime/pprof` CPU profile attributes samples to the goroutine that was running, and `go tool trace` is built specifically to show goroutine-level concurrency: when one parks, when the scheduler hands its P @@ -363,12 +363,12 @@ just-returned `allocChurn` call is already garbage; the default *in-use* view would show almost nothing live. `--alloc_space` switches to pprof's cumulative-since-start counter, which remembers every byte `mallocgc` ever handed out whether or not it was later freed, which is the view that -actually answers "where did the allocations happen": `main.allocChurn` +answers "where did the allocations happen": `main.allocChurn` itself directly owns 53.98% (the map's own bucket storage), and `fmt.Sprintf`, called twice per iteration to build the key and value strings, owns another 38.34% underneath it. `go tool trace` opens an interactive browser UI by default, but `-d=parsed` dumps the same event -stream as text, enough to confirm the tracer really captured `hotSpin` +stream as text, enough to confirm the tracer captured `hotSpin` executing: ```console @@ -380,8 +380,8 @@ app: mode=hot n=800000 result=63951 ms=114 main.hotSpin @ 0x4d272c ``` -Delve is the interactive debugger this chapter ties back to Chapter 28's -`gdb` work: same idea (breakpoints, stack frames, live variables), native to +Delve is the interactive debugger this chapter ties back to the `gdb` work +above: same idea (breakpoints, stack frames, live variables), native to Go's runtime instead of DWARF-only. It was not preinstalled on this host; `go install github.com/go-delve/delve/cmd/dlv@latest` pulled it from the Go module proxy in a few seconds, and it scripts the same way `gdb --batch` @@ -427,8 +427,8 @@ needing no `extern "C"` trick to stay readable) and reported as inlined at release optimization; the frame boundary survives as debug-info metadata even though the call itself was folded away. `cargo-flamegraph` was already on this host (`cargo install flamegraph --locked` is what would put it there -otherwise), so I ran it against the live binary rather than falling back to -the tabular report: +otherwise); running it against the live binary skips the tabular-report +fallback: ```console [host]$ cargo flamegraph --output rust-hot.svg -- hot @@ -458,9 +458,8 @@ clean on this source, no findings, no diff. `tokio-console` deserves a brief look before being set aside: it attaches to a running **async runtime** and shows live task/waker state, which only exists once a program has `tokio` tasks to show. `app` has none; no `.await` anywhere in this chapter's source, so -`tokio-console` has nothing to display here; it belongs with -Chapter 27's async runtimes, where `chatterd`'s tokio tasks are the thing -worth watching live. +`tokio-console` has nothing to display here; it belongs with async-runtime +debugging, where `chatterd`'s tokio tasks are the thing worth watching live. > **CLion's built-in profiler** unverified: > CLion wraps this same `perf record`/`perf report` pipeline (and, on @@ -507,7 +506,7 @@ hardware sampler reading raw ELF symbols, an in-process Go library sampling its own goroutine stacks, the same kernel sampler again reading Rust's DWARF) converge on the same name in every language's own spelling of it, at the same overwhelming share of samples. That agreement is the cross-check: if -`spin_hot`/`hotSpin` were *not* actually where the time went, three +`spin_hot`/`hotSpin` were *not* where the time went, three independently-implemented tools would have no reason to all point at the same place. diff --git a/_docs/32-namespaces-and-cgroups.md b/_docs/32-namespaces-and-cgroups.md index ffa28fd..1fad23a 100644 --- a/_docs/32-namespaces-and-cgroups.md +++ b/_docs/32-namespaces-and-cgroups.md @@ -22,7 +22,7 @@ By the end of this chapter you can: "Container" names no single kernel object: it is namespaces and cgroups, two independent primitives any userspace program can assemble by hand. Seeing them built from raw syscalls, rather than through `podman run`, is what makes the pid-1 reparenting rules, the per-thread nature of `unshare`, and a cgroup's enforcement all concrete instead of folklore. -The Go `ENOMEM` failure in this chapter is a real, reproduced defect, not a hypothetical: `unshare(CLONE_NEWPID)` caps the caller to exactly one child in the new namespace, and `os/exec`'s internal probe fork silently spends it. Knowing that restriction, and the `SysProcAttr.Cloneflags` fix that avoids it entirely, is the difference between a container tool that works on the host you tested and one that fails unpredictably on every other kernel. +The Go `ENOMEM` failure in this chapter happened on real hardware, not in theory: `unshare(CLONE_NEWPID)` caps the caller to exactly one child in the new namespace, and `os/exec`'s internal probe fork silently spends it. Knowing that restriction, and the `SysProcAttr.Cloneflags` fix that avoids it entirely, is the difference between a container tool that works on the host you tested and one that fails unpredictably on every other kernel. Every `pmon` version so far has supervised, signaled, and identified a child, but the child has always lived in the same world as its parent: same @@ -95,7 +95,7 @@ The three syscalls answer three different questions: - **`unshare(2)`**: "take *me*, right now, out of my current namespaces and into brand-new ones." For most types this applies to the calling - process/thread immediately. `CLONE_NEWPID` is the deliberate exception: + process/thread immediately. `CLONE_NEWPID` is the exception: unshare(2) can never move an already-running process into a new PID namespace (a process's own pid can't change mid-life), so it only arranges that the *next* child the caller creates becomes pid 1 of a @@ -131,8 +131,8 @@ whatever it does next. That is a simplification, not an oversight: a supervisor that expects to run *multiple* processes inside one PID namespace (the way `podman run` with more than one process, or a process manager container, would) needs an init that actually reaps, -which is exactly what chapters 11 and 12 already built, just not wired to -this namespace yet. +which is exactly what the process-lifecycle reaping and signal-handling +work already built, just not wired to this namespace yet. ## Cgroup v2: the unified hierarchy @@ -196,14 +196,14 @@ together by one userspace program: **namespaces** (this chapter, minus user: `containerize` never requests `CLONE_NEWUSER`, so its "root" inside the container is uid 0 on the host, which is why the demo needs real root rather than rootless mapped uids), **cgroups** (this chapter), -a **rootfs** (a different filesystem view via `pivot_root`/bind mounts; -chapter 34 builds this), **capabilities** (chapter 14's five sets, -dropped to a minimal set before `exec`), and a **seccomp/Landlock** -filter (chapter 33, next). `pmon containerize` deliberately stops after +a **rootfs** (a different filesystem view via `pivot_root`/bind mounts, +assembled when building a container image), **capabilities** (the five +capability sets, dropped to a minimal set before `exec`), and a +**seccomp/Landlock** filter. `pmon containerize` deliberately stops after the first two, on the same host filesystem CMD would otherwise see; the point is to make namespaces and cgroups *visible* as syscalls you could -write yourself, before chapter 34 shows what podman assembles from all -five at once inside a multi-stage Containerfile. +write yourself, before a Containerfile build shows what podman assembles +from all five at once inside a multi-stage build. {% include excalidraw.html file="32-cgroup-hierarchy" @@ -222,8 +222,8 @@ false-matches inside `"cpuset"`, `mkdir` the leaf, write `memory.max` then pid into `cgroup.procs`) runs **before** any namespace call in every language, by design: cgroup membership is inherited across `fork`/ `clone`/`exec`, so writing pmon's pid first means every later child (the -pid-1 task, and CMD after it execs) is under the limits from its very -first instruction, never after. +pid-1 task, and CMD after it execs) is under the limits from its first +instruction, never after. {% include codetabs.html langs="C++|Go|Rust" %} @@ -485,7 +485,8 @@ Concurrency lens below. inside, including `ps` run via `nsenter`, still see the host's process table (the cross-check below demonstrates this directly); a rootfs container fixes this with `mount -t proc proc /proc` after `pivot_root`, -which is chapter 34's job, not this one's. `memory.swap.max` is written +which belongs to assembling a full container image, not this chapter's +scope. `memory.swap.max` is written best-effort because not every kernel exposes swap accounting identically. And `deploy-to-vm.sh` rebuilds the remote command line with a bare `$*`, which flattens quoting, so every `CMD` in this chapter's runs is @@ -530,8 +531,9 @@ exit code alone, no log-scraping required. caption="Figure 32.3: C++/Rust's unshare-then-clone3 shape beside Go's two attempts: a real ENOMEM, then the SysProcAttr.Cloneflags fix that passes" %} Namespace changes made by `unshare(2)`/`setns(2)` are per-**thread** -kernel state, not per-process: the same fact chapter 14 hit with -capabilities. A Go program is multithreaded before `main` even runs, and +kernel state, not per-process, the same per-thread behavior that +capability dropping runs into. A Go program is multithreaded before +`main` even runs, and the goroutine scheduler is free to migrate any goroutine onto any OS thread between two lines of Go code. The textbook answer is `runtime.LockOSThread()`: pin the calling goroutine to its current OS @@ -569,8 +571,9 @@ with which thread is running: it is about how many times the *same* thread group is allowed to fork into a namespace it merely unshared rather than was born into. -The fix is the same move chapter 14 made for capabilities: stop calling -the namespace syscall directly from live, threaded Go code, and describe +The fix mirrors how dropping capabilities handles the same per-thread +hazard: stop calling the namespace syscall directly from live, threaded +Go code, and describe it declaratively instead. `exec.Cmd.SysProcAttr.Cloneflags` asks the runtime's own fork/exec trampoline to create the namespaces **and** the real child in one `clone(2)` call: there is no separate `unshare()` @@ -683,10 +686,10 @@ root 2 0 0 14:42 ? 00:00:00 [kthreadd] namespaces before forking `ps`, but `ps` reads `/proc`, and the mount namespace it entered never got a *fresh* procfs mounted after `unshare`; it still resolves to the host's original procfs instance, so `ps` prints -the host's entire process table instead of a scoped one. This is not a -bug in `containerize`; it is precisely why chapter 34's Containerfile -build includes a `pivot_root` and a fresh `/proc` mount; a namespace -without a matching filesystem view is only half a container. +the host's entire process table instead of a scoped one. That gap is +precisely why a container image's Containerfile build includes a +`pivot_root` and a fresh `/proc` mount: a namespace without a matching +filesystem view is only half a container. Third, the kernel's own OOM log, read independently of anything `pmon` printed: @@ -721,7 +724,7 @@ reported, confirmed at the source that actually made the decision. internal probe fork silently spends, producing a real `ENOMEM` this chapter reproduced live; the fix is `SysProcAttr.Cloneflags`, pushing the whole operation into the runtime's own fork/exec trampoline, the - same move chapter 14 made for capabilities. + same move that handles the per-thread capability-dropping hazard. - A container is namespaces + cgroups + rootfs + capabilities + seccomp, assembled by userspace, not one kernel object, and a namespace without its matching filesystem view is observably incomplete: `nsenter`'s `ps @@ -730,7 +733,8 @@ reported, confirmed at the source that actually made the decision. Next, **seccomp and Landlock**: filtering which syscalls a process may even attempt, and restricting which files it may reach, as the two -primitives chapter 32 left out of "what a container runtime assembles." +primitives left out of "what a container runtime actually assembles" +above. --- @@ -752,9 +756,9 @@ kill of pid 27013; nsenter -p -m -u -n ps -ef was run live and showed the host's full process table, confirming the missing /proc remount. The Go ENOMEM failure quoted in the Concurrency lens is this session's own strace output -against the pre-fix build, kept as evidence of a real, reproduced defect +against the pre-fix build, kept as evidence of the real failure mode before the Cloneflags fix that now ships and passes. A -one-line verify.lua bug this chapter also surfaced and fixed -(an unescaped - in a Lua pattern silently failing the +one-line verify.lua fix this chapter also surfaced +(an unescaped - in a Lua pattern that silently failed the hostname assertion) is recorded in the example's own history, not restated here as a finding about the chapter's subject matter.

diff --git a/_docs/33-seccomp-and-landlock.md b/_docs/33-seccomp-and-landlock.md index 8b340a3..c05d4c2 100644 --- a/_docs/33-seccomp-and-landlock.md +++ b/_docs/33-seccomp-and-landlock.md @@ -2,7 +2,7 @@ title: "Seccomp and Landlock" order: 33 part: "Containers and Virtualization" -description: "fwatch sandboxes itself with two independent, unprivileged kernel layers -- a seccomp-bpf syscall allowlist built from a real strace profile, and a Landlock ruleset that restricts filesystem reads to one directory -- probing the Landlock ABI at runtime instead of assuming it, and proving both with negative controls that watch the exact denial happen." +description: "fwatch sandboxes itself with two independent, unprivileged kernel layers (a seccomp-bpf syscall allowlist built from a real strace profile, and a Landlock ruleset that restricts filesystem reads to one directory), probing the Landlock ABI at runtime instead of assuming it, and proving both with negative controls that watch the exact denial happen." duration: "55 minutes" --- @@ -20,26 +20,26 @@ By the end of this chapter you can: ## Why this matters -seccomp-bpf and Landlock let an unprivileged process restrict itself, no root, no namespace, no VM required. That makes them two of the cheapest isolation techniques available to any program this book has built, and they compose directly with the namespaces and cgroups from Chapter 32 as part of what a real container runtime assembles. +seccomp-bpf and Landlock let an unprivileged process restrict itself, no root, no namespace, no VM required. That makes them two of the cheapest isolation techniques available to any program this book has built, and they compose directly with namespaces and cgroups as part of what a real container runtime assembles. -Both are also easy to get wrong in ways that look fine until tested: a filter built from a guessed syscall list breaks the program it was meant to protect, an ABI assumed rather than probed misbehaves on a different kernel, and a restriction applied to only one thread in a multithreaded runtime leaves every other thread unsandboxed. The design discipline this chapter teaches, profile first, probe the ABI, verify with a negative control, is what turns seccomp and Landlock from documentation into a sandbox you can trust. +Both also go wrong in ways that look fine until tested: a filter built from a guessed syscall list breaks the program it was meant to protect, an ABI assumed rather than probed misbehaves on a different kernel, and a restriction applied to only one thread in a multithreaded runtime leaves every other thread unsandboxed. The design discipline this chapter teaches, profile first, probe the ABI, verify with a negative control, is what turns seccomp and Landlock from documentation into a sandbox you can trust. -Chapter 32 gave `fwatch` its own view of the process tree and its own -resource limits through namespaces and cgroup v2 -- isolation the kernel -grants to a group of processes. This chapter hands the same idea to a +Namespaces and cgroup v2 already gave `fwatch` its own view of the process +tree and its own resource limits: isolation the kernel grants to a group +of processes. This chapter hands the same idea to a *single* process, no grouping required: `watch --sandbox DIR` now installs a syscall allowlist and a filesystem allowlist against itself, using two mechanisms any unprivileged process can invoke directly. **seccomp-bpf** decides, syscall by syscall, whether the calling process is allowed to make this call at all. **Landlock** decides, path by path, which parts of the filesystem that process may still read. Neither needs root, a namespace, or -a VM -- which is also why every number in this chapter was reproduced on +a VM, which is also why every number in this chapter was reproduced on the plain host and confirmed unchanged on the `systems-target` lab guest. The one new idea worth sitting with: seccomp-bpf is **classic BPF**, the same in-kernel bytecode VM whose modern descendant, eBPF, this book has so far only used as an *observation* tool (`bpftrace`, `bcc-tools`, `bpftool` -in Part 8). A seccomp filter is the mirror image -- you are not attaching a +in Part 8). A seccomp filter is the mirror image: you are not attaching a program that watches a syscall and reports it, you are attaching a program that *decides* the syscall's outcome, on the hot path, before the kernel does the work. Same VM, same verifier discipline, opposite job. @@ -50,7 +50,7 @@ contract, and the paired positive/negative verification in full. {% include excalidraw.html file="33-seccomp-filter-flow" - alt="Two stacked bands. The top userspace band shows fwatch issuing a syscall such as epoll_wait, read, or socket, beside a note that this is classic BPF -- you write the filter, Part 8's bpftrace only observes. The bottom kernel band shows the filter running on every syscall entry: a BPF_LD_ABS load of the syscall number and architecture, a compiled jump table of 26 entries for cpp and rust or 39 for go, a match arrow to SECCOMP_RET_ALLOW that returns control to the caller, and a no-match arrow down to the filter's default action. Two default-action boxes sit side by side: SCMP_ACT_ERRNO(EPERM), the one fwatch actually installs, where the process keeps running and sees EPERM on socket(2); and a dashed, unused SCMP_ACT_KILL_PROCESS box, where a separate probe program's getpid() call ends the whole process with SIGSYS. Footer notes report the real guest transcript: socket(AF_INET,...) = -1 EPERM under fwatch's filter, and the ACT_KILL_PROCESS probe's getpid() ending in killed by SIGSYS (core dumped)." + alt="Two stacked bands. The top userspace band shows fwatch issuing a syscall such as epoll_wait, read, or socket, beside a note that this is classic BPF: you write the filter, Part 8's bpftrace only observes. The bottom kernel band shows the filter running on every syscall entry: a BPF_LD_ABS load of the syscall number and architecture, a compiled jump table of 26 entries for cpp and rust or 39 for go, a match arrow to SECCOMP_RET_ALLOW that returns control to the caller, and a no-match arrow down to the filter's default action. Two default-action boxes sit side by side: SCMP_ACT_ERRNO(EPERM), the one fwatch actually installs, where the process keeps running and sees EPERM on socket(2); and a dashed, unused SCMP_ACT_KILL_PROCESS box, where a separate probe program's getpid() call ends the whole process with SIGSYS. Footer notes report the real guest transcript: socket(AF_INET,...) = -1 EPERM under fwatch's filter, and the ACT_KILL_PROCESS probe's getpid() ending in killed by SIGSYS (core dumped)." caption="Figure 33.1: a syscall's path through fwatch's seccomp-bpf filter: one cBPF program, run again on every entry, deciding ALLOW versus its default action" %} > **Tools used**: `strace` (host and `systems-target` VM, to build and @@ -66,18 +66,18 @@ contract, and the paired positive/negative verification in full. `seccomp(2)` installs a cBPF program that the kernel runs on *every* syscall the process makes from that point on. The program reads two fields -out of `struct seccomp_data` -- the architecture and the syscall number -- +out of `struct seccomp_data` (the architecture and the syscall number) and returns one of a small set of verdicts: `SECCOMP_RET_ALLOW` (run it), `SECCOMP_RET_ERRNO` (fail it with a chosen errno, without ever entering the real syscall handler), or `SECCOMP_RET_KILL_PROCESS` (end the process with `SIGSYS`, no return). Once a filter is loaded it can only ever be -*tightened* -- filters stack, and a later one can narrow what an earlier +*tightened*: filters stack, and a later one can narrow what an earlier one allowed but never widen it. `fwatch`'s allowlist did not come from guessing. The comment in the C++ source says it directly: the syscall set is "empirically confirmed with `strace -f` across a full watch session." That is the practical way to -build a seccomp profile -- run the real workload under `strace -f -c`, +build a seccomp profile: run the real workload under `strace -f -c`, read off the syscalls it actually issues, and allowlist exactly those. Reproduced on the host against the unsandboxed watch loop: @@ -116,14 +116,14 @@ That is exactly the source of `kWatchSyscalls` in `examples/33-seccomp-and-landl calls, the allocator's `mmap`/`munmap`/`brk`, and `fstat`/`rt_sigprocmask`). The rows elided above (`execve`, `openat`, `pread64`, `access`, `arch_prctl`, `set_tid_address`, `set_robust_list`, `prlimit64`, `rseq`) are -the dynamic linker resolving `libc`/`libstdc++` *before* `main` runs -- +the dynamic linker resolving `libc`/`libstdc++` *before* `main` runs: real syscalls this same process makes, but not ones the watch loop itself issues, so they are outside what the filter needs to allow once it loads after startup. `install_seccomp` (same file) calls `seccomp_init(SCMP_ACT_ERRNO(deny_errno))` --- that argument is the filter's **default action**, applied to anything -not explicitly allowed -- then `seccomp_rule_add(ctx, SCMP_ACT_ALLOW, nr, 0)` +(that argument is the filter's **default action**, applied to anything +not explicitly allowed), then `seccomp_rule_add(ctx, SCMP_ACT_ALLOW, nr, 0)` once per syscall, and `seccomp_load(ctx)` to install it. `seccomp_load` requires `no_new_privs` first (set via `prctl`), a kernel guard against a sandboxed process regaining privilege through a setuid binary it execs. @@ -133,7 +133,7 @@ sandboxed process regaining privilege through a setuid binary it execs. `fwatch` chose `SCMP_ACT_ERRNO(EPERM)`, not `SCMP_ACT_KILL_PROCESS`, as its default action, and the difference is observable, not cosmetic. With `ACT_ERRNO`, a denied syscall returns `-1` -with `errno` set and the calling code continues to run -- the process can +with `errno` set and the calling code continues to run: the process can log the denial, clean up, and exit on its own terms; that is what makes `probe --forbidden-syscall`'s own `EPERM`-and-continue behavior possible. With `ACT_KILL_PROCESS`, a denied syscall never returns at all: the kernel @@ -150,7 +150,7 @@ getpid() = 39 +++ killed by SIGSYS (core dumped) +++ ``` -`strace` shows the syscall entering the kernel and *never returning* -- +`strace` shows the syscall entering the kernel and *never returning*: the trailing line is `strace` reporting the process's death, not a normal return value. `ACT_ERRNO` is the right choice whenever the caller has useful cleanup to do or a human is watching the diagnostic; `ACT_KILL` is @@ -162,17 +162,17 @@ continuing at all is the greater risk. Landlock restricts what a process (and everything it execs from here on) may still do to the filesystem, expressed as three calls with no -convenience wrapper in any of the three languages' standard libraries -- +convenience wrapper in any of the three languages' standard libraries: `landlock_create_ruleset`, `landlock_add_rule`, `landlock_restrict_self`, reached through a raw `syscall(2)` in C++ and Go and through the `landlock` crate (which itself wraps the same three numbers) in Rust. A **ruleset** -declares *which access rights this program intends to handle at all* -- +declares *which access rights this program intends to handle at all*: `fwatch` handles only `LANDLOCK_ACCESS_FS_READ_FILE` and `_READ_DIR`, and leaves `handled_access_net` at zero. A **rule** then grants a subset of those handled rights to one filesystem object, here a `PATH_BENEATH` rule binding read rights to a directory fd. `restrict_self` is the point of no return: it activates the ruleset against the *calling* process and every -descendant, and there is no syscall to lift it -- the only way out is +descendant, and there is no syscall to lift it: the only way out is `execve`-ing a program that never asked for the restriction to begin with, and even then the new program inherits the same ruleset. @@ -190,7 +190,7 @@ fwatch: probe outside /etc/hostname: EACCES (Permission denied) Same distribution, same syscall numbers, and yet the host (kernel `7.1`) reports ABI 9 while `systems-target` (kernel -`6.19.10-300.fc44.x86_64`) reports ABI 7 -- two real Fedora 44 kernels a +`6.19.10-300.fc44.x86_64`) reports ABI 7, two real Fedora 44 kernels a few point releases apart, disagreeing about what Landlock supports, while agreeing completely on the observable behavior the ruleset enforces. Every call in this example reaches that number the only real way: calling @@ -201,30 +201,30 @@ misbehave the moment it ran on the guest. {% include excalidraw.html file="33-landlock-ruleset" - alt="Two stacked bands. The top band builds the ruleset: landlock_create_ruleset called first with NULL/0/VERSION to probe the ABI, returning 9 on the host versus 7 on the systems-target guest despite both being Fedora 44 kernels; then landlock_create_ruleset(attr) with handled_access_fs set to READ_FILE and READ_DIR, feeding into landlock_add_rule binding a PATH_BENEATH rule to one directory's O_PATH file descriptor; then prctl(PR_SET_NO_NEW_PRIVS,1) feeding into landlock_restrict_self, marked irrevocable, with a note that Go's build also passes the TSYNC flag here to cover every OS thread already running. The bottom band shows the filesystem view after restrict_self: the sandboxed directory /tmp/lsp33-watch-sandbox with READ_FILE and READ_DIR still allowed and the watch loop working normally, beside /tmp/lsp33-outside-target/secret.txt where open(O_RDONLY) returns EACCES, with an arrow from the ruleset's restrict_self box down to each outcome, solid to the grant and dashed to the denial. A footer note across the bottom states that handled_access_net was left at zero in this ruleset, and that Landlock's TCP bind/connect rights have existed since ABI 4 and UDP rights only since ABI 10 -- so even at ABI 7 or 9 a Landlock-only sandbox does not cover the network by itself." + alt="Two stacked bands. The top band builds the ruleset: landlock_create_ruleset called first with NULL/0/VERSION to probe the ABI, returning 9 on the host versus 7 on the systems-target guest despite both being Fedora 44 kernels; then landlock_create_ruleset(attr) with handled_access_fs set to READ_FILE and READ_DIR, feeding into landlock_add_rule binding a PATH_BENEATH rule to one directory's O_PATH file descriptor; then prctl(PR_SET_NO_NEW_PRIVS,1) feeding into landlock_restrict_self, marked irrevocable, with a note that Go's build also passes the TSYNC flag here to cover every OS thread already running. The bottom band shows the filesystem view after restrict_self: the sandboxed directory /tmp/lsp33-watch-sandbox with READ_FILE and READ_DIR still allowed and the watch loop working normally, beside /tmp/lsp33-outside-target/secret.txt where open(O_RDONLY) returns EACCES, with an arrow from the ruleset's restrict_self box down to each outcome, solid to the grant and dashed to the denial. A footer note across the bottom states that handled_access_net was left at zero in this ruleset, and that Landlock's TCP bind/connect rights have existed since ABI 4 and UDP rights only since ABI 10, so even at ABI 7 or 9 a Landlock-only sandbox does not cover the network by itself." caption="Figure 33.2: the Landlock ruleset/rule model, with the ABI probed live rather than assumed and the real grant/deny split from this chapter's probes" %} -**What Landlock can and cannot do.** Filesystem mediation -- -open/read/write/execute/rename, scoped to a path -- has been solid since +**What Landlock can and cannot do.** Filesystem mediation +(open/read/write/execute/rename, scoped to a path) has been solid since ABI 1 (Linux 5.13). Network mediation is newer and arrived in pieces: `LANDLOCK_ACCESS_NET_BIND_TCP`/`_CONNECT_TCP` since ABI 4, but UDP bind/connect rights not until ABI 10; `fwatch` targets neither, so on both of these machines a Landlock-restricted process can still open arbitrary -sockets outside the tree entirely -- that gap is exactly why `fwatch` +sockets outside the tree entirely: that gap is exactly why `fwatch` layers seccomp *underneath* it rather than relying on Landlock alone. IPC-scoping flags (restricting signals and abstract-socket connections between sandboxed and unsandboxed processes) arrived at ABI 6, so both machines here have them even though neither example uses them. The practical rule this chapter teaches: **probe the ABI, then gate each -feature on the version number it actually needs** -- never gate on "did +feature on the version number it actually needs**: never gate on "did `landlock_create_ruleset` return non-negative." ## How the code works Both mechanisms share one setup shape across all three languages: probe the Landlock ABI, build and enforce the ruleset, then install the seccomp -filter, each step logging a status line before the identical epoll watch -loop from chapter 09 runs unmodified. C++ and Rust do this **in-process** +filter, each step logging a status line before the identical epoll-and- +inotify watch loop runs unmodified. C++ and Rust do this **in-process** with raw syscalls; Go does something structurally different, and the concurrency lens below explains why. @@ -308,15 +308,15 @@ fn install_seccomp(allowed: &[i64], deny_errno: i32) -> Result { C++'s `apply_landlock` is the clearest read of the three-call shape: build the `landlock_ruleset_attr` naming the *rights this program is willing to have restricted at all* (`handled_access_fs`, leaving `handled_access_net` -at zero -- an explicit, auditable "we do not sandbox network here", not an -oversight), open the target directory `O_PATH`-only (no read/write access -needed just to name it as a rule target), bind the rule with +at zero, an explicit, auditable "we do not sandbox network here" rather +than an oversight), open the target directory `O_PATH`-only (no read/write +access needed just to name it as a rule target), bind the rule with `landlock_add_rule`, and only then call `prctl(PR_SET_NO_NEW_PRIVS)` and -`landlock_restrict_self` -- in that order, because `restrict_self` demands +`landlock_restrict_self`, in that order, because `restrict_self` demands `no_new_privs` already be set. Rust's `landlock` crate (source in -`rust/src/main.rs`) compresses the same four calls into one builder chain --- `Ruleset::default().handle_access(...).create()?.add_rule(...)?.restrict_self()?` --- and adds one thing the raw syscall path does not give you automatically: it +`rust/src/main.rs`) compresses the same four calls into one builder chain, +`Ruleset::default().handle_access(...).create()?.add_rule(...)?.restrict_self()?`, +and adds one thing the raw syscall path does not give you automatically: it checks the returned `RulesetStatus`, so a kernel that silently best-effort-ignores an unsupported right is caught (`bail!` if `status.ruleset != RulesetStatus::FullyEnforced`) instead of quietly @@ -325,11 +325,11 @@ running unsandboxed. Go's `installSeccomp` is the odd one out for a reason worth reading closely: `golang.org/x/sys/unix` exposes the raw `seccomp(2)` syscall number but no BPF assembler, so `buildAllowlist` (same file) hand-emits the -cBPF program byte for byte -- `BPF_LD|BPF_W|BPF_ABS` to load the syscall +cBPF program byte for byte: `BPF_LD|BPF_W|BPF_ABS` to load the syscall number, one `BPF_JMP|BPF_JEQ` comparison per allowed syscall that jumps to `RET_ALLOW` on a match, and a fall-through `RET_ERRNO`. That is, in miniature, exactly what `libseccomp` and `seccompiler` compile down to -underneath their friendlier APIs -- Go's version just makes the compiled +underneath their friendlier APIs; Go's version just makes the compiled form visible because nothing hides it. The other detail to notice is `SECCOMP_FILTER_FLAG_TSYNC` in the syscall arguments and its Landlock counterpart in `applyLandlock`: both matter only because of what the Go @@ -339,23 +339,23 @@ runtime does before `main` ever runs, covered next. Every failure path in this example distinguishes three outcomes, and the distinction is what makes a negative control trustworthy rather than -decorative. A **setup failure** -- Landlock unsupported, `seccomp_load` +decorative. A **setup failure** (Landlock unsupported, `seccomp_load` rejecting a malformed filter, a permission error opening the sandbox -directory -- is a real bug and exits `1` with the syscall name and +directory) exits `1` with the syscall name and `strerror` text, identically formatted across languages (`SysErr`-style in -C++/Rust, a wrapped `error` in Go). A **confirmed denial** -- the sandbox -did exactly what it claims, `EPERM` on the forbidden syscall or `EACCES` on -the outside path -- exits `20`, `fwatch`'s own "probe passed" convention. -An **unconfirmed denial** -- the probed operation *succeeded* when it -should have been blocked -- exits `21`: a real sandboxing bug, distinct +C++/Rust, a wrapped `error` in Go). A **confirmed denial** (the sandbox +did exactly what it claims: `EPERM` on the forbidden syscall or `EACCES` on +the outside path) exits `20`, `fwatch`'s own "probe passed" convention. +An **unconfirmed denial** (the probed operation *succeeded* when it +should have been blocked) exits `21`, distinct from both a clean pass and an unrelated error. Reusing the ordinary `0/1/2` contract for this would erase exactly the distinction the chapter is about: a script that only checks "exit code is nonzero" cannot tell "the sandbox -worked" from "the sandbox is broken." +worked" from "the sandbox failed to enforce." ## Concurrency lens -Both Landlock and seccomp are, by default, **per-thread** kernel state -- +Both Landlock and seccomp are, by default, **per-thread** kernel state: they restrict the calling thread, not automatically every thread in the process. That is invisible in C++ and Rust here because both builds stay single-threaded before the sandbox goes on: `main` calls @@ -366,19 +366,20 @@ are the same thing. Go cannot make that assumption, and its `main.go` says so in a comment worth quoting directly: "the Go runtime schedules goroutines across OS threads it creates and destroys on its own schedule (GC workers, sysmon, -threads parked in blocking syscalls) -- applying either sandbox to only the +threads parked in blocking syscalls): applying either sandbox to only the calling thread would leave those other threads unrestricted, and a goroutine the runtime later migrates onto one of them would silently run -outside the sandbox." The fix is two-part, mirroring the C++/Rust identity -lesson from chapter 14 almost exactly: `runtime.LockOSThread()` pins the +outside the sandbox." The fix is two-part, mirroring the C++/Rust +thread-identity lesson for per-thread privilege state almost exactly: +`runtime.LockOSThread()` pins the sandboxing goroutine to one OS thread so the thread issuing the Landlock and seccomp syscalls never changes out from under it mid-sequence, and both `landlock_restrict_self`'s `TSYNC` flag and `SECCOMP_FILTER_FLAG_TSYNC` apply the new restriction to *every* thread already running in the process atomically, not just the caller's. `applyLandlock` tries `TSYNC` first and -falls back to per-thread-only only if the running kernel rejects the flag --- a second, smaller instance of "probe, don't assume." Confirmed -behaviorally, not just by reading the flag names: a goroutine the runtime +falls back to per-thread-only only if the running kernel rejects the flag, +a second, smaller instance of "probe, don't assume." Confirmed +behaviorally, rather than merely by reading the flag names: a goroutine the runtime schedules onto a different OS thread *after* the sandbox is installed still gets `EACCES`/`EPERM` on the restricted operations, because `TSYNC` already covered that thread before it existed as a goroutine destination. @@ -463,11 +464,11 @@ Seccomp_filters: 1 ``` `Seccomp: 2` is `SECCOMP_MODE_FILTER` (mode 1 is the older, filter-less -"strict" mode; 0 is unsandboxed) -- `/proc` confirms the filter is live +"strict" mode; 0 is unsandboxed): `/proc` confirms the filter is live independently of anything the process prints about itself. `ip_unprivileged_port_start`-style sysctl shortcuts do not apply here since neither mechanism has a global toggle to check against; the closest -equivalent negative control not shown above is simply repeating the +equivalent negative control not shown above is repeating the `outside` probe against a path *inside* the sandboxed tree, which the demo script's third block already exercises implicitly by watching successfully there. @@ -480,23 +481,23 @@ there. observing it, built from a real `strace -f -c` profile of the workload, not a guess. - `SCMP_ACT_ERRNO` lets a denied call return and the caller keep running; - `SCMP_ACT_KILL_PROCESS` ends the process on `SIGSYS` with no return -- + `SCMP_ACT_KILL_PROCESS` ends the process on `SIGSYS` with no return, confirmed by watching `strace` show `getpid()` never return in the kill-mode probe. - Landlock's ruleset/rule/restrict_self model is layered access control you build yourself, and the ABI must be probed at runtime - (`landlock_create_ruleset(NULL, 0, VERSION)`) rather than assumed -- this + (`landlock_create_ruleset(NULL, 0, VERSION)`) rather than assumed: this session's own host and `systems-target` guest returned different versions (9 vs 7) from the same distribution. - Both mechanisms are per-thread by default; a single-threaded process never notices, but Go's always-threaded runtime needs `runtime.LockOSThread()` plus the `TSYNC` flags to cover threads the - scheduler creates around the sandboxing call -- the same "identity is - per-thread, not per-process" lesson chapter 14 taught for capabilities. + scheduler creates around the sandboxing call, the same "identity is + per-thread, not per-process" lesson that capability dropping teaches. Next, **`examples/34-our-programs-in-containers`**: the same daemons this book has built, running inside rootless Podman containers built from -multi-stage UBI 10 Containerfiles -- where seccomp and Landlock profiles +multi-stage UBI 10 Containerfiles, where seccomp and Landlock profiles like this chapter's stop being something a program installs on itself and start being something a container runtime installs *for* it. diff --git a/_docs/34-our-programs-in-containers.md b/_docs/34-our-programs-in-containers.md index 7dbcb86..2400bc4 100644 --- a/_docs/34-our-programs-in-containers.md +++ b/_docs/34-our-programs-in-containers.md @@ -2,7 +2,7 @@ title: "Our programs in containers" order: 34 part: "Containers and Virtualization" -description: "The same three daemons this book has built all along, now shipped as multi-stage UBI10 Containerfiles and run under rootless podman -- pid 1's signal and reaping duties, the GOMAXPROCS-from-cgroup story that catches C++'s hardware_concurrency() lying about its own container, debugging across the namespace boundary with podman exec/nsenter/gdb, and the seccomp default that quietly denies io_uring." +description: "The same three daemons this book has built all along, now shipped as multi-stage UBI10 Containerfiles and run under rootless podman: pid 1's signal and reaping duties, the GOMAXPROCS-from-cgroup story that catches C++'s hardware_concurrency() lying about its own container, debugging across the namespace boundary with podman exec/nsenter/gdb, and the seccomp default that quietly denies io_uring." duration: "60 minutes" --- @@ -22,16 +22,16 @@ By the end of this chapter you can: Shipping a daemon as a container changes facts the program itself can observe and must handle correctly: what counts as CPU parallelism, what happens to an unhandled signal on PID 1, and which syscalls are even reachable. Getting any of these wrong doesn't fail loudly, a C++ build that trusts `hardware_concurrency()` inside a `--cpus=2` container just oversubscribes silently, and a naive entrypoint just sits through a full `podman stop` grace period before being killed. -This chapter also closes the loop on two mechanisms the book built from raw syscalls in Chapters 32 and 33: a container runtime assembles namespaces, cgroups, and a seccomp profile for you, but the pid-1 obligations and the per-thread signal-mask hazards underneath them are the same ones you'd hit building a container tool by hand. +This chapter also closes the loop on two mechanisms the book built from raw syscalls: namespace/cgroup assembly and seccomp sandboxing. A container runtime assembles namespaces, cgroups, and a seccomp profile for you, but the pid-1 obligations and the per-thread signal-mask hazards underneath them are the same ones you'd hit building a container tool by hand. -Chapter 32 gave one of our programs its own namespaces and cgroup directly, -by calling `unshare`/`clone3` itself. Chapter 33 had `fwatch` sandbox itself +Earlier, one of our programs built its own namespaces and cgroup directly, +by calling `unshare`/`clone3` itself, and `fwatch` sandboxed itself with seccomp and Landlock. This chapter hands both jobs to something else: a container runtime. `app`, a tiny stand-in entrypoint built three times, -runs as `podman`'s idea of a container -- and that changes what "process 1" +runs as `podman`'s idea of a container, and that changes what "process 1" means, what "how many CPUs do I have" means, and what a debugger needs to cross to reach it. None of this is exotic: it is exactly the daemons this -book already writes, deployed the way most of them actually ship. +book already writes, deployed the way most of them ship. The code is in `examples/34-our-programs-in-containers/`. The run script there builds/sets up and runs it; its `README.md` covers the CLI, the @@ -39,10 +39,10 @@ output contract, and the full verification story. {% include excalidraw.html file="34-image-layering" - alt="Two side-by-side bands. The left, discarded builder stage starts from registry.access.redhat.com/ubi10/ubi:10.2 at 220 MB measured, runs dnf install of gcc-c++, golang, or rustup, and statically links the binary (cpp -static-libstdc++ -static-libgcc, go CGO_ENABLED=0, rust rustup 1.97.1). An amber COPY --from=builder arrow crosses into the right, shipped runtime stage, which starts from registry.access.redhat.com/ubi10-micro:10.2 at 24.3 MB measured, copies only the one binary to /usr/local/bin/app, and sets ENTRYPOINT/CMD. A footer note reports the real measured final image sizes -- cpp and go both 27.2 MB, rust 24.9 MB -- barely above the bare micro base, and explains that static linking is what makes copying one binary sufficient." - caption="Figure 34.1 -- app's multi-stage Containerfile: the 220 MB builder never ships, only the binary crosses into the 24.3 MB runtime base" %} + alt="Two side-by-side bands. The left, discarded builder stage starts from registry.access.redhat.com/ubi10/ubi:10.2 at 220 MB measured, runs dnf install of gcc-c++, golang, or rustup, and statically links the binary (cpp -static-libstdc++ -static-libgcc, go CGO_ENABLED=0, rust rustup 1.97.1). An amber COPY --from=builder arrow crosses into the right, shipped runtime stage, which starts from registry.access.redhat.com/ubi10-micro:10.2 at 24.3 MB measured, copies only the one binary to /usr/local/bin/app, and sets ENTRYPOINT/CMD. A footer note reports the real measured final image sizes (cpp and go both 27.2 MB, rust 24.9 MB), barely above the bare micro base, and explains that static linking is what makes copying one binary sufficient." + caption="Figure 34.1: app's multi-stage Containerfile; the 220 MB builder never ships, only the binary crosses into the 24.3 MB runtime base" %} -> **Tools used** -- `podman` build/run/exec/inspect/unshare/stop (host, +> **Tools used:** `podman` build/run/exec/inspect/unshare/stop (host, > rootless), `nsenter` (host, both directly and via `podman unshare`), > `gdb` (host, attached across the container's pid namespace), `ldd`/`file` > (host, on binaries copied out of the built images), `python3`+`ctypes` @@ -52,11 +52,11 @@ output contract, and the full verification story. ## Multi-stage images: the builder never ships -A **Containerfile** -- podman's name for the same syntax Dockerfiles use -- +A **Containerfile** (podman's name for the same syntax Dockerfiles use) can declare more than one `FROM`. Each `FROM` starts a fresh, unrelated build stage; a later stage can `COPY --from=` specific files out of it, and only the *last* stage's filesystem becomes the image podman -actually tags. `app`'s C++ build is the clearest read of the shape: +tags. `app`'s C++ build is the clearest read of the shape: ```dockerfile FROM registry.access.redhat.com/ubi10/ubi:10.2 AS builder @@ -75,30 +75,30 @@ ENTRYPOINT ["/usr/local/bin/app"] CMD ["serve"] ``` -The builder stage is `ubi10/ubi:10.2` -- measured on this host at 220 MB -- +The builder stage is `ubi10/ubi:10.2` (measured on this host at 220 MB) with a full `dnf` toolchain installed into it. The runtime stage is `ubi10-micro:10.2`, measured at 24.3 MB: no shell, no package manager, only `glibc` and the handful of libraries UBI's micro variant ships. `COPY --from=builder` reaches back into the discarded stage for exactly one file. Building all three languages' images fresh on this host: `lsp34-cpp` -and `lsp34-go` land at 27.2 MB, `lsp34-rust` at 24.9 MB -- a few MB above +and `lsp34-go` land at 27.2 MB, `lsp34-rust` at 24.9 MB, a few MB above the bare micro base, because the whole 220 MB builder is gone from the final image. -That gap only closes because of the static-vs-dynamic linking recap -chapter 4 built with `sysprobe`: pulling the three languages' binaries back -out of these images with `podman cp` and running `ldd` on them tells the -same story `ldd` told there, sharpened by one deliberate choice. Go's -binary is `not a dynamic executable` -- `CGO_ENABLED=0` gives a fully -static binary, so the runtime stage needs nothing beyond the file itself. -Rust's binary still links `libgcc_s.so.1` and `libc.so.6` -- both already -present in `ubi10-micro`, so no extra `RUN` is needed. C++'s binary is the -interesting middle case: `ldd` lists `libm.so.6`, `libc.so.6`, and the vDSO, -but *not* `libstdc++.so.6` -- the Containerfile's `-static-libstdc++ +That gap only closes because of the syscalls-and-ABI chapter's +static-vs-dynamic linking recap, built with `sysprobe`: pulling the three +languages' binaries back out of these images with `podman cp` and running +`ldd` on them tells the same story `ldd` told there, sharpened by one +deliberate choice. Go's binary is `not a dynamic executable` (`CGO_ENABLED=0` +gives a fully static binary), so the runtime stage needs nothing beyond the +file itself. Rust's binary still links `libgcc_s.so.1` and `libc.so.6`, both +already present in `ubi10-micro`, so no extra `RUN` is needed. C++'s binary +is the interesting middle case: `ldd` lists `libm.so.6`, `libc.so.6`, and +the vDSO, but *not* `libstdc++.so.6`. The Containerfile's `-static-libstdc++ -static-libgcc` flags embed exactly those two libraries into the binary, which is what lets the runtime stage skip installing them. A plain, -dynamically-linked C++ build (chapter 4's default) would need `libstdc++` -copied or installed into the runtime stage too; this chapter's Containerfile +dynamically-linked C++ build (the syscalls-and-ABI chapter's default) would +need `libstdc++` copied or installed into the runtime stage too; this chapter's Containerfile avoids that by choosing static linking specifically for the two libraries the micro base does not carry. Rust's builder has its own wrinkle worth naming: UBI10's `dnf` `rust` package is 1.92.0, older than this book's @@ -107,9 +107,9 @@ exact toolchain via `rustup` instead of `dnf install rust`. ## Rootless podman: one user, one private uid range -Every container here runs **rootless** -- no `sudo`, no privileged daemon --- which is possible because of a second Linux mechanism from chapter 32's -namespace tour: the **user namespace**. `podman` maps container uid 0 to +Every container here runs **rootless**: no `sudo`, no privileged daemon. +That is possible because of a second Linux mechanism from the +namespaces-and-cgroups tour: the **user namespace**. `podman` maps container uid 0 to the invoking host user and a further range of container uids to a private, per-user block of otherwise-unused host uids, declared in `/etc/subuid` and `/etc/subgid`. On this host: @@ -121,7 +121,7 @@ per-user block of otherwise-unused host uids, declared in `/etc/subuid` and ``` That line grants `user` a private range of 65,536 host uids starting at -524288 -- ids no other account owns. `podman unshare cat /proc/self/uid_map`, +524288, ids no other account owns. `podman unshare cat /proc/self/uid_map`, run in podman's own user namespace, shows exactly how the two ranges combine: @@ -131,24 +131,24 @@ combine: 1 524288 65536 ``` -Container uid 0 (root, inside) maps to host uid 1000 (`user`, outside) -- +Container uid 0 (root, inside) maps to host uid 1000 (`user`, outside): one id. Container uids 1 through 65536 map to the private subuid block. The consequence worth sitting with: a process that is root *inside* the container has, outside it, only the privileges of the ordinary host user -who launched podman -- root in the container cannot write to another +who launched podman; root in the container cannot write to another user's files, bind privileged host ports, or load kernel modules, because outside the namespace it never had uid 0 to begin with. This is the same -"identity is a kernel-side mapping, not a label" lesson chapter 14 taught -for capabilities, applied to the uid itself: the mapping table, not the -number a `ps` inside the container prints, is what the kernel actually -checks. +"identity is a kernel-side mapping, not a label" lesson identity and +privilege taught for capabilities, applied to the uid itself: the mapping +table, not the number a `ps` inside the container prints, is what the +kernel checks. ## PID 1: signal duties naive skips Every container's first process inherits a role most programs never occupy: it is PID 1 of its own PID namespace. The kernel treats PID 1 -specially -- a signal for which it has installed **no explicit handler** -is simply dropped instead of running that signal's default action. An +specially: a signal for which it has installed **no explicit handler** +is dropped instead of running that signal's default action. An ordinary process ignoring `SIGTERM` still dies, because "terminate" is the default action when no handler exists; PID 1 with no handler does not even get that. Only `SIGKILL` and `SIGSTOP` still work. `app naive` demonstrates @@ -156,19 +156,19 @@ this by installing nothing: it calls no `sigaction`, `signalfd`, or `signal.Notify` at all before its heartbeat loop. `app serve` is the fix: it blocks `SIGTERM`/`SIGINT`/`SIGCHLD`, receives them explicitly, stops spawning new work, forwards `SIGTERM` to the worker process it supervises, -waits for that worker to exit, and only then exits itself -- so nothing is +waits for that worker to exit, and only then exits itself, so nothing is left running or unreaped when the container stops. {% include excalidraw.html file="34-pid1-signal-flow" - alt="Two side-by-side bands contrasting serve and naive receiving SIGTERM as a container's PID 1. The serve band: podman stop sends SIGTERM, the signalfd/Notify/Signals wake fires, job_spawner.request_stop() runs, kill(worker, SIGTERM) plus waitpid/Wait() reaps it, a shutting-down line prints, and the process exits 0 -- measured at cpp 192ms, go 60 to 87ms, rust 101ms, all well inside a 3000ms grace budget. The naive band: podman stop sends SIGTERM into a kernel that finds no explicit handler on PID 1, so the signal is simply dropped for cpp and rust -- podman waits out the full grace period and falls back to SIGKILL, measured at cpp 2108ms and rust 2074ms. A muted box notes the Go exception: the Go runtime installs its own fatal-signal handler at startup regardless of user code, so naive still dies in about 60ms -- but with no shutting-down line and no cleanup, in every language." - caption="Figure 34.2 -- serve's graceful pid-1 shutdown sequence beside naive's kernel-masked signal, with this session's real podman stop timings" %} + alt="Two side-by-side bands contrasting serve and naive receiving SIGTERM as a container's PID 1. The serve band: podman stop sends SIGTERM, the signalfd/Notify/Signals wake fires, job_spawner.request_stop() runs, kill(worker, SIGTERM) plus waitpid/Wait() reaps it, a shutting-down line prints, and the process exits 0, measured at cpp 192ms, go 60 to 87ms, rust 101ms, all well inside a 3000ms grace budget. The naive band: podman stop sends SIGTERM into a kernel that finds no explicit handler on PID 1, so the signal is dropped for cpp and rust; podman waits out the full grace period and falls back to SIGKILL, measured at cpp 2108ms and rust 2074ms. A muted box notes the Go exception: the Go runtime installs its own fatal-signal handler at startup regardless of user code, so naive still dies in about 60ms, but with no shutting-down line and no cleanup, in every language." + caption="Figure 34.2: serve's graceful pid-1 shutdown sequence beside naive's kernel-masked signal, with this session's real podman stop timings" %} Reaping is the other pid-1 duty, and it splits into two real implementation strategies across the three languages, because Go's and Rust's runtimes are not safe to `fork()` once other threads or goroutines exist. C++ uses a raw `fork(2)` for both the worker and each short-lived -job -- no `exec`, just a different function running in the child -- because +job (no `exec`, just a different function running in the child) because `app`'s C++ build is single-threaded by design up to that point, so the forking thread and the whole process are the same thing. Go and Rust instead **self-reexec**: `serve` resolves its own binary path @@ -184,7 +184,7 @@ builds. The headline trap: three ways of asking "how many CPUs do I have" give two different answers under the same constraint. `podman run --cpus=2 --memory=128m` sets the container's cgroup v2 `cpu.max` to `200000 -100000` (200ms of CPU time per 100ms period -- two cores' worth) and +100000` (200ms of CPU time per 100ms period, two cores' worth) and `memory.max` to `134217728` bytes. Reading those files back from inside each language's own process and reporting them alongside a parallelism estimate is `app`'s first printed line: @@ -220,7 +220,7 @@ fn print_container_line() { ``` All three share the same `cpu_max_display()`/`mem_max_display()` helpers -(not shown above -- they resolve `/proc/self/cgroup`'s `"0::"` line +(not shown above; they resolve `/proc/self/cgroup`'s `"0::"` line to find *this process's own* cgroup, then read `cpu.max`/`memory.max` under `/sys/fs/cgroup` at that path, falling back to the root cgroup file if the per-path one is missing). Those helpers agree completely across @@ -236,20 +236,20 @@ rust: container: cpu.max=200000/100000 effective_parallelism=2 mem.max=13421772 ``` `std::thread::hardware_concurrency()` is `sysconf(_SC_NPROCESSORS_ONLN)` -under the hood -- it asks how many CPUs the *host* has online and has no +under the hood: it asks how many CPUs the *host* has online and has no concept of a cgroup, so it reports 16 regardless of `--cpus=2`. As of the `go 1.26` language version pinned in `go.mod`, `runtime.GOMAXPROCS(0)`'s *default* (unset by the environment or a prior call) divides the cgroup's CPU bandwidth quota by its period whenever that is lower than the host's core count, and re-checks periodically in case an orchestrator changes the -limit live -- confirmed here as `2`, matching the actual constraint +limit live, confirmed here as `2`, matching the actual constraint (`GODEBUG=containermaxprocs=0` opts back out, for anyone who needs the old host-count behavior). Rust's `std::thread::available_parallelism()` on Linux performs the same quota-over-period division against the calling thread's affinity mask, landing on the same correct `2`. A C++ thread pool sized off `hardware_concurrency()` in this container would spawn sixteen -threads to share two cores' worth of CPU time -- oversubscribed, thrashing -the scheduler, and never once reading the number that was actually true. +threads to share two cores' worth of CPU time: oversubscribed, thrashing +the scheduler, and never once reading the number that was true. ## How the code works @@ -258,8 +258,9 @@ one shape in all three languages: block/register the shutdown signals, start the supervised worker, start a periodic job spawner, then block until a signal arrives. C++ blocks `SIGTERM`/`SIGINT`/`SIGCHLD` with `sigprocmask(SIG_BLOCK, ...)` and receives them as `signalfd(2)` reads -inside an ordinary `poll(2)` loop -- the same pattern chapter 12 built for -`pmon`, so no code ever runs in real signal-handler context. Go and Rust +inside an ordinary `poll(2)` loop, the same pattern built earlier for +`pmon`'s signal handling, so no code ever runs in real signal-handler +context. Go and Rust skip that machinery entirely: Go's `signal.Notify` delivers onto a channel, and Rust's `signal_hook::iterator::Signals` delivers as a blocking iterator on the main thread; both are ordinary, safely-scheduled @@ -267,7 +268,7 @@ code, not async-signal-restricted handlers, because neither language's runtime lets you install a raw POSIX handler as directly as C does. The `std::jthread` (C++) and background goroutine/thread (Go/Rust) that spawn jobs every second all take the same shape: loop, sleep in short slices so -a stop request is noticed quickly, spawn one job, repeat -- and every one +a stop request is noticed quickly, spawn one job, repeat, and every one is cooperatively stopped (`stop_token`, an atomic flag, a channel close) before `serve` waits on the worker, so no job-spawn races the shutdown sequence. @@ -276,17 +277,17 @@ One hardcoded assumption deserves a direct look: `cpu_max_display()` only handles cgroup v2's single-line `cpu.max` format (`"max"` or `" "`); a host still running cgroup v1 would need the older `cpu.cfs_quota_us`/`cpu.cfs_period_us` pair instead, and none of these -builds falls back to that -- they print `"unknown"` from the empty-file +builds falls back to that; they print `"unknown"` from the empty-file path instead of guessing. ## Errors, three ways -Setup failures -- a `fork`/`spawn` that fails, a `signalfd`/`Signals` that -cannot be installed -- exit `1` with the syscall name and its error text, +Setup failures (a `fork`/`spawn` that fails, a `signalfd`/`Signals` that +cannot be installed) exit `1` with the syscall name and its error text, formatted identically across languages. Usage errors (an unrecognized subcommand, missing arguments to `job`) exit `2` before anything is spawned. There is no third "confirmed denial" exit code in this example -the way chapter 33's sandbox probes had one -- `app`'s only externally +the way the seccomp-and-Landlock sandbox probes had one: `app`'s only externally visible contract is its stdout lines and its exit code on a clean shutdown (`0`), and the *absence* of the `shutting down` line from `naive`'s log is itself the evidence `verify.lua` checks, not a distinct @@ -296,19 +297,18 @@ language needs, covered next. ## Concurrency lens -The one real bug this example produced during development lived in C++'s -raw `fork(2)` path, and it is the same class of mistake chapter 14 warned -about for credentials: **signal masks are per-thread kernel state, and -`fork()` copies only the calling thread into the child.** `serve` blocks -`SIGTERM`/`SIGINT`/`SIGCHLD` with `sigprocmask(SIG_BLOCK, ...)` so it can -receive them via `signalfd`. The worker, forked from that same thread, -inherited the *blocked* mask -- so when `serve` later forwarded `SIGTERM` -to it with `kill(worker_pid, SIGTERM)`, the signal did not run its default -(terminate) action. It just sat **pending**, forever: the worker was -unkillable by anything short of `SIGKILL`, exactly the failure mode -`naive` demonstrates at the container's pid-1 layer, now -reproduced accidentally one process down. The fix is one line in the -child, immediately after `fork()`: +C++'s raw `fork(2)` path hits the same class of mistake that identity and +privilege warned about for credentials: **signal masks are per-thread +kernel state, and `fork()` copies only the calling thread into the +child.** `serve` blocks `SIGTERM`/`SIGINT`/`SIGCHLD` with +`sigprocmask(SIG_BLOCK, ...)` so it can receive them via `signalfd`. The +worker, forked from that same thread, inherited the *blocked* mask, so +when `serve` later forwarded `SIGTERM` to it with `kill(worker_pid, +SIGTERM)`, the signal did not run its default (terminate) action. It just +sat **pending**, forever: the worker was unkillable by anything short of +`SIGKILL`, exactly the failure mode `naive` demonstrates at the +container's pid-1 layer, now reproduced one process down. The fix is one +line in the child, immediately after `fork()`: ```cpp sigset_t empty; @@ -316,16 +316,16 @@ child, immediately after `fork()`: ::sigprocmask(SIG_SETMASK, &empty, nullptr); ``` -which mirrors `pmon`'s reset-before-`execvp` from chapter 12 almost -exactly -- the difference is that there is no `exec` here to reset the -mask automatically, so the reset has to be written explicitly. Go and Rust -never meet this bug at all, and not by luck: both languages avoid +which mirrors `pmon`'s reset-before-`execvp` from the signals chapter +almost exactly: the difference is that there is no `exec` here to reset +the mask automatically, so the reset has to be written explicitly. Go and +Rust never hit this failure mode: both languages avoid `fork()`-without-`exec()` entirely, because forking a multithreaded runtime (goroutine scheduler threads, Rust's allocator and async executors) risks a forked child whose only surviving thread might be holding a lock some other, vanished thread owned at the moment of the fork. Self-reexec sidesteps the whole hazard class by never forking a -live multithreaded process in the first place -- the new process starts +live multithreaded process in the first place; the new process starts clean, mask and all, from `execve`. ## Debugging across the namespace boundary @@ -335,7 +335,7 @@ Three tools reach the same PID 1 from three different starting points. view, so `podman exec cat /proc/1/status` just works. Reaching the same process from the **bare host** needs a different approach, because a rootless container's namespaces are private to podman's own unprivileged -user namespace -- plain `nsenter` targeting the container's host-side pid +user namespace: plain `nsenter` targeting the container's host-side pid fails outright: ```bash @@ -354,8 +354,8 @@ the container's pid and mount namespaces on top: {% include excalidraw.html file="34-ns-debugging-map" - alt="A host band above a podman-user-namespace band. podman exec reaches the container PID 1 directly from inside the container runtime. podman unshare nsenter first joins podman's own user namespace, then nsenter --target --pid --mount reaches the same PID 1 from there -- both confirming Name: app and /proc/1/exe resolving to /usr/local/bin/app. A dashed, failing arrow shows plain nsenter without podman unshare returning EPERM: reassociate to namespaces failed. Below, real evidence boxes show the subuid mapping user:524288:65536 and a live zombie scan of /proc/[0-9]*/stat reporting zero." - caption="Figure 34.3 -- podman exec and podman unshare nsenter reach the same container PID 1 from two different starting namespaces; plain nsenter cannot" %} + alt="A host band above a podman-user-namespace band. podman exec reaches the container PID 1 directly from inside the container runtime. podman unshare nsenter first joins podman's own user namespace, then nsenter --target --pid --mount reaches the same PID 1 from there, both confirming Name: app and /proc/1/exe resolving to /usr/local/bin/app. A dashed, failing arrow shows plain nsenter without podman unshare returning EPERM: reassociate to namespaces failed. Below, real evidence boxes show the subuid mapping user:524288:65536 and a live zombie scan of /proc/[0-9]*/stat reporting zero." + caption="Figure 34.3: podman exec and podman unshare nsenter reach the same container PID 1 from two different starting namespaces; plain nsenter cannot" %} A live `gdb` attach is the real surprise: because rootless podman's user namespace is *owned by the same host uid* that is @@ -371,13 +371,13 @@ exe = '/usr/local/bin/app' ``` `gdb` attaches and correctly resolves the executable, but its own warning -names the limitation precisely: it is reading `/proc//*` from +names the boundary precisely: it is reading `/proc//*` from *outside* the container's pid namespace, so anything that depends on pid numbers matching what the container sees (`info threads`, breakpoints set by a pid the container reports) is unreliable from here. A `gdbserver` started with `podman exec` inside the container, with `gdb`'s `target remote` reaching it from the host, is the reliable version of this same -cross-boundary attach -- the same shape chapter 28's remote-debugging +cross-boundary attach, the same shape the gdb-and-remote-debugging topology already covers. ## Performance and security trade-offs @@ -385,9 +385,9 @@ topology already covers. Two costs are worth pricing accurately against what a container buys. The **default seccomp profile** podman ships (`/usr/share/containers/seccomp.json` on this host) declares `defaultAction: SCMP_ACT_ERRNO` with -`defaultErrnoRet: 38` (`ENOSYS`) for anything not on its allowlist -- and +`defaultErrnoRet: 38` (`ENOSYS`) for anything not on its allowlist, and `io_uring_setup`, `io_uring_enter`, and `io_uring_register` are not on it. -That is directly relevant after chapter 10: a program built to use +That is directly relevant to io_uring: a program built to use `io_uring` for its I/O will silently fail to set up a ring inside a default-profile container. Probing the same call three ways on this host makes the boundary observable rather than assumed: @@ -400,13 +400,14 @@ makes the boundary observable rather than assumed: ``` `EFAULT` (14, "Bad address") is what the real kernel handler returns for a -null `params` pointer -- proof the call reached it. `ENOSYS` (38) under +null `params` pointer, proof the call reached it. `ENOSYS` (38) under the default profile is seccomp's own default-deny return, generated without the syscall ever running; `seccomp=unconfined` removes the filter and the exact same `EFAULT` reappears, isolating the profile as the cause. -Namespace and cgroup accounting add their own small, measured tax -- -every `read`/`write` a namespaced process makes still walks the same -cgroup memory accounting chapter 32 introduced, and every `cpu.max` +Namespace and cgroup accounting add their own small, measured tax: every +`read`/`write` a namespaced process makes still walks the same cgroup +memory accounting the namespaces-and-cgroups chapter introduced, and +every `cpu.max` lookup this chapter's own `app` performs is one extra `/proc`+`/sys` read at startup, not a per-syscall cost. What a container buys in return is concrete and already demonstrated above: one image that carries its own @@ -415,7 +416,7 @@ compromised process cannot see or touch the rest of the host's process tree or filesystem by default, and a resource ceiling the kernel enforces whether or not the program inside ever asks about it. The next chapter's `overheadbench` puts hard numbers on the remaining rungs of this same -ladder -- container versus full KVM guest versus bare host. +ladder: container versus full KVM guest versus bare host. ## Build, run, observe @@ -462,7 +463,7 @@ app: shutting down (SIGTERM) The `naive` half of the trap, run the same way with a 2s grace instead of 3s, on all three images: `podman stop -t 2` measured `cpp` at 2108ms and -`rust` at 2074ms -- the full grace period, both `SIGKILL`ed -- while `go` +`rust` at 2074ms (the full grace period, both `SIGKILL`ed), while `go` stopped in 60ms because its runtime installs its own fatal-`SIGTERM` handler regardless of user code. Every one of the three logs, checked with `grep -c "shutting down"`, reported zero: none of the three ever ran the @@ -480,13 +481,13 @@ CPUQuota=200000 CPUPeriod=100000 Memory=134217728 ``` `200000/100000` from `podman inspect` matches `app`'s own -`cpu.max=200000/100000` line, and `134217728` matches `mem.max=134217728` --- two independent readers of the same cgroup files agreeing. The `SIGTERM` +`cpu.max=200000/100000` line, and `134217728` matches `mem.max=134217728`: +two independent readers of the same cgroup files agreeing. The `SIGTERM` grace behavior is its own cross-check in miniature: `serve`'s stop time (87-192ms across languages) sits nowhere near the 3-second grace budget `podman stop -t 3` allows, while `naive`'s cpp/rust stop time (2074-2108ms) sits right at the 2-second grace budget `podman stop -t 2` allows before -`SIGKILL` -- the timing itself is the proof that one path ran a graceful +`SIGKILL`: the timing itself is the proof that one path ran a graceful shutdown and the other waited out a timeout. ## What you learned @@ -496,30 +497,30 @@ shutdown and the other waited out a timeout. runtime (24.3 MB measured) and copying one static-linked binary across keeps the final image within a few MB of the bare runtime base. - A container's PID 1 gets no default signal action for anything it has - not explicitly handled -- `naive`'s absent handler is why `podman stop` + not explicitly handled: `naive`'s absent handler is why `podman stop` has to wait out the full grace period and fall back to `SIGKILL`, and why `serve`'s explicit forward-then-reap sequence is not optional ceremony. - "How many CPUs do I have" has three real answers under the same `--cpus=2` constraint: C++'s `hardware_concurrency()` reports the host's 16, while Go's cgroup-aware `GOMAXPROCS` default and Rust's - `available_parallelism()` both report the true 2 -- confirmed live on + `available_parallelism()` both report the true 2, confirmed live on this host, not merely documented. - `podman exec`, `podman unshare nsenter`, and even a direct `gdb -p` attach (because rootless podman's user namespace shares the host uid) all reach the same container PID 1 from different starting points, while - the default seccomp profile silently denies `io_uring` calls that a - chapter-10-style program might depend on. + the default seccomp profile silently denies `io_uring` calls that an + io_uring-based program might depend on. -Next, **virtualization and KVM**: the same daemons run a rung further out --- inside a full guest kernel instead of sharing the host's -- and +Next, **virtualization and KVM**: the same daemons run a rung further out, +inside a full guest kernel instead of sharing the host's, and `overheadbench` prices the difference in nanoseconds, gigabytes per second, and operations per second, across the bare host, this chapter's containers, and a real `systems-target` guest. --- -

verified -- every +

verified: every transcript and number above was produced this session on this host (Fedora Linux 44, kernel 7.1, 16 cpus, podman 5.8.4 rootless): python3 scripts/test-all-examples.py --only diff --git a/_docs/35-virtualization-and-kvm.md b/_docs/35-virtualization-and-kvm.md index 85e7d1d..94ee102 100644 --- a/_docs/35-virtualization-and-kvm.md +++ b/_docs/35-virtualization-and-kvm.md @@ -20,24 +20,23 @@ By the end of this chapter you can: ## Why this matters -Choosing between a process, a container, a microVM, and a full VM is a cost decision, not an abstract security ranking, and the cost only becomes real once you measure it in the same units this book has used throughout: nanoseconds per syscall, gigabytes per second, operations per second. `overheadbench` turns "which deployment option" into a question with a number attached, run on the exact hardware and kernel you'll actually ship on. +Choosing between a process, a container, a microVM, and a full VM is a cost decision, not an abstract security ranking, and the cost only becomes real once you measure it in the same units this book has used throughout: nanoseconds per syscall, gigabytes per second, operations per second. `overheadbench` turns "which deployment option" into a question with a number attached, run on the exact hardware and kernel you'll ship on. The VMX mental model matters beyond benchmarking too: knowing that a VM exit is triggered by touching state the guest doesn't own, not merely by "leaving the guest", explains why a program's syscall-heavy and I/O-heavy paths can have wildly different overhead profiles inside the same guest, a distinction that shapes where to spend optimization effort. Every daemon in this book has, until now, run either directly on the host or inside a Podman container sharing the host's kernel. This chapter puts a third option on the table: a full KVM guest with its own kernel, and asks the -one question a systems programmer actually needs answered before choosing +one question a systems programmer needs answered before choosing between them: not "which is more secure" in the abstract, but *what does each one cost*, measured in the same units this book has used throughout: nanoseconds per syscall, gigabytes per second, operations per second. `overheadbench` is a tri-language microbenchmark built for exactly this: the same binary, unmodified, runs on the bare host, inside the `systems-target` KVM guest, and inside a rootless Podman container, and the three runs land in -one table. The isolation spectrum this chapter walks (process, container, -microVM, full VM) is not a taxonomy exercise; it is the same spectrum every -one of this book's daemons could be deployed onto, and this is the chapter -that prices each rung. +one table. The isolation spectrum spanning process, container, microVM, and +full VM is not a taxonomy exercise; it is the same spectrum every one of +this book's daemons could be deployed onto, and each rung gets a price here. The code is in `examples/35-virtualization-and-kvm/`. The run script there builds and runs it; its `README.md` covers the CLI and the sanity ranges @@ -46,7 +45,7 @@ builds and runs it; its `README.md` covers the CLI and the sanity ranges {% include excalidraw.html file="35-kvm-virtio-stack" alt="Three horizontal bands. Top: host, VMX root mode, showing kvm.ko and kvm_intel.ko managing EPT tables, a QEMU/VMM userspace box running the vCPU thread and virtio-blk backend, and host NVMe/btrfs where a real fsync lands. Middle: guest, VMX non-root mode, showing the guest kernel serving getpid in-guest with no host trap, overheadbench in guest ring 3 issuing the syscall bench's raw getpid loop and the io bench's open/write/fsync calls, a virtio-blk front-end driver, and a virtqueue descriptor ring. Bottom: the trapped path for fsync: fsync in the guest causes a VM exit, the KVM exit handler hands off to the QEMU vCPU thread, the virtio-blk backend issues the real host fsync and triggers VM enter back into the guest. A fourth band gives the measured overheadbench deltas: syscall about 88ns/call on both host and guest because getpid never triggers a vmexit, mem about 14 to 15 GB/s on both because the EPT walk cost is amortized after the TLB warms up, and io about 1.3k ops/s on the host against about 0.5k ops/s on the guest, the real vmexit tax." - caption="Figure 35.1: the VMX root/non-root boundary, the virtio-blk round trip that fsync actually pays, and where overheadbench's own measured host-vs-guest deltas land" %} + caption="Figure 35.1: the VMX root/non-root boundary, the virtio-blk round trip that fsync pays, and where overheadbench's own measured host-vs-guest deltas land" %} > **Tools used**: `overheadbench` itself, `podman` (build and run, host), > `ssh`/`scp` via `scripts/lab/deploy-to-vm.sh` and the Python runner @@ -56,7 +55,7 @@ builds and runs it; its `README.md` covers the CLI and the sanity ranges > checked by `scripts/check-host.sh` or ship with Fedora's cloud image on the > lab VMs. -## What actually runs a VM +## What runs a VM A container's isolation is namespaces and cgroups layered over *one* kernel; every syscall a containerized process makes is a syscall the host kernel @@ -76,13 +75,13 @@ instructions execute directly on the real CPU (no instruction-level emulation) until something the guest cannot handle itself forces a **VM exit** back to root mode. -The key fact this chapter measures is *what actually causes an exit*. -Ordinary instructions (arithmetic, memory loads and stores, and crucially +The key fact this chapter measures is *what causes an exit*. +Ordinary instructions (arithmetic, memory loads and stores, and **syscalls the guest kernel can serve itself**, like `getpid(2)`) never leave non-root mode at all. The guest kernel's syscall entry point runs exactly like it would on bare metal; there is no hypervisor in that path to trap into. What *does* exit are instructions that touch state the guest does -not really own: port I/O, certain MSR accesses, and, the one this chapter's +not own: port I/O, certain MSR accesses, and, the one this chapter's `io` bench hits on every iteration, an access to an emulated device like `virtio-blk`, the paravirtualized block device virtio defines so guest I/O doesn't have to trap on every disk instruction the way full hardware @@ -107,7 +106,7 @@ beyond native. Four points, not two. A **bare process** has no isolation boundary at all: a syscall it makes *is* a host syscall, and an escape is "you are already root, or you exploit a bug to become root." A **container** -(Chapter 34's Podman UBI images) adds namespaces and cgroup v2 as a soft +(the Podman UBI images built in the containers chapter) adds namespaces and cgroup v2 as a soft boundary layered over the *same* kernel: still one syscall table, so `overheadbench`'s `syscall` and `mem` numbers inside a container should track the host almost exactly, and the `io` bench, run against a directory bind- @@ -215,13 +214,14 @@ fn bench_syscall(iters: u64) -> f64 { } ``` -Three deliberate choices, identical in all three: the loop calls a **raw** +Three choices repeat, identical in all three: the loop calls a **raw** syscall rather than a libc-cached identity (glibc does not cache `getpid` today, but the explicit `syscall(SYS_getpid)`/`unix.Getpid`/`rustix::getpid` spelling makes that guarantee visible in the source, not implicit in a libc version); the clock (`std::chrono::steady_clock`, `time.Now`/`Since`, -`Instant::now`) is a **monotonic** one, because Chapter 24 already -established that a wall-clock read can step backward mid-measurement; and +`Instant::now`) is a **monotonic** one, because the time-and-deadlines +material already established that a wall-clock read can step backward +mid-measurement; and each language guards the compiler from proving the loop is dead: `black_box` in Rust wraps the return value, C++ takes no extra step here because the syscall itself is an unremovable side effect, and the Go build @@ -288,7 +288,7 @@ already made writable for that uid. ## Errors, three ways `overheadbench`'s error contract is the same shape this book has used since -Chapter 5: **argument-parsing failures exit 2** with the usage line on +the errors-three-ways chapter: **argument-parsing failures exit 2** with the usage line on stderr, and **bench-execution failures exit 1** without it, kept as two distinct exit codes so a caller can tell "you asked wrong" from "it tried and the environment failed." An unknown `--bench` value, a flag missing its @@ -299,7 +299,7 @@ the first bucket in all three languages: C++'s `parse_args` returns plain `error` that `main` prints beside the same `usage` constant; Rust's `parse_args` returns `Result` and the `main` match arm does the identical two-line print. The second bucket (the `io` bench's -`open`/`write`/`fsync` calls actually failing, which none of the runs in +`open`/`write`/`fsync` calls failing, which none of the runs in this chapter hit but which a full disk or a read-only bind mount would trigger) propagates through `std::expected`, `(float64, error)`, and `Result` respectively, each wrapped one @@ -311,7 +311,7 @@ were fine; the environment was not. This chapter's isolation spectrum is built entirely from **per-thread kernel state**, and that constraint shows up on both the container and the VM side -in the same shape Chapter 14 already met with capabilities. On the VM side: +in the same shape identity and privilege already met with capabilities. On the VM side: a vCPU file descriptor returned by `KVM_CREATE_VCPU` is only valid to drive with `ioctl(KVM_RUN, ...)` **from the thread that created it**: call it from any other thread and the ioctl fails. That is why QEMU pins exactly one host @@ -321,19 +321,19 @@ in-kernel scheduling context are thread-local, not process-wide. On the container side, entering an existing set of namespaces with `setns(2)` has the *identical* constraint: namespace membership is per-thread state in the kernel's `task_struct`, not a process-wide property, exactly like the uids -and capability sets Chapter 14 walked. A single-threaded C++ or Rust build +and capability sets identity and privilege walked. A single-threaded C++ or Rust build that calls `setns` then immediately does its namespaced work never notices; a Go program is the case that bites, because the Go runtime is multi-threaded and goroutines migrate between OS threads by default: a goroutine that calls `setns` and then, without more, tries to act as though it is still in the new namespace can be rescheduled onto a *different* OS -thread that never made that call. The fix is the same shape as Chapter 14's -Go build pushing the capability dance into the fork/exec child: +thread that never made that call. The fix is the same shape as identity and +privilege's Go build pushing the capability dance into the fork/exec child: `runtime.LockOSThread()` before `setns`, pinning the calling goroutine to its current OS thread for the rest of its life (or until `UnlockOSThread`), so the namespace membership and the goroutine that believes it holds it stay -attached to the same kernel thread. The general rule repeats from Chapter -14: whenever a kernel API's state lives on the thread rather than the +attached to the same kernel thread. The general rule repeats from identity +and privilege: whenever a kernel API's state lives on the thread rather than the process (credentials, capabilities, namespace membership, a KVM vCPU), a runtime that hides its threading from you (Go's goroutines, but the same risk applies to any green-thread or fiber scheduler) needs an explicit pin, @@ -386,7 +386,7 @@ ops/s; guest: 87.7–88.1 ns/call, 14.3–15.0 GB/s, 463–521 ops/s; container: noise: **`syscall` and `mem` land in the same band across all three environments**: a container shares the host's syscall table entirely, and the guest's `getpid` never triggers a VM exit at all, matching Figure 35.1's -claim exactly. **`io` is where the VM actually pays**: roughly 2.9–3× slower +claim exactly. **`io` is where the VM pays**: roughly 2.9–3× slower in the guest than on the host or in the container, the measured cost of every `fsync` taking the virtio-blk round trip this chapter's diagram walks (VM exit, QEMU's backend thread, the real host `fsync`, VM entry) instead @@ -405,7 +405,7 @@ example cpp go rust ## Cross-check `overheadbench` prints its own timing, so an independent check has to verify -the printed numbers correspond to real syscalls actually made, not just +the printed numbers correspond to real syscalls made, not just trust the arithmetic. `strace -c`, run on both sides of the VMX boundary, does exactly that by counting syscalls a second way. On the host, wrapping `--bench io --iters 50`: @@ -424,7 +424,7 @@ bench=io metric=1233.15 unit=ops/s ``` Exactly 50 `fsync` calls for `--iters 50`: the count `overheadbench` claims -to have run is the count the kernel actually saw, independent of the +to have run is the count the kernel saw, independent of the program's own bookkeeping. Run identically on `systems-target`: ```bash @@ -446,7 +446,7 @@ magnitude above the untraced ~85 ns this chapter reports, because `ptrace` stops the tracee on every syscall entry and exit; that overhead is exactly why `overheadbench` times itself with its own monotonic clock instead of trusting a tracer's numbers, the same "the observation tool changes what it -measures" caveat Chapter 14 hit with `strace` across a privilege drop. There +measures" caveat identity and privilege hit with `strace` across a privilege drop. There is no dashboard panel for this chapter's numbers by design: `overheadbench` is a one-shot CLI, and the table above *is* the observation surface. @@ -467,15 +467,15 @@ is a one-shot CLI, and the table above *is* the observation surface. - Kernel state that lives per-thread, not per-process, is a recurring trap: a KVM vCPU fd must be driven from the thread that created it, and `setns(2)` namespace membership is equally per-thread: a Go program must - call `runtime.LockOSThread()` before either, the same lesson Chapter 14's - capability drop taught for uids and capability sets. + call `runtime.LockOSThread()` before either, the same lesson identity and + privilege's capability drop taught for uids and capability sets. - A benchmark's own numbers need an independent check: `strace -c`'s exact syscall counts confirmed `overheadbench`'s claimed iteration counts on both host and guest, while also showing where that same tool's overhead (`ptrace`'s stop-on-every-syscall cost) makes it unfit to trust for timing. Next, Part 10 opens with **observability**: turning `/proc`, `/sys`, and -cgroup statistics into the kind of live picture this chapter had to build by +cgroup statistics into the kind of live picture this chapter built by hand, one `overheadbench` run at a time. --- diff --git a/_docs/36-proc-sys-and-the-agent.md b/_docs/36-proc-sys-and-the-agent.md index fcb88a4..54fcfd2 100644 --- a/_docs/36-proc-sys-and-the-agent.md +++ b/_docs/36-proc-sys-and-the-agent.md @@ -22,21 +22,22 @@ By the end of this chapter you can: A metrics agent that's going to run continuously in production can't afford to fork a subprocess or shell out to a tool for every sample; reading `/proc` and `/sys/fs/cgroup` directly is the only approach cheap enough to run every second for months between deploys. Getting the `/proc/stat` delta wrong, or treating a cumulative counter as an instantaneous reading, is one of the most common correctness bugs in hand-rolled monitoring code. -The cross-language schema discipline here also matters beyond this one example: a dashboard, a `jq` filter, or a test suite that has to understand three different field-naming and serialization conventions across a C++, Go, and Rust build of the same tool is not a tool anyone will actually standardize on. Byte-identical output is what makes the three builds interchangeable in practice, not just in principle. - -Chapter 35's `overheadbench` measured syscall, memory, and I/O overhead across -a bare process, a container, and a VM, but every one of those numbers came -from watching `strace -c` and hand-timed loops wrapped around each run. That's -the right instrument for a one-off measurement campaign; it is not how a -program watches a system continuously, in production, for the months between -deploys. This chapter builds the tool that does: `sysagent`, a USE-method -metrics collector that reads `/proc` and `/sys/fs/cgroup` directly (no root, -no subprocess, no tool you'd have to trust is still installed on the box next -year). Chapter 37's `loadmix` builds the opposite kind of tool: an `analyze` +The cross-language schema discipline here also matters beyond this one example: a dashboard, a `jq` filter, or a test suite that has to understand three different field-naming and serialization conventions across a C++, Go, and Rust build of the same tool is not a tool anyone will standardize on. Byte-identical output is what makes the three builds interchangeable in practice, not just in principle. + +The virtualization-overhead benchmark, `overheadbench`, measured syscall, +memory, and I/O overhead across a bare process, a container, and a VM, but +every one of those numbers came from watching `strace -c` and hand-timed +loops wrapped around each run. That's the right instrument for a one-off +measurement campaign; it is not how a program watches a system continuously, +in production, for the months between deploys. This chapter builds the tool +that does: `sysagent`, a USE-method metrics collector that reads `/proc` and +`/sys/fs/cgroup` directly (no root, no subprocess, no tool you'd have to +trust is still installed on the box next year). The USE-method checklist +tool covered next, `loadmix`, builds the opposite kind of tool: an `analyze` mode that shells out to `vmstat`, `mpstat`, `iostat`, `sar`, and `pidstat` exactly the way an on-call engineer would, and parses their plain-text tables by column name instead of reading kernel counters directly. The split between -the two chapters is deliberate: this one is the case for reading the +the two is deliberate: this one is the case for reading the kernel's own accounting straight from the source, and the numbers it produces are the ones a long-running agent can afford to compute every second without forking a process to do it. @@ -55,8 +56,8 @@ drive it. > no root. The only tool this chapter reaches for to observe it from the > outside is `curl` (host), against the `serve --port P` `/metrics` endpoint; > `curl` ships in Fedora's base install and isn't separately checked by -> `scripts/check-host.sh`, the same way Chapter 39 notes for `cpupower` and -> `objdump`. +> `scripts/check-host.sh`, the same way the benchmarking-without-lies chapter +> notes for `cpupower` and `objdump`. ## Six sources, one deterministic schema @@ -215,7 +216,7 @@ never becomes the interesting part of the diff. `cmd_sample` calls `cmd_serve` hands `port` and `interval_ms` straight to `serve`, which owns the HTTP loop and calls `take_snapshot` again on every `GET /metrics`. -`to_json`/`ToJSON`/`to_json` is the piece that actually delivers the +`to_json`/`ToJSON`/`to_json` is the piece that delivers the cross-language guarantee the schema promises, and it earns that guarantee by staying unglamorous by design: none of the three languages hands the `Snapshot` to a generic serializer. C++ builds the object with an @@ -227,11 +228,11 @@ the *same* key order or the same float formatting as the others out of the box, and a schema that's supposed to be byte-identical across languages can't leave that up to three different libraries' defaults. Hand-writing the same fixed field order three times, with the same `%.2f`-style two-decimal -formatting (`fmt2`/`f2`/`{:.2}`) for every float, is what actually makes +formatting (`fmt2`/`f2`/`{:.2}`) for every float, is what makes `verify.lua`'s key-by-key checks, and a `diff` between any two languages' output, meaningful. -The HTTP layer is the one place the three implementations actually diverge +The HTTP layer is the one place the three implementations diverge in mechanism rather than just syntax, and it follows the book's established chatterd pattern: Go leans on `net/http`'s stdlib server, registering `/metrics` on a `ServeMux` and letting `http.Server.Shutdown` handle a clean @@ -239,8 +240,9 @@ chatterd pattern: Go leans on `net/http`'s stdlib server, registering instead (a raw listening socket, `poll(2)` waiting on both the listener and a `signalfd`/`SignalFd` doorbell, and one inline HTTP/1.1 response writer) because pulling in a framework for one endpoint would hide the exact -mechanism this book has been building toward since Chapter 21's `chatterd` -and Chapter 24's signal-driven shutdown, not simplify it. +mechanism this book has been building toward since `chatterd`'s raw TCP +sockets and the signal-driven shutdown pattern from the timers-and-deadlines +chapter, not simplify it. ## Errors, three ways @@ -270,7 +272,7 @@ never does, its `if` guards only the success path. Go hardcodes the same asymmetry directly into the fan-in loop, which explicitly `continue`s past `r.source == "psi"` before counting it toward the required sources or a possible `firstErr`. Three different enforcement mechanisms (a type the -compiler checks, a code path that's simply never written, a loop that +compiler checks, a code path that's never written, a loop that explicitly skips one name) for the same one-sentence rule: a missing PSI source degrades the snapshot, it never fails it. @@ -284,7 +286,7 @@ the wall-clock cost of one `sample` is `max(interval_ms, gather_time)`, not (`loadavg`, `meminfo`, `diskstats`, `netdev`, `psi`), sleeps, and joins before `t1` is read, guaranteeing every field is filled in before the delta is computed, but *how* each language gets a result out of a worker back into -the shared `Snapshot` is where the three actually part ways: +the shared `Snapshot` is where the three part ways: {% include codetabs.html langs="C++|Go|Rust" %} @@ -508,7 +510,7 @@ five spawned closures (`s.spawn(read_loadavg)`, etc.) don't capture `snap` at all, because two closures each holding `&mut snap` to write disjoint fields would still require *two simultaneous mutable borrows of the same value*, something the borrow checker refuses regardless of whether the underlying -fields actually overlap, since it reasons about the whole object, not the +fields overlap, since it reasons about the whole object, not the fields a closure happens to touch. So Rust's workers return owned values through their `JoinHandle`s instead, and every assignment into `snap` happens in the *scope* thread itself, sequentially, immediately after each @@ -593,7 +595,7 @@ noise, since nothing about this number is supposed to move between two reads a few milliseconds apart. The second cross-check backs a **dynamic** claim instead: that `cpu_util_pct` -actually tracks real CPU load rather than returning a constant or a +tracks real CPU load rather than returning a constant or a miscomputed delta. `verify.lua` takes an idle `sample --json` baseline, spins up `nproc` tight busy-loop children for slightly longer than a `--interval-ms 500` window, samples again, and checks that `cpu_util_pct` @@ -602,7 +604,7 @@ floor (or an outright ≥55% reading, for the case where the host is already loaded before the check even starts). The margin and the retries exist because this runs on a real, possibly-noisy development box, not an isolated benchmark rig; the check has to distinguish "this host happened to be busy -for an unrelated reason" from "the parser is broken," and a single +for an unrelated reason" from "the sampler is miscomputing," and a single non-retried comparison couldn't. ## What you learned @@ -636,15 +638,15 @@ non-retried comparison couldn't. - **`mem_total_kb` served over HTTP matched a direct `/proc/meminfo` read exactly**, and a busy-loop child measurably raised `cpu_util_pct` over an idle baseline: two cross-checks confirming the schema reports what the - kernel actually has, not what the code merely claims. - -Chapter 37 keeps the same Utilization/Saturation/Errors lens but inverts the -source of truth entirely: instead of reading `/proc` directly the way this -chapter does, `loadmix analyze` shells out to `uptime`, `vmstat`, `mpstat`, -`iostat`, `sar`, and `pidstat` exactly as an on-call engineer would, and -parses each tool's plain-text table by column name rather than reading -kernel counters itself. Read the two chapters back to back: same checklist, -same USE method, two structurally opposite ways of answering it. + kernel has, not what the code merely claims. + +The next USE-method checklist keeps the same Utilization/Saturation/Errors +lens but inverts the source of truth entirely: instead of reading `/proc` +directly the way this chapter does, `loadmix analyze` shells out to +`uptime`, `vmstat`, `mpstat`, `iostat`, `sar`, and `pidstat` exactly as an +on-call engineer would, and parses each tool's plain-text table by column +name rather than reading kernel counters itself. Same checklist, same USE +method, two structurally opposite ways of answering it. --- diff --git a/_docs/37-use-method-60s-analysis.md b/_docs/37-use-method-60s-analysis.md index 95b0305..faec6d1 100644 --- a/_docs/37-use-method-60s-analysis.md +++ b/_docs/37-use-method-60s-analysis.md @@ -16,19 +16,20 @@ By the end of this chapter you can: - Spawn five interval samplers concurrently and join them once so an N-second checklist costs N seconds, not five times N - Scale a saturation threshold to the machine it runs on (`run_queue >= cpus + 2`) where that is the accurate signal, instead of a flat constant - Rank resources by Saturation-value-to-threshold ratio rather than raw Saturation value, so a high-scale metric doesn't always win -- Diagnose a `std::jthread` RAII ordering bug (`request_stop()` firing before a worker's own deadline) and fix it by holding the parent scope open with `sleep_until` +- Reason about `std::jthread`'s RAII order (`request_stop()` runs before `join()`) and hold the parent scope open with `sleep_until` so a worker pool runs its full deadline instead of stopping at scope exit ## Why this matters -Most engineers triaging a live system reach for `uptime`, `vmstat`, `mpstat`, `iostat`, and `pidstat`, not a custom collector, because those tools are already trusted and already parse a running kernel correctly. `loadmix analyze` encodes that exact workflow as a program, so understanding how it parses tool output and scales its thresholds is understanding how an on-call engineer's checklist actually works under the hood. +Most engineers triaging a live system reach for `uptime`, `vmstat`, `mpstat`, `iostat`, and `pidstat`, not a custom collector, because those tools are already trusted and already parse a running kernel correctly. `loadmix analyze` encodes that exact workflow as a program, so understanding how it parses tool output and scales its thresholds is understanding how an on-call engineer's checklist works under the hood. The column-name parsing discipline and the concurrent-sampler design generalize past this one tool: any program that wraps an external CLI's plain-text output needs the same defense against a version bump reordering fields, and any program running several blocking interval samples needs the same concurrency shape to avoid turning a one-minute checklist into a five-minute one. -Chapter 36's `sysagent` built a USE-method collector from first principles: -reading `/proc/stat`, `/proc/net`, `/proc/diskstats`, and cgroup PSI directly, -no external process, no tool you'd have to trust. That is the right call for a -program that runs continuously in production. But it is not how most engineers -actually triage a system that's in trouble *right now*: they SSH in and reach +The proc/sys-reading collector built earlier, `sysagent`, built a USE-method +collector from first principles: reading `/proc/stat`, `/proc/net`, +`/proc/diskstats`, and cgroup PSI directly, no external process, no tool +you'd have to trust. That is the right call for a program that runs +continuously in production. But it is not how most engineers triage a +system that's in trouble *right now*: they SSH in and reach for `uptime`, `vmstat 1`, `mpstat -P ALL 1`, `iostat -xz 1`, `pidstat 1`, and `free`, running the checklist Brendan Gregg popularized as "the USE method": for every resource, ask whether it's **Utilized**, **Saturated**, or @@ -70,8 +71,8 @@ app analyze [--seconds N] run the checklist and name `--resource cpu` spins a pool of worker threads doing nothing but xorshift arithmetic, no syscalls, no allocation in the hot loop, pure CPU. `--resource -mem` touches 1.35× the guest's `MemTotal` across a worker pool, deliberately -overshooting available RAM to push real pages into swap. `--resource io` +mem` touches 1.35× the guest's `MemTotal` across a worker pool, overshooting +available RAM to push real pages into swap. `--resource io` opens one file per worker with `O_DSYNC` and `pwrite`s a 16 KiB buffer of xorshift-generated (non-compressible) bytes to the same offset in a loop, forcing a synchronous write to the block device on every call. `--resource @@ -91,7 +92,7 @@ that every parser reads a tool's header row first and looks fields up by *name* against that row, never by a hardcoded index. `Row` (C++), `row` (Go), and `Row` (Rust) all express the same idea: a table row is a name list zipped with a value list, and `get`/`getOr` walks the name list linearly to find the -column a caller actually asked for. +column a caller asked for. {% include codetabs.html langs="C++|Go|Rust" %} @@ -365,7 +366,7 @@ line exists so the U/S/E triple is complete and so a future chapter that The verdict is whichever resource has the highest `Saturation value / Saturation threshold` ratio, not the highest raw Saturation value, which would let net's `8000`-scale threshold always lose to cpu's `4`-scale one -regardless of which resource is actually worse off. +regardless of which resource is worse off. ## Errors, three ways @@ -376,8 +377,9 @@ error** (no `--resource`/`--seconds` given at all, an unrecognized funnels every parse path through `usage()` and returns `2` from `main`; Go calls `usage()` then `os.Exit(2)`; Rust calls `usage()` then `std::process::exit(2)`. There is no separate "runtime error" exit code here, -unlike Chapter 39's codec-bug path or Chapter 40's connection-refused path; -`analyze`'s subprocess calls are designed to degrade gracefully instead of +unlike the benchmarking chapter's codec-bug path or the low-latency-fastpath +chapter's connection-refused path; `analyze`'s subprocess calls are designed +to degrade gracefully instead of failing the whole run: a denied `dmesg`, an empty `ss -s`, or a sampler that produces no data rows all show up as a `0.00`/`denied`/empty field in the output rather than a non-zero exit, because the entire point of a diagnostic @@ -398,13 +400,13 @@ identical atomic counters accumulated across the pool. Where the three languages diverge is in how the parent thread stays alive long enough for that deadline to matter. C++'s `std::jthread` runs -`request_stop()` *before* `join()` in its destructor, which means the instant +`request_stop()` *before* `join()` in its destructor, so the instant the enclosing scope's spawn loop finishes, every worker is told to stop, before the deadline the workers are checking has had any chance to arrive. -Left alone, this was a real bug in the first cut of this example: every -saturator exited in single-digit milliseconds regardless of `--seconds`, -because `request_stop()` fired essentially at thread-creation time. The fix -is `std::this_thread::sleep_until(deadline)` in the parent, holding the +Left to that default, every saturator would exit in single-digit +milliseconds regardless of `--seconds`, since `request_stop()` fires at +thread-creation time. The pattern that works is +`std::this_thread::sleep_until(deadline)` in the parent, holding the enclosing scope open for the full requested duration before it's allowed to close and trigger the RAII-driven stop-then-join: @@ -438,7 +440,7 @@ close and trigger the RAII-driven stop-then-join: Go and Rust never needed this hack, because in both ports each worker checks its *own* deadline directly inside its loop condition (`time.Now().Before(deadline)`, `Instant::now() < deadline`) rather than waiting on an external stop signal -raised by the parent; the parent simply calls `wg.Wait()` / lets +raised by the parent; the parent calls `wg.Wait()` / lets `thread::scope` return, which blocks until every worker's own loop condition has already become false on its own. There is no analogue of `jthread`'s stop-before-join ordering to fight, because nothing external tells these @@ -489,7 +491,7 @@ analyze: done `dmesg status=denied` is expected: the guest's unprivileged `fedora` user gets `Operation not permitted` from `dmesg`, and `analyze` reports it rather than failing. This run finished in **10.417s** wall-clock for -a `--seconds 10` window, confirming the five samplers really do run +a `--seconds 10` window, confirming the five samplers run concurrently; five serial ` 1 10` invocations would have taken close to 50 seconds. @@ -546,7 +548,7 @@ example cpp go rust The load-attribution run above is itself a cross-check: `analyze`'s verdict is only trustworthy if it agrees with what you already know is true, since you launched the saturator yourself. But it's worth pairing `analyze`'s own -report against `top`'s independent view, captured in the very same run: `top +report against `top`'s independent view, captured in the same run: `top -b -n1`'s `%Cpu(s)` line read `97.8 us, 2.2 sy, 0.0 id` while `mpstat`'s `busy_pct` (100.0 − `%idle`) read `100.00`. The two numbers come from different tools sampling at different instants over different windows @@ -555,7 +557,7 @@ different tools sampling at different instants over different windows place, and they don't; what they agree on is the story: this guest's CPUs are saturated, not idle, not lightly loaded. A verdict that `mpstat` alone disagreed with `top` about (one reporting near-idle while the other reported -saturated) would be the sign of a parsing bug, not measurement noise, since +saturated) would be the sign of a parsing error, not measurement noise, since both tools are reading the same underlying `/proc/stat` counters through different sampling windows. @@ -580,20 +582,21 @@ different sampling windows. guest's own core count, unlike net's flat `8000 pkts/s`, because "threads waiting for a CPU" only means something relative to how many CPUs exist. - **`std::jthread`'s RAII order (`request_stop()` before `join()`) can silence - a worker before its own deadline ever fires**: the fix is holding the - parent scope open with `sleep_until(deadline)`; Go and Rust never needed - the workaround because each worker checks its own deadline directly rather - than waiting on an externally raised stop signal. + a worker before its own deadline ever fires**: holding the parent scope + open with `sleep_until(deadline)` keeps every worker alive for its full + duration; Go and Rust don't need that pattern because each worker checks + its own deadline directly rather than waiting on an externally raised stop + signal. - **`analyze`, run concurrently against a background `--resource cpu` saturator it knows nothing about, correctly attributed the load**: `busy_pct` 1.34 → 100.00, `run_queue` 3.00 → 7.00–8.00, Utilization fired true, verdict `cpu`, stable across three independent attempts on the lab guest. -Chapter 38 takes the numbers this checklist prints to a terminal and sends -them somewhere durable: OpenTelemetry into the LGTM stack Chapter 3 stood up, -so the next time a resource saturates you don't have to be watching when it -happens. +The OpenTelemetry pipeline covered next takes the numbers this checklist +prints to a terminal and sends them somewhere durable: into the LGTM stack +stood up early in the book, so the next time a resource saturates you don't +have to be watching when it happens. --- diff --git a/_docs/38-opentelemetry-to-lgtm.md b/_docs/38-opentelemetry-to-lgtm.md index f93fc6c..4ce91df 100644 --- a/_docs/38-opentelemetry-to-lgtm.md +++ b/_docs/38-opentelemetry-to-lgtm.md @@ -2,7 +2,7 @@ title: "OpenTelemetry to LGTM: instrument a service, then prove the spans and metrics actually arrived" order: 38 part: "Observability" -description: "lsp-otel is a traced request/response server that exports OpenTelemetry traces and metrics over OTLP/HTTP to the LGTM stack from Chapter 3: a parent request span wrapping work and respond children, a requests_total counter and a request_duration histogram, and the verify step proves it by querying Tempo and Prometheus through Grafana rather than trusting the program's own word, across C++, Go, and Rust each tagged with its own service.name." +description: "lsp-otel is a traced request/response server that exports OpenTelemetry traces and metrics over OTLP/HTTP to the local LGTM stack: a parent request span wrapping work and respond children, a requests_total counter and a request_duration histogram, and the verify step proves it by querying Tempo and Prometheus through Grafana rather than trusting the program's own word, across C++, Go, and Rust each tagged with its own service.name." duration: "50 minutes" --- @@ -20,24 +20,24 @@ By the end of this chapter you can: ## Why this matters -A program that claims to export telemetry and a program that actually exports telemetry look identical from the terminal: both print a clean exit and move on. The only way to tell them apart is to go read the backend the export was supposed to reach, which is exactly what this chapter's verification step does instead of trusting `serve`'s own stdout. +A program that claims to export telemetry and a program that exports telemetry look identical from the terminal: both print a clean exit and move on. The only way to tell them apart is to go read the backend the export was supposed to reach, which is exactly what this chapter's verification step does instead of trusting `serve`'s own stdout. -That distinction matters because the usual failure mode here is silent: a wrong OTLP endpoint path, a default SDK log flooding the output contract, or a batch that never flushes before the process exits all produce a program that looks healthy and ships nothing. Querying Tempo and Prometheus back is what turns "I instrumented the service" into a claim you can actually stand behind. +That distinction matters because the usual failure mode here is silent: a wrong OTLP endpoint path, a default SDK log flooding the output contract, or a batch that never flushes before the process exits all produce a program that looks healthy and ships nothing. Querying Tempo and Prometheus back is what turns "I instrumented the service" into a claim you can stand behind. -Chapter 37's `loadmix analyze` printed a USE-method verdict to a terminal, and -Chapter 36's `sysagent` served one JSON snapshot per HTTP request. Both make you -*present* to read the number: someone has to be running the command, or scraping -the endpoint, at the moment the system is in trouble. This chapter closes that +The USE-method verdict that `loadmix analyze` prints to a terminal, and the one +JSON snapshot per HTTP request that `sysagent` serves, both make you *present* +to read the number: someone has to be running the command, or scraping the +endpoint, at the moment the system is in trouble. This chapter closes that gap. `lsp-otel` is a small traced server that emits the three OpenTelemetry signal shapes: a distributed **trace** per request, a monotonic **counter**, and a latency **histogram**, and exports them over OTLP/HTTP into the LGTM stack -(Loki, Grafana, Tempo, Mimir plus an OpenTelemetry collector) that Chapter 3 -stood up, so the next time a request is slow you can go *look* at when it +(Loki, Grafana, Tempo, Mimir plus an OpenTelemetry collector) already running +on the host, so the next time a request is slow you can go *look* at when it happened instead of needing to have been watching. And because "I sent telemetry" is the easiest claim in observability to get quietly wrong, the whole point of the verification here is that it queries the stack back (Tempo for the traces, Prometheus for the metrics) and refuses to believe the export happened -until the data is actually there to read. +until the data is there to read. The code is in `examples/38-opentelemetry-to-lgtm/`. It is a **local-mode example that requires the LGTM stack**: `serve` and `drive` run on the host, and @@ -52,7 +52,7 @@ the exporter posts to the collector on `localhost:4318`. > OTLP on `:4318`, Grafana on `:3000`) and `curl` (host, the only tool the > verification needs: it queries Prometheus and Tempo through Grafana's > datasource proxy). The stack is the same `lsp-lgtm` container the -> `lgtm-podman-stack` setup from Chapter 3 provides; nothing else is required on +> `lgtm-podman-stack` setup provides; nothing else is required on > the host. ## Three signals, one request @@ -63,7 +63,7 @@ one "request": - A **trace** describes the shape and timing of that request as a tree of spans. `lsp-otel` starts a parent span named `request`, and *inside* it two - child spans: `work` (a deliberately tiny, variable `1 + seq%4` ms sleep + child spans: `work` (a tiny, variable `1 + seq%4` ms sleep standing in for actual computation) and `respond` (writing the `ok ` reply). The parent/child nesting is what lets Tempo later show you that the request's time was spent in `work`, not `respond`, or vice versa. @@ -233,7 +233,7 @@ defaults (5-second span batches, 60-second metric batches) are fine for a long-running service but far too slow for a demo that starts, drives ten requests, and stops within a couple of seconds. A short interval plus a **flush on shutdown** is what guarantees the last batch (often *all* the batches, at -this scale) actually leaves the process before it exits. That shutdown path is +this scale) leaves the process before it exits. That shutdown path is the subject of the Concurrency lens below, because it turned out to be where the three languages diverge most. @@ -252,7 +252,7 @@ A **runtime error** is exit **1** with an `app: error: …` message: the one `verify.lua` exercises is `drive` against an address nothing is listening on, which fails to connect and prints `app: error: dial …: connection refused`. This is the one message that is *not* byte-identical across the three languages: -each surfaces its own dial/`errno` wording, the same way Chapter 40's `connect` +each surfaces its own dial/`errno` wording, the same way a low-latency fastpath's `connect` errors carry each platform's `strerror` text, so `verify.lua` asserts only the shared `app: error:` prefix and the exit code, not the full string. A clean run of either subcommand exits **0**: `serve` after a signal-driven shutdown, `drive` @@ -444,9 +444,9 @@ from the program telling us about itself. `SIG_IGN` a background shell hands a child before `sigwait` can ever see the signal. -Chapter 39 turns from *collecting* numbers to *trusting* them: how to benchmark -your own code without the warmup, coordinated-omission, and dead-code traps that -make most microbenchmarks quietly measure the wrong thing. +Benchmarking without lies turns from *collecting* numbers to *trusting* them: +how to benchmark your own code without the warmup, coordinated-omission, and +dead-code traps that make most microbenchmarks quietly measure the wrong thing. --- diff --git a/_docs/39-benchmarking-without-lies.md b/_docs/39-benchmarking-without-lies.md index 8f8131a..3c359b5 100644 --- a/_docs/39-benchmarking-without-lies.md +++ b/_docs/39-benchmarking-without-lies.md @@ -2,7 +2,7 @@ title: "Benchmarking without lies: warmup, coordinated omission, and the percentile the mean hides" order: 39 part: "Performance and Low Latency" -description: "benchlab measures the ch21 chatterd frame codec two ways in the same binary: a real run with warmup discard, per-iteration timing, and a coordinated-omission-corrected p99, and --lie's single unwarmed wall-clock sample, cross-checked with perf stat cycles/instructions and an objdump proof that an unwatched microbenchmark just gets compiled away." +description: "benchlab measures the chatterd frame codec two ways in the same binary: a real run with warmup discard, per-iteration timing, and a coordinated-omission-corrected p99, and --lie's single unwarmed wall-clock sample, cross-checked with perf stat cycles/instructions and an objdump proof that an unwatched microbenchmark just gets compiled away." duration: "50 minutes" --- @@ -22,10 +22,10 @@ By the end of this chapter you can: A microbenchmark you wrote yourself is the easiest number in the whole book to get quietly wrong, because nothing stops it from compiling cleanly and printing a confident, reproducible figure for code that never ran, or for a stall a closed loop recorded once when a real caller would have felt it hundreds of times over. `benchlab` is built to show both traps happening in the same binary, so the gap between a misleading number and a real one is something you can watch rather than take on faith. -The payoff carries past this chapter: Chapter 40 spends the discipline earned here, pinning a core and reusing a buffer to shrink exactly the tail this chapter learned to measure accurately. +The payoff carries past this chapter: a low-latency fastpath spends the discipline earned here, pinning a core and reusing a buffer to shrink exactly the tail this chapter learned to measure accurately. -Chapter 37 gave you the USE method's 60-second checklist for a system already -in trouble, and Chapter 38 wired that telemetry into the LGTM stack you can +The USE method gives a 60-second checklist for a system already +in trouble, and OpenTelemetry wires that telemetry into the LGTM stack you can query back. Both assume the numbers arriving are real. This chapter asks the harder question: when *you* produce the number (a microbenchmark of your own function, timed on your own laptop), what makes it trustworthy? The uncomfortable @@ -146,7 +146,7 @@ distribution the tail exists to describe. {% include excalidraw.html file="39-coordinated-omission" alt="Two stacked bands. Top, benchlab's raw closed-loop timer: iterations 1 through 3 each a small box around 40 nanoseconds, iteration 4 an accent box labeled STALL page fault slash sched, 1057 nanoseconds, one sample. Bottom, what a fixed-rate caller would have seen: the same three small requests, the same stalling request 4, then two dashed ghost boxes labeled request 5 queued and request 6 queued at roughly 1018 and 979 nanoseconds, backfilled synthetic samples continuing down toward the expected interval floor. A dashed vertical line connects the two request 4 boxes labeled same stall. Captions read the raw p99 over a 1000-sample window buries the single stall sample, while the corrected co_p99 backfills the queued requests so co_p99_ns is always at least p99_ns." - caption="Figure 39.2: one stall, recorded once by the closed loop (top) versus what a fixed-rate caller actually experiences (bottom); co_correct backfills the difference" %} + caption="Figure 39.2: one stall, recorded once by the closed loop (top) versus what a fixed-rate caller experiences (bottom); co_correct backfills the difference" %} The correction (HdrHistogram calls it `recordValueWithExpectedInterval`) is mechanical once you see the picture: for every raw sample `v` that exceeds an @@ -187,7 +187,7 @@ tens of nanoseconds, and a small minority (the ones a scheduler, a page fault, or, in a GC'd runtime, a collection pause touches) finish orders of magnitude slower. Average those together and the mean is dragged toward the tail by exactly the outliers a reader most needs to see reported accurately, -while simultaneously hiding *how bad* the tail actually gets: a mean of 200 ns +while hiding *how bad* the tail gets: a mean of 200 ns is consistent with "every call took 200 ns" and with "999 calls took 40 ns and one took 160,200 ns," and those are utterly different systems to operate. A **percentile** answers a sharper question: "what latency do 99% of calls beat?" @@ -209,7 +209,7 @@ loop. Every `run_once` call in `benchlab` folds its result into a running `checksum` for exactly this reason: the accumulation is a real, observable side effect (it's printed at the end), so nothing upstream of it can be proven dead. To see the failure mode this guards against, compile a stripped -version of the same idea and read what GCC actually produced: +version of the same idea and read what GCC produced: ```console [host]$ g++ -std=c++23 -O2 -c dce.cpp -o dce.o && objdump -d --no-show-raw-insn dce.o @@ -360,7 +360,7 @@ fixed workload and checks the identical invariants (`co_p99_ns >= p99_ns`, `percentile()` is the other half: given a *sorted* array and a fraction `p`, nearest-rank indexing (`sorted[floor(p * (len - 1))]`) picks the sample at that rank directly: no interpolation, no bucketing, correct as long as the -array really is sorted first, which `main` guarantees by calling `sort` +array is sorted first, which `main` guarantees by calling `sort` (C++'s `std::sort`, Go's `sort.Slice`, Rust's `sort_unstable`) immediately before computing `median_ns`, `p99_ns`, and again after `co_correct` before computing `co_p99_ns`. The warmup loop and the timed loop share the same @@ -408,7 +408,7 @@ between any of this host's 16 logical CPUs between iterations, and a migration that lands it on a core the kernel had to first bring back up from a low C-state, or evicts its L1/L2 working set entirely, produces exactly the multi-microsecond single-iteration stall this chapter measures -and corrects for. Chapter 26's `taskset -c 0,2` pinning is the mitigation: +and corrects for. CPU pinning with `taskset -c 0,2` is the mitigation: running `taskset -c 4 ./cpp/build/release/app ...` confines the benchmark to one core for the whole run, trading the ability to migrate away from a noisy neighbor for the elimination of migration-induced stalls as a source @@ -495,7 +495,7 @@ explicitly: Same run, same 100,000 timed iterations, same single 56,595 ns stall visible in `max_ns`: one number treats that stall as a footnote, the other treats it -as the ~197x-worse tail a real fixed-rate caller would have actually felt. +as the ~197x-worse tail a real fixed-rate caller would have felt. Neither number is "wrong" in isolation; reporting only the left column while implying it describes what users experience is the mistake this chapter warns against. @@ -525,7 +525,7 @@ warns against. between samples, and pinning to `performance` before benchmarking removes that confound from the tail you're trying to measure. -Chapter 40 takes the discipline this chapter earned (real percentiles, a +The low-latency fastpath takes the discipline this chapter earned (real percentiles, a pinned clock, a tail you trust) and spends it: stripping a hot path down to pinned cores, zero allocation, and the low-latency techniques that make the corrected p99 small in the first place. diff --git a/_docs/40-low-latency-fastpath.md b/_docs/40-low-latency-fastpath.md index 16b87f9..d624a06 100644 --- a/_docs/40-low-latency-fastpath.md +++ b/_docs/40-low-latency-fastpath.md @@ -2,7 +2,7 @@ title: "The low-latency fast path: pin a core, reuse one buffer, and stop asking the scheduler for permission" order: 40 part: "Performance and Low Latency" -description: "chatterd-fastpath strips the ch21 echo hot path down to a pinned core, one reused 64-byte buffer, and an optional busy-poll spin, then proves each move at the kernel level: Cpus_allowed_list and voluntary_ctxt_switches read from /proc//status, not the program's own claim. The three languages reach identical behavior by three different shutdown mechanisms, and Go's CPU count lies about its container through a taskset mask rather than a cgroup." +description: "chatterd-fastpath strips chatterd's echo hot path down to a pinned core, one reused 64-byte buffer, and an optional busy-poll spin, then proves each move at the kernel level: Cpus_allowed_list and voluntary_ctxt_switches read from /proc//status, not the program's own claim. The three languages reach identical behavior by three different shutdown mechanisms, and Go's CPU count lies about its container through a taskset mask rather than a cgroup." duration: "50 minutes" --- @@ -20,17 +20,17 @@ By the end of this chapter you can: ## Why this matters -Chapter 39 earned the right to trust a `p99`. This chapter spends that trust on the next question: once you believe the number, how do you make it small? Pinning, zero allocation, and busy-poll each remove one specific, nameable source of tail latency, and each carries a real cost, most of all busy-poll's whole dedicated core, so the chapter's real lesson is picking the right trade for a latency-critical path rather than applying every trick everywhere. +Benchmarking without lies earned the right to trust a `p99`. This chapter spends that trust on the next question: once you believe the number, how do you make it small? Pinning, zero allocation, and busy-poll each remove one specific, nameable source of tail latency, and each carries a real cost, most of all busy-poll's whole dedicated core, so the chapter's real lesson is picking the right trade for a latency-critical path rather than applying every trick everywhere. The proof discipline matters as much as the techniques. `Cpus_allowed_list` and `voluntary_ctxt_switches` come from the kernel's own accounting in `/proc//status`, not from anything the server prints about itself, which is what turns "this server is pinned and never sleeps" from a claim into something you can check. -Chapter 39 taught you to trust a latency number: warm the code, time each +Benchmarking without lies covers how to trust a latency number: warm the code, time each iteration, correct for coordinated omission, and report the tail instead of the mean. This chapter spends that discipline. If you now believe a `p99` you measured, the next question is how to make it *small*, and the answer, on Linux, is a short list of moves that each remove one source of the multi-microsecond -stalls Chapter 39's `co_p99` was busy accounting for. `chatterd-fastpath` is the -same echo hot path from Chapter 21's `chatterd`, stripped to exactly those moves: +stalls `co_p99` was busy accounting for. `chatterd-fastpath` is the +same echo hot path from `chatterd`, stripped to exactly those moves: pin the process to one CPU, reuse a single stack buffer for every message so the loop never allocates, and, optionally, spin on a non-blocking `recv` instead of blocking, trading a whole core for the elimination of the sleep/wake round trip @@ -84,7 +84,7 @@ front: The rest of the program is ordinary by design: a fixed 64-byte wire frame (magic `CF`, a version and type byte, a big-endian `seq`, and zero padding), a `measure` subcommand that runs `N` synchronous ping-pong round trips against -either server and prints the same percentile table Chapter 39 built, and a +either server and prints the same coordinated-omission-corrected percentile table, and a `run-latency-bench.sh` driver that races the two under identical `taskset` placement so the *only* variable is the in-server discipline. @@ -173,7 +173,7 @@ the lab guest) and is blind to the caller's affinity mask. The obvious Go equivalent, `runtime.NumCPU()`, is *not* the same thing: it returns the size of the process's **CPU affinity mask**. Those two numbers agree on an idle machine and diverge the instant something narrows the mask, which is precisely what the -benchmark driver does. Phase B of `run-latency-bench.sh` launches each server +benchmark driver does. The second stage of `run-latency-bench.sh` launches each server under `taskset -c 1`, so the fastpath process starts life with a one-CPU mask, `runtime.NumCPU()` returns **1**, and the Go server's own range check rejects the `--pin 1` it was told to use: @@ -186,7 +186,7 @@ This is the same failure mode as `hardware_concurrency()` reporting a container' cgroup quota instead of the host's core count (the running theme of this part of the book) reached through an unexpected door. Here the count lies not because a cgroup capped it but because a `taskset` affinity mask narrowed it, and the fix -is the same in spirit: ask the kernel the question you actually mean. +is the same in spirit: ask the kernel the question you mean. `onlineCPUs()` in the Go port reads `/sys/devices/system/cpu/online` and counts the ranges there, matching the `sysconf` semantics the other two languages get automatically: @@ -208,7 +208,7 @@ goroutines over OS threads (Ms) and will migrate the one running your loop at will. `runtime.LockOSThread()` nails the hot loop's goroutine to one M and `GOMAXPROCS(1)` stops the runtime from parking worker threads on other cores; only then does pinning that thread mean anything. And the affinity call is made -*twice*: once with `0` (the calling thread, which is what actually restricts the +*twice*: once with `0` (the calling thread, which is what restricts the loop) and once with `os.Getpid()` (the thread-group leader, which is the mask `/proc//status` reports and the benchmark reads back). C++ and Rust are single-threaded here, so one `sched_setaffinity(0, …)` both pins the loop and @@ -295,7 +295,7 @@ There is nothing in the `EAGAIN` branch but a stop-flag check and `continue`. No `sched_yield`, no `nanosleep`, no `poll`: those would all, in different ways, hand the CPU back to the kernel, which is the exact cost busy-poll exists to avoid. The thread stays runnable and on its core, re-issuing `recv` as fast as the -CPU will let it, so the instant a frame lands the very next `recv` returns it with +CPU will let it, so the instant a frame lands the next `recv` returns it with no wakeup latency at all. The blocking path (`read_full_blocking`, not shown) is the ordinary alternative: a plain blocking `read` that the kernel parks on until data arrives. @@ -336,14 +336,14 @@ it straight back; the naive loop `make_unique`s (C++), `make([]byte, …)`s (Go) `vec![0u8; …]`s (Rust) a fresh buffer every iteration and lets it fall out of scope, the deliberate pessimization the benchmark is built to expose. -`measure` is the client half, and its percentile math is exactly Chapter 39's: +`measure` is the client half, and its percentile math is the same nearest-rank approach: sort the samples, then nearest-rank index with `idx = ceil(p/100 · n)` clamped to `[1, n]`. It builds each frame with `seq = i`, times the round trip between two `CLOCK_MONOTONIC` reads, and, a correctness check, not a timing one, verifies the echo came back with the same `seq` before recording the sample, so a cross-wired or corrupted connection fails loudly rather than reporting fast garbage. Warmup round trips (500 by default in the benchmark) are discarded before -recording starts, for the reasons Chapter 39 spent a section on. +recording starts, to avoid measuring first-call warmup costs rather than the steady-state path. ## Errors, three ways @@ -444,7 +444,7 @@ in flight to interrupt. So Go reaches identical behavior (`SIGINT`, then a clean `app: … shutting down` and exit 0) by a different route entirely: the `signal.Notify` goroutine sets the stop flag and then **closes the listener and the live connection**. Closing an fd is what wakes a poller-parked read; it is the -`shutdown(2)`-driven wakeup Chapter 21's `chatterd` used, and it is the idiomatic +same `shutdown(2)`-driven wakeup `chatterd` uses, and it is the idiomatic Go answer to "interrupt a blocked I/O wait." The busy-poll path needs one extra touch: its socket is a dup'd raw fd outside the poller, so `trigger` also calls `shutdown(SHUT_RDWR)` on it to break the spin. Three mechanisms (non-`SA_RESTART` @@ -458,10 +458,10 @@ that *fails* to shut down surfaces as a clear failure rather than a hung run. [host]$ cd examples/40-low-latency-fastpath && ./demo.sh build ``` -The `run-latency-bench.sh` driver does the real work. Phase A brings up a naive +The `run-latency-bench.sh` driver does the real work. It first brings up a naive and a fastpath server with no external pinning and reads each one's `Cpus_allowed_list` from `/proc//status`: this is the pinning proof, taken -from the kernel rather than from anything the program prints about itself. Phase B +from the kernel rather than from anything the program prints about itself. It then races the two under identical `taskset` placement (both servers pinned to the same CPU, the client to the other), running three interleaved trials of each so a lone scheduling stall on this shared nested-KVM guest can't decide the outcome: @@ -546,9 +546,9 @@ busy-poll buys and what it costs. network poller means no syscall is in flight to interrupt, so it closes the fd instead: same `SIGINT`, same exit 0, three different routes there. -Chapter 41 takes everything from this part and the last (the USE checklist, the +The fleet final project takes everything from this part and the last (the USE checklist, the LGTM pipeline, benchmarks you can trust, and this chapter's pinned, allocation-free -hot path) and assembles it into the final project: a supervised, sandboxed, fully +hot path) and assembles it into a supervised, sandboxed, fully observed two-VM fleet. --- diff --git a/_docs/41-capstone-fleet.md b/_docs/41-capstone-fleet.md index 5a5a55e..79d0f3b 100644 --- a/_docs/41-capstone-fleet.md +++ b/_docs/41-capstone-fleet.md @@ -2,7 +2,7 @@ title: "The final project fleet: pmon supervises chatterd, sysagent, and fwatch across two capability-dropped, Landlock-sandboxed, OTLP-observed hosts" order: 41 part: "Performance and Low Latency" -description: "41-capstone-fleet is one binary with four subcommands that assembles four of the book's six recurring programs into a single supervised system: pmon drops the capability bounding set and sets no_new_privs, then self-re-execs chatterd, sysagent, and fwatch as restartable children, forwarding SIGTERM/SIGINT and printing a pmon: health line every tick; chatterd (ch21-27) is bridged across two hosts so a message sent on one node is delivered on the other tagged from=NICK@node; sysagent (ch36) exports cpu%/mem%/load1 as OTel gauges over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; and fwatch (ch07-09, 33) watches its sandbox directory under an optional Landlock ruleset whose enforced ABI (7 on the lab guest's 6.19.10 kernel, 9 on the Fedora 44 host) is itself a live cross-check of the kernel-version probing discipline Chapter 33 established. Verified on the systems-target/systems-peer two-VM lab, with OTLP reaching the host's Grafana LGTM stack over the libvirt gateway." +description: "41-capstone-fleet is one binary with four subcommands that assembles four of the book's six recurring programs into a single supervised system: pmon drops the capability bounding set and sets no_new_privs, then self-re-execs chatterd, sysagent, and fwatch as restartable children, forwarding SIGTERM/SIGINT and printing a pmon: health line every tick; chatterd is bridged across two hosts so a message sent on one node is delivered on the other tagged from=NICK@node; sysagent exports cpu%/mem%/load1 as OTel gauges over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; and fwatch watches its sandbox directory under an optional Landlock ruleset whose enforced ABI (7 on the lab guest's 6.19.10 kernel, 9 on the Fedora 44 host) is itself a live cross-check of the kernel-version probing discipline established in the seccomp-and-landlock chapter. Verified on the systems-target/systems-peer two-VM lab, with OTLP reaching the host's Grafana LGTM stack over the libvirt gateway." duration: "60 minutes" --- @@ -24,15 +24,15 @@ This is the final project: four of the book's six recurring programs, assembled The chapter also closes the loop this book keeps returning to: prove the claim, don't just print it. A healthy-looking `pmon: health` line is not proof the fleet works; a message that crosses two real hosts and a gauge that lands in the host's own Prometheus through the libvirt gateway are. -Chapter 0's course map promised six programs that would grow chapter by +This book's course map promised six programs that would grow chapter by chapter into real systems software. By now four of them have grown enough to stand on their own: `pmon` learned to fork, exec, wait, drop privilege, and -walk namespaces (Chapters 11–14, 18–19, 32, 34); `chatterd` learned TCP, then +walk namespaces; `chatterd` learned TCP, then epoll at scale, then UDP peer discovery, then async, then a pinned-core fast -path (21–27, 34, 38, 40); `sysagent` learned to read `/proc` and -`/sys/fs/cgroup` directly and then to export what it read (36–38); and +path; `sysagent` learned to read `/proc` and +`/sys/fs/cgroup` directly and then to export what it read; and `fwatch` learned to watch a directory with polling, then inotify, then a -Landlock sandbox around itself (07–10, 33). This chapter does not teach a new +Landlock sandbox around itself. This chapter does not teach a new syscall. It assembles: one binary, four subcommands, where `pmon` is the fleet's own init: it drops the capability bounding set for itself and every descendant, then self-re-execs `chatterd`, `sysagent`, and `fwatch` as @@ -40,11 +40,11 @@ children it restarts on unexpected exit, forwards its own signals to, and reports on with a `pmon: health` line every tick. Two copies of that fleet run in this chapter: one on loopback, for -`verify.lua`, and one on the actual two-VM lab Chapter 2 built. +`verify.lua`, and one on the actual two-VM lab built earlier in the book. `chatterd` is bridged between `systems-target` and `systems-peer` so a message sent on one host is delivered on the other; each guest's `sysagent` exports gauges over OTLP, through the libvirt gateway, to the LGTM stack -Chapter 3 stood up on the host; and `fwatch` runs sandboxed under Landlock +already running on the host; and `fwatch` runs sandboxed under Landlock inside each fleet, its enforced ABI itself differing between the host kernel and the guest kernel: a real, verified number, not a claim. Nearly every part of this book from Setting Up through Performance and Low Latency shows @@ -52,7 +52,7 @@ up somewhere in the run below. {% include excalidraw.html file="41-fleet-topology" - alt="Two side-by-side VM boxes labeled systems-target (192.168.124.7) and systems-peer (192.168.124.95), each drawn as a large rounded rectangle representing one fleet. Inside each box, a top band labeled 'pmon: capability bounding set dropped, no_new_privs=1' encloses three child boxes side by side: chatterd, sysagent, and fwatch, each connected upward to a small pmon box with a supervise/restart arrow labeled 'fork+exec, restart on unexpected exit, forward SIGTERM'. The fwatch box additionally sits inside its own smaller nested box labeled 'Landlock ruleset: reads restricted to sandbox dir' with its enforced ABI number shown as a variable (7 on this guest's 6.19.10 kernel). A bidirectional arrow runs directly between the two chatterd boxes, labeled 'bridge: bridge@target / bridge@peer, DELIVER re-broadcast with from=NICK@node'. Below both VM boxes, a horizontal band represents the libvirt virbr0 bridge and its gateway address 192.168.124.1; two upward arrows from each guest's sysagent box pass through the gateway band labeled 'OTLP/HTTP :4318' and converge on a host-side box labeled 'LGTM stack (Grafana + Prometheus + Tempo), Chapter 3' sitting outside and above both VM boxes. A small caption near the gateway notes 'same route deploy-to-vm.sh forwards OTEL_EXPORTER_OTLP_ENDPOINT over.'" + alt="Two side-by-side VM boxes labeled systems-target (192.168.124.7) and systems-peer (192.168.124.95), each drawn as a large rounded rectangle representing one fleet. Inside each box, a top band labeled 'pmon: capability bounding set dropped, no_new_privs=1' encloses three child boxes side by side: chatterd, sysagent, and fwatch, each connected upward to a small pmon box with a supervise/restart arrow labeled 'fork+exec, restart on unexpected exit, forward SIGTERM'. The fwatch box additionally sits inside its own smaller nested box labeled 'Landlock ruleset: reads restricted to sandbox dir' with its enforced ABI number shown as a variable (7 on this guest's 6.19.10 kernel). A bidirectional arrow runs directly between the two chatterd boxes, labeled 'bridge: bridge@target / bridge@peer, DELIVER re-broadcast with from=NICK@node'. Below both VM boxes, a horizontal band represents the libvirt virbr0 bridge and its gateway address 192.168.124.1; two upward arrows from each guest's sysagent box pass through the gateway band labeled 'OTLP/HTTP :4318' and converge on a host-side box labeled 'LGTM stack (Grafana + Prometheus + Tempo)' sitting outside and above both VM boxes. A small caption near the gateway notes 'same route deploy-to-vm.sh forwards OTEL_EXPORTER_OTLP_ENDPOINT over.'" caption="Figure 41.1: the two-VM fleet: pmon supervises chatterd/sysagent/fwatch on each of systems-target and systems-peer inside a capability-bounding-set-drop boundary, fwatch additionally sandboxed under Landlock, chatterd bridged directly between the two guests, and both guests' sysagent exporting OTLP over the libvirt gateway to the host's LGTM stack" %} > **Tools used**: `ssh`/`scp` (host, to stage the built binary onto both @@ -63,10 +63,10 @@ up somewhere in the run below. > route guest-side OTLP traffic takes to reach the host's LGTM stack: a > network path, not a tool. Capability-bounding-set drop and Landlock are > syscalls the program makes itself (`prctl(2)`, `landlock_create_ruleset(2)` -> and friends); seccomp was already proven alongside Landlock in Chapter 33 -> and is not re-derived here. `ssh`/`scp`/`curl` ship in Fedora's base -> install and aren't separately checked by `scripts/check-host.sh`, the same -> way Chapter 36 notes for `curl` itself. +> and friends); seccomp was already proven alongside Landlock earlier in the +> book and is not re-derived here. `ssh`/`scp`/`curl` ship in Fedora's base +> install and aren't separately checked by `scripts/check-host.sh`, consistent +> with how `curl` is treated elsewhere in the book. ## One binary, four programs: pmon as fleet init @@ -86,7 +86,7 @@ fwatch watch DIR [--sandbox] [--timeout-ms T] `pmon` never execs a *different* binary for chatterd, sysagent, or fwatch: it re-execs **itself**, with a different `argv[0]` command word, the same -self-re-exec technique Chapter 34's container entrypoint used. Concretely: +self-re-exec technique a container entrypoint uses elsewhere in this book. Concretely: `mkdir` the sandbox directory, drop the capability bounding set once, resolve the running binary's own path (`os.Executable()` in Go, `readlink("/proc/self/exe")` in C++ and Rust), then fork+exec that same path @@ -345,7 +345,7 @@ supervisor's job is to keep trying, not to give up on the first bad attempt. ## chatterd, bridged across two hosts -`chatterd` (Chapters 21–27) speaks the same wire frame it always has: +`chatterd` speaks the same wire frame it always has: magic `"CH"`, version 1, a type byte, a big-endian `u16` length, then payload (`proto.go`/`.hpp`/`.rs`, unchanged), and this chapter adds exactly one new idea on top: with `--peer HOST:PORT` set on both sides, a message sent on @@ -526,15 +526,15 @@ back over the connection it opened, sees `dave` has no `@` in it yet, tags it `dave@target`, and re-broadcasts locally on `peer` with `includeBridges=false`. A listener on `peer` sees `chatterd: received from=dave@target text=…`: the `@target` suffix is the -proof the message actually crossed the wire, not that it was echoed +proof the message crossed the wire, not that it was echoed somewhere local. The reconnect loop (dial, backoff, retry) means either side can start first, or the other side can restart under `pmon`, without the bridge ever needing a fixed startup order. ## sysagent and fwatch: what's observed, what's sandboxed -`sysagent` under the fleet is the same `/proc`-reading collector Chapter 36 -built, trimmed to the three signals a fleet operator watches continuously: +`sysagent` under the fleet is the same `/proc`-reading collector built +earlier in the book, trimmed to the three signals a fleet operator watches continuously: `cpu_pct` (a `/proc/stat` tick delta), `mem_pct` (`/proc/meminfo`), and `load1` (`/proc/loadavg`), printed every `--interval-ms` as `sysagent: node=… cpu_pct=… mem_pct=… load1=… ts=…`. New in this chapter: @@ -551,22 +551,22 @@ nothing but arithmetic, or holds a touched, resident memory allocation, so a concurrently-running `sysagent`'s own `cpu_pct`/`mem_pct` visibly moves: the USE-method callback this fleet can point at itself. -`fwatch` is Chapter 07–10's file watcher (`inotify_init1`, one +`fwatch` is the file watcher introduced earlier in the book (`inotify_init1`, one non-recursive watch on a directory, `create`/`modify`/`delete` lines) with -Chapter 33's Landlock ruleset available as `--sandbox`. The ruleset +a Landlock ruleset available as `--sandbox`. The ruleset restricts filesystem *reads* to the watched directory: `pmon` passes fwatch `watch SANDBOX_DIR --sandbox`, and before the inotify loop starts, `fwatch` probes the running kernel's Landlock ABI with `landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, builds a ruleset scoped to that one directory, and calls `landlock_restrict_self` -after setting `no_new_privs`, the identical sequence Chapter 33 proved, +after setting `no_new_privs`, the identical sequence proved earlier in the book, reused rather than re-derived. `fwatch: landlock ABI=N enforced dir=…` is -printed only once the kernel has actually accepted the restriction, and, as +printed only once the kernel has accepted the restriction, and, as the cross-check below shows, `N` is not a constant baked into the program; it is whatever the kernel underneath happens to support. Capability-bounding- set drop (applied once, by `pmon`, before any child exists) and Landlock (applied per-process, by `fwatch` itself) are this final project's two sandbox -layers; seccomp, Chapter 33's third layer, was proven there and is not +layers; seccomp, this book's third sandboxing layer, was proven earlier and is not reapplied here. ## How the code works @@ -629,7 +629,7 @@ restart loop shown above, all three joined with a `sync.WaitGroup`; the main goroutine waits on a `select` between the `WaitGroup`'s completion (via a `done` channel) and a health-interval `time.Ticker`, printing `pmon: health …` each time the ticker fires and returning once every child -has actually stopped. +has stopped. C++ and Rust reach for the same alternative to a signal *handler*: a dedicated **`sigwait`-consuming thread**. Both block `SIGTERM` and `SIGINT` @@ -648,9 +648,9 @@ child clears its own inherited copy of the blocked signal mask thread_set_mask()` in Rust) immediately after `fork()` and before `execv()`: without this, the child would inherit a *blocked* `SIGTERM`/`SIGINT` and its own shutdown-signal handling (the same `install_signal_flag` -mechanism `chatterd`/`sysagent`/`fwatch` use standalone) could never fire, -the exact bug Chapter 34's container entrypoint first ran into and -documented. Where C++ and Rust part ways is the health-tick wait: C++'s main +mechanism `chatterd`/`sysagent`/`fwatch` use standalone) could never fire: +exactly the failure a container entrypoint elsewhere in this book first ran +into. Where C++ and Rust part ways is the health-tick wait: C++'s main thread holds a `std::condition_variable` and calls `wait_for(lock, interval, predicate)`, waking early the moment every worker thread's `active` counter reaches zero and otherwise printing health once per timeout; Rust's main @@ -665,7 +665,7 @@ health interval) for not needing a shared mutex/condvar pair at all. [host]$ cd examples/41-capstone-fleet && ./demo.sh build ``` -`verify.lua` follows the convention Chapter 23 established for `vm-peer` +`verify.lua` follows the convention established elsewhere in the book for `vm-peer` examples: the assertion itself runs entirely on loopback (the fleet, the cross-node bridge across two `127.0.0.1` ports, and telemetry to a local LGTM stack) while the real two-host run is the chapter's demo. A loopback @@ -748,15 +748,15 @@ comes from a line the program prints about its own intentions. The first is Landlock's enforced ABI. `fwatch --sandbox` never hardcodes a version: it asks the running kernel with `landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, the -exact probe Chapter 33 established, and prints whatever comes back. On the +exact probe established earlier in the book, and prints whatever comes back. On the Fedora 44 reference host (kernel `7.1`), where this chapter's loopback `verify.lua` runs, that probe returns **ABI 9**. On the `systems-target` lab guest (kernel `6.19.10-300.fc44`) this session, the identical code, the identical syscall, returned **ABI 7**. That is not a bug and not a discrepancy to reconcile; it is the accurate, verified consequence of two different kernel versions supporting two different sets -of Landlock features, and it is the same lesson Chapter 33 taught about -attach targets and struct offsets, reached through a sandbox ABI instead: a +of Landlock features, and it is the same lesson about +attach targets and struct offsets taught earlier in the book, reached through a sandbox ABI instead: a program that hardcodes "ABI 9" would be *wrong* on the guest, and `fwatch`'s own probe is what keeps it right on both. @@ -768,7 +768,7 @@ Grafana's Prometheus proxy) was present with value **1.75**, a gauge that started life inside an unprivileged process on `systems-peer`, was exported over OTLP/HTTP to `http://192.168.124.1:4318`, crossed the libvirt bridge's gateway, and landed in the host's own Prometheus, labeled by the -node that produced it. That is the guest→host telemetry path Chapter 3's +node that produced it. That is the guest→host telemetry path the LGTM stack exists to receive, closed end to end. ## What you learned @@ -787,11 +787,11 @@ LGTM stack exists to receive, closed end to end. bridging needed no protocol change: `bridge@` is an ordinary connection in the peer's client registry, `includeBridges=false` on re-broadcast is what stops a two-node ping-pong, and the `@node` suffix on - a delivered nick is the observable proof a message actually crossed. + a delivered nick is the observable proof a message crossed. - **A sandbox's enforced feature set is a kernel fact, not a program's assumption.** The identical Landlock probe returned ABI 9 on the Fedora 44 host and ABI 7 on the lab guest's 6.19.10 kernel this session, the - same class of version drift Chapter 33 taught, reached through a + same class of version drift taught earlier in the book, reached through a capability-negotiation syscall instead of an attach point. - **Telemetry is additive, and the guest→host OTLP path is real, not assumed.** Every service ran identically with `OTEL_EXPORTER_OTLP_ENDPOINT` @@ -807,7 +807,7 @@ LGTM stack exists to receive, closed end to end. polls `is_finished()` on a sleep: the same restart/health/shutdown contract, four distinctly different mechanisms underneath. -This closes the six-program arc that began in Chapter 7: four of the six +This closes the six-program arc that began early in the book: four of the six (`pmon`, `chatterd`, `sysagent`, `fwatch`) now run together, supervised, sandboxed, and observed, across two real hosts. The book continues into three deep dives (embedding Lua, Rust macros for systems code, the Go @@ -823,8 +823,8 @@ reference host (kernel 7.1) and the two lab guests python3 scripts/test-all-examples.py --only 41-capstone-fleet --mode vm printed PASS PASS PASS (3 passed, 0 failed; verify.lua reported PASS 31 / FAIL 0 for each of -cpp/go/rust, exercised entirely on loopback per the Chapter 23 vm-peer -convention). On the real two-host run, pmon on both +cpp/go/rust, exercised entirely on loopback per the vm-peer +convention established elsewhere in the book). On the real two-host run, pmon on both systems-target and systems-peer reached pmon: health chatterd=up sysagent=up fwatch=up; a message sent on target (nick dave, text "hello across hosts") @@ -837,8 +837,8 @@ difference; and with OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1: 4318 set on both guests, sysagent_cpu_pct{node="peer"} reached the host's Prometheus (queried through Grafana's proxy) with value 1.75, confirming the guest→host OTLP path over the libvirt -gateway. Not exercised: seccomp filtering of any fleet child (Chapter 33 -proved seccomp alongside Landlock; this final project reuses only the +gateway. Not exercised: seccomp filtering of any fleet child (seccomp was +proven alongside Landlock earlier in the book; this final project reuses only the capability-bounding-set-drop and Landlock layers, not seccomp) and a run where the LGTM stack was unreachable (that path reports SKIP for the telemetry assertions, not a failing example, per the manifest's diff --git a/_docs/42-embedded-lua-scripting.md b/_docs/42-embedded-lua-scripting.md index bddade0..c5c97e8 100644 --- a/_docs/42-embedded-lua-scripting.md +++ b/_docs/42-embedded-lua-scripting.md @@ -2,7 +2,7 @@ title: "Embedding Lua as a policy engine: one policy.lua, three host runtimes" order: 42 part: "Deep Dives" -description: "lua-policy moves pmon's restart/backoff decision out of the compiled binary and into a single sandboxed policy.lua, embedded three structurally different ways -- C++23 via sol2, Go via gopher-lua (Lua 5.1), Rust via mlua (Lua 5.4) -- and reloaded live on a real SIGHUP with no process restart. Verified on the Fedora 44 host: byte-identical decisions across all three languages (diff-confirmed), PASS 19/FAIL 0 per language, and three distinct sandbox-rejection diagnostics for the same malicious policy." +description: "lua-policy moves pmon's restart/backoff decision out of the compiled binary and into a single sandboxed policy.lua, embedded three structurally different ways (C++23 via sol2, Go via gopher-lua (Lua 5.1), Rust via mlua (Lua 5.4)) and reloaded live on a real SIGHUP with no process restart. Verified on the Fedora 44 host: byte-identical decisions across all three languages (diff-confirmed), PASS 19/FAIL 0 per language, and three distinct sandbox-rejection diagnostics for the same malicious policy." duration: "45 minutes" --- @@ -20,13 +20,13 @@ By the end of this chapter you can: ## Why this matters -A restart policy baked into compiled C++, Go, or Rust means a tuning change costs a rebuild and a redeploy. Moving that decision into one sandboxed `policy.lua` instead means `pmon` can change its crash-looping threshold or its backoff curve on a live `SIGHUP`, with the same process and the same PID throughout, which is the capability an operator actually wants during an incident. +A restart policy baked into compiled C++, Go, or Rust means a tuning change costs a rebuild and a redeploy. Moving that decision into one sandboxed `policy.lua` instead means `pmon` can change its crash-looping threshold or its backoff curve on a live `SIGHUP`, with the same process and the same PID throughout, which is the capability an operator wants during an incident. -The harder question this chapter answers is whether "the same policy" really means the same behavior once it is embedded three structurally different ways. Proving sol2, gopher-lua, and mlua reach byte-identical restart decisions from one unmodified file, across a Lua 5.1/5.4 version gap, is what makes "one policy.lua, three host runtimes" more than a slogan. +The harder question this chapter answers is whether "the same policy" means the same behavior once it is embedded three structurally different ways. Proving sol2, gopher-lua, and mlua reach byte-identical restart decisions from one unmodified file, across a Lua 5.1/5.4 version gap, is what makes "one policy.lua, three host runtimes" more than a slogan. -Chapter 12 taught `pmon` to treat a signal as data instead of an +Earlier in the book, `pmon` learned to treat a signal as data instead of an interruption: block `SIGCHLD` process-wide, then read it off a `signalfd` on -the next turn of an ordinary `poll` loop. Chapter 41's final-project fleet leaned +the next turn of an ordinary `poll` loop. The fleet final project leaned on the same discipline for `SIGTERM`/`SIGINT`. This chapter reaches for a different signal (`SIGHUP`, "please re-read your configuration") to reload something `pmon` has never had before: a restart policy that is not @@ -45,7 +45,7 @@ decisions from it, live reload included. {% include excalidraw.html file="42-host-script-boundary" - alt="Three stacked host-runtime boxes on the left -- C++23/sol2 against PUC-Rio Lua 5.4, Go/gopher-lua against Lua 5.1, Rust/mlua against a vendored PUC-Rio Lua 5.4 -- each with a bidirectional arrow into a center box labeled policy.lua, one file, marked on_exit(info) going in and decision{action,delay_ms,reason} coming back. The policy.lua box sits inside a shaded sandbox-boundary band with a ghost box reading blocked: os, io, package, debug (never opened); nilled: load/loadfile/dofile/require/collectgarbage/loadstring. A host table box on the right connects to policy.lua with a bidirectional arrow labeled host.log / host.now_ms. A caption note reads Lua 5.1 (gopher-lua) vs Lua 5.4 (sol2, mlua) forces the shared subset -- one policy.lua drives all three." + alt="Three stacked host-runtime boxes on the left: C++23/sol2 against PUC-Rio Lua 5.4, Go/gopher-lua against Lua 5.1, Rust/mlua against a vendored PUC-Rio Lua 5.4, each with a bidirectional arrow into a center box labeled policy.lua, one file, marked on_exit(info) going in and decision{action,delay_ms,reason} coming back. The policy.lua box sits inside a shaded sandbox-boundary band with a ghost box reading blocked: os, io, package, debug (never opened); nilled: load/loadfile/dofile/require/collectgarbage/loadstring. A host table box on the right connects to policy.lua with a bidirectional arrow labeled host.log / host.now_ms. A caption note reads Lua 5.1 (gopher-lua) vs Lua 5.4 (sol2, mlua) forces the shared subset: one policy.lua drives all three." caption="Figure 42.1: the host↔script boundary: three differently-built sandboxed Lua states, one shared policy.lua, and the host table it calls back into" %} > **Tools used**: `lua` (host, drives `verify.lua` and is the PUC-Rio 5.4 @@ -498,7 +498,7 @@ error and keep serving whatever policy was already loaded; a ## Concurrency lens Every host consumes the *same* `SIGHUP` through three different primitives: -the same divergence Chapter 41 mapped for `SIGTERM`, reached here through a +the same divergence the fleet final project mapped for `SIGTERM`, reached here through a reload path with one new wrinkle: after the signal lands, **every** binding discards the whole interpreter and builds a brand-new one. @@ -574,7 +574,7 @@ void wait_for_sighup(int sigfd) { .context("reloading policy")?; ``` -C++ reuses ch12/ch13's pattern outright: `SIGHUP` is blocked process-wide +C++ reuses the pattern established earlier in the book outright: `SIGHUP` is blocked process-wide with `sigprocmask` up front (in `make_sighup_fd`, not shown), then consumed as ordinary readability on a `signalfd` inside a `poll` loop; no signal handler runs at all. Go reaches for the idiomatic equivalent: `signal.Notify` @@ -661,12 +661,12 @@ Each language's `verify.lua` reported `PASS 19 / FAIL 0`. {% include excalidraw.html file="42-sighup-reload-lifecycle" - alt="A horizontal sequence of six boxes inside a band labeled one pmon process -- PID never changes: load v1 (base=500ms max_restarts=8), phase-1 decisions (web: stop; worker: restart 500ms to 1000ms), awaiting SIGHUP (a dashed ghost box), reload (fresh Lua state rebuilt, on_load logs v1.1.0), phase-2 decisions (api: restart 100ms; worker: escalate after 2 restarts), and exit 0. Above the awaiting-SIGHUP box, a ghost box shows the operator action -- edit the policy file, then kill -HUP $(pgrep app) -- with an arrow into the flow. Below it, three sub-boxes show the three per-language consumption mechanisms: C++ signalfd + poll, Go signal.Notify(ch), Rust signal_hook flag (AtomicBool + 20ms poll). A bottom note reads: the process is NOT restarted -- only a fresh, sandboxed Lua state is built in place and the decision logic changes underneath it." + alt="A horizontal sequence of six boxes inside a band labeled one pmon process: PID never changes: load v1 (base=500ms max_restarts=8), phase-1 decisions (web: stop; worker: restart 500ms to 1000ms), awaiting SIGHUP (a dashed ghost box), reload (fresh Lua state rebuilt, on_load logs v1.1.0), phase-2 decisions (api: restart 100ms; worker: escalate after 2 restarts), and exit 0. Above the awaiting-SIGHUP box, a ghost box shows the operator action: edit the policy file, then kill -HUP $(pgrep app), with an arrow into the flow. Below it, three sub-boxes show the three per-language consumption mechanisms: C++ signalfd + poll, Go signal.Notify(ch), Rust signal_hook flag (AtomicBool + 20ms poll). A bottom note reads: the process is not restarted; only a fresh, sandboxed Lua state is built in place and the decision logic changes underneath it." caption="Figure 42.2: the SIGHUP reload lifecycle: v1's decisions, a real SIGHUP consumed three different ways, a fresh Lua state built in place, and v1.1.0's tighter decisions, same process throughout" %} ## Cross-check: one script, three interpreters, byte-identical decisions -The claim this chapter makes is falsifiable and was actually falsified +The claim this chapter makes is falsifiable and was tested against: `policy.lua`, unmodified, drives sol2/PUC-Lua 5.4, gopher-lua's pure-Go 5.1 VM, and mlua/PUC-Lua 5.4 to the *exact same sequence of restart decisions*, including the exact backoff numbers and the exact `reason` @@ -708,8 +708,8 @@ not three green exit codes agreeing by coincidence. `pmon`'s PID and child-tracking state survive untouched; only the Lua state is thrown away and rebuilt from `policy-v2.lua`, so no closure or global from the old policy leaks into the new one. -- **Three signals, one outcome, three primitives.** C++ reuses ch12's - `sigprocmask` + `signalfd` + `poll`; Go hands the signal to the runtime's +- **Three signals, one outcome, three primitives.** C++ reuses the + `sigprocmask` + `signalfd` + `poll` pattern established earlier in the book; Go hands the signal to the runtime's own machinery via `signal.Notify` and blocks on a channel; Rust's `signal-hook` flips an `AtomicBool` from a minimal `sigaction` handler a separate thread polls every 20 ms, armed before the "awaiting" line prints diff --git a/_docs/43-rust-macros-for-systems.md b/_docs/43-rust-macros-for-systems.md index eb8d580..dcb1478 100644 --- a/_docs/43-rust-macros-for-systems.md +++ b/_docs/43-rust-macros-for-systems.md @@ -2,7 +2,7 @@ title: "Rust macros for systems code: proc-macros, syn/quote, and what miri proves about unsafe" order: 43 part: "Deep Dives" -description: "sysmacros is chapter 43's proc-macro crate, powering two Rust-only techniques in the app binary -- #[derive(SysError)], a miniature thiserror built with syn 2.0's AST and quote's token emission, and #[instrument], a compile-time tracing attribute that times a function with Instant -- plus a sound unsafe unaligned read that cargo +nightly miri validates clean and, under the --oob-read path, catches as undefined behaviour. The book's first single-language deep dive, verified on the Fedora 44 host with verify.lua PASS 8/FAIL 0, the exact six-line stable stdout, and miri's own out-of-bounds diagnostic reproduced byte for byte." +description: "sysmacros is this chapter's proc-macro crate, powering two Rust-only techniques in the app binary (#[derive(SysError)], a miniature thiserror built with syn 2.0's AST and quote's token emission, and #[instrument], a compile-time tracing attribute that times a function with Instant) plus a sound unsafe unaligned read that cargo +nightly miri validates clean and, under the --oob-read path, catches as undefined behaviour. The book's first single-language deep dive, verified on the Fedora 44 host with verify.lua PASS 8/FAIL 0, the exact six-line stable stdout, and miri's own out-of-bounds diagnostic reproduced byte for byte." duration: "35 minutes" --- @@ -31,8 +31,9 @@ equally. This one breaks that pattern by design. Proc-macros are a Rust compiler feature with no C++ or Go equivalent worth building a codetab around, and `miri`, the interpreter that catches undefined behaviour a normal build would run right past, only exists because Rust's `unsafe` -carries obligations the type system stops checking. Chapter 12 gave -`pmon` a `PmonError`-shaped hierarchy by hand, written out field by field; +carries obligations the type system stops checking. Signal-safe error +handling earlier gave `pmon` a `PmonError`-shaped hierarchy by hand, written +out field by field; this chapter asks how much of that boilerplate a macro can write for you, and how you prove the `unsafe` block sitting a few lines below it is still doing exactly what its comment claims. @@ -50,7 +51,7 @@ skips the check, which is exactly the distinction `miri` exists to draw. {% include excalidraw.html file="43-proc-macro-pipeline" - alt="Two horizontal bands. Top band: the app crate, main.rs's annotated source (#[derive(SysError)] enum PmonError, #[instrument] fn hash_chunk) feeding rustc, which packages a TokenStream and hands it off; on the right, rustc splices tokens back into the AST, typechecks, and compiles the result into the app binary (generated Display + Error impls, the instrumented hash_chunk). Bottom band: sysmacros, a separate proc-macro crate compiled FIRST, containing two nodes -- syn::parse_macro_input! turning the TokenStream into a real DeriveInput/ItemFn AST, and quote! emitting a new TokenStream2 (the Display impl or the wrapped function). Arrows cross the band boundary labeled TokenStream (macro input) going down and TokenStream (macro output) coming back up. A note at the bottom reads: sysmacros never runs at app runtime -- it is pure tokens-in/tokens-out, executed only while rustc is compiling app." + alt="Two horizontal bands. Top band: the app crate, main.rs's annotated source (#[derive(SysError)] enum PmonError, #[instrument] fn hash_chunk) feeding rustc, which packages a TokenStream and hands it off; on the right, rustc splices tokens back into the AST, typechecks, and compiles the result into the app binary (generated Display + Error impls, the instrumented hash_chunk). Bottom band: sysmacros, a separate proc-macro crate compiled first, containing two nodes: syn::parse_macro_input! turning the TokenStream into a real DeriveInput/ItemFn AST, and quote! emitting a new TokenStream2 (the Display impl or the wrapped function). Arrows cross the band boundary labeled TokenStream (macro input) going down and TokenStream (macro output) coming back up. A note at the bottom reads: sysmacros never runs at app runtime; it is pure tokens-in/tokens-out, executed only while rustc is compiling app." caption="Figure 43.1: the compile-time-only pipeline: annotated source becomes a TokenStream, the separately-compiled sysmacros crate parses it with syn and re-emits new tokens with quote, and rustc splices the result into app" %} > **Tools used**: `cargo` (host, build/run the pinned stable toolchain, @@ -75,7 +76,7 @@ execute the finished `app` binary; every line of it runs once, at build time, and produces tokens that rustc splices into the program in its place. -The macro `PmonError` actually uses is the plainer of the two ideas: +The macro `PmonError` uses is the plainer of the two ideas: each variant just needs a message: ```rust @@ -126,11 +127,11 @@ fn read_record(buf: &[u8], oob: bool) -> Record { An `unsafe` block is a promise to the compiler: "I checked something the type system can't, and this operation is valid anyway." `read_unaligned` needs three things to be true: the pointer is readable for -`size_of::()` bytes, those bytes form a valid `Record` (satisfied -trivially here, since `Record` is `repr(C)` and made only of integers), +`size_of::()` bytes, those bytes form a valid `Record` (true here +because `Record` is `repr(C)` and made only of integers), and the read doesn't race another write. The `assert!` right above the -`unsafe` block is what makes the sound path's promise true: it checks the -buffer really does have 8 bytes before offset 0 is read from. `--oob-read` +`unsafe` block is what makes the sound path's promise true: it checks that +the buffer has 8 bytes before offset 0 is read from. `--oob-read` sets `off = 4` and, by design, skips that `assert!`; the read now reaches for offset `4..12` against an 8-byte `buf`, and nothing in a normal `cargo build`/`cargo run` will tell you that happened. That silence is the @@ -140,7 +141,7 @@ the instant a read or write steps outside them. {% include excalidraw.html file="43-miri-bounds-check" - alt="An 8-byte buffer BUF: [u8; 8] drawn as eight numbered byte boxes 0 through 7, labeled above as magic: u32 (bytes 0..4) and slots: u32 (bytes 4..8), followed by four dashed ghost boxes numbered 8 through 11 marked past the end (never allocated). Below the real bytes, a blue SOUND window box spans bytes 0 through 7, labeled: read_record(buf, oob=false), read_unaligned::(ptr.add(0)), window [0..8) fully inside the 8-byte alloc, cargo +nightly miri run: exit 0, no UB. Below and offset right, an amber window box spans bytes 4 through 11 (crossing from the real buffer into the ghost bytes), labeled: --oob-read: read_record(buf, oob=true), read_unaligned::(ptr.add(4)), window [4..12) -- 4 bytes past the 8-byte alloc, miri: only 4 bytes from the end of the allocation (main.rs:57). Dashed amber boundary markers point from the OOB window up into the byte row at the offset-4 and offset-12 boundaries." + alt="An 8-byte buffer BUF: [u8; 8] drawn as eight numbered byte boxes 0 through 7, labeled above as magic: u32 (bytes 0..4) and slots: u32 (bytes 4..8), followed by four dashed ghost boxes numbered 8 through 11 marked past the end (never allocated). Below the real bytes, a blue SOUND window box spans bytes 0 through 7, labeled: read_record(buf, oob=false), read_unaligned::(ptr.add(0)), window [0..8) fully inside the 8-byte alloc, cargo +nightly miri run: exit 0, no UB. Below and offset right, an amber window box spans bytes 4 through 11 (crossing from the real buffer into the ghost bytes), labeled: --oob-read: read_record(buf, oob=true), read_unaligned::(ptr.add(4)), window [4..12), 4 bytes past the 8-byte alloc, miri: only 4 bytes from the end of the allocation (main.rs:57). Dashed amber boundary markers point from the OOB window up into the byte row at the offset-4 and offset-12 boundaries." caption="Figure 43.2: the same 8-byte Record read from two offsets: the SOUND window [0..8) miri accepts, and the --oob-read window [4..12) that runs 4 bytes past the allocation, exactly what miri's diagnostic names" %} ## How the code works @@ -248,7 +249,7 @@ capture, so the arm binds synthetic identifiers (`f0`, `f1`, ...) and passes them to `write!` explicitly as `#(#idents),*` after the message. The whole function returns one `quote!` block containing both the `Display` impl and the `Error` impl, and that block (not `PmonError`'s -own source) is what rustc actually type-checks and compiles into `app`. +own source) is what rustc type-checks and compiles into `app`. `expand_instrument` takes the same shape and applies it to a function instead of an enum: @@ -328,7 +329,7 @@ itself; the caller never sees a difference except the printed lines. Every earlier chapter's "errors, three ways" contrasted three languages; this one has only Rust, so the three ways are the three moments an error -in this program can actually surface. **Compile time**: a malformed macro +in this program can surface. **Compile time**: a malformed macro input (`#[derive(SysError)]` on a struct, or a variant missing `#[error("...")]`) becomes a `syn::Error`, and `derive_sys_error` turns that into real tokens with `err.to_compile_error().into()`, so a bad @@ -433,7 +434,7 @@ in its interpreted heap; `+0x4` is the offset the failing read started at (`off = 4` on the `--oob-read` path); "only 4 bytes from the end of the allocation" is `miri` doing arithmetic a native run never does: `BUF` is 8 bytes, the read needs 8 more bytes starting at offset 4, and offset 4 -plus 8 is 12, four bytes past where the allocation actually ends. The +plus 8 is 12, four bytes past where the allocation ends. The line and column (`src/main.rs:57:14`) point at the exact `unsafe` block quoted above. On stable Rust this same `--oob-read` invocation runs to completion, silently reading whatever garbage bytes happen to sit past @@ -463,8 +464,8 @@ the exact three lines `PmonError`'s variants must produce `policy rejected: sandbox blocked os.execute`), plus the `#[instrument]` entry/exit shape and the parsed `Record` fields, for `PASS 8 / FAIL 0`: eight assertions against behaviour, not "the binary exited 0." The second -is that the sound `read_record` path really is sound and the -`--oob-read` path really is a bug, and that claim is checked by an +is that the sound `read_record` path is sound and the +`--oob-read` path is a bug, and that claim is checked by an interpreter rather than taken on faith: the identical scenario run under `cargo +nightly miri run` exits 0 with no UB reported on the default path, and reports the precise out-of-bounds access (4 bytes past an 8-byte diff --git a/_docs/44-go-runtime-for-systems.md b/_docs/44-go-runtime-for-systems.md index 2f349f0..76078ef 100644 --- a/_docs/44-go-runtime-for-systems.md +++ b/_docs/44-go-runtime-for-systems.md @@ -2,7 +2,7 @@ title: "The Go runtime from a systems programmer's seat: the scheduler, the collector, and the netpoller" order: 44 part: "Deep Dives" -description: "go-runtime-for-systems is a single Go binary that observes its own runtime across four themes -- the GMP scheduler and signal-based async preemption, GC pacing under GOGC/GOMEMLIMIT with Go 1.26's Green Tea collector on by default, the epoll-backed netpoller that turns 2,000 blocked goroutines into roughly a dozen OS threads, and the effective knobs a process is actually running under -- each theme printing a self-computed ok/FAIL verdict. Verified on the Fedora 44 host on go1.26.5 with verify.lua PASS 16/FAIL 0, and the same preemption check flipped to FAIL on demand by GODEBUG=asyncpreemptoff=1." +description: "go-runtime-for-systems is a single Go binary that observes its own runtime across four themes (the GMP scheduler and signal-based async preemption, GC pacing under GOGC/GOMEMLIMIT with Go 1.26's Green Tea collector on by default, the epoll-backed netpoller that turns 2,000 blocked goroutines into roughly a dozen OS threads, and the effective knobs a process is actually running under), each theme printing a self-computed ok/FAIL verdict. Verified on the Fedora 44 host on go1.26.5 with verify.lua PASS 16/FAIL 0, and the same preemption check flipped to FAIL on demand by GODEBUG=asyncpreemptoff=1." duration: "45 minutes" --- @@ -24,24 +24,24 @@ Every earlier Go chapter in this book leaned on the runtime without looking at i The netpoller theme is the clearest payoff: 2,000 goroutines blocked on a read and a thread count that stays in the double digits is the concrete, measured reason Go programs can hold far more concurrent work in flight than their OS thread count would suggest, and the chapter proves it against the kernel's own `/proc/self/status` accounting, not the runtime's self-report. -Chapter 43 spent its whole budget on Rust's compiler doing extra work at -build time -- a proc-macro crate that runs once, produces tokens, and gets -out of the way. This chapter is the mirror image: a runtime that never gets -out of the way, because it is doing something every second the program -runs. Every earlier chapter that used Go leaned on its runtime without -looking at it -- goroutines that seemed to run in parallel, garbage that -seemed to collect itself, sockets that seemed to block without blocking -anything. `go-runtime-for-systems` is the chapter where that runtime stops -being invisible: a single stdlib-only binary that asks its own scheduler, -collector, and netpoller direct questions and prints the answers as -computed verdicts -- `ok` or `FAIL`, never just an exit code. It's also the +The previous chapter on Rust macros spent its whole budget on Rust's +compiler doing extra work at build time: a proc-macro crate that runs once, +produces tokens, and gets out of the way. This chapter is the mirror image: a +runtime that never gets out of the way, because it is doing something every +second the program runs. Every earlier chapter that used Go leaned on its +runtime without looking at it: goroutines that seemed to run in parallel, +garbage that seemed to collect itself, sockets that seemed to block without +blocking anything. `go-runtime-for-systems` is the chapter where that runtime +stops being invisible: a single stdlib-only binary that asks its own +scheduler, collector, and netpoller direct questions and prints the answers +as computed verdicts: `ok` or `FAIL`, never just an exit code. It's also the last chapter of Part 12: the Rust-only dive into macros and `miri`, and this Go-only look under the runtime's hood, are the two language-specific excursions the tri-language chapters made room for. {% include excalidraw.html file="44-gmp-scheduler" - alt="A global run queue band at top holds overflow G's. Below, two P columns (P0.M0 running a spinner G, P1.M1 running a ticker G) each show a local run queue (256 slots + runnext), plus a box for P2 through P15 on this 16-CPU host. A dashed amber steal half arrow runs P1 into P0. To the right, a dashed sysmon box (no P, its own M) lists retake, forcegc, netpoll backstop, and the Go 1.25+ GOMAXPROCS recheck, with a solid amber SIGURG to asyncPreempt arrow into P0's running M. Caption: M:N -- many G onto few M via P; GOMAXPROCS=1 still preempts since sysmon's signal needs no spare P." + alt="A global run queue band at top holds overflow G's. Below, two P columns (P0.M0 running a spinner G, P1.M1 running a ticker G) each show a local run queue (256 slots + runnext), plus a box for P2 through P15 on this 16-CPU host. A dashed amber steal half arrow runs P1 into P0. To the right, a dashed sysmon box (no P, its own M) lists retake, forcegc, netpoll backstop, and the Go 1.25+ GOMAXPROCS recheck, with a solid amber SIGURG to asyncPreempt arrow into P0's running M. Caption: M:N: many G onto few M via P; GOMAXPROCS=1 still preempts since sysmon's signal needs no spare P." caption="Figure 44.1: the GMP model: G's queued locally (256 slots + runnext) and globally, GOMAXPROCS P's bound to M's, work-stealing between them, and sysmon's SIGURG reaching in from outside the P pool" %} > **Tools used**: `go build` / `go run` (host, the `go 1.26.x` check in @@ -251,7 +251,7 @@ temporarily *becoming* the netpoller thread. {% include excalidraw.html file="44-goroutines-not-threads" - alt="Top band: 2000 goroutines blocked on os.Pipe reads feed one shared netpoller (epoll_pwait, per-fd pollDesc) labeled parked, no M; the OS-thread box reads threads=12, bound=80. Bottom band, the contrast: N goroutines calling syscall.Read on a raw, never-epoll-registered fd each block one M in read(2); the thread box shows the count climbing toward N via sysmon retake, handoffp, startm/newm. Caption: goroutines != threads -- 2000 parked, 12 threads, bound 80." + alt="Top band: 2000 goroutines blocked on os.Pipe reads feed one shared netpoller (epoll_pwait, per-fd pollDesc) labeled parked, no M; the OS-thread box reads threads=12, bound=80. Bottom band, the contrast: N goroutines calling syscall.Read on a raw, never-epoll-registered fd each block one M in read(2); the thread box shows the count climbing toward N via sysmon retake, handoffp, startm/newm. Caption: goroutines != threads: 2000 parked, 12 threads, bound 80." caption="Figure 44.2: the netpoller contrast: 2,000 goroutines parked in one shared epoll with no M held, against a raw blocking read that pins one M per blocked goroutine" %} The `netpoll` theme measures exactly this: 2,000 `os.Pipe()` pairs, one @@ -327,7 +327,7 @@ fails if something regresses to one-thread-per-blocked-goroutine. Go 1.26 also removes something from this picture: a dedicated `_Psyscall` P status. A P whose M was in a syscall used to carry that status directly; as of 1.26, `pp.status` only ever takes -`_Pidle`/`_Prunning`/`_Pgcstop`/`_Pdead`, and whether a P's M is really in +`_Pidle`/`_Prunning`/`_Pgcstop`/`_Pdead`, and whether a P's M is in a syscall is read off the **goroutine's** status instead (`setBlockOnExitSyscall`); the release notes attribute a ~30% cut in baseline cgo-call overhead to this, a narrower change than the still-open @@ -341,7 +341,7 @@ end the line with a verdict word the program itself decides: `fmt.Printf("... %s\n", verdict(cond))`, where `verdict` renders `"ok"` or `"FAIL"` from a boolean. `verify.lua` matches the label text plus the literal `ok`; if the condition flips, `FAIL` prints instead and the -pattern simply stops matching. The check is the program's own +pattern stops matching. The check is the program's own arithmetic, not the test's, and three techniques above make that arithmetic trustworthy rather than merely plausible: `debug.SetGCPercent(-1)` turns "allocate then measure" into an exact @@ -374,18 +374,16 @@ signal-based preemption is what let the ticker progress, and disabling it reproduces the pre-1.14 world where a non-yielding loop starves everything else on its P. -That contrast run almost didn't terminate while this example was being -built. With async preemption disabled, the only way the ticker gets to -run is if the spinner voluntarily yields its P, which it never does by -design. Launch the spinner first and the ticker second under -`asyncpreemptoff=1`, and the ticker claims `runnext`, starts, and blocks -forever on a flag only the never-scheduled spinner can set: a true -livelock, not a slow test. The fix is the launch order in the excerpt -above (ticker first, spinner second), so the ticker observes `tick` -incrementing on a normal run, or never runs at all under -`asyncpreemptoff=1`, correctly reporting `ticker_incr=0`. The bug and -the fix teach the same lesson as SIGURG itself: a scheduler decision -that looks arbitrary usually isn't. +With async preemption disabled, the only way the ticker runs is if the +spinner voluntarily yields its P, which it never does by design. Launching +the spinner first and the ticker second under `asyncpreemptoff=1` would let +the ticker claim `runnext`, start, and block forever on a flag only the +never-scheduled spinner can set: a true livelock, not a slow check. The +launch order in the excerpt above (ticker first, spinner second) avoids +that: the ticker observes `tick` incrementing on a normal run, or never +runs at all under `asyncpreemptoff=1`, correctly reporting `ticker_incr=0`. +The ordering choice teaches the same lesson as SIGURG itself: a scheduler +decision that looks arbitrary usually isn't. ## Concurrency lens @@ -582,9 +580,9 @@ none of them "the binary exited 0." has to match the fixed label text plus the literal word `ok`. Part 12 closes here. Two deep dives, two languages, one shared discipline: -Chapter 43 proved Rust's `unsafe` promises with `miri`; this chapter proved -Go's runtime promises with the runtime's own instrumentation, turned back -on itself. +the Rust macros chapter proved Rust's `unsafe` promises with `miri`; this +chapter proved Go's runtime promises with the runtime's own instrumentation, +turned back on itself. --- @@ -611,7 +609,7 @@ followed by a sane 4 MB goal once a real percent was restored; a fresh GODEBUG=schedtrace=1000,scheddetail=1 capture showed the startup threads=5 summary line and per-P/M/G detail including the Go 1.25+ -GOMAXPROCS updater (idle) goroutine. The deliberate contrast +GOMAXPROCS updater (idle) goroutine. This contrast run, GODEBUG=asyncpreemptoff=1 ./go/bin/app sched, reproduced ticker_incr=0 ticker_incr>0 FAIL and sched: FAIL exactly, with the other three sched checks unaffected. Green @@ -620,5 +618,5 @@ Tea GC's default-on status in Go 1.26 and the removal of the dedicated notes and runtime source, not measured directly; Green Tea leaves no signal in gctrace output by design. Not exercised: examples/manifest.yaml marks this mode: local: -no VM or LGTM path applies; the asyncpreemptoff=1 run is a -deliberate FAIL demo, never asserted green in verify.lua.

+no VM or LGTM path applies; the asyncpreemptoff=1 run is +designed to fail, never asserted green in verify.lua.

diff --git a/_docs/45-linux-analysis-suites.md b/_docs/45-linux-analysis-suites.md index 31f6e13..a91bb41 100644 --- a/_docs/45-linux-analysis-suites.md +++ b/_docs/45-linux-analysis-suites.md @@ -2,7 +2,7 @@ title: "Linux analysis suites: Cockpit, SystemTap, and PCP watch a whole fleet" order: 45 part: "Appendices: Tooling" -description: "linux-analysis-suites builds a single Fedora-based, multi-stage Containerfile (lsp45-pcp, 265 MB) shipping Performance Co-Pilot's pmcd with two live PMDAs -- openmetrics and podman -- run rootless with podman, alongside Cockpit's zero-scripting web console and SystemTap's kernel-module tracing on the systems-target VM: verified PASS 10/0 with a real pminfo value 42, live podman-socket container visibility, a working SystemTap kernel-module smoke test, and Cockpit's socket answering HTTP 200." +description: "linux-analysis-suites builds a single Fedora-based, multi-stage Containerfile (lsp45-pcp, 265 MB) shipping Performance Co-Pilot's pmcd with two live PMDAs (openmetrics and podman) run rootless with podman, alongside Cockpit's zero-scripting web console and SystemTap's kernel-module tracing on the systems-target VM: verified PASS 10/0 with a real pminfo value 42, live podman-socket container visibility, a working SystemTap kernel-module smoke test, and Cockpit's socket answering HTTP 200." duration: "35 minutes" --- @@ -15,15 +15,15 @@ appendices, and it pivots from single-process tools to three fleet-wide **suites**, each built on a different philosophy for the same underlying question: what is this system doing right now. Cockpit answers it with a web console you never script. SystemTap answers it by compiling a kernel -module per probe, bcc's harder-to-build sibling from Chapter 30. And PCP -(Performance Co-Pilot) answers it by pulling metrics out of things this book -already built: a `podman.sock` (Chapter 34) and an OpenMetrics document -(Chapter 38). +module per probe, the harder-to-build sibling to bcc-tools and bpftrace. And +PCP (Performance Co-Pilot) answers it by pulling metrics out of things this +book already built: a `podman.sock` from its container fleet work and an +OpenMetrics document from its OpenTelemetry pipeline. {% include excalidraw.html file="45-analysis-suite-landscape" - alt="Three side-by-side boxes under a banner reading three philosophies for fleet-wide observation. Left: a browser icon into a Cockpit box, labeled zero-scripting web console, cockpit-ws multiplexing sessions over a socket answering on port 9090. Middle: a stap CLI box compiling into a kernel-module icon, labeled SystemTap -- bcc's harder-to-build kernel-module-compiling sibling from Chapter 30, needing matching kernel-devel. Right: a pmcd hub box with two labeled PMDA arms -- pmda-openmetrics reading a file:// document and pmda-podman reading a podman.sock -- both arrows curving back to two smaller ghost boxes captioned Chapter 34 containers and Chapter 38 OpenMetrics, showing PCP pulls metrics out of things this book already built. Caption: three ways to observe a fleet -- click, script, or scrape." - caption="Figure 45.1: three philosophies for fleet-wide observation: Cockpit's zero-scripting console, SystemTap's kernel-module scripting (bcc's harder-to-build sibling from Chapter 30), and PCP pulling metrics out of two things this book already built" %} + alt="Three side-by-side boxes under a banner reading three philosophies for fleet-wide observation. Left: a browser icon into a Cockpit box, labeled zero-scripting web console, cockpit-ws multiplexing sessions over a socket answering on port 9090. Middle: a stap CLI box compiling into a kernel-module icon, labeled SystemTap: bcc's harder-to-build kernel-module-compiling sibling, needing matching kernel-devel. Right: a pmcd hub box with two labeled PMDA arms: pmda-openmetrics reading a file:// document and pmda-podman reading a podman.sock, both arrows curving back to two smaller ghost boxes captioned containers and OpenMetrics, showing PCP pulls metrics out of things this book already built. Caption: three ways to observe a fleet: click, script, or scrape." + caption="Figure 45.1: three philosophies for fleet-wide observation: Cockpit's zero-scripting console, SystemTap's kernel-module scripting (bcc's harder-to-build sibling), and PCP pulling metrics out of two things this book already built" %} > **Tools used**: `podman` (host, build/run/exec, in > `scripts/check-host.sh`); `pmcd`/`pminfo`/`pmrep`/`pcp` (a third location: @@ -43,8 +43,8 @@ running the check once rather than three times for no behavioral gain. Whichever of the three is given, the image built and the container run are byte-identical. Code excerpts below are therefore plain fenced blocks, never the tri-language `codetabs` include this book uses everywhere else; there -is only one path through this example, the same handling Chapter 44 used for -its single-language runtime deep dive. +is only one path through this example, the same handling the Go runtime +chapter used for its single-language deep dive. ## Cockpit: a web console with no scripting required @@ -60,9 +60,9 @@ answering before this chapter's demo ever runs. ## SystemTap: bcc's kernel-module-compiling sibling -Chapter 30 put SystemTap side by side with bcc-tools and bpftrace and found -its kernel-module backend blocked by a real `kernel-devel`/running-kernel -mismatch, a friction point flagged, not resolved, in that chapter. This +Putting SystemTap side by side with bcc-tools and bpftrace surfaced its +kernel-module backend blocked by a real `kernel-devel`/running-kernel +mismatch, a friction point flagged but not resolved at the time. This appendix is where it gets resolved: once `kernel-devel-$(uname -r)` matches the running kernel, `stap` compiles a real, loadable kernel module per probe script, runs it, and unloads it when the script exits, the same @@ -83,9 +83,10 @@ namespace, queryable with `pminfo`, `pmrep`, or the summary command `pcp` itself. This chapter's image layers exactly two PMDAs, and both pull their numbers from things this book already built rather than from anything new: `pmda-openmetrics` bridges an OpenMetrics/Prometheus-style document, the -same format Chapter 38 wired a real exporter around, into PCP's namespace, -and `pmda-podman` reads container and pod state over a `podman.sock`, the -same rootless socket Chapter 34 built a container fleet around. PCP doesn't +same format this book's OpenTelemetry pipeline wired a real exporter around, +into PCP's namespace, and `pmda-podman` reads container and pod state over a +`podman.sock`, the same rootless socket this book's container fleet work +built around. PCP doesn't introduce a new thing to observe; it gives a uniform namespace to metrics this book's own examples already produce. @@ -107,7 +108,7 @@ scrape of a real file, every run, with zero network dependency. ### Why Fedora, not UBI -Every other container example in this book, Chapter 34's, builds on a +Every other container example in this book builds on a UBI 10 base, per this project's own convention. This one doesn't, and the reason is direct: checked directly against a real `dnf install`, `pcp` and its PMDA packages are **not resolvable on UBI 9 or UBI @@ -118,18 +119,18 @@ exists, but it needs Red Hat portal authentication, and there is no conventions, are assumed **not** to have a RHEL subscription. Fedora's own repositories carry `pcp 7.1.5` and every PMDA package this chapter needs, fully unauthenticated, so `registry.fedoraproject.org/fedora:44` is the -base that actually builds for everyone, not just readers with a Red Hat +base that builds for everyone, not just readers with a Red Hat account. One PMDA is absent from this image: `pmda-bcc`, PCP's own -bridge into the eBPF world Chapter 30 covered directly. Activating it needs -a working BCC/eBPF stack underneath: raw `BPF` syscalls, `/sys/kernel/ +bridge into the eBPF world bcc-tools and bpftrace cover directly. Activating +it needs a working BCC/eBPF stack underneath: raw `BPF` syscalls, `/sys/kernel/ debug`, a kernel-devel package matching the exact running kernel, which in turn needs a **super-privileged container**, the opposite of the rootless posture every other example in this book targets. It's covered here only as a name: PCP's privileged sibling PMDA, demonstrated properly on the -`systems-target` VM where Chapter 30's real eBPF tooling already runs, not -shipped inside this rootless image. +`systems-target` VM where real bcc-tools and bpftrace tooling already runs, +not shipped inside this rootless image. ## How the code works @@ -211,13 +212,13 @@ ENTRYPOINT ["/usr/libexec/pcp/bin/pmcd", "-f"] ``` `FROM smoke AS runtime` both reuses `smoke`'s already-activated PMDA state -(no third `./Install` re-run needed) and forces `podman build` to actually -build `smoke` first: the gate stage only runs because something real -depends on it existing. +(no third `./Install` re-run needed) and forces `podman build` to build +`smoke` first: the gate stage only runs because something real depends on +it existing. {% include excalidraw.html file="45-pcp-build-gate" - alt="A vertical three-stage pipeline. Stage one, builder, installs pcp packages, copies lsp45.prom, starts pmcd, and runs two ./Install calls against that live pmcd. An arrow labeled FROM builder AS smoke descends into stage two, smoke, which restarts pmcd and asserts both PMDAs are listed plus a real value 42 and a real kernel.all.load sample, gated by grep -q calls that can fail the whole podman build. A ghost, disconnected box beside stage two reads FROM builder AS runtime -- the skipped path, dashed and crossed out, with a note that podman silently never builds a stage nothing chains onto. The real arrow, solid and amber, is labeled FROM smoke AS runtime and descends into stage three, runtime, which reuses the smoke stage's already-activated PMNS/pmcd.conf and carries only EXPOSE 44321 and the pmcd entrypoint. Caption: chaining onto smoke, not builder, is what forces the gate stage to actually build." + alt="A vertical three-stage pipeline. Stage one, builder, installs pcp packages, copies lsp45.prom, starts pmcd, and runs two ./Install calls against that live pmcd. An arrow labeled FROM builder AS smoke descends into stage two, smoke, which restarts pmcd and asserts both PMDAs are listed plus a real value 42 and a real kernel.all.load sample, gated by grep -q calls that can fail the whole podman build. A ghost, disconnected box beside stage two reads FROM builder AS runtime: the skipped path, dashed and crossed out, with a note that podman silently never builds a stage nothing chains onto. The real arrow, solid and amber, is labeled FROM smoke AS runtime and descends into stage three, runtime, which reuses the smoke stage's already-activated PMNS/pmcd.conf and carries only EXPOSE 44321 and the pmcd entrypoint. Caption: chaining onto smoke, not builder, is what forces the gate stage to build." caption="Figure 45.2: the PCP image's build gate: chaining runtime onto smoke forces the gate stage to build, and a PMDA that fails to register fails the build, not just the demo" %} ## Errors, three ways @@ -256,16 +257,17 @@ environment-dependent optional feature look like a broken build. Each suite here answers "how many things are running at once" a different way. `pmcd` is one reactor process fronting **N independent PMDA processes**, each talking to `pmcd` over its own pipe, the same -process-isolation discipline Chapter 11 built `pmon` around, applied here -to metrics collection instead of service supervision: a PMDA that hangs or -crashes takes down only its own slice of the namespace, never `pmcd` -itself or any sibling agent. SystemTap's `<<<` operator is the sharpest -concurrency idea in this chapter: it's a per-CPU-safe **statistical -aggregation**: every CPU accumulates into its own private bucket with no -shared-memory contention at all, and only the `end` probe's `@count()` -call folds those per-CPU buckets together into the single number a script -prints. That's the same lock-free instinct Chapter 26 built atomics and -CAS loops around, reached here through a completely different mechanism: +process-isolation discipline this book's process-lifecycle work built +`pmon` around, applied here to metrics collection instead of service +supervision: a PMDA that hangs or crashes takes down only its own slice of +the namespace, never `pmcd` itself or any sibling agent. SystemTap's `<<<` +operator is the sharpest concurrency idea in this chapter: it's a +per-CPU-safe **statistical aggregation**: every CPU accumulates into its own +private bucket with no shared-memory contention at all, and only the `end` +probe's `@count()` call folds those per-CPU buckets together into the single +number a script prints. That's the same lock-free instinct behind this +book's atomics and CAS loops, reached here through a completely different +mechanism: avoid the race by never sharing the counter in the first place, rather than serializing access to one. Cockpit's concurrency story is the simplest of the three: one `cockpit-ws` process multiplexes every browser @@ -333,7 +335,7 @@ podman.container.running (`root pmcd proc pmproxy xfs linux mmv kvm jbd2`), all eleven active. The openmetrics PMDA hands back the exact `42` `config/lsp45.prom` ships, every run. The podman PMDA, once the container's own `podman.sock` is mounted -into it, sees `lsp45-pcp-demo` (the very container running the query), +into it, sees `lsp45-pcp-demo`, the same container running the query, reporting itself as `podman.container.running value 1`. `pmrep`'s three load-average columns come from `kernel.all.load`, a core PCP metric with nothing to do with either PMDA this chapter added, confirming that `pmcd`'s @@ -369,7 +371,7 @@ stap-module-ok That one line is a real kernel module compiled against `kernel-devel-$(uname -r)` (matched to the running `6.19.10-300.fc44` -kernel), loaded, run once, and unloaded: the payoff Chapter 30 flagged as +kernel), loaded, run once, and unloaded: the payoff flagged earlier as blocked by a kernel-devel mismatch, resolved here once the matching package is installed. A slightly larger script aggregates real syscall activity from a controlled workload: @@ -399,7 +401,8 @@ PASS 10 / FAIL 0 ``` Image size on this host: **265 MB** (`podman images lsp45-pcp`), much -larger than Chapter 34's few-tens-of-MB runtime images, because PCP's own +larger than this book's other few-tens-of-MB container runtime images, +because PCP's own dependency chain (Python, `pcp-libs`, `pcp-system-tools`) is the size driver, and there's no further multi-stage trim available when `runtime` needs the exact same userspace `smoke` just proved works. @@ -441,11 +444,11 @@ shipped, and started cold. Neither check trusts the other's result. - **Three suites, three philosophies for the same question.** Cockpit answers "what is this system doing" with a browser and zero scripting; - SystemTap answers it by compiling a real kernel module per probe, bcc's - harder-to-build sibling from Chapter 30; PCP answers it by giving a - uniform namespace to metrics pulled out of things this book already - built: a `podman.sock` (Chapter 34) and an OpenMetrics document - (Chapter 38). + SystemTap answers it by compiling a real kernel module per probe, the + harder-to-build sibling to bcc-tools and bpftrace; PCP answers it by + giving a uniform namespace to metrics pulled out of things this book + already built: a `podman.sock` from its container fleet work and an + OpenMetrics document from its OpenTelemetry pipeline. - **Fedora, not UBI, because the packages aren't there.** `pcp` and its PMDAs are unresolvable on UBI 9/10 without a RHEL entitlement this book's readers are assumed not to have; @@ -453,7 +456,7 @@ shipped, and started cold. Neither check trusts the other's result. - **A disconnected build stage is a build stage that never runs.** `FROM builder AS runtime` compiles cleanly and silently skips `smoke` entirely; `FROM smoke AS runtime` is what forces `podman build` to - actually execute the gate. + execute the gate. - **A PMDA's own `Install` script can silently revert itself.** With no systemd in a build layer, `systemctl restart pmcd.service` fails and `Install`'s failure path un-does its own `pmcd.conf` edit: starting @@ -467,7 +470,7 @@ shipped, and started cold. Neither check trusts the other's result. - **`pmda-bcc` is excluded by design, not by oversight.** Real eBPF attachment needs a super-privileged container, the opposite of every rootless posture in this book, so it's named here as PCP's privileged - sibling PMDA and left to Chapter 30's real eBPF toolkit on + sibling PMDA and left to the real bcc-tools/bpftrace toolkit on `systems-target`. Part 13's appendices continue from here, tooling this book leaned on diff --git a/_docs/46-cpp-toolbox.md b/_docs/46-cpp-toolbox.md index 5d63880..0596c0e 100644 --- a/_docs/46-cpp-toolbox.md +++ b/_docs/46-cpp-toolbox.md @@ -2,18 +2,19 @@ title: "The C++ toolbox: CMake presets, Conan lockfiles, and GCC-vs-clang parity" order: 46 part: "Appendices: Tooling" -description: "cpp-toolbox is one C++23 binary and the full toolchain wrapped around it -- seven CMakePresets.json configurations sharing one _base, a committed Conan 2 lockfile for an isolated fmt sub-target, and a GCC-vs-clang parity gate on toolbox report -- read by clang-tidy, clang-format, and a custom gdb Python pretty-printer, with ccache and mold demonstrated but absent on the reference host. Verified on the Fedora 44 host: verify.lua PASS 20/FAIL 0, GCC and clang release builds byte-identical (digest=0x481984990deee5ff, diff empty), UBSan catching a seeded signed-integer overflow at toolbox.cpp:43, and gdb rendering Digest(0x481984990deee5ff) through toolbox-printers.py." +description: "cpp-toolbox is one C++23 binary and the full toolchain wrapped around it (seven CMakePresets.json configurations sharing one _base, a committed Conan 2 lockfile for an isolated fmt sub-target, and a GCC-vs-clang parity gate on toolbox report) read by clang-tidy, clang-format, and a custom gdb Python pretty-printer, with ccache and mold demonstrated but absent on the reference host. Verified on the Fedora 44 host: verify.lua PASS 20/FAIL 0, GCC and clang release builds byte-identical (digest=0x481984990deee5ff, diff empty), UBSan catching a seeded signed-integer overflow at toolbox.cpp:43, and gdb rendering Digest(0x481984990deee5ff) through toolbox-printers.py." duration: "50 minutes" --- -Chapter 45 opened this part with three suites that watch a whole fleet from -outside (Cockpit's console, SystemTap's kernel modules, PCP's pull-based -metrics), and none of them cared what language produced the process they -were watching. This chapter turns all the way around: one language, one -small program, and every tool that touches it before it ever runs. Chapter -31 already gave C++ one paragraph of toolbelt each (`perf`, `clang-tidy`, -GCC's `-fanalyzer`) on the way to covering Go and Rust too. This chapter -reopens exactly that toolchain and does not move on. `toolbox` is a single +The linux-analysis-suites chapter opened this part with three suites that +watch a whole fleet from outside (Cockpit's console, SystemTap's kernel +modules, PCP's pull-based metrics), and none of them cared what language +produced the process they were watching. This chapter turns all the way +around: one language, one small program, and every tool that touches it +before it ever runs. The per-language toolbelts chapter already gave C++ +one paragraph of toolbelt each (`perf`, `clang-tidy`, GCC's `-fanalyzer`) on +the way to covering Go and Rust too. This chapter reopens exactly that +toolchain and does not move on. `toolbox` is a single C++23 binary, engineered so that every tool pointed at it (two compilers, a build-system generator, a package manager, a static analyzer, a formatter, a debugger that needs help understanding a custom type) has @@ -31,7 +32,7 @@ whether they agree. {% include excalidraw.html file="46-cmake-preset-graph" - alt="A tree diagram. At the top, a hidden _base configure preset (generator Ninja, binaryDir sourceDir/build/presetName) fans out to six visible presets: release (RelWithDebInfo, GCC), debug (Debug, GCC), asan (Debug + -fsanitize=address,undefined), and ubsan (Debug + -fsanitize=undefined -fno-sanitize-recover=undefined) sit in one column; release-clang and debug-clang sit beside them, each drawn inheriting from release/debug respectively with an arrow labeled inherits plus a CMAKE_CXX_COMPILER=clang++ override. Each of the six feeds its own build/ directory box below it. A seventh preset, conan, also inherits _base but is drawn held apart in a dashed box labeled isolated -- only reachable when TOOLBOX_ENABLE_CONAN=ON, needed by no other preset, so the default build never touches Conan. A caption note names which preset the parity gate uses (release vs release-clang), which the UBSan gate uses (ubsan), and which gdb uses (debug)." + alt="A tree diagram. At the top, a hidden _base configure preset (generator Ninja, binaryDir sourceDir/build/presetName) fans out to six visible presets: release (RelWithDebInfo, GCC), debug (Debug, GCC), asan (Debug + -fsanitize=address,undefined), and ubsan (Debug + -fsanitize=undefined -fno-sanitize-recover=undefined) sit in one column; release-clang and debug-clang sit beside them, each drawn inheriting from release/debug respectively with an arrow labeled inherits plus a CMAKE_CXX_COMPILER=clang++ override. Each of the six feeds its own build/ directory box below it. A seventh preset, conan, also inherits _base but is drawn held apart in a dashed box labeled isolated: only reachable when TOOLBOX_ENABLE_CONAN=ON, needed by no other preset, so the default build never touches Conan. A caption note names which preset the parity gate uses (release vs release-clang), which the UBSan gate uses (ubsan), and which gdb uses (debug)." caption="Figure 46.1: the CMakePresets inheritance graph, one _base config fanning out to release/debug/asan/ubsan and their clang twins, each into build/${presetName}, with the Conan sub-target preset held apart so the default build needs no Conan" %} > **Tools used**: `cmake` and `ninja` (host; both are hard requirements in @@ -51,9 +52,9 @@ whether they agree. Because the C++ toolchain is itself the subject, `examples/46-cpp-toolbox/` ships `langs: [cpp]` only: a Go or Rust build would have nothing new to say about a `CMakePresets.json` file or a `.clang-tidy` config. Every code -block below is a plain fenced `cpp`/`console`/`json` block, the way Chapter -44's Go runtime chapter presented Go alone; there is no language-tab -include to reach for when there is only one language in the room. +block below is a plain fenced `cpp`/`console`/`json` block, the way the Go +runtime chapter presented Go alone; there is no language-tab include to +reach for when there is only one language in the room. ## CMake presets in depth @@ -403,7 +404,7 @@ about code written to please one compiler over another. {% include excalidraw.html file="46-toolbox-pipeline" - alt="One source box, toolbox.cpp plus digest and smell translation units, splits into two build arrows -- one through GCC, one through clang -- each landing in its own build/ binary. Both binaries feed into a report command box, and their outputs converge on an amber diff box marked empty, the byte-identical parity gate, digest=0x481984990deee5ff. Below the source box, a static-analysis arm branches off into clang-tidy (flagging performance-unnecessary-value-param in smell.cpp) and clang-format --dry-run --Werror (clean on tracked sources, nonzero on a misformatted fixture). A separate runtime arm branches into the ubsan build, which traps the seeded signed-integer overflow with an abort icon, and the debug build, which feeds gdb plus a Python pretty-printer producing Digest(0x481984990deee5ff). Two dashed accelerator boxes, ccache and mold, sit off to the side labeled shown, not verified on this host." + alt="One source box, toolbox.cpp plus digest and smell translation units, splits into two build arrows (one through GCC, one through clang) each landing in its own build/ binary. Both binaries feed into a report command box, and their outputs converge on an amber diff box marked empty, the byte-identical parity gate, digest=0x481984990deee5ff. Below the source box, a static-analysis arm branches off into clang-tidy (flagging performance-unnecessary-value-param in smell.cpp) and clang-format --dry-run --Werror (clean on tracked sources, nonzero on a misformatted fixture). A separate runtime arm branches into the ubsan build, which traps the seeded signed-integer overflow with an abort icon, and the debug build, which feeds gdb plus a Python pretty-printer producing Digest(0x481984990deee5ff). Two dashed accelerator boxes, ccache and mold, sit off to the side labeled shown, not verified on this host." caption="Figure 46.2: the toolbox pipeline, one source built by two compilers into byte-identical output (the parity gate), with static tools (clang-tidy, clang-format) and runtime tools (UBSan, gdb pretty-printers) around it, and ccache/mold as optional accelerators" %} ## Errors, three ways @@ -465,7 +466,7 @@ FormatStyle: file ``` The third is the UBSan runtime abort already introduced above, this time -as the pinned gate `verify.lua` actually runs: +as the pinned gate `verify.lua` runs: ```lua local ubsan_run = checks.run("./cpp/build/ubsan/toolbox defect overflow") @@ -483,9 +484,9 @@ a reason grounded in how that particular tool works. ## Concurrency lens: the build graph's own parallelism -Every earlier concurrency lens in this book (Chapter 25's threads, -Chapter 44's GMP scheduler) asked how a *running program* multiplexes -work. This chapter's build has no threads of its own to watch, but the +Every earlier concurrency lens in this book, from a futex-synchronized +thread pool to the Go runtime's GMP scheduler, asked how a *running +program* multiplexes work. This chapter's build has no threads of its own to watch, but the tool building it does: Ninja's default job count is bounded by the host's CPU count, and every one of `toolbox`'s translation units that has no dependency on another can compile in parallel: `cmake --build --preset @@ -503,8 +504,9 @@ The link step itself is where a second, differently-parallel tool would matter more directly: `mold`, "a Modern Linker," parallelizes symbol resolution and section layout internally, the way a traditional single-threaded linker (`ld.bfd`) never does. This book's own earlier -concurrency lenses (Chapter 44's M:N scheduler, Chapter 25's thread pool) -are about overlapping *independent* units of work, and `mold`'s internal +concurrency lenses, the Go runtime's M:N scheduler and a futex-synchronized +thread pool, are about overlapping *independent* units of work, and +`mold`'s internal design is the identical idea applied to the one step Ninja itself cannot parallelize away, because a link has to see every object file at once. `mold` is demonstrated, not verified, in "lint, format, sanitize, @@ -625,7 +627,7 @@ re-invoking the compiler on an unchanged translation unit: the second build of an unchanged tree turning into a cache lookup instead of a recompile is the effect a real run would show. `mold` swaps in as the linker with `"CMAKE_EXE_LINKER_FLAGS": "-fuse-ld=mold"`, and a binary it -actually linked would carry its own name in the `.comment` ELF section, +linked would carry its own name in the `.comment` ELF section, checkable with `readelf -p .comment build/release/toolbox | grep mold` with no debugger needed at all. Both recipes are shown here exactly as they would be run; neither produced output on this host, and the status @@ -720,21 +722,21 @@ needed. Three earlier chapters already covered pieces of this ground, and this chapter does not re-teach any of them: -- **Chapter 31** ("Per-Language Toolbelts") gave C++ one column of a +- **Per-Language Toolbelts** gave C++ one column of a three-language *profiling* comparison (`perf`, `clang-tidy`, `-fanalyzer`) next to Go's `pprof` and Rust's `cargo flamegraph`. This chapter is the opposite shape: one language, covered in depth, and it never profiles anything. -- **Chapter 28** ("gdb and Remote Debugging") is where `gdb`'s core +- **gdb and Remote Debugging** is where `gdb`'s core mechanics live: reading a core file, `gdbserver`/`target remote`, `rust-gdb`'s type-aware rendering. This chapter assumes all of that and adds exactly one new idea: a pretty-printer *you write* for *your own* type, not one that ships with libstdc++. -- **Chapter 29** ("valgrind, Sanitizers, and Miri") built the full five-bug +- **valgrind, Sanitizers, and Miri** built the full five-bug ASan/UBSan/TSan/valgrind/miri catch-or-miss matrix. This chapter uses exactly one corner of that matrix (UBSan on a signed-integer overflow) and spends its real depth budget on CMake presets and Conan lockfiles - instead, ground Chapter 29 never touches. + instead, ground that chapter never touches. What is new here (preset inheritance mechanics, a committed Conan 2 lockfile, cross-compiler byte parity, a hand-written gdb Python diff --git a/_docs/47-go-toolbox.md b/_docs/47-go-toolbox.md index 60e5226..f906432 100644 --- a/_docs/47-go-toolbox.md +++ b/_docs/47-go-toolbox.md @@ -2,11 +2,11 @@ title: "The Go toolbox: go.mod toolchain pins, go generate discipline, and the pprof-to-benchstat workflow" order: 47 part: "Appendices: Tooling" -description: "go-toolbox is one Go binary and the toolchain wrapped around it -- go.mod's go/toolchain directives resolved through GOTOOLCHAIN, a self-contained go generate pipeline producing a DO-NOT-EDIT tool table, and gofmt/go vet/pprof as go-native hard gates -- with committed configs for gofumpt, golangci-lint, staticcheck, delve, and benchstat shown as reference but absent on the reference host. Verified on the Fedora 44 host: verify.lua PASS 16/FAIL 0 offline, toolbox report digest=0x481984990deee5ff (byte-identical to ch46's C++ digest), go generate regenerating tools_generated.go byte-for-byte, go vet -tags vetdemo catching a printf mismatch at vetdemo.go:13, and go tool pprof -top naming hotLoop." +description: "go-toolbox is one Go binary and the toolchain wrapped around it (go.mod's go/toolchain directives resolved through GOTOOLCHAIN, a self-contained go generate pipeline producing a DO-NOT-EDIT tool table, and gofmt/go vet/pprof as go-native hard gates) with committed configs for gofumpt, golangci-lint, staticcheck, delve, and benchstat shown as reference but absent on the reference host. Verified on the Fedora 44 host: verify.lua PASS 16/FAIL 0 offline, toolbox report digest=0x481984990deee5ff (byte-identical to the C++ toolbox's digest), go generate regenerating tools_generated.go byte-for-byte, go vet -tags vetdemo catching a printf mismatch at vetdemo.go:13, and go tool pprof -top naming hotLoop." duration: "50 minutes" --- -Chapter 46 turned all the way around on C++: one language, one small binary, +The C++ toolbox chapter turned all the way around on C++: one language, one small binary, and every tool that touches it before it ever runs: two compilers, a build-system generator, a package manager, a static analyzer, a debugger taught a custom type. Go gets the identical treatment here, and the shape of @@ -40,8 +40,8 @@ catch or confirm, never just a clean exit. > `examples/47-go-toolbox` is `mode: local` in `examples/manifest.yaml`. Because the Go toolchain is itself the subject, `examples/47-go-toolbox/` -ships `langs: [go]` only, the same way Chapter 44's Go-runtime chapter and -Chapter 46's C++ chapter each stayed inside one language: a C++ or Rust +ships `langs: [go]` only, the same way the Go-runtime chapter and +the C++ toolbox chapter each stayed inside one language: a C++ or Rust build would have nothing new to say about a `go.mod` toolchain directive or a `.golangci.yml`. Every code block below is a plain fenced `go`/`console`/ `yaml` block; there is no `codetabs` include to reach for when only one @@ -83,7 +83,7 @@ itself enforces, and on this host it resolves to the binary already installed without a single byte crossing the network. That is also the cleanest point of contrast with the two sibling toolchain -chapters. C++ has no equivalent single-file pin: Chapter 46 needed seven +chapters. C++ has no equivalent single-file pin: the C++ toolbox chapter needed seven named `CMakePresets.json` configurations plus a Conan lockfile to make "which compiler, which build type" and "which exact package revision" explicit, because C++ leaves compiler choice and dependency resolution as @@ -163,7 +163,7 @@ $ ./demo.sh go run report toolbox report: payload_len=16 digest=0x481984990deee5ff ``` -`digest=0x481984990deee5ff` is not a coincidence shared with Chapter 46: it +`digest=0x481984990deee5ff` is not a coincidence shared with the C++ toolbox chapter: it is the same 16-byte payload, hashed by the same 64-bit FNV-1a algorithm, computed by an entirely different compiler and runtime. FNV-1a has no floating-point rounding, no pointer values, and no unordered-container @@ -179,7 +179,7 @@ twice by hand. `go generate` is Go's answer to "keep a derived file accurate against its source of truth": it runs a directive comment, once, on demand, and never as part -of `go build`. `main.go`'s directive is one line at the very top of the +of `go build`. `main.go`'s directive is one line at the top of the file: ```go @@ -300,7 +300,7 @@ checks.expect_exit(gen, 0, Run directly, both directions agree: the copy-aside diff is empty, and the idiomatic form a reader would actually put in CI, `go generate ./... && git diff --exit-code`, exits 0 on this checkout too. -Either way, the claim is the same one Chapter 46 made about GCC-vs-clang +Either way, the claim is the same one the C++ toolbox chapter made about GCC-vs-clang parity, aimed at a different kind of determinism: not "two compilers agree," but "the generator, run twice against unchanged input, agrees with itself." @@ -471,13 +471,13 @@ func unused_smell_name() int { ``` Three functions, three distinct designed-for findings, none of them -overlapping: `unusedHelper` (line 15) is simply dead code: **U1000**, +overlapping: `unusedHelper` (line 15) is dead code: **U1000**, staticcheck's unused-declaration check. `deadStore` (line 24) is dead *and* its first assignment (`x := 10`) is overwritten before ever being read: **U1000** again for the whole function, plus **SA4006** for the specific -overwritten value, a pattern `go vet` does not catch because `x` actually +overwritten value, a pattern `go vet` does not catch because `x` is read eventually, just not at its first value. `unused_smell_name` -(line 34) is dead a third time (a third **U1000**) and also violates Go's +(line 34) is dead a third time (a third **U1000**) and also breaks Go's `mixedCaps` naming convention on its own: **ST1003**, staticcheck's `stylecheck` companion. None of the three is ever referenced from `main.go` or anywhere else, so all three compile cleanly and none of them can ever @@ -495,7 +495,7 @@ its default check set. `govulncheck` answers a question neither `gofmt`, `go vet`, nor `staticcheck` can: not "is this code suspicious," but "does anything this -module actually calls sit inside a function the Go vulnerability database +module calls sit inside a function the Go vulnerability database has an advisory against." That is a fundamentally different kind of check: it needs a reachable network (the vulnerability database lives at `vuln.go.dev`) and it only has something to say about a *vulnerable @@ -503,7 +503,7 @@ dependency*, which this module does not have: `go/go.mod` pulls in nothing outside the standard library, so there is no `go.sum` and nothing for `govulncheck` to cross-reference here at all. -The workflow a reader would actually use is `govulncheck ./...` (or +The workflow a reader would use is `govulncheck ./...` (or `govulncheck -mode=binary ./bin/toolbox` against an already-built binary), run from inside the module whose dependencies are in question. Because this module has no third-party dependency to demonstrate against, the practical way @@ -542,7 +542,7 @@ The two details worth carrying away from that shape: `govulncheck` reports a stable `GO-xxxx-xxxxx` identifier per advisory (the same kind of durable token `staticcheck`'s `U1000`/`SA4006` and `golangci-lint`'s `(unused)` are (a name to grep for that survives wording changes across releases), -and it only ever flags a symbol your own code actually reaches (the +and it only ever flags a symbol your own code reaches (the "Example traces found" call chain). A vulnerable function sitting unused in a dependency you happen to import produces no finding at all, since nothing calls it. There is no `verify.lua` gate for `govulncheck` in this @@ -552,8 +552,8 @@ example, by design: it is prose and workflow here, not a gated check. Delve is Go's native debugger, DWARF-aware like `gdb`, but built with goroutines as a first-class concept rather than bolted on. `go/.dlvinit` is -a committed, four-line headless-batch script, the same idea as Chapter 46's -`gdb` batch invocation, expressed in delve's own init-script format: +a committed, four-line headless-batch script, the same idea as the C++ toolbox +chapter's `gdb` batch invocation, expressed in delve's own init-script format: ``` break main.go:33 @@ -578,7 +578,7 @@ its own line prints anything, which is exactly why `print d.FNV` has something meaningful to show: delve renders `Digest`'s one field as a plain 64-bit unsigned decimal, `5195329438047200767`, the same bits as `0x481984990deee5ff`, the identical value `toolbox report`'s own `%016x` -formatting and Chapter 46's `gdb` pretty-printer both already surfaced, this +formatting and the C++ toolbox chapter's `gdb` pretty-printer both already surfaced, this time with no pretty-printer at all, because a single-field `uint64` needs no help to read. The designed-for invocation a reader with `dlv` installed would run: @@ -601,7 +601,7 @@ above prints `5195329438047200767` and exits 0. `main.go`'s whole dispatch is a three-way `switch` on `os.Args[1]`: `report` computes and prints the FNV-1a digest above, `tools` prints the `go generate`-produced table, and `defect divzero` seeds a deliberate -runtime panic, Go's analog of Chapter 46's UBSan-caught signed-integer +runtime panic, Go's analog of the C++ toolbox chapter's UBSan-caught signed-integer overflow, using the failure mode Go itself provides instead: ```go @@ -621,7 +621,7 @@ func cmdDefectDivZero() int { ``` `divisor` is a variable initialized to its zero value, not a literal `0`, -the same reason Chapter 46's `max_value` was declared `volatile`: a constant +the same reason the C++ toolbox chapter's `max_value` was declared `volatile`: a constant division by a literal zero is a compile-time error in Go (`invalid operation: division by zero`), so the defect has to survive as far as run time to be the thing this section demonstrates, and a plain variable the @@ -629,10 +629,10 @@ compiler cannot constant-fold does exactly that. `digest.go` holds the `Digest` type and the `fnv1a` implementation quoted above; `tools_generated.go` holds the generator's committed output; -`smells.go` and `vetdemo.go` hold the deliberate, never-called findings for +`smells.go` and `vetdemo.go` hold the never-called findings for the static-analysis and `go vet` gates. Every one of those last three files is isolated from the digest and from each other by a different mechanism: -`smells.go`'s functions are simply unreferenced, `vetdemo.go` sits behind a +`smells.go`'s functions are unreferenced, `vetdemo.go` sits behind a `//go:build vetdemo` tag excluded from the default build entirely, and the `testdata/` fixtures are ignored by `go build`/`go vet`/`go test` by directory-name convention. Nothing about running `toolbox report` or @@ -641,7 +641,7 @@ has not been "fixed." {% include excalidraw.html file="47-go-tool-gates" - alt="One Go module box splits into two gate lanes. The amber hard lane lists gofmt -l, go generate diff-clean, go vet -tags vetdemo naming a printf finding, go tool pprof -top naming hotLoop, and the FNV-1a digest -- all shipping inside the go binary and verifying offline. A dashed ghost lane beside it lists gofumpt, golangci-lint, staticcheck, and delve as gated-if-present, govulncheck marked network required, and benchstat, all shown as reference and not verified on this host." + alt="One Go module box splits into two gate lanes. The amber hard lane lists gofmt -l, go generate diff-clean, go vet -tags vetdemo naming a printf finding, go tool pprof -top naming hotLoop, and the FNV-1a digest: all shipping inside the go binary and verifying offline. A dashed ghost lane beside it lists gofumpt, golangci-lint, staticcheck, and delve as gated-if-present, govulncheck marked network required, and benchstat, all shown as reference and not verified on this host." caption="Figure 47.2: one module, two gate lanes: the go-native hard gates (gofmt, go generate, go vet, pprof, digest) that ship with the go binary and verify offline, beside the gated-if-present tools (gofumpt, golangci-lint, staticcheck, delve, govulncheck, benchstat) shown as reference" %} ## Errors, three ways @@ -659,7 +659,7 @@ question that tool answers. The second surface is `go vet`, which ships with `go` itself and reads *usage patterns the compiler allows but almost never means*. `vetdemo.go`, -excluded from the default build by its own build tag, is one deliberate +excluded from the default build by its own build tag, is one designed-for mismatch: ```go @@ -682,7 +682,7 @@ func printBadFormat() { The Go compiler accepts this without complaint: `Printf`'s format string and its variadic arguments are only strings and `interface{}` values at the type level, so there is nothing for the type checker itself to reject. `go -vet`'s printf analyzer is what actually reads the format verb against the +vet`'s printf analyzer is what reads the format verb against the argument's real type and flags the mismatch, run directly on this host: ```console @@ -702,7 +702,7 @@ the others do. ## Concurrency lens -Chapter 44 spent a whole chapter on the GMP model: `GOMAXPROCS` P's are the +The Go-runtime chapter already covered the GMP model in full: `GOMAXPROCS` P's are the scarce resource that lets an M run Go code at all, goroutines are cheap, and work-stealing moves half a queue at a time between P's that run dry. This chapter's benchmark is the tooling side of that same picture: not *how* @@ -740,7 +740,7 @@ func BenchmarkHotLoopParallel(b *testing.B) { } ``` -`b.RunParallel` is not doing anything Chapter 44 didn't already name: each +`b.RunParallel` is not doing anything the Go-runtime chapter didn't already name: each of the goroutines it launches is a G, competing for one of `GOMAXPROCS` P's the same way any other goroutines would, and `hotLoop`'s pure, allocation-free loop is close to the best case for that model: no channel sends, no shared @@ -774,16 +774,16 @@ $ go tool pprof -top -nodecount=15 toolbox.test cpu.prof intends, and the gate this chapter asserts is that name, `hotLoop`, appearing in the top frames, never the timing next to it. Sample counts and elapsed seconds vary with host load and iteration count from run to run; the -*identity* of the function actually spending the cycles does not, which is +*identity* of the function spending the cycles does not, which is why `verify.lua` gate E matches the string `hotLoop` and nothing else. That same profile also shows `BenchmarkHotLoop` (7.43% cumulative) and `BenchmarkHotLoopParallel.func1` (92.57% cumulative) as the two call paths into `hotLoop`: both benchmarks ran during profiling, and `pprof` attributes -each sample to whichever one was actually on the stack at that instant. +each sample to whichever one was on the stack at that instant. `benchstat` is the tool that turns two sets of raw benchmark numbers into a statistical comparison instead of an eyeballed diff: the workflow a reader -would use to ask "did this change actually help, or is that just noise": +would use to ask "did this change help, or is that just noise": ```console $ go test -bench=BenchmarkHotLoop -run=^$ -count=10 . > old.txt @@ -801,8 +801,8 @@ scheduling noise, and this chapter's gates never assert a raw timing as if it were a fixed literal. A reader comparing `-cpu=1` against `-cpu=N` for `BenchmarkHotLoopParallel` would read the same table shape, with `benchstat`'s own statistics (median, confidence interval) doing the job of saying whether -an observed difference is real or within noise, the same discipline Chapter -44's `GOGC`/`GOMEMLIMIT` comparisons used exact `NumGC` deltas to sidestep +an observed difference is real or within noise, the same discipline the +Go-runtime chapter's `GOGC`/`GOMEMLIMIT` comparisons used exact `NumGC` deltas to sidestep entirely, here handled by a purpose-built statistics tool instead because wall-clock benchmark timings have no exact-integer shortcut. @@ -846,7 +846,7 @@ not have installed. ## Cross-check: the same digest, two languages apart The clearest falsifiable claim in this chapter is not internal to Go at -all: it is the one this chapter shares with Chapter 46. `toolbox report` +all: it is the one this chapter shares with the C++ toolbox chapter. `toolbox report` here and `toolbox report` there hash a byte-for-byte identical 16-byte payload with the byte-for-byte identical FNV-1a algorithm, and land on the identical 64-bit value: @@ -867,9 +867,9 @@ every conforming language implementation is required to agree on bit for bit. The second cross-check is internal: the copy-aside `go generate` diff and the idiomatic `git diff --exit-code` form check the identical property two different ways: one without assuming a git checkout exists, one the -way a reader's own CI would actually gate it. Both report clean on +way a reader's own CI would gate it. Both report clean on this tree, which is the same "two independent paths to the same answer" -discipline Chapter 46's GCC-vs-clang `diff` used, aimed here at a generator +discipline the C++ toolbox chapter's GCC-vs-clang `diff` used, aimed here at a generator agreeing with itself instead of two compilers agreeing with each other. ## Where this sits next to the book's other tooling chapters @@ -877,7 +877,7 @@ agreeing with itself instead of two compilers agreeing with each other. Two earlier chapters already covered pieces of this ground, and this chapter does not re-teach either: -- **Chapter 31** ("Per-Language Toolbelts") gave Go one column of a +- **The earlier Per-Language Toolbelts coverage** gave Go one column of a three-language *profiling breadth* comparison: `runtime/pprof`, `go tool trace`, and `dlv`, one paragraph each, next to C++'s `perf` and Rust's `cargo flamegraph`. This chapter is the opposite shape: it never @@ -885,16 +885,16 @@ chapter does not re-teach either: `dlv`'s **config authoring** (a committed `.dlvinit` headless-batch script, not just "delve exists") and the `pprof`-to-`benchstat` **workflow**, going from a raw CPU profile to a statistically - defensible before/after comparison, ground Chapter 31 never had room for. -- **Chapter 44** ("The Go Runtime from a Systems Programmer's Seat") is - where the GMP scheduler, the pacer, and the netpoller actually live: - what `GOMAXPROCS`, work-stealing, and `SIGURG`-based preemption mean and - why they behave as they do. This chapter assumes all of that and asks a - narrower, tooling-shaped question on top of it: given that model, how does - a reader *measure* whether a change to `hotLoop` actually changed - anything, with `b.RunParallel` as the harness and `pprof`/`benchstat` as - the instruments: the tooling that measures the runtime, not the runtime - itself. + defensible before/after comparison, ground that earlier coverage never + had room for. +- **The Go-runtime chapter** is where the GMP scheduler, the pacer, and the + netpoller live: what `GOMAXPROCS`, work-stealing, and `SIGURG`-based + preemption mean and why they behave as they do. This chapter assumes all + of that and asks a narrower, tooling-shaped question on top of it: given + that model, how does a reader *measure* whether a change to `hotLoop` + changed anything, with `b.RunParallel` as the harness and + `pprof`/`benchstat` as the instruments: the tooling that measures the + runtime, not the runtime itself. What is new here: the `go`/`toolchain` directive pair and `GOTOOLCHAIN`'s resolution rule, `go generate` as a diff-clean CI discipline @@ -916,21 +916,21 @@ has no earlier chapter to point back to instead. `sort.Strings` plus `go/format`'s `format.Source` are what make `tools_generated.go` byte-identical across regenerations, which is the property `go generate ./... && git diff --exit-code` (and this chapter's - copy-aside equivalent) actually gates. + copy-aside equivalent) gates. - **`gofmt`, `gofumpt`, `go vet`, and `staticcheck`/`golangci-lint` read successively more context before saying anything**: pure text shape, one function call's argument types (`vetdemo.go:13`'s printf mismatch), and a whole package's data flow and naming (`smells.go`'s `U1000`/`SA4006`/ `ST1003`): three static surfaces, not three languages, and none of them ever executes the program. -- **`b.RunParallel` is Chapter 44's GMP model, applied as a measurement +- **`b.RunParallel` is the Go-runtime chapter's GMP model, applied as a measurement harness**: `hotLoop`'s allocation-free, lock-free loop is close to the best case for `GOMAXPROCS` P's to scale cleanly, which is exactly why it is the right benchmark to hand to `benchstat` for a statistically defensible `-cpu=1` vs `-cpu=N` comparison rather than an eyeballed one. - **The same digest twice, in two languages that share nothing else**: - `toolbox report`'s `0x481984990deee5ff` here matches Chapter 46's C++ - `toolbox report` byte for byte, because FNV-1a constrains only integer + `toolbox report`'s `0x481984990deee5ff` here matches the C++ toolbox + chapter's `toolbox report` byte for byte, because FNV-1a constrains only integer arithmetic over a fixed payload: one of the few things every conforming language implementation is required to agree on exactly. - **Six real, committed tool configs, none of them run on this host**: @@ -940,7 +940,7 @@ has no earlier chapter to point back to instead. and the footer below marks exactly that gap and nothing more. Part 13 continues with Rust's own toolchain, getting the same reference-depth -treatment this chapter and Chapter 46 gave C++ and Go. +treatment this chapter and the C++ toolbox chapter gave C++ and Go. --- @@ -955,7 +955,7 @@ folded into that PASS count, the go-native hard gates A-E: go.mod pins go 1.26 and toolchain go1.26.5; toolbox report printed toolbox report: payload_len=16 digest=0x481984990deee5ff, -byte-identical to Chapter 46's C++ digest; go generate ./... +byte-identical to the C++ toolbox chapter's C++ digest; go generate ./... regenerated tools_generated.go byte-for-byte against the committed copy (copy-aside diff empty) and toolbox tools printed the generated table's count=10 plus its sorted @@ -970,7 +970,7 @@ class="status status--unverified">unverified: gofumpt, dlv, and benchstat are all absent from this reference host (confirmed by command -v for each), so gates F, G, H, I, and K each printed an informational SKIP: rather -than actually running, and their sections above show committed configs +than running, and their sections above show committed configs (.golangci.yml, .dlvinit) and designed-for findings, the exact tokens ((unused), U1000, SA4006, the decimal 5195329438047200767, diff --git a/_docs/48-rust-toolbox.md b/_docs/48-rust-toolbox.md index e841d75..1966e4e 100644 --- a/_docs/48-rust-toolbox.md +++ b/_docs/48-rust-toolbox.md @@ -2,19 +2,20 @@ title: "The Rust toolbox: rust-toolchain.toml pins, rustfmt and clippy as toolchain components, and the cargo-test-to-nextest workflow" order: 48 part: "Appendices: Tooling" -description: "rust-toolbox is one zero-dependency crate and the toolchain wrapped around it -- rust-toolchain.toml pinning channel and components so rustup resolves the same rustc, rustfmt, and clippy for everyone, an isolated misformatted fixture and a feature-gated clippy bait that make the format and lint gates assert real findings, and cargo test asserting the digest literal -- with committed deny.toml and clippy.toml for the cargo-nextest/cargo-deny/cargo-llvm-cov/cargo-audit tier shown as reference but absent on the host. Verified on the Fedora 44 host: verify.lua PASS 20/FAIL 0 fully offline, toolbox report digest=0x481984990deee5ff (byte-identical to ch46's C++ and ch47's Go digest), cargo fmt --check clean while rustfmt flags the fixture, cargo clippy -D warnings clean by default and naming needless_return under --features lintbait, and cargo test passing the digest unit test." +description: "rust-toolbox is one zero-dependency crate and the toolchain wrapped around it (rust-toolchain.toml pinning channel and components so rustup resolves the same rustc, rustfmt, and clippy for everyone, an isolated misformatted fixture and a feature-gated clippy bait that make the format and lint gates assert real findings, and cargo test asserting the digest literal) with committed deny.toml and clippy.toml for the cargo-nextest/cargo-deny/cargo-llvm-cov/cargo-audit tier shown as reference but absent on the host. Verified on the Fedora 44 host: verify.lua PASS 20/FAIL 0 fully offline, toolbox report digest=0x481984990deee5ff (byte-identical to the C++ toolbox's and the Go toolbox's digest), cargo fmt --check clean while rustfmt flags the fixture, cargo clippy -D warnings clean by default and naming needless_return under --features lintbait, and cargo test passing the digest unit test." duration: "50 minutes" --- -Chapter 46 pointed every C++ tool it could find at one small binary; Chapter -47 did the same for Go and found the depth had moved: out of the build -system and into `go.mod`'s own toolchain directives. Rust closes the arc, and -it moves the line one step further in the same direction. Go pins its -compiler in the module file; Rust pins its compiler *and its formatter and -its linter*, in a file `rustup` reads before `cargo` has done anything at -all. That single fact reorganizes this whole chapter. In Chapter 47, -`gofumpt`, `golangci-lint`, and `staticcheck` had to be gated-if-present, -because they are separate downloads that a given host may simply not have. In +The C++ toolbox chapter pointed every C++ tool it could find at one small +binary; the Go toolbox chapter did the same for Go and found the depth had +moved: out of the build system and into `go.mod`'s own toolchain directives. +Rust closes the arc, and it moves the line one step further in the same +direction. Go pins its compiler in the module file; Rust pins its compiler +*and its formatter and its linter*, in a file `rustup` reads before `cargo` +has done anything at all. That single fact reorganizes this whole chapter. In +the Go toolbox chapter, `gofumpt`, `golangci-lint`, and `staticcheck` had to +be gated-if-present, because they are separate downloads that a given host +may not have. In Rust, `rustfmt` and `clippy` are *components of the pinned channel*: if the pin resolves, they are there, at exactly the version the pin names. So the formatter and the linter graduate out of the "shown as reference" tier and @@ -45,8 +46,8 @@ never just a clean exit. > `examples/48-rust-toolbox` is `mode: local` in `examples/manifest.yaml`. Because the Rust toolchain is itself the subject, `examples/48-rust-toolbox/` -ships `langs: [rust]` only, the same way Chapter 43's macro chapter, Chapter -46's C++ chapter, and Chapter 47's Go chapter each stayed inside one +ships `langs: [rust]` only, the same way the Rust-macros chapter, the C++ +toolbox chapter, and the Go toolbox chapter each stayed inside one language: a C++ or Go build has nothing to say about a `rust-toolchain.toml` component list or a `clippy.toml` MSRV. Every code block below is a plain fenced `rust`/`toml`/`console` block; there is no @@ -106,8 +107,9 @@ $ cd /tmp && rustup show active-toolchain stable-x86_64-unknown-linux-gnu (default) ``` -That is the property Chapter 47 got from `GOTOOLCHAIN` and Chapter 46 had to -build by hand out of seven `CMakePresets.json` configurations: the version +That is the property the Go toolbox chapter got from `GOTOOLCHAIN` and the +C++ toolbox chapter had to build by hand out of seven `CMakePresets.json` +configurations: the version decision lives with the source, not with whoever checked it out. What Rust adds on top is that the *tools* travel with it too. Every binary in the chain reports the same release: @@ -136,7 +138,7 @@ as the inclusions. `llvm-tools` (needed by `cargo-llvm-cov`) and `rust-src` (needed by miri and rust-analyzer) are left out, because `rustup` fetches any missing component **over the network, on the first `cargo` invocation in this directory**. Pinning a component a reader does not have turns `cargo build ---offline` into a download: exactly the failure this example exists +--offline` into a download: the failure this example exists to avoid. The chapter's own draft hit it: an earlier `components` list included `llvm-tools`, and the first build in a fresh checkout printed `info: downloading component llvm-tools` before compiling anything. The file @@ -204,13 +206,13 @@ mechanism for changing language semantics without breaking old code: a 2024 crate and a 2015 crate can link together in one binary, each compiled under its own rules. Editions are per-crate, the channel is per-repository, and they are set in different files: `Cargo.toml` and `rust-toolchain.toml` -respectively. Both of ch48's gates check both, because getting one right and +respectively. Both of this chapter's gates check both, because getting one right and the other wrong is the common failure: a 2024 crate compiled by a toolchain too old to know that edition fails with a confusing error about an unknown edition, not an obviously-wrong one. -The payload and the hash are lifted byte-for-byte from Chapter 46's C++ and -Chapter 47's Go toolbox, which is the whole point: +The payload and the hash are lifted byte-for-byte from the C++ and Go +toolbox chapters, which is the whole point: ```rust /// ch46/ch47's exact 16-byte payload (`"The quick brown."`), reused @@ -241,9 +243,9 @@ pub fn fnv1a(data: &[u8]) -> Digest { `*` on `u64` panics on overflow in a debug build and wraps in a release build, so the same source would compute the digest in release and abort in `cargo test`, which builds with debug assertions on. `wrapping_mul` states -the intent the algorithm actually requires (FNV-1a is defined modulo 2⁶⁴) and +the intent the algorithm requires (FNV-1a is defined modulo 2⁶⁴) and makes the two profiles agree. C++ got this behavior by using `uint64_t`, -where wrapping is simply what the language does; Go got it the same way. Rust +where wrapping is what the language does; Go got it the same way. Rust is the only one of the three where you must ask for it, and the only one where forgetting is caught rather than silent. @@ -343,7 +345,7 @@ that output: nonzero exit, a `Diff in …fixtures/misformatted.rs` line, and the specific reformatted signature `pub fn classify(n: i64) -> &'static str {`, because any one of them alone could pass for the wrong reason. A nonzero exit could be a missing file; a `Diff in` line could name some other file; -only the reformatted signature proves `rustfmt` actually parsed this source +only the reformatted signature proves `rustfmt` parsed this source and produced this rewrite. ## `clippy`: clean by default, loud behind a feature @@ -492,7 +494,7 @@ test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; fini ``` The gate asserts the named test line and the `test result: ok. 1 passed; 0 -failed` summary, not the process exit code — a `cargo test` run that compiled +failed` summary, not the process exit code: a `cargo test` run that compiled zero tests also exits 0, and would sail past a weaker check. Watching the gate fail matters too, because a gate nobody has @@ -523,8 +525,8 @@ test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; One byte, and `assert_eq!` prints both sides in decimal because that is how `Digest`'s derived `Debug` formats a `u64`. `5195329438047200767` is -`0x481984990deee5ff`: the identical decimal Chapter 47's `delve` gate -printed when it read the Go struct's field out of a live process. Two +`0x481984990deee5ff`: the identical decimal the Go toolbox chapter's `delve` +gate printed when it read the Go struct's field out of a live process. Two languages, two entirely different tools, one number. This is also why the test gate is separate from the report gate rather than @@ -593,7 +595,7 @@ not been "fixed." {% include excalidraw.html file="48-rust-tool-gates" - alt="One Rust crate box splits into two gate lanes. The amber hard lane lists the rustup pin at channel 1.97.1 and edition 2024, the FNV-1a digest gate at 0x481984990deee5ff, cargo fmt --check clean on tracked sources and nonzero on the fixture, cargo clippy -D warnings with --features lintbait naming needless_return, and cargo test asserting the digest literal -- all running offline from the pinned toolchain alone because rustfmt and clippy ship as toolchain components. A dashed ghost lane beside it lists cargo-nextest, cargo-deny with its deny.toml, cargo-llvm-cov asserting a filename, and cargo-audit marked advisory-db slash network, plus cargo-watch, cargo-flamegraph, and sccache marked not gated, all shown as reference and not verified on this host." + alt="One Rust crate box splits into two gate lanes. The amber hard lane lists the rustup pin at channel 1.97.1 and edition 2024, the FNV-1a digest gate at 0x481984990deee5ff, cargo fmt --check clean on tracked sources and nonzero on the fixture, cargo clippy -D warnings with --features lintbait naming needless_return, and cargo test asserting the digest literal: all running offline from the pinned toolchain alone because rustfmt and clippy ship as toolchain components. A dashed ghost lane beside it lists cargo-nextest, cargo-deny with its deny.toml, cargo-llvm-cov asserting a filename, and cargo-audit marked advisory-db slash network, plus cargo-watch, cargo-flamegraph, and sccache marked not gated, all shown as reference and not verified on this host." caption="Figure 48.2: one crate, two gate lanes: the pinned-toolchain hard gates (rustup pin, digest, cargo fmt, cargo clippy, cargo test) that need no cargo install, beside the gated-if-present tools (cargo-nextest, cargo-deny, cargo-llvm-cov, cargo-audit) and the shown-not-gated accelerators (cargo-watch, cargo-flamegraph, sccache)" %} ## Errors, three ways @@ -639,9 +641,9 @@ panic. `toolbox` has no `Result` in it, which is itself the design decision: the report path has nothing that can fail (a fixed array, an infallible hash, a `println!`), so introducing a `Result` would be inventing an error case to have one. The panics that do appear are all in test code, where a panic *is* -the reporting mechanism. Chapter 5 (the book's own "Errors, three ways") -and Chapter 43 cover `Result` propagation and `?` at length for code that has -real fallible I/O; a tooling chapter's +the reporting mechanism. The errors chapter (the book's own "Errors, three +ways" section) and the Rust-macros chapter cover `Result` propagation and +`?` at length for code that has real fallible I/O; a tooling chapter's contribution is the opposite observation: that a deterministic gate should have no error paths to test, because every branch it does not have is a branch that cannot drift. @@ -661,8 +663,8 @@ reproducibility but not for speed: the graph's *shape* is fixed by single crate, `rustc` parallelizes differently again, by splitting the crate into codegen units (16 in the release profile by default, 1 under `lto = "fat"`), which trades compile time against optimization quality: the -same trade `mold` and `ccache` addressed from the outside in Chapter 46, made -from the inside here. +same trade `mold` and `ccache` addressed from the outside in the C++ +toolbox chapter, made from the inside here. The second is the test harness. `cargo test` builds one test binary and libtest runs the `#[test]` functions **in threads, inside that one process**, @@ -694,8 +696,9 @@ rather than the whole run, which for a book full of `mmap`, signal, and `rlimit` examples is not a marginal benefit. The cost is process-spawn overhead per test, which matters for thousands of microsecond-scale unit tests and does not matter here. Go made the opposite structural choice -(one process per *package*, tests within a package sharing it), and Chapter -47's `b.RunParallel` benchmark lived entirely inside that shared process. Same +(one process per *package*, tests within a package sharing it), and the Go +toolbox chapter's `b.RunParallel` benchmark lived entirely inside that shared +process. Same problem, two different places to put the isolation boundary. ## The gated-if-present tier: `cargo-nextest`, `cargo-deny`, `cargo-llvm-cov`, `cargo-audit` @@ -783,8 +786,8 @@ Solution: Upgrade to >=0.4.3 ``` The durable token there is the `RUSTSEC-YYYY-NNNN` identifier: stable, -citable, and the thing a policy actually references. Chapter 47 made the same -call for `govulncheck` and for the same two reasons: network, and no +citable, and the thing a policy references. The Go toolbox chapter made the +same call for `govulncheck` and for the same two reasons: network, and no committed vulnerability. ## Shown, not gated: `cargo-watch`, `cargo-flamegraph`, `sccache` @@ -799,26 +802,28 @@ A check that never returns is not a check. `cargo flamegraph --bin toolbox -- report` produces an SVG of where CPU time went. It needs `perf`, which needs `kernel.perf_event_paranoid` lowered or -root, and it produces output that differs run to run by construction. Chapter -31 already ran `cargo-flamegraph` against a Rust binary and taught -how to read the result, and Chapter 26 hit the `perf_event_paranoid=2` -privilege wall head-on; the reason it is not gated *here* is the same reason -Chapter 46 did not gate `ccache`: sampling output is not a deterministic -assertion. +root, and it produces output that differs run to run by construction. The +earlier Per-Language Toolbelts coverage already ran `cargo-flamegraph` +against a Rust binary and taught how to read the result, and the atomics +chapter hit the `perf_event_paranoid=2` privilege wall head-on; the reason it +is not gated *here* is the same reason the C++ toolbox chapter did not gate +`ccache`: sampling output is not a deterministic assertion. `sccache` caches compilation artifacts across builds (`RUSTC_WRAPPER=sccache`). It is a pure accelerator: a correct build and a correct cached build produce identical output, so there is nothing about correctness to assert. Its own metric (cache hit rate) depends on what you -compiled five minutes ago, which is exactly the kind of number this book -refuses to gate on. `ccache` and `mold` got the same treatment in Chapter 46. +compiled five minutes ago, which is the kind of number this book +refuses to gate on. `ccache` and `mold` got the same treatment in the C++ +toolbox chapter. **miri** deserves a pointer rather than a section: it interprets Rust at the MIR level and catches undefined behavior in `unsafe` code that no amount of formatting, linting, or testing will. It is not re-taught here because -Chapters 29 and 43 already own it, and `toolbox` has no `unsafe` block for it -to find anything in: which is the correct reason to omit a tool, as opposed -to omitting it because it is inconvenient. +earlier coverage of unsafe Rust and the Rust-macros chapter already own it, +and `toolbox` has no `unsafe` block for it to find anything in: which is the +correct reason to omit a tool, as opposed to omitting it because it is +inconvenient. ## Build, run, observe @@ -894,8 +899,8 @@ $ examples/48-rust-toolbox/rust/target/release/toolbox report toolbox report: payload_len=16 digest=0x481984990deee5ff ``` -(The C++ line carries an extra `label=[toolbox]` field from Chapter 46's -argument-echo demo; the `payload_len` and `digest` fields are identical +(The C++ line carries an extra `label=[toolbox]` field from the C++ toolbox +chapter's argument-echo demo; the `payload_len` and `digest` fields are identical across all three.) FNV-1a is language-independent (a shift, an xor, and a multiply modulo @@ -912,43 +917,45 @@ none of them did. ## Where this sits next to the book's other tooling chapters Part 13 covers tooling in four chapters and this is the last of them, so here -is the arc. Chapter 45 handled the *analysis suites*: tools that observe a -running system from outside it. Chapters 46, 47, and 48 -each took one language and pointed its whole toolchain at one small binary, +is the arc. The Linux-analysis-suites chapter handled the *analysis suites*: +tools that observe a running system from outside it. The C++, Go, and Rust +toolbox chapters each took one language and pointed its whole toolchain at +one small binary, and the interesting result is how differently the three languages distribute the same responsibilities: -- **C++ (ch46)** puts almost nothing in the language distribution. CMake, +- **C++** puts almost nothing in the language distribution. CMake, Conan, clang-tidy, clang-format, the sanitizers, gdb, ccache, and mold are eight separate projects with eight separate versions, and `CMakePresets.json` exists to make "which of these, configured how" an explicit, committed choice. The chapter's parity gate (GCC and clang producing byte-identical output) is a check the other two languages have no reason to run. -- **Go (ch47)** puts the compiler, the formatter, the vetter, the test +- **Go** puts the compiler, the formatter, the vetter, the test runner, the profiler, and the generator inside one `go` binary, and pins that binary's version in `go.mod`. What it leaves outside (gofumpt, - golangci-lint, staticcheck, delve, benchstat) is precisely what Chapter 47 - had to gate as if-present. -- **Rust (ch48)** pins a *toolchain* rather than a binary, and that toolchain + golangci-lint, staticcheck, delve, benchstat) is precisely what the Go + toolbox chapter had to gate as if-present. +- **Rust** pins a *toolchain* rather than a binary, and that toolchain includes the formatter and the linter as components. The consequence is this chapter's structure: the hard, offline core covers formatting and - linting, which Chapter 47's could not, while the if-present tier is a - different set of tools entirely (nextest, deny, llvm-cov, audit), every - one of them a `cargo install` rather than a component. + linting, which the Go toolbox chapter's could not, while the if-present + tier is a different set of tools entirely (nextest, deny, llvm-cov, audit), + every one of them a `cargo install` rather than a component. -Chapter 31 sits underneath all three: it ran each language's *profiling* -toolbelt (`perf` for C++, the `pprof` ecosystem for Go, `cargo-flamegraph` -and clippy for Rust) against one shared workload, which is why these three -chapters could concentrate on the build-and-check side instead of re-teaching -how to read a flamegraph. +The earlier Per-Language Toolbelts coverage sits underneath all three: it ran +each language's *profiling* toolbelt (`perf` for C++, the `pprof` ecosystem +for Go, `cargo-flamegraph` and clippy for Rust) against one shared workload, +which is why these three chapters could concentrate on the build-and-check +side instead of re-teaching how to read a flamegraph. The dependency and supply-chain story splits the same way. C++ needed Conan lockfiles to have a dependency graph worth checking at all; Go's module system makes the graph free and `govulncheck` the tool that reads it; Rust gets the graph free too and splits the reading in two: `cargo-deny` for policy answerable from the lockfile, `cargo-audit` for the RustSec advisory -database over the network. Chapters 47 and 48 made the identical call about -that network half: show it, name the durable identifier, never gate it. +database over the network. The Go and Rust toolbox chapters made the +identical call about that network half: show it, name the durable +identifier, never gate it. ## What you learned @@ -957,7 +964,7 @@ that network half: show it, name the durable identifier, never gate it. installed. Because `rustfmt` and `clippy` are components, pinning the channel pins the formatter and the linter at the same commit as `rustc`: which is why this chapter's format and lint gates are hard and offline - where Chapter 47's equivalents could not be. + where the Go toolbox chapter's equivalents could not be. - **`rustup show` states the override.** `active because: overridden by '…/rust-toolchain.toml'` is the mechanism reported by the tool, and it disappears one directory outside the crate. @@ -1016,8 +1023,8 @@ sets edition = "2024"; rustc --version resolved through the pin to rustc 1.97.1 (8bab26f4f 2026-07-14); cargo tree --offline printed exactly one node; ./demo.sh rust run report printed toolbox report: -payload_len=16 digest=0x481984990deee5ff, byte-identical to Chapter -46's C++ and Chapter 47's Go digest; cargo fmt --check exited 0 +payload_len=16 digest=0x481984990deee5ff, byte-identical to the C++ +and Go toolbox chapters' digest; cargo fmt --check exited 0 with no output while rustfmt --edition 2024 --check fixtures/misformatted.rs exited 1 with the diff quoted above; cargo clippy --offline --all-targets -- -D warnings exited 0 @@ -1045,6 +1052,7 @@ design and no vulnerable dependency will be committed to it. this host but not gated (never terminates / needs perf privileges and produces non-deterministic output), and sccache's cache-hit rate is not a correctness property; miri is -cross-referenced to Chapters 29 and 43 and has no unsafe block -here to examine. examples/manifest.yaml marks this example +cross-referenced to the earlier unsafe-Rust coverage and the Rust-macros +chapter and has no unsafe block here to examine. +examples/manifest.yaml marks this example mode: local: no VM or LGTM path applies.

diff --git a/_docs/49-concurrency-vs-parallelism.md b/_docs/49-concurrency-vs-parallelism.md index 8ad6556..7d100f2 100644 --- a/_docs/49-concurrency-vs-parallelism.md +++ b/_docs/49-concurrency-vs-parallelism.md @@ -2,7 +2,7 @@ title: "Concurrency vs parallelism: one workload, four execution models, and observables that are counts rather than clocks" order: 49 part: "Compendium: C++ Concurrency" -description: "conc runs one identical workload under four execution models -- sequential, concurrent (pinned to one CPU by sched_setaffinity), parallel (one thread per CPU), and both (2x oversubscribed) -- and reports structure rather than time: cpus_allowed from sched_getaffinity as a permission, distinct_cpus unioned from sched_getcpu as an outcome, a max_inflight high-water mark, and interleaves counted from gaps in a monotonic ticket sequence. All four converge on digest=0x481984990deee5ff, the same value ch46's C++, ch47's Go, and ch48's Rust toolbox produce. Verified on the Fedora 44 host: verify.lua PASS 43/FAIL 0, concurrent reporting cpus_allowed=1 distinct_cpus=1 max_inflight=16 interleaves=16 while parallel reports distinct_cpus=16, C++26 Contracts built four times for four distinct behaviors under GCC 16, and a hardened build trapping on __n < this->size() where the unchecked build is asserted statically to contain no bounds check at all." +description: "conc runs one identical workload under four execution models (sequential, concurrent (pinned to one CPU by sched_setaffinity), parallel (one thread per CPU), and both (2x oversubscribed)) and reports structure rather than time: cpus_allowed from sched_getaffinity as a permission, distinct_cpus unioned from sched_getcpu as an outcome, a max_inflight high-water mark, and interleaves counted from gaps in a monotonic ticket sequence. All four converge on digest=0x481984990deee5ff, the same value the C++ toolbox, the Go toolbox, and the Rust toolbox produce. Verified on the Fedora 44 host: verify.lua PASS 43/FAIL 0, concurrent reporting cpus_allowed=1 distinct_cpus=1 max_inflight=16 interleaves=16 while parallel reports distinct_cpus=16, C++26 Contracts built four times for four distinct behaviors under GCC 16, and a hardened build trapping on __n < this->size() where the unchecked build is asserted statically to contain no bounds check at all." duration: "55 minutes" --- diff --git a/_docs/50-pthreads.md b/_docs/50-pthreads.md index 5067c36..3d31a2c 100644 --- a/_docs/50-pthreads.md +++ b/_docs/50-pthreads.md @@ -2,7 +2,7 @@ title: "pthreads: the thread under std::thread, its two identities, and the six controls the standard does not give you" order: 50 part: "Compendium: C++ Concurrency" -description: "pthreads runs one facet per subcommand over the per-thread controls POSIX offers and the C++ standard library does not expose: gettid() versus the opaque pthread_t (and why the main thread's tid equals the pid), pthread_setname_np landing in /proc/self/task//comm with a 16-character name refused as ERANGE, pthread_attr_setstacksize giving exactly 262144 against an 8388608 default with sub-PTHREAD_STACK_MIN requests refused as EINVAL, per-thread pthread_setaffinity_np putting two workers on two CPUs while the caller's mask is untouched, SCHED_OTHER at priority 0 with an unprivileged SCHED_FIFO refused as EPERM, and pthread_cancel unwinding C++ destructors -- plus the trap where a catch (...) that does not rethrow aborts the process with FATAL: exception not rethrown. Verified on the Fedora 44 host with glibc 2.43: verify.lua PASS 45/FAIL 0, local, unprivileged, offline, with std::thread::native_handle_type static_asserted to be pthread_t." +description: "pthreads runs one facet per subcommand over the per-thread controls POSIX offers and the C++ standard library does not expose: gettid() versus the opaque pthread_t (and why the main thread's tid equals the pid), pthread_setname_np landing in /proc/self/task//comm with a 16-character name refused as ERANGE, pthread_attr_setstacksize giving exactly 262144 against an 8388608 default with sub-PTHREAD_STACK_MIN requests refused as EINVAL, per-thread pthread_setaffinity_np putting two workers on two CPUs while the caller's mask is untouched, SCHED_OTHER at priority 0 with an unprivileged SCHED_FIFO refused as EPERM, and pthread_cancel unwinding C++ destructors, plus the trap where a catch (...) that does not rethrow aborts the process with FATAL: exception not rethrown. Verified on the Fedora 44 host with glibc 2.43: verify.lua PASS 45/FAIL 0, local, unprivileged, offline, with std::thread::native_handle_type static_asserted to be pthread_t." duration: "55 minutes" --- diff --git a/_docs/51-std-threading.md b/_docs/51-std-threading.md index 0d0cf46..0d017a0 100644 --- a/_docs/51-std-threading.md +++ b/_docs/51-std-threading.md @@ -2,7 +2,7 @@ title: "The standard threading library: what a lock costs, and why the answer is zero until it isn't" order: 51 part: "Compendium: C++ Concurrency" -description: "stdthread performs the identical 200,000 increments three ways -- with no mutex, with an uncontended std::mutex, and with the same mutex across eight threads -- and counts the syscalls each one issues with strace. The uncontended case matches the no-mutex baseline exactly at 1 futex call, which is the dynamic loader rather than a lock, while the contended case costs 1844-3111. std::condition_variable, std::latch, std::barrier, std::counting_semaphore, and std::atomic::wait are then each shown reaching the same futex(2), and a std::lock_guard deadlock is reproduced under timeout(1) with every task's /proc wchan reading futex_do_wait, against a std::scoped_lock version that completes 100,000 acquisitions in opposing orders. Verified on the Fedora 44 host with glibc 2.43: verify.lua PASS 38/FAIL 0, local, unprivileged, offline." +description: "stdthread performs the identical 200,000 increments three ways (with no mutex, with an uncontended std::mutex, and with the same mutex across eight threads) and counts the syscalls each one issues with strace. The uncontended case matches the no-mutex baseline exactly at 1 futex call, which is the dynamic loader rather than a lock, while the contended case costs 1844-3111. std::condition_variable, std::latch, std::barrier, std::counting_semaphore, and std::atomic::wait are then each shown reaching the same futex(2), and a std::lock_guard deadlock is reproduced under timeout(1) with every task's /proc wchan reading futex_do_wait, against a std::scoped_lock version that completes 100,000 acquisitions in opposing orders. Verified on the Fedora 44 host with glibc 2.43: verify.lua PASS 38/FAIL 0, local, unprivileged, offline." duration: "60 minutes" --- diff --git a/_docs/52-boost-thread.md b/_docs/52-boost-thread.md index 5a86d79..3bc84c1 100644 --- a/_docs/52-boost-thread.md +++ b/_docs/52-boost-thread.md @@ -2,7 +2,7 @@ title: "Boost.Thread: what the standard took, what it left, and the third way to cancel a thread" order: 52 part: "Compendium: C++ Concurrency" -description: "boostthread asks what survived C++11's near-wholesale standardization of Boost.Thread, and answers with four capabilities std:: still lacks in C++23 on this toolchain: future::then() and when_all composing without blocking, where a concept evaluated against std::future and boost::future on every run reports no and yes; upgrade_lock promoting a reader to a writer in place, where std::upgrade_lock is not merely absent but undeclared; thread::interrupt() throwing an ordinary boost::thread_interrupted at defined interruption points; and thread::attributes::set_stack_size yielding exactly 262144, byte-identical to ch50's pthread_attr_setstacksize. The cancellation arc closes here -- catch (...) without a rethrow aborts the process under ch50's pthread_cancel (exit 134) and leaves it alive under Boost (exit 0), while a busy loop with no interruption point ignores interrupt() entirely. Verified against system Boost 1.90.0 on the Fedora 44 host: verify.lua PASS 37/FAIL 0, no Conan, no network." +description: "boostthread asks what survived C++11's near-wholesale standardization of Boost.Thread, and answers with four capabilities std:: still lacks in C++23 on this toolchain: future::then() and when_all composing without blocking, where a concept evaluated against std::future and boost::future on every run reports no and yes; upgrade_lock promoting a reader to a writer in place, where std::upgrade_lock is not merely absent but undeclared; thread::interrupt() throwing an ordinary boost::thread_interrupted at defined interruption points; and thread::attributes::set_stack_size yielding exactly 262144, byte-identical to the pthreads chapter's pthread_attr_setstacksize. The cancellation arc closes here: catch (...) without a rethrow aborts the process under the pthreads chapter's pthread_cancel (exit 134) and leaves it alive under Boost (exit 0), while a busy loop with no interruption point ignores interrupt() entirely. Verified against system Boost 1.90.0 on the Fedora 44 host: verify.lua PASS 37/FAIL 0, no Conan, no network." duration: "55 minutes" --- diff --git a/_docs/53-coroutines.md b/_docs/53-coroutines.md index 3b33851..f8ea91b 100644 --- a/_docs/53-coroutines.md +++ b/_docs/53-coroutines.md @@ -2,7 +2,7 @@ title: "C++20 coroutines: what a suspended computation costs, and the allocation that may not happen" order: 53 part: "Compendium: C++ Concurrency" -description: "coro measures what Chapter 27 built but never priced. A coroutine frame is measured by overloading operator new inside promise_type -- 32 bytes trivial, 304 carrying four ints and a char[256], 4128 carrying a char[4096] -- against the 8388608-byte thread stack Chapter 50 measured with pthread_getattr_np, a ratio of 262144x. Heap allocation elision turns out to be a permission rather than a guarantee: across 1000 calls to the easiest possible case, g++ 16.1.1 elides none at -O0 through -O3 while clang++ 22.1.8 elides every one from -O1 up, so verify.lua gates that the count is measured rather than that elision occurs. Plus C++23 std::generator exercised to the 20th Fibonacci term and the coroutine_handle lifetime trap counted through operator delete. Verified on the Fedora 44 host: verify.lua PASS 32/FAIL 0, C++23 stdlib only, no threads, no network." +description: "coro measures what the async-runtimes chapter built but never priced. A coroutine frame is measured by overloading operator new inside promise_type (32 bytes trivial, 304 carrying four ints and a char[256], 4128 carrying a char[4096]) against the 8388608-byte thread stack the pthreads chapter measured with pthread_getattr_np, a ratio of 262144x. Heap allocation elision turns out to be a permission rather than a guarantee: across 1000 calls to the easiest possible case, g++ 16.1.1 elides none at -O0 through -O3 while clang++ 22.1.8 elides every one from -O1 up, so verify.lua gates that the count is measured rather than that elision occurs. Plus C++23 std::generator exercised to the 20th Fibonacci term and the coroutine_handle lifetime trap counted through operator delete. Verified on the Fedora 44 host: verify.lua PASS 32/FAIL 0, C++23 stdlib only, no threads, no network." duration: "55 minutes" --- diff --git a/_docs/54-boost-fiber.md b/_docs/54-boost-fiber.md index c50c7f0..2b46e36 100644 --- a/_docs/54-boost-fiber.md +++ b/_docs/54-boost-fiber.md @@ -2,7 +2,7 @@ title: "Boost.Fiber: stackful suspension, the guard page, and the scheduler you have to choose" order: 54 part: "Compendium: C++ Concurrency" -description: "fiberdemo measures what a real stack costs and what it buys. boost::context::stack_traits reports a 131072-byte default fiber stack, sitting between ch53's measured 32-byte coroutine frame and ch50's measured 8388608-byte thread stack -- 8 MiB to 128 KiB to 32 bytes, with both ratios computed from measured numbers. The capability that stack buys is demonstrated rather than described: a fiber suspends from inside an ordinary non-coroutine function six frames deep and resumes there, which no stackless coroutine can do. protected_fixedsize_stack is shown costing exactly one page more than fixedsize_stack, gated against page_size(), and a 60x overflow dies of SIGSEGV on that guard page rather than corrupting the next mapping. The same 16 fibers report one distinct tid under round_robin and four under shared_work, making ch49's grid a scheduler choice. Verified against system Boost 1.90.0 on the Fedora 44 host: verify.lua PASS 32/FAIL 0, no Conan, no network." +description: "fiberdemo measures what a real stack costs and what it buys. boost::context::stack_traits reports a 131072-byte default fiber stack, sitting between the coroutines chapter's measured 32-byte coroutine frame and the pthreads chapter's measured 8388608-byte thread stack: 8 MiB to 128 KiB to 32 bytes, with both ratios computed from measured numbers. The capability that stack buys is demonstrated rather than described: a fiber suspends from inside an ordinary non-coroutine function six frames deep and resumes there, which no stackless coroutine can do. protected_fixedsize_stack is shown costing exactly one page more than fixedsize_stack, gated against page_size(), and a 60x overflow dies of SIGSEGV on that guard page rather than corrupting the next mapping. The same 16 fibers report one distinct tid under round_robin and four under shared_work, making the concurrency-vs-parallelism chapter's grid a scheduler choice. Verified against system Boost 1.90.0 on the Fedora 44 host: verify.lua PASS 32/FAIL 0, no Conan, no network." duration: "55 minutes" --- diff --git a/_docs/55-boost-asio.md b/_docs/55-boost-asio.md index a4cc576..a4e59dc 100644 --- a/_docs/55-boost-asio.md +++ b/_docs/55-boost-asio.md @@ -2,7 +2,7 @@ title: "Boost.Asio: the strand, the work guard, and who actually runs your handler" order: 55 part: "Compendium: C++ Concurrency" -description: "asiodemo measures what a library adds on top of the reactor Chapters 9, 22, and 27 built by hand -- and re-derives none of it. A freshly constructed io_context::run() returns after zero handlers, because it returns on no outstanding work rather than on completion. The same handler posted through a strand by 8 threads leaves a deliberately unsynchronized counter at exactly 2000 with max_inflight=1 and overlaps=0; posted straight to the io_context it overlaps 1492 times and loses increments -- a data race visible as a wrong number, whose magnitude is reported and never gated because it is UB. A strand and a std::mutex both produce the correct 20000, at 52 futex calls against 1022 counted with Chapter 51's strace method: a mutex blocks a thread, a strand defers a handler. Four buffers written as one sequence cost one sendmsg carrying 4 iovecs against four sendto, Part 5's writev machinery without an iovec array. Four shapes of run() report 1, 8, 4, and 4 distinct tids by gettid(). Cancellation arrives as a completion carrying operation_aborted -- a third shape after Chapter 51's flag and Chapter 52's exception. Verified against system Boost 1.90.0 on the Fedora 44 host: verify.lua PASS 48/FAIL 0, no Conan, no network, nothing bound to a port." +description: "asiodemo measures what a library adds on top of the reactor the event-loop, server-scaling, and async chapters built by hand, and re-derives none of it. A freshly constructed io_context::run() returns after zero handlers, because it returns on no outstanding work rather than on completion. The same handler posted through a strand by 8 threads leaves a deliberately unsynchronized counter at exactly 2000 with max_inflight=1 and overlaps=0; posted straight to the io_context it overlaps 1492 times and loses increments: a data race visible as a wrong number, whose magnitude is reported and never gated because it is UB. A strand and a std::mutex both produce the correct 20000, at 52 futex calls against 1022 counted with the std-threading chapter's strace method: a mutex blocks a thread, a strand defers a handler. Four buffers written as one sequence cost one sendmsg carrying 4 iovecs against four sendto, Part 5's writev machinery without an iovec array. Four shapes of run() report 1, 8, 4, and 4 distinct tids by gettid(). Cancellation arrives as a completion carrying operation_aborted: a third shape after the std-threading chapter's flag and the Boost.Thread chapter's exception. Verified against system Boost 1.90.0 on the Fedora 44 host: verify.lua PASS 48/FAIL 0, no Conan, no network, nothing bound to a port." duration: "60 minutes" --- diff --git a/_docs/56-capstone-comparison.md b/_docs/56-capstone-comparison.md index d82ebd3..87166f4 100644 --- a/_docs/56-capstone-comparison.md +++ b/_docs/56-capstone-comparison.md @@ -2,7 +2,7 @@ title: "The final project: one workload, seven models, and the two instruments that make them comparable" order: 56 part: "Compendium: C++ Concurrency" -description: "This final project closes Part 14 by fixing the flaw in every chapter before it: ch50-ch55 each measured one model, with its own instrument, on its own workload, so no two numbers in the compendium were strictly comparable. Here one workload -- 8 tasks x 5000 rounds folding into a single accumulator, shared verbatim by every arm -- runs through all six models plus a sequential control, and all of them produce the identical total 0xb75768f6610642a0, which is gated before any comparison is drawn. Measured with gettid(), the three single-threaded models hold eight tasks in flight on exactly one OS thread while the four multi-threaded ones report eight. Measured with ch51's strace method, the same 40000 folds cost 1 futex call under any single-threaded model, 230 to 451 under a lock, and 51 through an Asio strand. A seventh arm settles Chapter 49's conditional promise about P2300: __cpp_lib_senders is still undefined, so it links NVIDIA's stdexec reference implementation, pinned by an in-repo Conan recipe to tag nvhpc-26.05 by sha256, isolated so the six-model core needs no Conan and no network. Verified on the Fedora 44 host: verify.lua PASS 57/FAIL 0 with Conan present and PASS 52/FAIL 0 with it hidden." +description: "This final project closes Part 14 by fixing the flaw in every chapter before it: the preceding single-model chapters each measured one model, with its own instrument, on its own workload, so no two numbers in the compendium were strictly comparable. Here one workload (8 tasks x 5000 rounds folding into a single accumulator, shared verbatim by every arm) runs through all six models plus a sequential control, and all of them produce the identical total 0xb75768f6610642a0, which is gated before any comparison is drawn. Measured with gettid(), the three single-threaded models hold eight tasks in flight on exactly one OS thread while the four multi-threaded ones report eight. Measured with the std-threading chapter's strace method, the same 40000 folds cost 1 futex call under any single-threaded model, 230 to 451 under a lock, and 51 through an Asio strand. A seventh arm settles the concurrency-vs-parallelism chapter's conditional promise about P2300: __cpp_lib_senders is still undefined, so it links NVIDIA's stdexec reference implementation, pinned by an in-repo Conan recipe to tag nvhpc-26.05 by sha256, isolated so the six-model core needs no Conan and no network. Verified on the Fedora 44 host: verify.lua PASS 57/FAIL 0 with Conan present and PASS 52/FAIL 0 with it hidden." duration: "65 minutes" --- diff --git a/_parts/00-setting-up.md b/_parts/00-setting-up.md index c0f888b..b56d27f 100644 --- a/_parts/00-setting-up.md +++ b/_parts/00-setting-up.md @@ -2,5 +2,5 @@ title: "Setting Up" order: 0 part_name: "Setting Up" -blurb: "Prepare a Fedora 44 workstation for three toolchains, provision the two-VM KVM lab for anything privileged, and stand up the Podman LGTM observability stack — everything later chapters assume." +blurb: "Prepare a Fedora 44 workstation for three toolchains, provision the two-VM KVM lab for anything privileged, and stand up the Podman LGTM observability stack: everything later chapters assume." --- diff --git a/_parts/01-foundations.md b/_parts/01-foundations.md index a694207..fb610b8 100644 --- a/_parts/01-foundations.md +++ b/_parts/01-foundations.md @@ -2,5 +2,5 @@ title: "Foundations" order: 1 part_name: "Foundations" -blurb: "What a system call really is, and the two disciplines this book never stops practicing — how each language reports failure, and how each language shares work." +blurb: "What a system call really is, and the two disciplines this book never stops practicing: how each language reports failure, and how each language shares work." --- diff --git a/_parts/02-files-and-io.md b/_parts/02-files-and-io.md index a9af6ee..1d7147c 100644 --- a/_parts/02-files-and-io.md +++ b/_parts/02-files-and-io.md @@ -2,5 +2,5 @@ title: "Files and I/O" order: 2 part_name: "Files and I/O" -blurb: "From file descriptors and the VFS through the page cache, epoll event loops, and io_uring — building fwatch, a real file watcher, one I/O model at a time." +blurb: "From file descriptors and the VFS through the page cache, epoll event loops, and io_uring, building fwatch, a real file watcher, one I/O model at a time." --- diff --git a/_parts/03-processes-signals-privilege.md b/_parts/03-processes-signals-privilege.md index e0b1183..a98f2bb 100644 --- a/_parts/03-processes-signals-privilege.md +++ b/_parts/03-processes-signals-privilege.md @@ -2,5 +2,5 @@ title: "Processes, Signals, Privilege" order: 3 part_name: "Processes, Signals, Privilege" -blurb: "fork/exec/wait, safe signal handling, race-free pidfd supervision, and privilege dropping — building pmon, a mini process supervisor that stays with us to the capstone." +blurb: "fork/exec/wait, safe signal handling, race-free pidfd supervision, and privilege dropping, building pmon, a mini process supervisor that stays with us to the final project." --- diff --git a/_parts/04-memory.md b/_parts/04-memory.md index f468186..11d50c6 100644 --- a/_parts/04-memory.md +++ b/_parts/04-memory.md @@ -2,5 +2,5 @@ title: "Memory" order: 4 part_name: "Memory" -blurb: "Virtual address spaces, mmap and shared mappings, and what glibc malloc, Rust allocators, and the Go GC actually do — with shmkv, a cross-language mmap-backed key-value store." +blurb: "Virtual address spaces, mmap and shared mappings, and what glibc malloc, Rust allocators, and the Go GC do, with shmkv, a cross-language mmap-backed key-value store." --- diff --git a/_parts/05-ipc.md b/_parts/05-ipc.md index da7b13c..048d1fb 100644 --- a/_parts/05-ipc.md +++ b/_parts/05-ipc.md @@ -2,5 +2,5 @@ title: "IPC" order: 5 part_name: "IPC" -blurb: "Pipes, FIFOs, UNIX sockets with fd passing, and shared-memory coordination with futexes and eventfd — pmon grows a log pipeline and a control plane; shmkv goes multi-process." +blurb: "Pipes, FIFOs, UNIX sockets with fd passing, and shared-memory coordination with futexes and eventfd: pmon grows a log pipeline and a control plane; shmkv goes multi-process." --- diff --git a/_parts/06-networking.md b/_parts/06-networking.md index d921004..7d1828b 100644 --- a/_parts/06-networking.md +++ b/_parts/06-networking.md @@ -2,5 +2,5 @@ title: "Networking" order: 6 part_name: "Networking" -blurb: "TCP and UDP sockets from first principles to an epoll event loop and a two-VM peer-to-peer chat mesh — plus the time, timers, and deadline discipline every networked program needs." +blurb: "TCP and UDP sockets from first principles to an epoll event loop and a two-VM peer-to-peer chat mesh, plus the time, timers, and deadline discipline every networked program needs." --- diff --git a/_parts/07-concurrency-in-depth.md b/_parts/07-concurrency-in-depth.md index fd23306..ee33472 100644 --- a/_parts/07-concurrency-in-depth.md +++ b/_parts/07-concurrency-in-depth.md @@ -2,5 +2,5 @@ title: "Concurrency in Depth" order: 7 part_name: "Concurrency in Depth" -blurb: "Down to the futex, up to lock-free rings and memory ordering, and out to async — C++ coroutines on epoll, the Go scheduler, and tokio — and when plain threads still win. The C++ treatment of these topics has been extracted into its own dedicated, runnable book, C++ Concurrency and Parallelism (patterncatalyst.github.io/cpp-concurrency-and-parallelism); the chapters here now point to it." +blurb: "Down to the futex, up to lock-free rings and memory ordering, and out to async (C++ coroutines on epoll, the Go scheduler, and tokio), and when plain threads still win. The C++ treatment of these topics has been extracted into its own dedicated, runnable book, C++ Concurrency and Parallelism (patterncatalyst.github.io/cpp-concurrency-and-parallelism); the chapters here now point to it." --- diff --git a/_parts/09-containers-and-virtualization.md b/_parts/09-containers-and-virtualization.md index 4071ebc..54d27e4 100644 --- a/_parts/09-containers-and-virtualization.md +++ b/_parts/09-containers-and-virtualization.md @@ -2,5 +2,5 @@ title: "Containers and Virtualization" order: 9 part_name: "Containers and Virtualization" -blurb: "Namespaces, cgroup v2, seccomp, and Landlock as APIs our programs call — then our daemons living inside Podman containers and the real trade-offs of containers versus KVM virtual machines." +blurb: "Namespaces, cgroup v2, seccomp, and Landlock as APIs our programs call, then our daemons living inside Podman containers and the real trade-offs of containers versus KVM virtual machines." --- diff --git a/_parts/10-observability.md b/_parts/10-observability.md index e177ffc..8363ada 100644 --- a/_parts/10-observability.md +++ b/_parts/10-observability.md @@ -2,5 +2,5 @@ title: "Observability" order: 10 part_name: "Observability" -blurb: "From /proc, /sys, and cgroup stats through the USE method and eBPF tools to OpenTelemetry in all three languages, flowing into the local LGTM stack — with Cockpit and Performance Co-Pilot alongside." +blurb: "From /proc, /sys, and cgroup stats through the USE method and eBPF tools to OpenTelemetry in all three languages, flowing into the local LGTM stack, with Cockpit and Performance Co-Pilot alongside." --- diff --git a/_parts/12-deep-dives.md b/_parts/12-deep-dives.md index 6e61cd9..c9a3537 100644 --- a/_parts/12-deep-dives.md +++ b/_parts/12-deep-dives.md @@ -2,5 +2,5 @@ title: "Deep Dives" order: 12 part_name: "Deep Dives" -blurb: "Language-specific excursions — embedding Lua as a policy engine in all three languages, Rust macros for systems code, and the Go runtime from a systems programmer's seat." +blurb: "Language-specific excursions: embedding Lua as a policy engine in all three languages, Rust macros for systems code, and the Go runtime from a systems programmer's seat." --- diff --git a/_parts/13-appendices-tooling.md b/_parts/13-appendices-tooling.md index 23680de..2d03f4c 100644 --- a/_parts/13-appendices-tooling.md +++ b/_parts/13-appendices-tooling.md @@ -2,5 +2,5 @@ title: "Appendices: Tooling" order: 13 part_name: "Appendices: Tooling" -blurb: "Reference-depth guides to the analysis suites — Cockpit, SystemTap, and Performance Co-Pilot — and the full C++, Go, and Rust toolboxes that keep this book's examples sharp." +blurb: "Reference-depth guides to the analysis suites (Cockpit, SystemTap, and Performance Co-Pilot) and the full C++, Go, and Rust toolboxes that keep this book's examples sharp." --- diff --git a/_parts/14-compendium-cpp-concurrency.md b/_parts/14-compendium-cpp-concurrency.md index 24f0d6c..8fd2cd1 100644 --- a/_parts/14-compendium-cpp-concurrency.md +++ b/_parts/14-compendium-cpp-concurrency.md @@ -2,5 +2,5 @@ title: "Compendium: C++ Concurrency" order: 14 part_name: "Compendium: C++ Concurrency" -blurb: "A C++-only compendium that takes one workload through every concurrency model the language offers — pthreads, std threading, Boost.Thread, C++20 coroutines, Boost.Fiber, Boost.Asio. This compendium has been extracted into its own dedicated, runnable book, C++ Concurrency and Parallelism (patterncatalyst.github.io/cpp-concurrency-and-parallelism); the chapters here now point to it." +blurb: "A C++-only compendium that takes one workload through every concurrency model the language offers: pthreads, std threading, Boost.Thread, C++20 coroutines, Boost.Fiber, Boost.Asio. This compendium has been extracted into its own dedicated, runnable book, C++ Concurrency and Parallelism (patterncatalyst.github.io/cpp-concurrency-and-parallelism); the chapters here now point to it." --- diff --git a/assets/diagrams/19-scm-rights-transfer.excalidraw b/assets/diagrams/19-scm-rights-transfer.excalidraw index c3a767f..6f231a3 100644 --- a/assets/diagrams/19-scm-rights-transfer.excalidraw +++ b/assets/diagrams/19-scm-rights-transfer.excalidraw @@ -1491,13 +1491,13 @@ "updated": 1, "link": null, "locked": false, - "text": "ch07's three layers: per-process slot (cloexec bit lives here) \u2192 shared description (pos, flags) \u2192 inode", + "text": "The three layers: per-process slot (cloexec bit lives here) \u2192 shared description (pos, flags) \u2192 inode", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch07's three layers: per-process slot (cloexec bit lives here) \u2192 shared description (pos, flags) \u2192 inode", + "originalText": "The three layers: per-process slot (cloexec bit lives here) \u2192 shared description (pos, flags) \u2192 inode", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/19-scm-rights-transfer.svg b/assets/diagrams/19-scm-rights-transfer.svg index f263dd0..f7b2f85 100644 --- a/assets/diagrams/19-scm-rights-transfer.svg +++ b/assets/diagrams/19-scm-rights-transfer.svg @@ -49,6 +49,6 @@ kernel installs new slot kernel installs new slot SCM_RIGHTS: the fd number does not travel: a reference to the open file description does -ch07's three layers: per-process slot (cloexec bit lives here) → shared description (pos, flags) → inode +The three layers: per-process slot (cloexec bit lives here) → shared description (pos, flags) → inode numbers from the chapter's held-open run \ No newline at end of file diff --git a/assets/diagrams/21-frame-format.excalidraw b/assets/diagrams/21-frame-format.excalidraw index 8a8e464..8a32002 100644 --- a/assets/diagrams/21-frame-format.excalidraw +++ b/assets/diagrams/21-frame-format.excalidraw @@ -1470,13 +1470,13 @@ "updated": 1, "link": null, "locked": false, - "text": "JOIN name NUL (empty text) -- client sends exactly one at connect", + "text": "JOIN name NUL (empty text): client sends exactly one at connect", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "JOIN name NUL (empty text) -- client sends exactly one at connect", + "originalText": "JOIN name NUL (empty text): client sends exactly one at connect", "lineHeight": 1.25, "baseline": 11 }, @@ -1505,13 +1505,13 @@ "updated": 1, "link": null, "locked": false, - "text": "CHAT name NUL text -- one per non-empty stdin line", + "text": "CHAT name NUL text : one per non-empty stdin line", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "CHAT name NUL text -- one per non-empty stdin line", + "originalText": "CHAT name NUL text : one per non-empty stdin line", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/21-frame-format.svg b/assets/diagrams/21-frame-format.svg index 1e72d3a..5de998f 100644 --- a/assets/diagrams/21-frame-format.svg +++ b/assets/diagrams/21-frame-format.svg @@ -45,7 +45,7 @@ text = hello bob 0f 00 00 00 = 0x0000000f = 15 = len("alice") + 1 (NUL) + len("hello bob") Two frame kinds, told apart only by whether text is empty: -JOIN name NUL (empty text) -- client sends exactly one at connect -CHAT name NUL text -- one per non-empty stdin line +JOIN name NUL (empty text): client sends exactly one at connect +CHAT name NUL text : one per non-empty stdin line len == 0 and len > 1048576 (1 MiB) are rejected; the 4 header bytes are never counted. \ No newline at end of file diff --git a/assets/diagrams/23-two-vm-topology.excalidraw b/assets/diagrams/23-two-vm-topology.excalidraw index d010d50..5e74042 100644 --- a/assets/diagrams/23-two-vm-topology.excalidraw +++ b/assets/diagrams/23-two-vm-topology.excalidraw @@ -724,13 +724,13 @@ "updated": 1, "link": null, "locked": false, - "text": "TCP :9241: one ch21 chat frame each way", + "text": "TCP :9241: one chat frame each way", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "TCP :9241: one ch21 chat frame each way", + "originalText": "TCP :9241: one chat frame each way", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/23-two-vm-topology.svg b/assets/diagrams/23-two-vm-topology.svg index 69a280a..a827c6e 100644 --- a/assets/diagrams/23-two-vm-topology.svg +++ b/assets/diagrams/23-two-vm-topology.svg @@ -22,7 +22,7 @@ beacon 34 B beacon 34 B -TCP :9241: one ch21 chat frame each way -TCP :9241: one ch21 chat frame each way +TCP :9241: one chat frame each way +TCP :9241: one chat frame each way each guest sends one beacon to the group and hears the other's: dedup by name, then dial TCP \ No newline at end of file diff --git a/assets/diagrams/28-remote-debug-topology.excalidraw b/assets/diagrams/28-remote-debug-topology.excalidraw index d9cd8e1..24aa7dc 100644 --- a/assets/diagrams/28-remote-debug-topology.excalidraw +++ b/assets/diagrams/28-remote-debug-topology.excalidraw @@ -1265,13 +1265,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Left: the local path this chapter's evidence is built on -- app crashes, systemd-coredump", + "text": "Left: the local path this chapter's evidence is built on (app crashes, systemd-coredump)", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Left: the local path this chapter's evidence is built on -- app crashes, systemd-coredump", + "originalText": "Left: the local path this chapter's evidence is built on (app crashes, systemd-coredump)", "lineHeight": 1.25, "baseline": 11 }, @@ -1335,13 +1335,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Right: the remote path -- gdbserver attaches on the VM, the host's gdb drives it over", + "text": "Right: the remote path (gdbserver attaches on the VM, the host's gdb drives it over)", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Right: the remote path -- gdbserver attaches on the VM, the host's gdb drives it over", + "originalText": "Right: the remote path (gdbserver attaches on the VM, the host's gdb drives it over)", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/28-remote-debug-topology.svg b/assets/diagrams/28-remote-debug-topology.svg index 0037d8a..0687f96 100644 --- a/assets/diagrams/28-remote-debug-topology.svg +++ b/assets/diagrams/28-remote-debug-topology.svg @@ -38,9 +38,9 @@ gdb remote serial protocol, TCP:2345 -Left: the local path this chapter's evidence is built on -- app crashes, systemd-coredump +Left: the local path this chapter's evidence is built on (app crashes, systemd-coredump) catches it, coredumpctl extracts core.app, gdb opens the core file directly. -Right: the remote path -- gdbserver attaches on the VM, the host's gdb drives it over +Right: the remote path (gdbserver attaches on the VM, the host's gdb drives it over) TCP with 'target remote'. No systems-target VM was up this session; not run here. gdbserver is preinstalled on the lab VM image (not on this host). \ No newline at end of file diff --git a/assets/diagrams/28-stack-frame-anatomy.excalidraw b/assets/diagrams/28-stack-frame-anatomy.excalidraw index c58ec6e..0c87d9c 100644 --- a/assets/diagrams/28-stack-frame-anatomy.excalidraw +++ b/assets/diagrams/28-stack-frame-anatomy.excalidraw @@ -217,13 +217,13 @@ "updated": 1, "link": null, "locked": false, - "text": "/usr/include/c++/16/print:164 -- reads *child.exit_slot here", + "text": "/usr/include/c++/16/print:164 (reads *child.exit_slot here)", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "/usr/include/c++/16/print:164 -- reads *child.exit_slot here", + "originalText": "/usr/include/c++/16/print:164 (reads *child.exit_slot here)", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/28-stack-frame-anatomy.svg b/assets/diagrams/28-stack-frame-anatomy.svg index fa00b87..9f92edf 100644 --- a/assets/diagrams/28-stack-frame-anatomy.svg +++ b/assets/diagrams/28-stack-frame-anatomy.svg @@ -7,7 +7,7 @@ (readelf/objdump --dwarf=decodedline) #0 std::println<int,int> (crash PC) -/usr/include/c++/16/print:164 -- reads *child.exit_slot here +/usr/include/c++/16/print:164 (reads *child.exit_slot here) #1 (anonymous namespace)::deref(child) main.cpp:52 child = {pid=4242, exit_slot=0x0} diff --git a/assets/diagrams/29-shadow-memory.excalidraw b/assets/diagrams/29-shadow-memory.excalidraw index 67f8689..05593f9 100644 --- a/assets/diagrams/29-shadow-memory.excalidraw +++ b/assets/diagrams/29-shadow-memory.excalidraw @@ -56,13 +56,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Application memory -- real addresses from this chapter's ASan `uaf` run", + "text": "Application memory: real addresses from this chapter's ASan `uaf` run", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Application memory -- real addresses from this chapter's ASan `uaf` run", + "originalText": "Application memory: real addresses from this chapter's ASan `uaf` run", "lineHeight": 1.25, "baseline": 12 }, @@ -119,13 +119,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Shadow memory -- 1 shadow byte encodes the state of 8 application bytes", + "text": "Shadow memory: 1 shadow byte encodes the state of 8 application bytes", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Shadow memory -- 1 shadow byte encodes the state of 8 application bytes", + "originalText": "Shadow memory: 1 shadow byte encodes the state of 8 application bytes", "lineHeight": 1.25, "baseline": 12 }, @@ -385,13 +385,13 @@ "updated": 1, "link": null, "locked": false, - "text": "new'd, then delete'd -- *p=7 UB here", + "text": "new'd, then delete'd: *p=7 UB here", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "new'd, then delete'd -- *p=7 UB here", + "originalText": "new'd, then delete'd: *p=7 UB here", "lineHeight": 1.25, "baseline": 11 }, @@ -980,13 +980,13 @@ "updated": 1, "link": null, "locked": false, - "text": "shadow_addr = (app_addr >> 3) + offset -- one comparison before every load/store, not a full CPU emulation", + "text": "shadow_addr = (app_addr >> 3) + offset: one comparison before every load/store, not a full CPU emulation", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "shadow_addr = (app_addr >> 3) + offset -- one comparison before every load/store, not a full CPU emulation", + "originalText": "shadow_addr = (app_addr >> 3) + offset: one comparison before every load/store, not a full CPU emulation", "lineHeight": 1.25, "baseline": 14 }, diff --git a/assets/diagrams/29-shadow-memory.svg b/assets/diagrams/29-shadow-memory.svg index 7e5b288..c8d4883 100644 --- a/assets/diagrams/29-shadow-memory.svg +++ b/assets/diagrams/29-shadow-memory.svg @@ -1,9 +1,9 @@ -Application memory -- real addresses from this chapter's ASan `uaf` run +Application memory: real addresses from this chapter's ASan `uaf` run -Shadow memory -- 1 shadow byte encodes the state of 8 application bytes +Shadow memory: 1 shadow byte encodes the state of 8 application bytes left redzone 0x7b29be3dff80..ffff @@ -11,7 +11,7 @@ int* p (4 bytes) 0x7b29be3e0010 -new'd, then delete'd -- *p=7 UB here +new'd, then delete'd: *p=7 UB here right redzone 0x7b29be3e0018..007f @@ -29,7 +29,7 @@ "Heap left redzone" (next block's guard) -shadow_addr = (app_addr >> 3) + offset -- one comparison before every load/store, not a full CPU emulation +shadow_addr = (app_addr >> 3) + offset: one comparison before every load/store, not a full CPU emulation >>3 + offset diff --git a/assets/diagrams/29-tool-coverage-matrix.excalidraw b/assets/diagrams/29-tool-coverage-matrix.excalidraw index c94d003..2e63fed 100644 --- a/assets/diagrams/29-tool-coverage-matrix.excalidraw +++ b/assets/diagrams/29-tool-coverage-matrix.excalidraw @@ -567,13 +567,13 @@ "updated": 1, "link": null, "locked": false, - "text": "0 real signal -- see below)", + "text": "0 real signal, see below)", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "0 real signal -- see below)", + "originalText": "0 real signal, see below)", "lineHeight": 1.25, "baseline": 11 }, @@ -2303,13 +2303,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Every cell is backed by a command in \"Build, run, observe\" / \"Cross-check\" -- none of this is asserted from memory.", + "text": "Every cell is backed by a command in \"Build, run, observe\" / \"Cross-check\": none of this is asserted from memory.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Every cell is backed by a command in \"Build, run, observe\" / \"Cross-check\" -- none of this is asserted from memory.", + "originalText": "Every cell is backed by a command in \"Build, run, observe\" / \"Cross-check\": none of this is asserted from memory.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/29-tool-coverage-matrix.svg b/assets/diagrams/29-tool-coverage-matrix.svg index 1527430..4d00e92 100644 --- a/assets/diagrams/29-tool-coverage-matrix.svg +++ b/assets/diagrams/29-tool-coverage-matrix.svg @@ -17,7 +17,7 @@ self-detected: leaked=1 goroutine valgrind: CONFUSED (630 errors, -0 real signal -- see below) +0 real signal, see below) valgrind: CAUGHT (65,536B lost) + miri (nightly): "memory leaked" @@ -70,5 +70,5 @@ prevented: Send/Sync rules reject the unsync'd share amber = a tool run this session caught the defect · dark = the language prevents it outright, no tool needed · plain = a real blind spot this session measured -Every cell is backed by a command in "Build, run, observe" / "Cross-check" -- none of this is asserted from memory. +Every cell is backed by a command in "Build, run, observe" / "Cross-check": none of this is asserted from memory. \ No newline at end of file diff --git a/assets/diagrams/34-ns-debugging-map.excalidraw b/assets/diagrams/34-ns-debugging-map.excalidraw index 7314c43..40b426a 100644 --- a/assets/diagrams/34-ns-debugging-map.excalidraw +++ b/assets/diagrams/34-ns-debugging-map.excalidraw @@ -1090,13 +1090,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Both paths independently confirm the same PID 1 identity -- podman exec from inside the container runtime, nsenter from the bare host.", + "text": "Both paths independently confirm the same PID 1 identity: podman exec from inside the container runtime, nsenter from the bare host.", "fontSize": 13, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Both paths independently confirm the same PID 1 identity -- podman exec from inside the container runtime, nsenter from the bare host.", + "originalText": "Both paths independently confirm the same PID 1 identity: podman exec from inside the container runtime, nsenter from the bare host.", "lineHeight": 1.25, "baseline": 13 } diff --git a/assets/diagrams/34-ns-debugging-map.svg b/assets/diagrams/34-ns-debugging-map.svg index 69ed163..7b2c2c7 100644 --- a/assets/diagrams/34-ns-debugging-map.svg +++ b/assets/diagrams/34-ns-debugging-map.svg @@ -33,5 +33,5 @@ -Both paths independently confirm the same PID 1 identity -- podman exec from inside the container runtime, nsenter from the bare host. +Both paths independently confirm the same PID 1 identity: podman exec from inside the container runtime, nsenter from the bare host. \ No newline at end of file diff --git a/assets/diagrams/34-pid1-signal-flow.excalidraw b/assets/diagrams/34-pid1-signal-flow.excalidraw index 79711ac..11f459c 100644 --- a/assets/diagrams/34-pid1-signal-flow.excalidraw +++ b/assets/diagrams/34-pid1-signal-flow.excalidraw @@ -504,13 +504,13 @@ "updated": 1, "link": null, "locked": false, - "text": "exit 0 -- measured: cpp 192ms, go 60ms, rust 101ms", + "text": "exit 0 (measured: cpp 192ms, go 60ms, rust 101ms)", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "exit 0 -- measured: cpp 192ms, go 60ms, rust 101ms", + "originalText": "exit 0 (measured: cpp 192ms, go 60ms, rust 101ms)", "lineHeight": 1.25, "baseline": 14 }, @@ -959,13 +959,13 @@ "updated": 1, "link": null, "locked": false, - "text": "go 60ms (dies anyway -- but no shutdown line, no cleanup)", + "text": "go 60ms (dies anyway, but no shutdown line, no cleanup)", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "go 60ms (dies anyway -- but no shutdown line, no cleanup)", + "originalText": "go 60ms (dies anyway, but no shutdown line, no cleanup)", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/34-pid1-signal-flow.svg b/assets/diagrams/34-pid1-signal-flow.svg index 7b688af..9a07182 100644 --- a/assets/diagrams/34-pid1-signal-flow.svg +++ b/assets/diagrams/34-pid1-signal-flow.svg @@ -15,7 +15,7 @@ print "app: shutting down (SIGTERM)" -exit 0 -- measured: cpp 192ms, go 60ms, rust 101ms +exit 0 (measured: cpp 192ms, go 60ms, rust 101ms) (3000ms grace budget) podman stop -t 2 -> SIGTERM to PID 1 @@ -29,7 +29,7 @@ handler at startup regardless of signal.Notify measured: cpp 2108ms, rust 2074ms (full grace, SIGKILLed) -go 60ms (dies anyway -- but no shutdown line, no cleanup) +go 60ms (dies anyway, but no shutdown line, no cleanup) diff --git a/assets/diagrams/36-agent-architecture.excalidraw b/assets/diagrams/36-agent-architecture.excalidraw index 1718165..0201f37 100644 --- a/assets/diagrams/36-agent-architecture.excalidraw +++ b/assets/diagrams/36-agent-architecture.excalidraw @@ -812,13 +812,13 @@ "updated": 1, "link": null, "locked": false, - "text": "ch37 lab VM", + "text": "lab VM", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch37 lab VM", + "originalText": "lab VM", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/36-agent-architecture.svg b/assets/diagrams/36-agent-architecture.svg index 7dfb15a..e19a5e2 100644 --- a/assets/diagrams/36-agent-architecture.svg +++ b/assets/diagrams/36-agent-architecture.svg @@ -24,7 +24,7 @@ mpstat -P ALL 1 not installed here -ch37 lab VM +lab VM free -m MemAvailable vs available diff --git a/assets/diagrams/42-host-script-boundary.excalidraw b/assets/diagrams/42-host-script-boundary.excalidraw index 5f02320..d51a7b4 100644 --- a/assets/diagrams/42-host-script-boundary.excalidraw +++ b/assets/diagrams/42-host-script-boundary.excalidraw @@ -1474,13 +1474,13 @@ "updated": 1, "link": null, "locked": false, - "text": "no 5.2+-only stdlib -- one policy.lua drives all three.", + "text": "no 5.2+-only stdlib: one policy.lua drives all three.", "fontSize": 11.5, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "no 5.2+-only stdlib -- one policy.lua drives all three.", + "originalText": "no 5.2+-only stdlib: one policy.lua drives all three.", "lineHeight": 1.25, "baseline": 11.5 }, diff --git a/assets/diagrams/42-host-script-boundary.svg b/assets/diagrams/42-host-script-boundary.svg index a78fc8d..fbedc09 100644 --- a/assets/diagrams/42-host-script-boundary.svg +++ b/assets/diagrams/42-host-script-boundary.svg @@ -46,7 +46,7 @@ pmon (three ports, one behavior) Lua 5.1 (gopher-lua) vs Lua 5.4 (sol2, mlua) forces the shared subset: no //, no goto, no <close>, no bitwise ops, -no 5.2+-only stdlib -- one policy.lua drives all three. +no 5.2+-only stdlib: one policy.lua drives all three. every (re)load builds a fresh sandboxed state from scratch \ No newline at end of file diff --git a/assets/diagrams/42-sighup-reload-lifecycle.excalidraw b/assets/diagrams/42-sighup-reload-lifecycle.excalidraw index 23df3af..a94eb3c 100644 --- a/assets/diagrams/42-sighup-reload-lifecycle.excalidraw +++ b/assets/diagrams/42-sighup-reload-lifecycle.excalidraw @@ -56,13 +56,13 @@ "updated": 1, "link": null, "locked": false, - "text": "one pmon process -- PID never changes", + "text": "one pmon process: PID never changes", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "one pmon process -- PID never changes", + "originalText": "one pmon process: PID never changes", "lineHeight": 1.25, "baseline": 12 }, @@ -1617,13 +1617,13 @@ "updated": 1, "link": null, "locked": false, - "text": "The process is NOT restarted -- pmon keeps its PID and child-tracking state;", + "text": "The process is NOT restarted: pmon keeps its PID and child-tracking state;", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "The process is NOT restarted -- pmon keeps its PID and child-tracking state;", + "originalText": "The process is NOT restarted: pmon keeps its PID and child-tracking state;", "lineHeight": 1.25, "baseline": 12 }, diff --git a/assets/diagrams/42-sighup-reload-lifecycle.svg b/assets/diagrams/42-sighup-reload-lifecycle.svg index 281e406..87460d8 100644 --- a/assets/diagrams/42-sighup-reload-lifecycle.svg +++ b/assets/diagrams/42-sighup-reload-lifecycle.svg @@ -1,7 +1,7 @@ -one pmon process -- PID never changes +one pmon process: PID never changes load v1 policy.lua v1.0.0 @@ -48,6 +48,6 @@ each waits for the same each waits for the same -The process is NOT restarted -- pmon keeps its PID and child-tracking state; +The process is NOT restarted: pmon keeps its PID and child-tracking state; only a fresh, sandboxed Lua state is built in place and the decision logic changes underneath it. \ No newline at end of file diff --git a/assets/diagrams/43-miri-bounds-check.excalidraw b/assets/diagrams/43-miri-bounds-check.excalidraw index f69cbf5..0bef5ec 100644 --- a/assets/diagrams/43-miri-bounds-check.excalidraw +++ b/assets/diagrams/43-miri-bounds-check.excalidraw @@ -1036,13 +1036,13 @@ "updated": 1, "link": null, "locked": false, - "text": "window [0..8) -- fully inside the 8-byte alloc", + "text": "window [0..8): fully inside the 8-byte alloc", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "window [0..8) -- fully inside the 8-byte alloc", + "originalText": "window [0..8): fully inside the 8-byte alloc", "lineHeight": 1.25, "baseline": 11 }, @@ -1204,13 +1204,13 @@ "updated": 1, "link": null, "locked": false, - "text": "window [4..12) -- 4 bytes past the 8-byte alloc", + "text": "window [4..12): 4 bytes past the 8-byte alloc", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "window [4..12) -- 4 bytes past the 8-byte alloc", + "originalText": "window [4..12): 4 bytes past the 8-byte alloc", "lineHeight": 1.25, "baseline": 11 }, @@ -1446,13 +1446,13 @@ "updated": 1, "link": null, "locked": false, - "text": "BUF: [u8; 8] -- one 8-byte stack allocation (miri: \"alloc25\")", + "text": "BUF: [u8; 8], one 8-byte stack allocation (miri: \"alloc25\")", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "BUF: [u8; 8] -- one 8-byte stack allocation (miri: \"alloc25\")", + "originalText": "BUF: [u8; 8], one 8-byte stack allocation (miri: \"alloc25\")", "lineHeight": 1.25, "baseline": 12 }, diff --git a/assets/diagrams/43-miri-bounds-check.svg b/assets/diagrams/43-miri-bounds-check.svg index 3c33094..c618bed 100644 --- a/assets/diagrams/43-miri-bounds-check.svg +++ b/assets/diagrams/43-miri-bounds-check.svg @@ -32,18 +32,18 @@ SOUND: read_record(buf, oob=false) read_unaligned::<Record>(ptr.add(0)) -window [0..8) -- fully inside the 8-byte alloc +window [0..8): fully inside the 8-byte alloc cargo +nightly miri run: exit 0, no UB --oob-read: read_record(buf, oob=true) read_unaligned::<Record>(ptr.add(4)) -window [4..12) -- 4 bytes past the 8-byte alloc +window [4..12): 4 bytes past the 8-byte alloc miri: "...only 4 bytes from the end of the allocation" (main.rs:57) -BUF: [u8; 8] -- one 8-byte stack allocation (miri: "alloc25") +BUF: [u8; 8], one 8-byte stack allocation (miri: "alloc25") end of the allocation past the end (never allocated) \ No newline at end of file diff --git a/assets/diagrams/43-proc-macro-pipeline.excalidraw b/assets/diagrams/43-proc-macro-pipeline.excalidraw index bd5accf..b706047 100644 --- a/assets/diagrams/43-proc-macro-pipeline.excalidraw +++ b/assets/diagrams/43-proc-macro-pipeline.excalidraw @@ -56,13 +56,13 @@ "updated": 1, "link": null, "locked": false, - "text": "app crate -- main.rs (built around the sysmacros expansion)", + "text": "app crate: main.rs (built around the sysmacros expansion)", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "app crate -- main.rs (built around the sysmacros expansion)", + "originalText": "app crate: main.rs (built around the sysmacros expansion)", "lineHeight": 1.25, "baseline": 12 }, @@ -119,13 +119,13 @@ "updated": 1, "link": null, "locked": false, - "text": "sysmacros -- a separate proc-macro crate, compiled FIRST", + "text": "sysmacros: a separate proc-macro crate, compiled FIRST", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "sysmacros -- a separate proc-macro crate, compiled FIRST", + "originalText": "sysmacros: a separate proc-macro crate, compiled FIRST", "lineHeight": 1.25, "baseline": 12 }, @@ -889,13 +889,13 @@ "updated": 1, "link": null, "locked": false, - "text": "/ ItemFn -- a real, typed AST", + "text": "/ ItemFn: a real, typed AST", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "/ ItemFn -- a real, typed AST", + "originalText": "/ ItemFn: a real, typed AST", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/43-proc-macro-pipeline.svg b/assets/diagrams/43-proc-macro-pipeline.svg index 112c6b4..ead7df6 100644 --- a/assets/diagrams/43-proc-macro-pipeline.svg +++ b/assets/diagrams/43-proc-macro-pipeline.svg @@ -1,9 +1,9 @@ -app crate -- main.rs (built around the sysmacros expansion) +app crate: main.rs (built around the sysmacros expansion) -sysmacros -- a separate proc-macro crate, compiled FIRST +sysmacros: a separate proc-macro crate, compiled FIRST main.rs (annotated source) #[derive(SysError)] @@ -26,7 +26,7 @@ syn::parse_macro_input! TokenStream -> DeriveInput -/ ItemFn -- a real, typed AST +/ ItemFn: a real, typed AST quote! { .. } AST -> new TokenStream2 diff --git a/assets/diagrams/45-analysis-suite-landscape.excalidraw b/assets/diagrams/45-analysis-suite-landscape.excalidraw index 85b091e..067088a 100644 --- a/assets/diagrams/45-analysis-suite-landscape.excalidraw +++ b/assets/diagrams/45-analysis-suite-landscape.excalidraw @@ -182,13 +182,13 @@ "updated": 1, "link": null, "locked": false, - "text": "PCP: pmcd + PMDAs (container, ch45)", + "text": "PCP: pmcd + PMDAs (container)", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "PCP: pmcd + PMDAs (container, ch45)", + "originalText": "PCP: pmcd + PMDAs (container)", "lineHeight": 1.25, "baseline": 12 }, @@ -1295,13 +1295,13 @@ "updated": 1, "link": null, "locked": false, - "text": "podman.sock (ch34)", + "text": "podman.sock", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "podman.sock (ch34)", + "originalText": "podman.sock", "lineHeight": 1.25, "baseline": 11 }, @@ -1393,13 +1393,13 @@ "updated": 1, "link": null, "locked": false, - "text": "super-privileged; ch30 on the VM", + "text": "super-privileged; on the VM", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "super-privileged; ch30 on the VM", + "originalText": "super-privileged; on the VM", "lineHeight": 1.25, "baseline": 11 }, @@ -1877,13 +1877,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Contrast: ch30's dyninst backend was inconclusive on this host", + "text": "Contrast: the dyninst backend was inconclusive on this host", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Contrast: ch30's dyninst backend was inconclusive on this host", + "originalText": "Contrast: the dyninst backend was inconclusive on this host", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/45-analysis-suite-landscape.svg b/assets/diagrams/45-analysis-suite-landscape.svg index ba9e6d6..7062832 100644 --- a/assets/diagrams/45-analysis-suite-landscape.svg +++ b/assets/diagrams/45-analysis-suite-landscape.svg @@ -5,7 +5,7 @@ SystemTap: kernel-module tracing (VM) -PCP: pmcd + PMDAs (container, ch45) +PCP: pmcd + PMDAs (container) browser https://<vm>:9090/ @@ -39,10 +39,10 @@ central metrics daemon pmda-podman -podman.sock (ch34) +podman.sock pmda-bcc: NOT shipped -super-privileged; ch30 on the VM +super-privileged; on the VM HTTP 200 (verified) HTTP 200 (verified) @@ -55,6 +55,6 @@ Web panels (Overview / Podman / Logs): described only no browser automation in this session. -Contrast: ch30's dyninst backend was inconclusive on this host +Contrast: the dyninst backend was inconclusive on this host the kernel-module path needed matching kernel-devel; now verified. \ No newline at end of file diff --git a/assets/diagrams/46-cmake-preset-graph.excalidraw b/assets/diagrams/46-cmake-preset-graph.excalidraw index 2a4ab5a..24ab5b4 100644 --- a/assets/diagrams/46-cmake-preset-graph.excalidraw +++ b/assets/diagrams/46-cmake-preset-graph.excalidraw @@ -1344,13 +1344,13 @@ "updated": 1, "link": null, "locked": false, - "text": "cross-ref: ch29 matrix", + "text": "cross-ref: tool coverage matrix", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "cross-ref: ch29 matrix", + "originalText": "cross-ref: tool coverage matrix", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/46-cmake-preset-graph.svg b/assets/diagrams/46-cmake-preset-graph.svg index 526d155..5715dcc 100644 --- a/assets/diagrams/46-cmake-preset-graph.svg +++ b/assets/diagrams/46-cmake-preset-graph.svg @@ -39,7 +39,7 @@ same file, no inherits verify: parity gate ① gdb: .gdbinit + printers -cross-ref: ch29 matrix +cross-ref: tool coverage matrix verify: UBSan gate verify: parity gate ① clang diagnostics (shown) diff --git a/assets/diagrams/48-rust-toolchain-pipeline.excalidraw b/assets/diagrams/48-rust-toolchain-pipeline.excalidraw index 532b6a2..84d0c9d 100644 --- a/assets/diagrams/48-rust-toolchain-pipeline.excalidraw +++ b/assets/diagrams/48-rust-toolchain-pipeline.excalidraw @@ -2040,13 +2040,13 @@ "updated": 1, "link": null, "locked": false, - "text": "FNV-1a is language-independent: the same 0x481984990deee5ff digest appears in the ch46 C++ and ch47 Go toolbox reports.", + "text": "FNV-1a is language-independent: the same 0x481984990deee5ff digest appears in the C++ and Go toolbox reports.", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "FNV-1a is language-independent: the same 0x481984990deee5ff digest appears in the ch46 C++ and ch47 Go toolbox reports.", + "originalText": "FNV-1a is language-independent: the same 0x481984990deee5ff digest appears in the C++ and Go toolbox reports.", "lineHeight": 1.25, "baseline": 12 } diff --git a/assets/diagrams/48-rust-toolchain-pipeline.svg b/assets/diagrams/48-rust-toolchain-pipeline.svg index 45d1c34..07f9507 100644 --- a/assets/diagrams/48-rust-toolchain-pipeline.svg +++ b/assets/diagrams/48-rust-toolchain-pipeline.svg @@ -58,5 +58,5 @@ Both gates are rust-native: cargo test and the deterministic report digest run inside the pinned toolchain, verified offline on this host. -FNV-1a is language-independent: the same 0x481984990deee5ff digest appears in the ch46 C++ and ch47 Go toolbox reports. +FNV-1a is language-independent: the same 0x481984990deee5ff digest appears in the C++ and Go toolbox reports. \ No newline at end of file diff --git a/assets/diagrams/49-concurrency-vs-parallelism.excalidraw b/assets/diagrams/49-concurrency-vs-parallelism.excalidraw index 4a08242..b18722a 100644 --- a/assets/diagrams/49-concurrency-vs-parallelism.excalidraw +++ b/assets/diagrams/49-concurrency-vs-parallelism.excalidraw @@ -1600,13 +1600,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Numbers are from real runs on the 16-CPU reference host. Nothing here is a duration: every observable is a count or a set cardinality (see ch39).", + "text": "Numbers are from real runs on the 16-CPU reference host. Nothing here is a duration: every observable is a count or a set cardinality.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Numbers are from real runs on the 16-CPU reference host. Nothing here is a duration: every observable is a count or a set cardinality (see ch39).", + "originalText": "Numbers are from real runs on the 16-CPU reference host. Nothing here is a duration: every observable is a count or a set cardinality.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/49-concurrency-vs-parallelism.svg b/assets/diagrams/49-concurrency-vs-parallelism.svg index 06e1ae1..ef43be5 100644 --- a/assets/diagrams/49-concurrency-vs-parallelism.svg +++ b/assets/diagrams/49-concurrency-vs-parallelism.svg @@ -46,5 +46,5 @@ Concurrency is STRUCTURE (how many tasks are in flight); parallelism is EXECUTION (how many CPUs are occupied). They are independent axes, which is why all four cells exist. -Numbers are from real runs on the 16-CPU reference host. Nothing here is a duration: every observable is a count or a set cardinality (see ch39). +Numbers are from real runs on the 16-CPU reference host. Nothing here is a duration: every observable is a count or a set cardinality. \ No newline at end of file diff --git a/assets/diagrams/49-execution-model-lanes.excalidraw b/assets/diagrams/49-execution-model-lanes.excalidraw index bf1f27a..7dfc29b 100644 --- a/assets/diagrams/49-execution-model-lanes.excalidraw +++ b/assets/diagrams/49-execution-model-lanes.excalidraw @@ -182,13 +182,13 @@ "updated": 1, "link": null, "locked": false, - "text": "C++26 in ch49: measured, not assumed", + "text": "C++26: measured, not assumed", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "C++26 in ch49: measured, not assumed", + "originalText": "C++26: measured, not assumed", "lineHeight": 1.25, "baseline": 14 }, @@ -1204,13 +1204,13 @@ "updated": 1, "link": null, "locked": false, - "text": "live comparison deferred to ch56", + "text": "live comparison deferred", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "live comparison deferred to ch56", + "originalText": "live comparison deferred", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/49-execution-model-lanes.svg b/assets/diagrams/49-execution-model-lanes.svg index 483f2ed..0d4a3a7 100644 --- a/assets/diagrams/49-execution-model-lanes.svg +++ b/assets/diagrams/49-execution-model-lanes.svg @@ -5,7 +5,7 @@ Not implemented here: forward-looking prose, no gate, no committed dead code -C++26 in ch49: measured, not assumed +C++26: measured, not assumed Contracts: P2900 -fcontracts, __cpp_contracts=202502 @@ -35,7 +35,7 @@ GCC 16's <execution> is the C++17 parallel-algorithms header (__cpp_lib_execution = 201902) -live comparison deferred to ch56 +live comparison deferred Static reflection: P2996 __cpp_impl_reflection = undefined diff --git a/assets/diagrams/50-thread-two-identities.excalidraw b/assets/diagrams/50-thread-two-identities.excalidraw index 4821185..cefec54 100644 --- a/assets/diagrams/50-thread-two-identities.excalidraw +++ b/assets/diagrams/50-thread-two-identities.excalidraw @@ -1683,13 +1683,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Every value shown is from a real run on the 16-CPU reference host (glibc 2.43). Nothing here is a duration; see ch39.", + "text": "Every value shown is from a real run on the 16-CPU reference host (glibc 2.43). Nothing here is a duration.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Every value shown is from a real run on the 16-CPU reference host (glibc 2.43). Nothing here is a duration; see ch39.", + "originalText": "Every value shown is from a real run on the 16-CPU reference host (glibc 2.43). Nothing here is a duration.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/50-thread-two-identities.svg b/assets/diagrams/50-thread-two-identities.svg index 7ffeda1..44260ff 100644 --- a/assets/diagrams/50-thread-two-identities.svg +++ b/assets/diagrams/50-thread-two-identities.svg @@ -49,5 +49,5 @@ std::thread wraps a pthread and exposes almost none of it. Everything in the lower band is reachable, but only through native_handle(): which is why the standard put it there. pthread_t and gettid() are not two spellings of one id: one is a glibc handle you may only compare, the other is the kernel's task id that /proc, perf, and gdb all speak. -Every value shown is from a real run on the 16-CPU reference host (glibc 2.43). Nothing here is a duration; see ch39. +Every value shown is from a real run on the 16-CPU reference host (glibc 2.43). Nothing here is a duration. \ No newline at end of file diff --git a/assets/diagrams/51-futex-cost-model.excalidraw b/assets/diagrams/51-futex-cost-model.excalidraw index 7a89d7a..e686351 100644 --- a/assets/diagrams/51-futex-cost-model.excalidraw +++ b/assets/diagrams/51-futex-cost-model.excalidraw @@ -119,13 +119,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Kernel: futex(2), the syscall ch25 built by hand", + "text": "Kernel: futex(2), the syscall built by hand", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Kernel: futex(2), the syscall ch25 built by hand", + "originalText": "Kernel: futex(2), the syscall built by hand", "lineHeight": 1.25, "baseline": 12 }, @@ -1669,13 +1669,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Counted from outside with strace -f -c -e trace=futex. Nothing here is a duration; only the SIGN of the comparison is gated (see ch39).", + "text": "Counted from outside with strace -f -c -e trace=futex. Nothing here is a duration; only the SIGN of the comparison is gated.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Counted from outside with strace -f -c -e trace=futex. Nothing here is a duration; only the SIGN of the comparison is gated (see ch39).", + "originalText": "Counted from outside with strace -f -c -e trace=futex. Nothing here is a duration; only the SIGN of the comparison is gated.", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/51-futex-cost-model.svg b/assets/diagrams/51-futex-cost-model.svg index 6c37c53..1df0235 100644 --- a/assets/diagrams/51-futex-cost-model.svg +++ b/assets/diagrams/51-futex-cost-model.svg @@ -3,7 +3,7 @@ User space: atomic compare-exchange on the futex word, no kernel involved -Kernel: futex(2), the syscall ch25 built by hand +Kernel: futex(2), the syscall built by hand 200,000 lock/unlock pairs, two ways @@ -49,6 +49,6 @@ Same total work in all three cost cases: 200,000 increments: so only the contention differs, which is what makes the syscall counts comparable. An uncontended std::mutex costs no syscalls at all. What costs is contention: identical work, two orders of magnitude more kernel involvement. -Counted from outside with strace -f -c -e trace=futex. Nothing here is a duration; only the SIGN of the comparison is gated (see ch39). +Counted from outside with strace -f -c -e trace=futex. Nothing here is a duration; only the SIGN of the comparison is gated. Measured on the Fedora 44 reference host, glibc 2.43, GCC 16.1.1. \ No newline at end of file diff --git a/assets/diagrams/52-cancellation-three-ways.excalidraw b/assets/diagrams/52-cancellation-three-ways.excalidraw index d91ff20..1dc1978 100644 --- a/assets/diagrams/52-cancellation-three-ways.excalidraw +++ b/assets/diagrams/52-cancellation-three-ways.excalidraw @@ -119,13 +119,13 @@ "updated": 1, "link": null, "locked": false, - "text": "cancellation across ch50, ch52, ch51", + "text": "cancellation across three approaches", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "cancellation across ch50, ch52, ch51", + "originalText": "cancellation across three approaches", "lineHeight": 1.25, "baseline": 14 }, @@ -182,13 +182,13 @@ "updated": 1, "link": null, "locked": false, - "text": "ch50: pthread_cancel", + "text": "pthread_cancel", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch50: pthread_cancel", + "originalText": "pthread_cancel", "lineHeight": 1.25, "baseline": 14 }, @@ -385,13 +385,13 @@ "updated": 1, "link": null, "locked": false, - "text": "ch52: thread::interrupt()", + "text": "thread::interrupt()", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch52: thread::interrupt()", + "originalText": "thread::interrupt()", "lineHeight": 1.25, "baseline": 14 }, @@ -588,13 +588,13 @@ "updated": 1, "link": null, "locked": false, - "text": "ch51: stop_token", + "text": "stop_token", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch51: stop_token", + "originalText": "stop_token", "lineHeight": 1.25, "baseline": 14 }, @@ -1389,13 +1389,13 @@ "updated": 1, "link": null, "locked": false, - "text": "All three rows are real runs on this host. The ch50 abort and the ch52 survival are both hard-gated, in their own chapters' harnesses.", + "text": "All three rows are real runs on this host. The pthread_cancel abort and the thread::interrupt() survival are both hard-gated, in their own chapters' harnesses.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "All three rows are real runs on this host. The ch50 abort and the ch52 survival are both hard-gated, in their own chapters' harnesses.", + "originalText": "All three rows are real runs on this host. The pthread_cancel abort and the thread::interrupt() survival are both hard-gated, in their own chapters' harnesses.", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/52-cancellation-three-ways.svg b/assets/diagrams/52-cancellation-three-ways.svg index e683b45..7464e1b 100644 --- a/assets/diagrams/52-cancellation-three-ways.svg +++ b/assets/diagrams/52-cancellation-three-ways.svg @@ -3,21 +3,21 @@ One mistake: catch (...) without a rethrow: against three cancellation models -cancellation across ch50, ch52, ch51 +cancellation across three approaches -ch50: pthread_cancel +pthread_cancel glibc FORCED UNWIND destructors DO run but the unwind must propagate asynchronous: any cancellation point -ch52: thread::interrupt() +thread::interrupt() an ORDINARY C++ exception boost::thread_interrupted thrown at defined points only sleep_for, cv wait, join, … -ch51: stop_token +stop_token a FLAG, nothing thrown request_stop() sets it the target polls when it likes @@ -40,7 +40,7 @@ -All three rows are real runs on this host. The ch50 abort and the ch52 survival are both hard-gated, in their own chapters' harnesses. +All three rows are real runs on this host. The pthread_cancel abort and the thread::interrupt() survival are both hard-gated, in their own chapters' harnesses. The difference is not how careful the caller is: it is what the mechanism IS. A forced unwind must propagate; an ordinary exception need not; a flag cannot be caught. Power runs left to right; safety runs right to left. Boost.Thread is the only one of the three that is both catchable and non-fatal when mishandled. \ No newline at end of file diff --git a/assets/diagrams/52-what-the-standard-took.excalidraw b/assets/diagrams/52-what-the-standard-took.excalidraw index 726789e..4ddd431 100644 --- a/assets/diagrams/52-what-the-standard-took.excalidraw +++ b/assets/diagrams/52-what-the-standard-took.excalidraw @@ -1561,13 +1561,13 @@ "updated": 1, "link": null, "locked": false, - "text": "as ch50's POSIX call", + "text": "as the POSIX call", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "as ch50's POSIX call", + "originalText": "as the POSIX call", "lineHeight": 1.25, "baseline": 11 }, @@ -1850,13 +1850,13 @@ "updated": 1, "link": null, "locked": false, - "text": "C++26's std::execution is the standard's answer to the composition gap: forward-looking in ch49, compared directly in ch56.", + "text": "C++26's std::execution is the standard's answer to the composition gap: forward-looking, and compared directly against the other six models.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "C++26's std::execution is the standard's answer to the composition gap: forward-looking in ch49, compared directly in ch56.", + "originalText": "C++26's std::execution is the standard's answer to the composition gap: forward-looking, and compared directly against the other six models.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/52-what-the-standard-took.svg b/assets/diagrams/52-what-the-standard-took.svg index a8a30e2..f000134 100644 --- a/assets/diagrams/52-what-the-standard-took.svg +++ b/assets/diagrams/52-what-the-standard-took.svg @@ -46,7 +46,7 @@ set_stack_size(256 KiB) → observed 262144, the same byte-for-byte -as ch50's POSIX call +as the POSIX call std::upgrade_lock does not exist: not absent, undeclared @@ -54,5 +54,5 @@ C++11 standardized Boost.Thread's types nearly unchanged. What it did not take is specific and still missing 15 years later: composition, the upgrade lock, and per-thread attributes. The .then() contrast is checked by the compiler on every run: a concept evaluated against std::future<int> and boost::future<int>: rather than asserted by the author once. Measured on the Fedora 44 reference host: Boost 1.90.0 from boost-devel, BOOST_THREAD_VERSION=5, GCC 16.1.1. No Conan, no network. -C++26's std::execution is the standard's answer to the composition gap: forward-looking in ch49, compared directly in ch56. +C++26's std::execution is the standard's answer to the composition gap: forward-looking, and compared directly against the other six models. \ No newline at end of file diff --git a/assets/diagrams/53-frame-versus-stack.excalidraw b/assets/diagrams/53-frame-versus-stack.excalidraw index 5e5f838..5c9bde6 100644 --- a/assets/diagrams/53-frame-versus-stack.excalidraw +++ b/assets/diagrams/53-frame-versus-stack.excalidraw @@ -56,13 +56,13 @@ "updated": 1, "link": null, "locked": false, - "text": "A thread: ch50 measured this", + "text": "A thread: measured here", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "A thread: ch50 measured this", + "originalText": "A thread: measured here", "lineHeight": 1.25, "baseline": 12 }, @@ -1584,13 +1584,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Thread stack from ch50 (pthread_getattr_np); frame sizes from this chapter. GCC 16.1.1, C++23, Fedora 44. Nothing here is a duration.", + "text": "Thread stack (pthread_getattr_np); frame sizes from this chapter. GCC 16.1.1, C++23, Fedora 44. Nothing here is a duration.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Thread stack from ch50 (pthread_getattr_np); frame sizes from this chapter. GCC 16.1.1, C++23, Fedora 44. Nothing here is a duration.", + "originalText": "Thread stack (pthread_getattr_np); frame sizes from this chapter. GCC 16.1.1, C++23, Fedora 44. Nothing here is a duration.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/53-frame-versus-stack.svg b/assets/diagrams/53-frame-versus-stack.svg index 9acb10f..d6f55ce 100644 --- a/assets/diagrams/53-frame-versus-stack.svg +++ b/assets/diagrams/53-frame-versus-stack.svg @@ -1,7 +1,7 @@ -A thread: ch50 measured this +A thread: measured here A coroutine frame: measured here, via operator new in the promise @@ -46,5 +46,5 @@ Both hold one paused computation. Only one of them reserves 8 MiB of address space and a kernel task to do it: which is the whole reason coroutines exist. Frame sizes are measured by overloading operator new inside promise_type: the only portable way to see a size the language hides. -Thread stack from ch50 (pthread_getattr_np); frame sizes from this chapter. GCC 16.1.1, C++23, Fedora 44. Nothing here is a duration. +Thread stack (pthread_getattr_np); frame sizes from this chapter. GCC 16.1.1, C++23, Fedora 44. Nothing here is a duration. \ No newline at end of file diff --git a/assets/diagrams/54-scheduler-is-a-choice.excalidraw b/assets/diagrams/54-scheduler-is-a-choice.excalidraw index 20432a7..e9340f7 100644 --- a/assets/diagrams/54-scheduler-is-a-choice.excalidraw +++ b/assets/diagrams/54-scheduler-is-a-choice.excalidraw @@ -378,13 +378,13 @@ "updated": 1, "link": null, "locked": false, - "text": "ch49's vocabulary: 16 tasks in flight,", + "text": "Shared vocabulary: 16 tasks in flight,", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch49's vocabulary: 16 tasks in flight,", + "originalText": "Shared vocabulary: 16 tasks in flight,", "lineHeight": 1.25, "baseline": 14 }, @@ -742,13 +742,13 @@ "updated": 1, "link": null, "locked": false, - "text": "cell of ch49's grid, reached by choosing it", + "text": "cell of the grid, reached by choosing it", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "cell of ch49's grid, reached by choosing it", + "originalText": "cell of the grid, reached by choosing it", "lineHeight": 1.25, "baseline": 11 }, @@ -938,13 +938,13 @@ "updated": 1, "link": null, "locked": false, - "text": "one line: and it is yours to write, not a runtime's to decide (contrast ch44's Go)", + "text": "one line: and it is yours to write, not a runtime's to decide (contrast Go)", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "one line: and it is yours to write, not a runtime's to decide (contrast ch44's Go)", + "originalText": "one line: and it is yours to write, not a runtime's to decide (contrast Go)", "lineHeight": 1.25, "baseline": 11 }, @@ -1016,13 +1016,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Both columns are real runs, tids collected with gettid(): the same kernel task id ch50 established. shared_work reported 4 of 4 on five consecutive runs.", + "text": "Both columns are real runs, tids collected with gettid(): the same kernel task id established earlier. shared_work reported 4 of 4 on five consecutive runs.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Both columns are real runs, tids collected with gettid(): the same kernel task id ch50 established. shared_work reported 4 of 4 on five consecutive runs.", + "originalText": "Both columns are real runs, tids collected with gettid(): the same kernel task id established earlier. shared_work reported 4 of 4 on five consecutive runs.", "lineHeight": 1.25, "baseline": 11 }, @@ -1051,13 +1051,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Go's runtime makes this decision for you and mostly will not be argued with (ch44). Boost.Fiber makes you choose, which is more work and more control.", + "text": "Go's runtime makes this decision for you and mostly will not be argued with. Boost.Fiber makes you choose, which is more work and more control.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Go's runtime makes this decision for you and mostly will not be argued with (ch44). Boost.Fiber makes you choose, which is more work and more control.", + "originalText": "Go's runtime makes this decision for you and mostly will not be argued with. Boost.Fiber makes you choose, which is more work and more control.", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/54-scheduler-is-a-choice.svg b/assets/diagrams/54-scheduler-is-a-choice.svg index 168df3e..9735916 100644 --- a/assets/diagrams/54-scheduler-is-a-choice.svg +++ b/assets/diagrams/54-scheduler-is-a-choice.svg @@ -11,7 +11,7 @@ a fiber never leaves the thread that created it distinct_tids = 1 -ch49's vocabulary: 16 tasks in flight, +Shared vocabulary: 16 tasks in flight, one CPU occupied: concurrent, NOT parallel and that is a legitimate choice: no migration means @@ -22,15 +22,15 @@ distinct_tids = 4 concurrent AND parallel: the bottom-right -cell of ch49's grid, reached by choosing it +cell of the grid, reached by choosing it the price: your fiber's data is now touched by several threads, and you own that problem boost::fibers::use_scheduling_algorithm<algo::shared_work>(); -one line: and it is yours to write, not a runtime's to decide (contrast ch44's Go) +one line: and it is yours to write, not a runtime's to decide (contrast Go) -Both columns are real runs, tids collected with gettid(): the same kernel task id ch50 established. shared_work reported 4 of 4 on five consecutive runs. -Go's runtime makes this decision for you and mostly will not be argued with (ch44). Boost.Fiber makes you choose, which is more work and more control. +Both columns are real runs, tids collected with gettid(): the same kernel task id established earlier. shared_work reported 4 of 4 on five consecutive runs. +Go's runtime makes this decision for you and mostly will not be argued with. Boost.Fiber makes you choose, which is more work and more control. verify.lua gates round_robin at exactly 1 tid and shared_work at more than 1: the sign, never the magnitude, since the count is scheduler-dependent. \ No newline at end of file diff --git a/assets/diagrams/54-three-ways-to-pause.excalidraw b/assets/diagrams/54-three-ways-to-pause.excalidraw index 2c480e0..a246db3 100644 --- a/assets/diagrams/54-three-ways-to-pause.excalidraw +++ b/assets/diagrams/54-three-ways-to-pause.excalidraw @@ -308,13 +308,13 @@ "updated": 1, "link": null, "locked": false, - "text": "thread: ch50", + "text": "thread", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "thread: ch50", + "originalText": "thread", "lineHeight": 1.25, "baseline": 14 }, @@ -504,13 +504,13 @@ "updated": 1, "link": null, "locked": false, - "text": "fiber: ch54", + "text": "fiber", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "fiber: ch54", + "originalText": "fiber", "lineHeight": 1.25, "baseline": 14 }, @@ -770,13 +770,13 @@ "updated": 1, "link": null, "locked": false, - "text": "coroutine frame: ch53", + "text": "coroutine frame", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "coroutine frame: ch53", + "originalText": "coroutine frame", "lineHeight": 1.25, "baseline": 14 }, @@ -1227,13 +1227,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Thread stack from ch50 (pthread_getattr_np); frame from ch53 (operator new in promise_type); fiber from stack_traits::default_size(). Boost 1.90.0, Fedora 44.", + "text": "Thread stack (pthread_getattr_np); frame (operator new in promise_type); fiber from stack_traits::default_size(). Boost 1.90.0, Fedora 44.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Thread stack from ch50 (pthread_getattr_np); frame from ch53 (operator new in promise_type); fiber from stack_traits::default_size(). Boost 1.90.0, Fedora 44.", + "originalText": "Thread stack (pthread_getattr_np); frame (operator new in promise_type); fiber from stack_traits::default_size(). Boost 1.90.0, Fedora 44.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/54-three-ways-to-pause.svg b/assets/diagrams/54-three-ways-to-pause.svg index 4954509..c92b509 100644 --- a/assets/diagrams/54-three-ways-to-pause.svg +++ b/assets/diagrams/54-three-ways-to-pause.svg @@ -9,13 +9,13 @@ one paused computation, three prices -thread: ch50 +thread 8388608 bytes of stack preempted anywhere · a tid in /proc · a kernel task_struct suspends wherever the scheduler decides, with no cooperation needed -fiber: ch54 +fiber 131072 bytes (128 KiB) 64 fit in one thread stack @@ -23,7 +23,7 @@ no co_await, no coroutine, no function colouring. The stack is exactly what buys that. -coroutine frame: ch53 +coroutine frame 32 bytes 4096 fit in one fiber stack @@ -38,5 +38,5 @@ ÷4096 ÷4096 Cost and capability move together, and in opposite directions: the more a paused computation costs, the fewer restrictions there are on where it may pause. -Thread stack from ch50 (pthread_getattr_np); frame from ch53 (operator new in promise_type); fiber from stack_traits::default_size(). Boost 1.90.0, Fedora 44. +Thread stack (pthread_getattr_np); frame (operator new in promise_type); fiber from stack_traits::default_size(). Boost 1.90.0, Fedora 44. \ No newline at end of file diff --git a/assets/diagrams/55-strand-not-a-mutex.excalidraw b/assets/diagrams/55-strand-not-a-mutex.excalidraw index 12f8688..d951e86 100644 --- a/assets/diagrams/55-strand-not-a-mutex.excalidraw +++ b/assets/diagrams/55-strand-not-a-mutex.excalidraw @@ -1038,13 +1038,13 @@ "updated": 1, "link": null, "locked": false, - "text": "Futex counts measured with ch51's method (strace -f -c -e trace=futex). Gated as a ratio, never as magnitudes: they move with scheduling.", + "text": "Futex counts measured with the same method (strace -f -c -e trace=futex). Gated as a ratio, never as magnitudes: they move with scheduling.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Futex counts measured with ch51's method (strace -f -c -e trace=futex). Gated as a ratio, never as magnitudes: they move with scheduling.", + "originalText": "Futex counts measured with the same method (strace -f -c -e trace=futex). Gated as a ratio, never as magnitudes: they move with scheduling.", "lineHeight": 1.25, "baseline": 11 }, diff --git a/assets/diagrams/55-strand-not-a-mutex.svg b/assets/diagrams/55-strand-not-a-mutex.svg index 07d3d83..ec18161 100644 --- a/assets/diagrams/55-strand-not-a-mutex.svg +++ b/assets/diagrams/55-strand-not-a-mutex.svg @@ -30,7 +30,7 @@ Both mechanisms produce the correct answer. The difference is what happens to a thread that loses the race: a mutex puts it to sleep, a strand gives it something else to do. -Futex counts measured with ch51's method (strace -f -c -e trace=futex). Gated as a ratio, never as magnitudes: they move with scheduling. +Futex counts measured with the same method (strace -f -c -e trace=futex). Gated as a ratio, never as magnitudes: they move with scheduling. A strand costs no lock and no thread. What it costs instead is ordering: handlers on one strand can never run in parallel with each other, however many threads you have. Measured on the Fedora 44 reference host, Boost 1.90.0, GCC 16.1.1. \ No newline at end of file diff --git a/assets/diagrams/55-topologies-and-gather.excalidraw b/assets/diagrams/55-topologies-and-gather.excalidraw index 584887e..56e3d5d 100644 --- a/assets/diagrams/55-topologies-and-gather.excalidraw +++ b/assets/diagrams/55-topologies-and-gather.excalidraw @@ -917,13 +917,13 @@ "updated": 1, "link": null, "locked": false, - "text": "tids from gettid(): ch50's instrument and ch49's", + "text": "tids from gettid(): the same instrument used elsewhere", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "tids from gettid(): ch50's instrument and ch49's", + "originalText": "tids from gettid(): the same instrument used elsewhere", "lineHeight": 1.25, "baseline": 14 }, diff --git a/assets/diagrams/55-topologies-and-gather.svg b/assets/diagrams/55-topologies-and-gather.svg index 71c7ef4..79804a8 100644 --- a/assets/diagrams/55-topologies-and-gather.svg +++ b/assets/diagrams/55-topologies-and-gather.svg @@ -27,7 +27,7 @@ distinct_tids = 4 no run() call at all -tids from gettid(): ch50's instrument and ch49's +tids from gettid(): the same instrument used elsewhere vocabulary, applied to an I/O runtime asio::write(sock, array<const_buffer,4>) diff --git a/assets/diagrams/56-one-workload-seven-models.excalidraw b/assets/diagrams/56-one-workload-seven-models.excalidraw index ca27021..b3e43ee 100644 --- a/assets/diagrams/56-one-workload-seven-models.excalidraw +++ b/assets/diagrams/56-one-workload-seven-models.excalidraw @@ -4,7 +4,7 @@ "source": "https://excalidraw.com", "elements": [ { - "id": "r167974424", + "id": "r1526863660", "type": "rectangle", "x": 24, "y": 40, @@ -23,8 +23,8 @@ "roundness": { "type": 3 }, - "seed": 1121817902, - "versionNonce": 1637415825, + "seed": 523890650, + "versionNonce": 948966634, "isDeleted": false, "boundElements": [], "updated": 1, @@ -32,7 +32,7 @@ "locked": false }, { - "id": "t375371607", + "id": "t1969196893", "type": "text", "x": 36, "y": 46, @@ -49,8 +49,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 196965334, - "versionNonce": 1131880237, + "seed": 523982710, + "versionNonce": 1605942943, "isDeleted": false, "boundElements": [], "updated": 1, @@ -67,7 +67,7 @@ "baseline": 12 }, { - "id": "r876751167", + "id": "r955642466", "type": "rectangle", "x": 24, "y": 156, @@ -86,8 +86,8 @@ "roundness": { "type": 3 }, - "seed": 873789590, - "versionNonce": 308439798, + "seed": 720216250, + "versionNonce": 1044878403, "isDeleted": false, "boundElements": [], "updated": 1, @@ -95,11 +95,11 @@ "locked": false }, { - "id": "t188794020", + "id": "t901593011", "type": "text", "x": 36, "y": 162, - "width": 693.0, + "width": 640.2, "height": 15.0, "angle": 0, "strokeColor": "#555555", @@ -112,25 +112,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1853760776, - "versionNonce": 401983522, + "seed": 548173435, + "versionNonce": 1000189940, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "one OS thread: concurrent, not parallel (ch49). No lock anywhere, because there is nothing to race with.", + "text": "one OS thread: concurrent, not parallel. No lock anywhere, because there is nothing to race with.", "fontSize": 12, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "one OS thread: concurrent, not parallel (ch49). No lock anywhere, because there is nothing to race with.", + "originalText": "one OS thread: concurrent, not parallel. No lock anywhere, because there is nothing to race with.", "lineHeight": 1.25, "baseline": 12 }, { - "id": "r1033221426", + "id": "r1774738583", "type": "rectangle", "x": 24, "y": 340, @@ -149,8 +149,8 @@ "roundness": { "type": 3 }, - "seed": 410994584, - "versionNonce": 1925671363, + "seed": 1294486771, + "versionNonce": 781325685, "isDeleted": false, "boundElements": [], "updated": 1, @@ -158,11 +158,11 @@ "locked": false }, { - "id": "t743764797", + "id": "t1035004147", "type": "text", "x": 36, "y": 346, - "width": 567.6, + "width": 561.0, "height": 15.0, "angle": 0, "strokeColor": "#555555", @@ -175,8 +175,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1158581753, - "versionNonce": 147001409, + "seed": 1421560315, + "versionNonce": 183355870, "isDeleted": false, "boundElements": [], "updated": 1, @@ -193,7 +193,7 @@ "baseline": 12 }, { - "id": "r1636501502", + "id": "r1759347547", "type": "rectangle", "x": 24, "y": 536, @@ -212,8 +212,8 @@ "roundness": { "type": 3 }, - "seed": 211500241, - "versionNonce": 1282537832, + "seed": 448798583, + "versionNonce": 93294330, "isDeleted": false, "boundElements": [], "updated": 1, @@ -221,11 +221,11 @@ "locked": false }, { - "id": "t1707542695", + "id": "t229229469", "type": "text", "x": 36, "y": 542, - "width": 514.8000000000001, + "width": 508.20000000000005, "height": 15.0, "angle": 0, "strokeColor": "#555555", @@ -238,8 +238,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 811871611, - "versionNonce": 1138257526, + "seed": 26379313, + "versionNonce": 1370940784, "isDeleted": false, "boundElements": [], "updated": 1, @@ -256,7 +256,7 @@ "baseline": 12 }, { - "id": "r1720656908", + "id": "r1644227669", "type": "rectangle", "x": 180, "y": 70, @@ -275,8 +275,8 @@ "roundness": { "type": 3 }, - "seed": 668682772, - "versionNonce": 708247693, + "seed": 1060527565, + "versionNonce": 1554621548, "isDeleted": false, "boundElements": [], "updated": 1, @@ -284,7 +284,7 @@ "locked": false }, { - "id": "t1360268056", + "id": "t1933366757", "type": "text", "x": 190, "y": 78, @@ -301,8 +301,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 29261457, - "versionNonce": 1902850333, + "seed": 1333515802, + "versionNonce": 193510405, "isDeleted": false, "boundElements": [], "updated": 1, @@ -319,7 +319,7 @@ "baseline": 14 }, { - "id": "t1162649396", + "id": "t765097659", "type": "text", "x": 190, "y": 98, @@ -336,8 +336,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 859844436, - "versionNonce": 1024699016, + "seed": 839744307, + "versionNonce": 1938348952, "isDeleted": false, "boundElements": [], "updated": 1, @@ -354,7 +354,7 @@ "baseline": 11 }, { - "id": "r1430871690", + "id": "r197379885", "type": "rectangle", "x": 45, "y": 196, @@ -373,8 +373,8 @@ "roundness": { "type": 3 }, - "seed": 1905833941, - "versionNonce": 1808535231, + "seed": 1130324693, + "versionNonce": 1209420267, "isDeleted": false, "boundElements": [], "updated": 1, @@ -382,11 +382,11 @@ "locked": false }, { - "id": "t228067496", + "id": "t538976265", "type": "text", "x": 55, "y": 204, - "width": 192.50000000000003, + "width": 184.8, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -399,25 +399,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1278858314, - "versionNonce": 1523790003, + "seed": 25734013, + "versionNonce": 1445332811, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "sequential (the control)", + "text": "sequential (the control)", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "sequential (the control)", + "originalText": "sequential (the control)", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t1878607004", + "id": "t1859802204", "type": "text", "x": 55, "y": 224, @@ -434,8 +434,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1822900172, - "versionNonce": 1144562099, + "seed": 1950840694, + "versionNonce": 888789428, "isDeleted": false, "boundElements": [], "updated": 1, @@ -452,7 +452,7 @@ "baseline": 11 }, { - "id": "t115890586", + "id": "t1025444388", "type": "text", "x": 55, "y": 239, @@ -469,8 +469,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1524311250, - "versionNonce": 1302989627, + "seed": 1871072287, + "versionNonce": 1033409426, "isDeleted": false, "boundElements": [], "updated": 1, @@ -487,7 +487,7 @@ "baseline": 11 }, { - "id": "t1723014000", + "id": "t1115981681", "type": "text", "x": 55, "y": 254, @@ -504,8 +504,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 229928184, - "versionNonce": 1530188877, + "seed": 164853467, + "versionNonce": 1183569001, "isDeleted": false, "boundElements": [], "updated": 1, @@ -522,7 +522,7 @@ "baseline": 11 }, { - "id": "r565316136", + "id": "r907984220", "type": "rectangle", "x": 415, "y": 196, @@ -541,8 +541,8 @@ "roundness": { "type": 3 }, - "seed": 1537306129, - "versionNonce": 1388874134, + "seed": 1401914497, + "versionNonce": 338200483, "isDeleted": false, "boundElements": [], "updated": 1, @@ -550,11 +550,11 @@ "locked": false }, { - "id": "t464760957", + "id": "t709203523", "type": "text", "x": 425, "y": 204, - "width": 130.9, + "width": 69.30000000000001, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -567,25 +567,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1493788945, - "versionNonce": 512778867, + "seed": 798807069, + "versionNonce": 666014052, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "coroutine (ch53)", + "text": "coroutine", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "coroutine (ch53)", + "originalText": "coroutine", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t642048541", + "id": "t79921359", "type": "text", "x": 425, "y": 224, @@ -602,8 +602,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 405691406, - "versionNonce": 447587610, + "seed": 1953294502, + "versionNonce": 151624923, "isDeleted": false, "boundElements": [], "updated": 1, @@ -620,7 +620,7 @@ "baseline": 11 }, { - "id": "t399169544", + "id": "t1406352462", "type": "text", "x": 425, "y": 239, @@ -637,8 +637,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1814339704, - "versionNonce": 345545544, + "seed": 460239621, + "versionNonce": 370682201, "isDeleted": false, "boundElements": [], "updated": 1, @@ -655,7 +655,7 @@ "baseline": 11 }, { - "id": "t1621488498", + "id": "t686709391", "type": "text", "x": 425, "y": 254, @@ -672,8 +672,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 801344237, - "versionNonce": 1444634170, + "seed": 1412122829, + "versionNonce": 1773555042, "isDeleted": false, "boundElements": [], "updated": 1, @@ -690,7 +690,7 @@ "baseline": 11 }, { - "id": "r478256495", + "id": "r373521382", "type": "rectangle", "x": 785, "y": 196, @@ -709,8 +709,8 @@ "roundness": { "type": 3 }, - "seed": 565993739, - "versionNonce": 469576895, + "seed": 753536475, + "versionNonce": 1413861614, "isDeleted": false, "boundElements": [], "updated": 1, @@ -718,11 +718,11 @@ "locked": false }, { - "id": "t1405839658", + "id": "t1876621636", "type": "text", "x": 795, "y": 204, - "width": 100.10000000000001, + "width": 40, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -735,25 +735,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1374145194, - "versionNonce": 777379563, + "seed": 1421788823, + "versionNonce": 860519537, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "fiber (ch54)", + "text": "fiber", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "fiber (ch54)", + "originalText": "fiber", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t1142009846", + "id": "t1233089433", "type": "text", "x": 795, "y": 224, @@ -770,8 +770,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 556907854, - "versionNonce": 1025719859, + "seed": 58527072, + "versionNonce": 1085271147, "isDeleted": false, "boundElements": [], "updated": 1, @@ -788,7 +788,7 @@ "baseline": 11 }, { - "id": "t1978746231", + "id": "t738141626", "type": "text", "x": 795, "y": 239, @@ -805,8 +805,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1207348693, - "versionNonce": 659791550, + "seed": 214009943, + "versionNonce": 70714989, "isDeleted": false, "boundElements": [], "updated": 1, @@ -823,7 +823,7 @@ "baseline": 11 }, { - "id": "t1761736752", + "id": "t582688424", "type": "text", "x": 795, "y": 254, @@ -840,8 +840,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 736944923, - "versionNonce": 1690432419, + "seed": 1145006559, + "versionNonce": 1930796866, "isDeleted": false, "boundElements": [], "updated": 1, @@ -858,7 +858,7 @@ "baseline": 11 }, { - "id": "r1692623575", + "id": "r1132425667", "type": "rectangle", "x": 44, "y": 380, @@ -877,8 +877,8 @@ "roundness": { "type": 3 }, - "seed": 224126527, - "versionNonce": 1581235302, + "seed": 344685960, + "versionNonce": 1448712379, "isDeleted": false, "boundElements": [], "updated": 1, @@ -886,11 +886,11 @@ "locked": false }, { - "id": "t1180986277", + "id": "t533479734", "type": "text", "x": 54, "y": 388, - "width": 123.20000000000002, + "width": 61.60000000000001, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -903,25 +903,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 607757504, - "versionNonce": 311237588, + "seed": 1659497210, + "versionNonce": 1135478581, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "pthreads (ch50)", + "text": "pthreads", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "pthreads (ch50)", + "originalText": "pthreads", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t859413569", + "id": "t884389457", "type": "text", "x": 54, "y": 408, @@ -938,8 +938,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1643214542, - "versionNonce": 1827665208, + "seed": 1949204238, + "versionNonce": 596885069, "isDeleted": false, "boundElements": [], "updated": 1, @@ -956,7 +956,7 @@ "baseline": 11 }, { - "id": "t589344228", + "id": "t1241442666", "type": "text", "x": 54, "y": 423, @@ -973,8 +973,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 595461609, - "versionNonce": 188097404, + "seed": 1663095589, + "versionNonce": 1667953070, "isDeleted": false, "boundElements": [], "updated": 1, @@ -991,7 +991,7 @@ "baseline": 11 }, { - "id": "t685036582", + "id": "t842510568", "type": "text", "x": 54, "y": 438, @@ -1008,8 +1008,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 514779871, - "versionNonce": 1987285562, + "seed": 1799051935, + "versionNonce": 718994199, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1026,7 +1026,7 @@ "baseline": 11 }, { - "id": "r852845965", + "id": "r1431468574", "type": "rectangle", "x": 318, "y": 380, @@ -1045,8 +1045,8 @@ "roundness": { "type": 3 }, - "seed": 1403803402, - "versionNonce": 1196786748, + "seed": 261172575, + "versionNonce": 488683201, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1054,11 +1054,11 @@ "locked": false }, { - "id": "t1417759522", + "id": "t1840731161", "type": "text", "x": 328, "y": 388, - "width": 154.0, + "width": 92.4, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -1071,25 +1071,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 916004999, - "versionNonce": 1305372218, + "seed": 1968624128, + "versionNonce": 466991064, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "std::jthread (ch51)", + "text": "std::jthread", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "std::jthread (ch51)", + "originalText": "std::jthread", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t149168259", + "id": "t1907482462", "type": "text", "x": 328, "y": 408, @@ -1106,8 +1106,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 794371888, - "versionNonce": 1409422834, + "seed": 1061383537, + "versionNonce": 111425599, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1124,7 +1124,7 @@ "baseline": 11 }, { - "id": "t1629052694", + "id": "t1392991697", "type": "text", "x": 328, "y": 423, @@ -1141,8 +1141,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 758858122, - "versionNonce": 1204900600, + "seed": 680076519, + "versionNonce": 1312062470, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1159,7 +1159,7 @@ "baseline": 11 }, { - "id": "t1229875644", + "id": "t1932967927", "type": "text", "x": 328, "y": 438, @@ -1176,8 +1176,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 572783509, - "versionNonce": 1644030329, + "seed": 524426345, + "versionNonce": 759156076, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1194,7 +1194,7 @@ "baseline": 11 }, { - "id": "r1487902357", + "id": "r1071383349", "type": "rectangle", "x": 592, "y": 380, @@ -1213,8 +1213,8 @@ "roundness": { "type": 3 }, - "seed": 1165615215, - "versionNonce": 741999323, + "seed": 660223507, + "versionNonce": 930238331, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1222,11 +1222,11 @@ "locked": false }, { - "id": "t383250987", + "id": "t1838764572", "type": "text", "x": 602, "y": 388, - "width": 154.0, + "width": 92.4, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -1239,25 +1239,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1593474017, - "versionNonce": 358043581, + "seed": 163264979, + "versionNonce": 1576852379, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "Boost.Thread (ch52)", + "text": "Boost.Thread", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Boost.Thread (ch52)", + "originalText": "Boost.Thread", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t1972683816", + "id": "t1059786722", "type": "text", "x": 602, "y": 408, @@ -1274,8 +1274,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1363109251, - "versionNonce": 617760625, + "seed": 1954217074, + "versionNonce": 1158559991, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1292,7 +1292,7 @@ "baseline": 11 }, { - "id": "t703532624", + "id": "t985243158", "type": "text", "x": 602, "y": 423, @@ -1309,8 +1309,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1707271207, - "versionNonce": 1465354173, + "seed": 1611741150, + "versionNonce": 1025115441, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1327,7 +1327,7 @@ "baseline": 11 }, { - "id": "t313251171", + "id": "t666550162", "type": "text", "x": 602, "y": 438, @@ -1344,8 +1344,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 444392268, - "versionNonce": 1034168534, + "seed": 683919177, + "versionNonce": 1677210984, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1362,7 +1362,7 @@ "baseline": 11 }, { - "id": "r1434538375", + "id": "r1663611508", "type": "rectangle", "x": 866, "y": 380, @@ -1381,8 +1381,8 @@ "roundness": { "type": 3 }, - "seed": 600176345, - "versionNonce": 819601657, + "seed": 1190039060, + "versionNonce": 1828405320, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1390,11 +1390,11 @@ "locked": false }, { - "id": "t912510286", + "id": "t922367784", "type": "text", "x": 876, "y": 388, - "width": 177.10000000000002, + "width": 169.4, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -1407,25 +1407,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 499771524, - "versionNonce": 469810638, + "seed": 158698450, + "versionNonce": 646628339, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "P2300 senders (opt-in)", + "text": "P2300 senders (opt-in)", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "P2300 senders (opt-in)", + "originalText": "P2300 senders (opt-in)", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t633684193", + "id": "t131457300", "type": "text", "x": 876, "y": 408, @@ -1442,8 +1442,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1514801560, - "versionNonce": 734869156, + "seed": 462215858, + "versionNonce": 1854804944, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1460,7 +1460,7 @@ "baseline": 11 }, { - "id": "t1321854556", + "id": "t128091948", "type": "text", "x": 876, "y": 423, @@ -1477,8 +1477,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1688728804, - "versionNonce": 73816773, + "seed": 976071473, + "versionNonce": 1344746952, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1495,7 +1495,7 @@ "baseline": 11 }, { - "id": "t577171615", + "id": "t140981890", "type": "text", "x": 876, "y": 438, @@ -1512,8 +1512,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 480788608, - "versionNonce": 8016448, + "seed": 79409379, + "versionNonce": 1463142806, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1530,7 +1530,7 @@ "baseline": 11 }, { - "id": "r234006257", + "id": "r1086189563", "type": "rectangle", "x": 330, "y": 568, @@ -1549,8 +1549,8 @@ "roundness": { "type": 3 }, - "seed": 1728151709, - "versionNonce": 963399128, + "seed": 1206430622, + "versionNonce": 1591184273, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1558,11 +1558,11 @@ "locked": false }, { - "id": "t335401848", + "id": "t1908615941", "type": "text", "x": 340, "y": 576, - "width": 192.50000000000003, + "width": 130.9, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -1575,25 +1575,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 454700388, - "versionNonce": 1693441889, + "seed": 1686239908, + "versionNonce": 1909081222, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "Boost.Asio strand (ch55)", + "text": "Boost.Asio strand", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Boost.Asio strand (ch55)", + "originalText": "Boost.Asio strand", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t420761790", + "id": "t1773450013", "type": "text", "x": 340, "y": 596, @@ -1610,8 +1610,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 997520521, - "versionNonce": 768086011, + "seed": 167048626, + "versionNonce": 1625942570, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1628,7 +1628,7 @@ "baseline": 11 }, { - "id": "t812905222", + "id": "t1189477856", "type": "text", "x": 340, "y": 611, @@ -1645,8 +1645,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1641442500, - "versionNonce": 362959238, + "seed": 31413051, + "versionNonce": 1394315538, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1663,7 +1663,7 @@ "baseline": 11 }, { - "id": "a1076059551", + "id": "a1768069803", "type": "arrow", "x": 717, "y": 496, @@ -1682,8 +1682,8 @@ "roundness": { "type": 2 }, - "seed": 1807528438, - "versionNonce": 14781987, + "seed": 813931412, + "versionNonce": 963879073, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1706,7 +1706,7 @@ "endArrowhead": "arrow" }, { - "id": "t1116300542", + "id": "t885372972", "type": "text", "x": 717.0, "y": 517.0, @@ -1723,8 +1723,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 627769232, - "versionNonce": 525484873, + "seed": 777013905, + "versionNonce": 1409827888, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1741,11 +1741,11 @@ "baseline": 11 }, { - "id": "t808930511", + "id": "t743102912", "type": "text", "x": 24, "y": 682, - "width": 1312.8500000000001, + "width": 1210.0, "height": 13.75, "angle": 0, "strokeColor": "#1d1d1d", @@ -1758,25 +1758,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 494460235, - "versionNonce": 1707696744, + "seed": 372258323, + "versionNonce": 1397293544, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "Instrument 1: gettid(), ch50's. Instrument 2: strace -f -c -e trace=futex, ch51's. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations.", + "text": "Instrument 1: gettid(). Instrument 2: strace -f -c -e trace=futex. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "Instrument 1: gettid(), ch50's. Instrument 2: strace -f -c -e trace=futex, ch51's. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations.", + "originalText": "Instrument 1: gettid(). Instrument 2: strace -f -c -e trace=futex. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations.", "lineHeight": 1.25, "baseline": 11 }, { - "id": "t58672402", + "id": "t1482059497", "type": "text", "x": 24, "y": 700, @@ -1793,8 +1793,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 420529448, - "versionNonce": 1964282622, + "seed": 830119515, + "versionNonce": 1958666264, "isDeleted": false, "boundElements": [], "updated": 1, diff --git a/assets/diagrams/56-one-workload-seven-models.svg b/assets/diagrams/56-one-workload-seven-models.svg index fc16ac4..add2adf 100644 --- a/assets/diagrams/56-one-workload-seven-models.svg +++ b/assets/diagrams/56-one-workload-seven-models.svg @@ -3,7 +3,7 @@ ONE workload, shared verbatim by every model (workload.hpp) -one OS thread: concurrent, not parallel (ch49). No lock anywhere, because there is nothing to race with. +one OS thread: concurrent, not parallel. No lock anywhere, because there is nothing to race with. many OS threads + a LOCK: a thread that loses the race is parked, and has to be woken @@ -12,47 +12,47 @@ 8 tasks x 5000 rounds = 40000 folds into one accumulator every model below produced total = 0xb75768f6610642a0 (gate B, checked before any comparison is drawn) -sequential (the control) +sequential (the control) distinct_tids = 1 futex calls = 1 8 tasks, one after another -coroutine (ch53) +coroutine distinct_tids = 1 futex calls = 1 8 in flight, 32-byte frames -fiber (ch54) +fiber distinct_tids = 1 futex calls = 1 8 in flight, 128 KiB stacks -pthreads (ch50) +pthreads distinct_tids = 8 futex calls = 230 pthread_mutex_t -std::jthread (ch51) +std::jthread distinct_tids = 8 futex calls = 427 std::mutex -Boost.Thread (ch52) +Boost.Thread distinct_tids = 8 futex calls = 451 boost::mutex + when_all -P2300 senders (opt-in) +P2300 senders (opt-in) distinct_tids = 8 futex calls = 562 NVIDIA stdexec, NOT the stdlib -Boost.Asio strand (ch55) +Boost.Asio strand distinct_tids = 8 · futex calls = 51 same 8 threads as the row above, serialized without a lock 1/8 the futex traffic of the row above 1/8 the futex traffic of the row above -Instrument 1: gettid(), ch50's. Instrument 2: strace -f -c -e trace=futex, ch51's. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations. +Instrument 1: gettid(). Instrument 2: strace -f -c -e trace=futex. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations. Futex magnitudes move on every run and are never gated; the three TIERS are. Measured on the Fedora 44 reference host, Boost 1.90.0, GCC 16.1.1. \ No newline at end of file diff --git a/assets/diagrams/56-two-assembled-axes.excalidraw b/assets/diagrams/56-two-assembled-axes.excalidraw index 4c9c828..3557ccf 100644 --- a/assets/diagrams/56-two-assembled-axes.excalidraw +++ b/assets/diagrams/56-two-assembled-axes.excalidraw @@ -4,7 +4,7 @@ "source": "https://excalidraw.com", "elements": [ { - "id": "r227385313", + "id": "r1923022791", "type": "rectangle", "x": 24, "y": 40, @@ -23,8 +23,8 @@ "roundness": { "type": 3 }, - "seed": 979639629, - "versionNonce": 952389529, + "seed": 983124089, + "versionNonce": 239947400, "isDeleted": false, "boundElements": [], "updated": 1, @@ -32,7 +32,7 @@ "locked": false }, { - "id": "t1732497267", + "id": "t1807631029", "type": "text", "x": 36, "y": 46, @@ -49,8 +49,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1210738910, - "versionNonce": 326469431, + "seed": 1272927830, + "versionNonce": 1469720951, "isDeleted": false, "boundElements": [], "updated": 1, @@ -67,7 +67,7 @@ "baseline": 12 }, { - "id": "r1299711684", + "id": "r74374751", "type": "rectangle", "x": 596, "y": 40, @@ -86,8 +86,8 @@ "roundness": { "type": 3 }, - "seed": 1029964451, - "versionNonce": 1861672693, + "seed": 428851179, + "versionNonce": 99969854, "isDeleted": false, "boundElements": [], "updated": 1, @@ -95,7 +95,7 @@ "locked": false }, { - "id": "t1655993831", + "id": "t1114165232", "type": "text", "x": 608, "y": 46, @@ -112,8 +112,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 29992039, - "versionNonce": 625375542, + "seed": 1286323207, + "versionNonce": 611058508, "isDeleted": false, "boundElements": [], "updated": 1, @@ -130,7 +130,7 @@ "baseline": 12 }, { - "id": "r146342470", + "id": "r1280716498", "type": "rectangle", "x": 64, "y": 96, @@ -149,8 +149,8 @@ "roundness": { "type": 3 }, - "seed": 44252983, - "versionNonce": 1724707450, + "seed": 354409642, + "versionNonce": 966975572, "isDeleted": false, "boundElements": [], "updated": 1, @@ -158,11 +158,11 @@ "locked": false }, { - "id": "t239185316", + "id": "t410951658", "type": "text", "x": 74, "y": 104, - "width": 169.4, + "width": 161.70000000000002, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -175,8 +175,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1093339326, - "versionNonce": 637296230, + "seed": 953037006, + "versionNonce": 1290829463, "isDeleted": false, "boundElements": [], "updated": 1, @@ -193,11 +193,11 @@ "baseline": 14 }, { - "id": "t953663232", + "id": "t1905958870", "type": "text", "x": 74, "y": 124, - "width": 145.20000000000002, + "width": 108.9, "height": 13.75, "angle": 0, "strokeColor": "#555555", @@ -210,25 +210,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1223739328, - "versionNonce": 1204337110, + "seed": 907892804, + "versionNonce": 1721925153, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch50, pthread_getattr_np", + "text": "pthread_getattr_np", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch50, pthread_getattr_np", + "originalText": "pthread_getattr_np", "lineHeight": 1.25, "baseline": 11 }, { - "id": "t1487601003", + "id": "t442037344", "type": "text", "x": 74, "y": 139, @@ -245,8 +245,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 673154924, - "versionNonce": 1385951290, + "seed": 877476984, + "versionNonce": 1594509742, "isDeleted": false, "boundElements": [], "updated": 1, @@ -263,7 +263,7 @@ "baseline": 11 }, { - "id": "r1789417017", + "id": "r837132545", "type": "rectangle", "x": 64, "y": 246, @@ -282,8 +282,8 @@ "roundness": { "type": 3 }, - "seed": 1604790199, - "versionNonce": 933466920, + "seed": 60201224, + "versionNonce": 308500806, "isDeleted": false, "boundElements": [], "updated": 1, @@ -291,11 +291,11 @@ "locked": false }, { - "id": "t1561740925", + "id": "t428541404", "type": "text", "x": 74, "y": 254, - "width": 154.0, + "width": 146.3, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -308,8 +308,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 720076915, - "versionNonce": 1842328322, + "seed": 1517871384, + "versionNonce": 350109858, "isDeleted": false, "boundElements": [], "updated": 1, @@ -326,11 +326,11 @@ "baseline": 14 }, { - "id": "t1593653155", + "id": "t965995406", "type": "text", "x": 74, "y": 274, - "width": 205.70000000000002, + "width": 169.4, "height": 13.75, "angle": 0, "strokeColor": "#555555", @@ -343,25 +343,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1676652297, - "versionNonce": 467527824, + "seed": 917435129, + "versionNonce": 1737568511, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch54, stack_traits::default_size()", + "text": "stack_traits::default_size()", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch54, stack_traits::default_size()", + "originalText": "stack_traits::default_size()", "lineHeight": 1.25, "baseline": 11 }, { - "id": "t1525671233", + "id": "t1882981593", "type": "text", "x": 74, "y": 289, @@ -378,8 +378,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 187969345, - "versionNonce": 463017179, + "seed": 35551842, + "versionNonce": 442008607, "isDeleted": false, "boundElements": [], "updated": 1, @@ -396,7 +396,7 @@ "baseline": 11 }, { - "id": "r973350444", + "id": "r462391448", "type": "rectangle", "x": 64, "y": 396, @@ -415,8 +415,8 @@ "roundness": { "type": 3 }, - "seed": 1379262674, - "versionNonce": 1063368115, + "seed": 1331455621, + "versionNonce": 157586766, "isDeleted": false, "boundElements": [], "updated": 1, @@ -424,11 +424,11 @@ "locked": false }, { - "id": "t1640519890", + "id": "t430743308", "type": "text", "x": 74, "y": 404, - "width": 200.20000000000002, + "width": 192.50000000000003, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -441,8 +441,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 526174325, - "versionNonce": 1518048748, + "seed": 1727642085, + "versionNonce": 1458327812, "isDeleted": false, "boundElements": [], "updated": 1, @@ -459,11 +459,11 @@ "baseline": 14 }, { - "id": "t883375350", + "id": "t1800800826", "type": "text", "x": 74, "y": 424, - "width": 205.70000000000002, + "width": 169.4, "height": 13.75, "angle": 0, "strokeColor": "#555555", @@ -476,25 +476,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 41848629, - "versionNonce": 1892182775, + "seed": 1063447037, + "versionNonce": 1208068701, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch53, operator new in promise_type", + "text": "operator new in promise_type", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch53, operator new in promise_type", + "originalText": "operator new in promise_type", "lineHeight": 1.25, "baseline": 11 }, { - "id": "t802353626", + "id": "t1881249407", "type": "text", "x": 74, "y": 439, @@ -511,8 +511,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1924728028, - "versionNonce": 527439180, + "seed": 1146499941, + "versionNonce": 872664957, "isDeleted": false, "boundElements": [], "updated": 1, @@ -529,7 +529,7 @@ "baseline": 11 }, { - "id": "r1726238877", + "id": "r254081756", "type": "rectangle", "x": 636, "y": 96, @@ -548,8 +548,8 @@ "roundness": { "type": 3 }, - "seed": 608473055, - "versionNonce": 416957083, + "seed": 1073514954, + "versionNonce": 636325992, "isDeleted": false, "boundElements": [], "updated": 1, @@ -557,11 +557,11 @@ "locked": false }, { - "id": "t767004056", + "id": "t204297087", "type": "text", "x": 646, "y": 104, - "width": 154.0, + "width": 107.80000000000001, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -574,25 +574,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1747864629, - "versionNonce": 627868741, + "seed": 132535782, + "versionNonce": 1438677340, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch50 pthread_cancel", + "text": "pthread_cancel", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch50 pthread_cancel", + "originalText": "pthread_cancel", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t1223405368", + "id": "t1842860684", "type": "text", "x": 646, "y": 124, @@ -609,8 +609,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 405668115, - "versionNonce": 578282756, + "seed": 1247074567, + "versionNonce": 891020726, "isDeleted": false, "boundElements": [], "updated": 1, @@ -627,7 +627,7 @@ "baseline": 11 }, { - "id": "t868975668", + "id": "t149465807", "type": "text", "x": 646, "y": 139, @@ -644,8 +644,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 894999432, - "versionNonce": 1371994072, + "seed": 820049838, + "versionNonce": 857512371, "isDeleted": false, "boundElements": [], "updated": 1, @@ -662,7 +662,7 @@ "baseline": 11 }, { - "id": "r254633482", + "id": "r1091387446", "type": "rectangle", "x": 636, "y": 206, @@ -681,8 +681,8 @@ "roundness": { "type": 3 }, - "seed": 555888011, - "versionNonce": 216936259, + "seed": 47438301, + "versionNonce": 1636321181, "isDeleted": false, "boundElements": [], "updated": 1, @@ -690,11 +690,11 @@ "locked": false }, { - "id": "t1516470290", + "id": "t1708501205", "type": "text", "x": 646, "y": 214, - "width": 161.70000000000002, + "width": 115.50000000000001, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -707,25 +707,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1085204841, - "versionNonce": 1660514424, + "seed": 806065628, + "versionNonce": 827844788, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch51 std::stop_token", + "text": "std::stop_token", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch51 std::stop_token", + "originalText": "std::stop_token", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t998821839", + "id": "t60853212", "type": "text", "x": 646, "y": 234, @@ -742,8 +742,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1506684344, - "versionNonce": 1592192738, + "seed": 1482524920, + "versionNonce": 972021335, "isDeleted": false, "boundElements": [], "updated": 1, @@ -760,7 +760,7 @@ "baseline": 11 }, { - "id": "t794202869", + "id": "t347562190", "type": "text", "x": 646, "y": 249, @@ -777,8 +777,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 450657016, - "versionNonce": 26366265, + "seed": 1614002126, + "versionNonce": 760592349, "isDeleted": false, "boundElements": [], "updated": 1, @@ -795,7 +795,7 @@ "baseline": 11 }, { - "id": "r594077930", + "id": "r1987252594", "type": "rectangle", "x": 636, "y": 316, @@ -814,8 +814,8 @@ "roundness": { "type": 3 }, - "seed": 998247980, - "versionNonce": 1768348772, + "seed": 1708275809, + "versionNonce": 1173348105, "isDeleted": false, "boundElements": [], "updated": 1, @@ -823,11 +823,11 @@ "locked": false }, { - "id": "t1337134746", + "id": "t292661487", "type": "text", "x": 646, "y": 324, - "width": 192.50000000000003, + "width": 146.3, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -840,25 +840,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1856742368, - "versionNonce": 782926863, + "seed": 333240222, + "versionNonce": 694130056, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch52 thread::interrupt()", + "text": "thread::interrupt()", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch52 thread::interrupt()", + "originalText": "thread::interrupt()", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t1345276951", + "id": "t280409610", "type": "text", "x": 646, "y": 344, @@ -875,8 +875,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1905809665, - "versionNonce": 546533971, + "seed": 935632516, + "versionNonce": 1407327749, "isDeleted": false, "boundElements": [], "updated": 1, @@ -893,7 +893,7 @@ "baseline": 11 }, { - "id": "t1060321234", + "id": "t630584228", "type": "text", "x": 646, "y": 359, @@ -910,8 +910,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1664753510, - "versionNonce": 940731286, + "seed": 173587187, + "versionNonce": 947553220, "isDeleted": false, "boundElements": [], "updated": 1, @@ -928,7 +928,7 @@ "baseline": 11 }, { - "id": "r1569096549", + "id": "r805563383", "type": "rectangle", "x": 636, "y": 426, @@ -947,8 +947,8 @@ "roundness": { "type": 3 }, - "seed": 10669684, - "versionNonce": 803083070, + "seed": 1656208116, + "versionNonce": 1128031434, "isDeleted": false, "boundElements": [], "updated": 1, @@ -956,11 +956,11 @@ "locked": false }, { - "id": "t1600080809", + "id": "t908208857", "type": "text", "x": 646, "y": 434, - "width": 192.50000000000003, + "width": 146.3, "height": 17.5, "angle": 0, "strokeColor": "#111111", @@ -973,25 +973,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 285446942, - "versionNonce": 1712782016, + "seed": 1644600378, + "versionNonce": 1052949238, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "ch55 cancellation_signal", + "text": "cancellation_signal", "fontSize": 14, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "ch55 cancellation_signal", + "originalText": "cancellation_signal", "lineHeight": 1.25, "baseline": 14 }, { - "id": "t690971374", + "id": "t1956848207", "type": "text", "x": 646, "y": 454, @@ -1008,8 +1008,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 814262724, - "versionNonce": 546022092, + "seed": 562330396, + "versionNonce": 963320507, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1026,7 +1026,7 @@ "baseline": 11 }, { - "id": "t1725932996", + "id": "t213459217", "type": "text", "x": 646, "y": 469, @@ -1043,8 +1043,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1531697235, - "versionNonce": 602314647, + "seed": 1173223290, + "versionNonce": 107357992, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1061,7 +1061,7 @@ "baseline": 11 }, { - "id": "a46544817", + "id": "a783988376", "type": "arrow", "x": 294, "y": 188, @@ -1080,8 +1080,8 @@ "roundness": { "type": 2 }, - "seed": 1977528979, - "versionNonce": 1743860946, + "seed": 1021371161, + "versionNonce": 1590151022, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1104,7 +1104,7 @@ "endArrowhead": "arrow" }, { - "id": "t1164834831", + "id": "t1950411718", "type": "text", "x": 294.0, "y": 202.0, @@ -1121,8 +1121,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 667235924, - "versionNonce": 1722375273, + "seed": 551748343, + "versionNonce": 699569018, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1139,7 +1139,7 @@ "baseline": 11 }, { - "id": "a1143647852", + "id": "a787463775", "type": "arrow", "x": 294, "y": 338, @@ -1158,8 +1158,8 @@ "roundness": { "type": 2 }, - "seed": 1744557329, - "versionNonce": 1225754643, + "seed": 29353743, + "versionNonce": 454143074, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1182,7 +1182,7 @@ "endArrowhead": "arrow" }, { - "id": "t1813808585", + "id": "t239279394", "type": "text", "x": 294.0, "y": 352.0, @@ -1199,8 +1199,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1636104648, - "versionNonce": 155327002, + "seed": 915958969, + "versionNonce": 688149049, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1217,11 +1217,11 @@ "baseline": 11 }, { - "id": "t1296465552", + "id": "t650351433", "type": "text", "x": 24, "y": 568, - "width": 1282.6000000000001, + "width": 992.2, "height": 13.75, "angle": 0, "strokeColor": "#1d1d1d", @@ -1234,25 +1234,25 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 1175369593, - "versionNonce": 1424455077, + "seed": 1836771290, + "versionNonce": 522122347, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "NEITHER band is measured in ch56. Both are assembled from earlier chapters' own measurements, carried into the table arm as named constants beside the instrument that produced them: the pattern ch54 established.", + "text": "NEITHER band is measured here. Both are assembled from earlier measurements, carried into the table arm as named constants beside the instrument that produced them.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "NEITHER band is measured in ch56. Both are assembled from earlier chapters' own measurements, carried into the table arm as named constants beside the instrument that produced them: the pattern ch54 established.", + "originalText": "NEITHER band is measured here. Both are assembled from earlier measurements, carried into the table arm as named constants beside the instrument that produced them.", "lineHeight": 1.25, "baseline": 11 }, { - "id": "t238910032", + "id": "t858196072", "type": "text", "x": 24, "y": 586, @@ -1269,8 +1269,8 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 937213153, - "versionNonce": 1184463458, + "seed": 1150282616, + "versionNonce": 1494805449, "isDeleted": false, "boundElements": [], "updated": 1, @@ -1287,11 +1287,11 @@ "baseline": 11 }, { - "id": "t327315948", + "id": "t1239548179", "type": "text", "x": 24, "y": 604, - "width": 1113.2, + "width": 1089.0, "height": 13.75, "angle": 0, "strokeColor": "#555555", @@ -1304,20 +1304,20 @@ "groupIds": [], "frameId": null, "roundness": null, - "seed": 351945920, - "versionNonce": 1337236561, + "seed": 1627753484, + "versionNonce": 1944076407, "isDeleted": false, "boundElements": [], "updated": 1, "link": null, "locked": false, - "text": "P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so ch56's seventh arm links NVIDIA's reference implementation, not the standard library.", + "text": "P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so the seventh arm links NVIDIA's reference implementation, not the standard library.", "fontSize": 11, "fontFamily": 1, "textAlign": "left", "verticalAlign": "top", "containerId": null, - "originalText": "P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so ch56's seventh arm links NVIDIA's reference implementation, not the standard library.", + "originalText": "P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so the seventh arm links NVIDIA's reference implementation, not the standard library.", "lineHeight": 1.25, "baseline": 11 } diff --git a/assets/diagrams/56-two-assembled-axes.svg b/assets/diagrams/56-two-assembled-axes.svg index 948174f..3cfa9c6 100644 --- a/assets/diagrams/56-two-assembled-axes.svg +++ b/assets/diagrams/56-two-assembled-axes.svg @@ -6,30 +6,30 @@ how cancellation arrives thread: 8388608 bytes -ch50, pthread_getattr_np +pthread_getattr_np the kernel schedules it, so it costs kernel state fiber: 131072 bytes -ch54, stack_traits::default_size() +stack_traits::default_size() user-scheduled, but it keeps a real stack coroutine frame: 32 bytes -ch53, operator new in promise_type +operator new in promise_type no stack: only what crosses a suspension -ch50 pthread_cancel +pthread_cancel a FORCED UNWIND swallow it without rethrowing and the process aborts -ch51 std::stop_token +std::stop_token a FLAG the target polls nothing is thrown; there is nothing to swallow -ch52 thread::interrupt() +thread::interrupt() an ORDINARY EXCEPTION thrown at defined interruption points -ch55 cancellation_signal +cancellation_signal a COMPLETION carrying operation_aborted the handler still runs exactly once, so ownership never changes @@ -38,7 +38,7 @@ divide by 4096 divide by 4096 -NEITHER band is measured in ch56. Both are assembled from earlier chapters' own measurements, carried into the table arm as named constants beside the instrument that produced them: the pattern ch54 established. +NEITHER band is measured here. Both are assembled from earlier measurements, carried into the table arm as named constants beside the instrument that produced them. Cost and capability move together and in opposite directions: the more a paused computation costs, the fewer restrictions there are on where it may pause. -P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so ch56's seventh arm links NVIDIA's reference implementation, not the standard library. +P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so the seventh arm links NVIDIA's reference implementation, not the standard library. \ No newline at end of file diff --git a/assets/diagrams/specs/56.py b/assets/diagrams/specs/56.py index 9658cd0..81b4760 100644 --- a/assets/diagrams/specs/56.py +++ b/assets/diagrams/specs/56.py @@ -6,7 +6,7 @@ # ── Figure 56.1: one workload, seven models, two uniform instruments ────── bands = [ dict(x=24, y=40, w=1112, h=96, label="ONE workload, shared verbatim by every model (workload.hpp)", fill="#fbfbfb"), - dict(x=24, y=156, w=1112, h=164, label="one OS thread: concurrent, not parallel (ch49). No lock anywhere, because there is nothing to race with.", fill="#f7faf7"), + dict(x=24, y=156, w=1112, h=164, label="one OS thread: concurrent, not parallel. No lock anywhere, because there is nothing to race with.", fill="#f7faf7"), dict(x=24, y=340, w=1112, h=176, label="many OS threads + a LOCK: a thread that loses the race is parked, and has to be woken", fill="#faf7f7"), dict(x=24, y=536, w=1112, h=132, label="many OS threads + a STRAND: the handler is deferred, so no thread ever blocks", fill="#fff8ef"), ] @@ -15,22 +15,22 @@ "8 tasks x 5000 rounds = 40000 folds into one accumulator", "every model below produced total = 0xb75768f6610642a0 (gate B, checked before any comparison is drawn)"]), - dict(x=45, y=196, w=330, h=100, lines=["sequential (the control)", "distinct_tids = 1", "futex calls = 1", "8 tasks, one after another"]), - dict(x=415, y=196, w=330, h=100, lines=["coroutine (ch53)", "distinct_tids = 1", "futex calls = 1", "8 in flight, 32-byte frames"]), - dict(x=785, y=196, w=330, h=100, lines=["fiber (ch54)", "distinct_tids = 1", "futex calls = 1", "8 in flight, 128 KiB stacks"]), + dict(x=45, y=196, w=330, h=100, lines=["sequential (the control)", "distinct_tids = 1", "futex calls = 1", "8 tasks, one after another"]), + dict(x=415, y=196, w=330, h=100, lines=["coroutine", "distinct_tids = 1", "futex calls = 1", "8 in flight, 32-byte frames"]), + dict(x=785, y=196, w=330, h=100, lines=["fiber", "distinct_tids = 1", "futex calls = 1", "8 in flight, 128 KiB stacks"]), - dict(x=44, y=380, w=250, h=112, lines=["pthreads (ch50)", "distinct_tids = 8", "futex calls = 230", "pthread_mutex_t"]), - dict(x=318, y=380, w=250, h=112, lines=["std::jthread (ch51)", "distinct_tids = 8", "futex calls = 427", "std::mutex"]), - dict(x=592, y=380, w=250, h=112, lines=["Boost.Thread (ch52)", "distinct_tids = 8", "futex calls = 451", "boost::mutex + when_all"]), - dict(x=866, y=380, w=250, h=112, style="ghost", lines=["P2300 senders (opt-in)", "distinct_tids = 8", "futex calls = 562", "NVIDIA stdexec, NOT the stdlib"]), + dict(x=44, y=380, w=250, h=112, lines=["pthreads", "distinct_tids = 8", "futex calls = 230", "pthread_mutex_t"]), + dict(x=318, y=380, w=250, h=112, lines=["std::jthread", "distinct_tids = 8", "futex calls = 427", "std::mutex"]), + dict(x=592, y=380, w=250, h=112, lines=["Boost.Thread", "distinct_tids = 8", "futex calls = 451", "boost::mutex + when_all"]), + dict(x=866, y=380, w=250, h=112, style="ghost", lines=["P2300 senders (opt-in)", "distinct_tids = 8", "futex calls = 562", "NVIDIA stdexec, NOT the stdlib"]), - dict(x=330, y=568, w=500, h=88, style="accent", lines=["Boost.Asio strand (ch55)", "distinct_tids = 8 · futex calls = 51", "same 8 threads as the row above, serialized without a lock"]), + dict(x=330, y=568, w=500, h=88, style="accent", lines=["Boost.Asio strand", "distinct_tids = 8 · futex calls = 51", "same 8 threads as the row above, serialized without a lock"]), ] edges = [ dict(x1=717, y1=496, x2=717, y2=566, amber=True, label="1/8 the futex traffic of the row above", lx=0, ly=-8), ] notes = [ - dict(x=24, y=692, text="Instrument 1: gettid(), ch50's. Instrument 2: strace -f -c -e trace=futex, ch51's. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations."), + dict(x=24, y=692, text="Instrument 1: gettid(). Instrument 2: strace -f -c -e trace=futex. Both applied identically to all seven models: which is the only thing that makes this a comparison rather than a table of quotations."), dict(x=24, y=710, text="Futex magnitudes move on every run and are never gated; the three TIERS are. Measured on the Fedora 44 reference host, Boost 1.90.0, GCC 16.1.1.", color="#555555"), ] g.emit("56-one-workload-seven-models", 1160, 726, bands, nodes, edges, notes) @@ -41,22 +41,22 @@ dict(x=596, y=40, w=540, h=506, label="how cancellation arrives", fill="#fbfbfb"), ] nodes2 = [ - dict(x=64, y=96, w=460, h=92, lines=["thread: 8388608 bytes", "ch50, pthread_getattr_np", "the kernel schedules it, so it costs kernel state"]), - dict(x=64, y=246, w=460, h=92, style="accent", lines=["fiber: 131072 bytes", "ch54, stack_traits::default_size()", "user-scheduled, but it keeps a real stack"]), - dict(x=64, y=396, w=460, h=92, lines=["coroutine frame: 32 bytes", "ch53, operator new in promise_type", "no stack: only what crosses a suspension"]), + dict(x=64, y=96, w=460, h=92, lines=["thread: 8388608 bytes", "pthread_getattr_np", "the kernel schedules it, so it costs kernel state"]), + dict(x=64, y=246, w=460, h=92, style="accent", lines=["fiber: 131072 bytes", "stack_traits::default_size()", "user-scheduled, but it keeps a real stack"]), + dict(x=64, y=396, w=460, h=92, lines=["coroutine frame: 32 bytes", "operator new in promise_type", "no stack: only what crosses a suspension"]), - dict(x=636, y=96, w=460, h=92, lines=["ch50 pthread_cancel", "a FORCED UNWIND", "swallow it without rethrowing and the process aborts"]), - dict(x=636, y=206, w=460, h=92, lines=["ch51 std::stop_token", "a FLAG the target polls", "nothing is thrown; there is nothing to swallow"]), - dict(x=636, y=316, w=460, h=92, lines=["ch52 thread::interrupt()", "an ORDINARY EXCEPTION", "thrown at defined interruption points"]), - dict(x=636, y=426, w=460, h=92, style="accent", lines=["ch55 cancellation_signal", "a COMPLETION carrying operation_aborted", "the handler still runs exactly once, so ownership never changes"]), + dict(x=636, y=96, w=460, h=92, lines=["pthread_cancel", "a FORCED UNWIND", "swallow it without rethrowing and the process aborts"]), + dict(x=636, y=206, w=460, h=92, lines=["std::stop_token", "a FLAG the target polls", "nothing is thrown; there is nothing to swallow"]), + dict(x=636, y=316, w=460, h=92, lines=["thread::interrupt()", "an ORDINARY EXCEPTION", "thrown at defined interruption points"]), + dict(x=636, y=426, w=460, h=92, style="accent", lines=["cancellation_signal", "a COMPLETION carrying operation_aborted", "the handler still runs exactly once, so ownership never changes"]), ] edges2 = [ dict(x1=294, y1=188, x2=294, y2=244, amber=True, label="divide by 64", lx=0, ly=-4), dict(x1=294, y1=338, x2=294, y2=394, amber=True, label="divide by 4096", lx=0, ly=-4), ] notes2 = [ - dict(x=24, y=578, text="NEITHER band is measured in ch56. Both are assembled from earlier chapters' own measurements, carried into the table arm as named constants beside the instrument that produced them: the pattern ch54 established."), + dict(x=24, y=578, text="NEITHER band is measured here. Both are assembled from earlier measurements, carried into the table arm as named constants beside the instrument that produced them."), dict(x=24, y=596, text="Cost and capability move together and in opposite directions: the more a paused computation costs, the fewer restrictions there are on where it may pause.", color="#555555"), - dict(x=24, y=614, text="P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so ch56's seventh arm links NVIDIA's reference implementation, not the standard library.", color="#555555"), + dict(x=24, y=614, text="P2300 senders are absent from both bands by design: __cpp_lib_senders is still undefined here, so the seventh arm links NVIDIA's reference implementation, not the standard library.", color="#555555"), ] g.emit("56-two-assembled-axes", 1160, 636, bands2, nodes2, edges2, notes2) diff --git a/examples/01-hello-syscalls/README.md b/examples/01-hello-syscalls/README.md index 6c0f67f..a8fcd3d 100644 --- a/examples/01-hello-syscalls/README.md +++ b/examples/01-hello-syscalls/README.md @@ -1,7 +1,7 @@ # 01-hello-syscalls -Chapter 1's example — and the seed for every runnable example in the book. It -implements the same tiny program three times — C++23, Go, and Rust — each +This chapter's example is the seed for every runnable example in the book. It +implements the same tiny program three times, in C++23, Go, and Rust, each printing exactly one line: ``` @@ -32,9 +32,9 @@ built binaries. Every language directory has a `demo.sh` with the same interface, identical across the whole book: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. diff --git a/examples/04-syscalls-and-the-abi/README.md b/examples/04-syscalls-and-the-abi/README.md index 32f194d..06b9a7e 100644 --- a/examples/04-syscalls-and-the-abi/README.md +++ b/examples/04-syscalls-and-the-abi/README.md @@ -1,16 +1,16 @@ -# 04-syscalls-and-the-abi — sysprobe +# 04-syscalls-and-the-abi: sysprobe -A syscall specimen: the same small program three times — C++23, Go, and -Rust — performing a fixed, labeled sequence of system calls so you can watch +A syscall specimen: the same small program three times, in C++23, Go, and +Rust, performing a fixed, labeled sequence of system calls so you can watch each language's runtime hit the kernel ABI under `strace(1)`: -1. **open** — `openat(2)` of a scratch file in `$TMPDIR` (default `/tmp`), +1. **open**: `openat(2)` of a scratch file in `$TMPDIR` (default `/tmp`), preferring an anonymous `O_TMPFILE` inode, with a `mkstemp`-style create+unlink fallback on filesystems without `O_TMPFILE` support -2. **write** — `write(2)` of a 25-byte payload to that fd, then `close(2)` +2. **write**: `write(2)` of a 25-byte payload to that fd, then `close(2)` (RAII scope exit in C++, checked `unix.Close` in Go, `OwnedFd` drop in Rust) -3. **sleep** — `nanosleep(2)` for 10 ms, resuming on `EINTR` -4. **random** — `getrandom(2)` for 16 bytes, looping over partial reads +3. **sleep**: `nanosleep(2)` for 10 ms, resuming on `EINTR` +4. **random**: `getrandom(2)` for 16 bytes, looping over partial reads ## CLI and output @@ -37,18 +37,18 @@ The point of the program. For example: strace -e trace=openat,write,close,nanosleep,getrandom ./cpp/build/release/app ``` -shows the labeled sequence — `openat(... O_TMPFILE|O_RDWR|O_CLOEXEC)`, the +shows the labeled sequence: `openat(... O_TMPFILE|O_RDWR|O_CLOEXEC)`, the 25-byte `write`, the `close`, `nanosleep({tv_sec=0, tv_nsec=10000000})`, and -`getrandom(..., 16, 0) = 16` — interleaved with each runtime's own startup -syscalls, which is exactly the contrast the chapter is about. +`getrandom(..., 16, 0) = 16`, interleaved with each runtime's own startup +syscalls, which is the contrast the chapter is about. ## The demo contract Every language directory has a `demo.sh` with the same interface: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. diff --git a/examples/04-syscalls-and-the-abi/cpp/src/main.cpp b/examples/04-syscalls-and-the-abi/cpp/src/main.cpp index 1f53973..a31a130 100644 --- a/examples/04-syscalls-and-the-abi/cpp/src/main.cpp +++ b/examples/04-syscalls-and-the-abi/cpp/src/main.cpp @@ -1,5 +1,5 @@ -// sysprobe: a labeled syscall specimen — openat+write+close of an unlinked -// temp file, a 10 ms nanosleep, and 16 bytes of getrandom — built to be +// sysprobe: a labeled syscall specimen (openat+write+close of an unlinked +// temp file, a 10 ms nanosleep, and 16 bytes of getrandom), built to be // watched under strace(1). Prints "step= ok" per step, then a summary. #include diff --git a/examples/04-syscalls-and-the-abi/go/main.go b/examples/04-syscalls-and-the-abi/go/main.go index 1a0ad67..3220693 100644 --- a/examples/04-syscalls-and-the-abi/go/main.go +++ b/examples/04-syscalls-and-the-abi/go/main.go @@ -1,5 +1,5 @@ -// sysprobe: a labeled syscall specimen — openat+write+close of an unlinked -// temp file, a 10 ms nanosleep, and 16 bytes of getrandom — built to be +// sysprobe: a labeled syscall specimen (openat+write+close of an unlinked +// temp file, a 10 ms nanosleep, and 16 bytes of getrandom), built to be // watched under strace(1). Prints "step= ok" per step, then a summary. package main diff --git a/examples/04-syscalls-and-the-abi/rust/src/main.rs b/examples/04-syscalls-and-the-abi/rust/src/main.rs index 00bedf9..17f86c8 100644 --- a/examples/04-syscalls-and-the-abi/rust/src/main.rs +++ b/examples/04-syscalls-and-the-abi/rust/src/main.rs @@ -1,5 +1,5 @@ -//! sysprobe: a labeled syscall specimen — openat+write+close of an unlinked -//! temp file, a 10 ms nanosleep, and 16 bytes of getrandom — built to be +//! sysprobe: a labeled syscall specimen (openat+write+close of an unlinked +//! temp file, a 10 ms nanosleep, and 16 bytes of getrandom), built to be //! watched under strace(1). Prints "step= ok" per step, then a summary. use std::os::fd::{AsFd, BorrowedFd, OwnedFd}; diff --git a/examples/04-syscalls-and-the-abi/verify.lua b/examples/04-syscalls-and-the-abi/verify.lua index 0da0f28..0e2c18e 100644 --- a/examples/04-syscalls-and-the-abi/verify.lua +++ b/examples/04-syscalls-and-the-abi/verify.lua @@ -1,5 +1,5 @@ -- Verify sysprobe: run the demo for LSP_LANG and assert the labeled syscall --- sequence — four "step= ok" lines plus the summary, in order. +-- sequence: four "step= ok" lines plus the summary, in order. -- Invoked from the example directory with LSP_LANG (cpp|go|rust) and EXAMPLE_DIR set. local script_dir = arg[0]:match("(.*/)") or "./" diff --git a/examples/05-errors-three-ways/README.md b/examples/05-errors-three-ways/README.md index 7c29939..852fd07 100644 --- a/examples/05-errors-three-ways/README.md +++ b/examples/05-errors-three-ways/README.md @@ -1,6 +1,6 @@ -# 05-errors-three-ways — copyx +# 05-errors-three-ways: copyx -A file copier, written three times — C++23, Go, and Rust — to establish the +A file copier, written three times, in C++23, Go, and Rust, to establish the book's error taxonomy: every failure is classified by *which side of the copy broke*, the reason is printed in one fixed shape, and the exit code carries the classification for scripts and callers. @@ -28,7 +28,7 @@ copyx: write /dev/full: No space left on device # exit 3 copyx: close /dev/full: No space left on device # exit 3 ``` -Two policies are deliberately visible in each implementation: +Two policies are visible in each implementation: - **EINTR is retried.** `read(2)`/`write(2)` interrupted by a signal transferred nothing; each language has an explicit retry loop @@ -36,7 +36,7 @@ Two policies are deliberately visible in each implementation: - **Short writes are resumed.** `write_all`/`writeAll` advances past the bytes the kernel accepted and reissues the rest until the buffer is gone. The write side also *observes* `close(2)`, because that is where deferred - IO errors surface — the read side's close result carries nothing actionable + IO errors surface; the read side's close result carries nothing actionable and is released by RAII/defer. The copy loop uses a 64 KiB buffer; `DST` is created `0644` and truncated. @@ -63,11 +63,11 @@ The copy loop uses a 64 KiB buffer; `DST` is created `0644` and truncated. Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary with `args` +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary with `args` (with env `TARGET` set, deploys to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally) -- `./demo.sh` — build, then run a local walkthrough of the taxonomy: +- `./demo.sh`: build, then run a local walkthrough of the taxonomy: a real 200 000-byte copy verified with `cmp`, a missing source (exit 2), and a write to `/dev/full` (exit 3) diff --git a/examples/05-errors-three-ways/cpp/src/main.cpp b/examples/05-errors-three-ways/cpp/src/main.cpp index abfba7f..a2bb698 100644 --- a/examples/05-errors-three-ways/cpp/src/main.cpp +++ b/examples/05-errors-three-ways/cpp/src/main.cpp @@ -1,8 +1,8 @@ -// copyx SRC DST — file copier that establishes the book's error taxonomy. +// copyx SRC DST: file copier that establishes the book's error taxonomy. // -// exit 0 — success; "copied bytes" on stdout -// exit 2 — source-side failure (open/read SRC); "copyx: " on stderr -// exit 3 — destination-side failure (open/write/close DST); same shape +// exit 0: success; "copied bytes" on stdout +// exit 2: source-side failure (open/read SRC); "copyx: " on stderr +// exit 3: destination-side failure (open/write/close DST); same shape // // EINTR on read(2)/write(2) is retried (the syscall was interrupted before // completing; reissuing it is the only correct policy), and short writes are diff --git a/examples/05-errors-three-ways/go/main.go b/examples/05-errors-three-ways/go/main.go index 55c8006..f07d4f0 100644 --- a/examples/05-errors-three-ways/go/main.go +++ b/examples/05-errors-three-ways/go/main.go @@ -1,8 +1,8 @@ -// copyx SRC DST — file copier that establishes the book's error taxonomy. +// copyx SRC DST: file copier that establishes the book's error taxonomy. // -// exit 0 — success; "copied bytes" on stdout -// exit 2 — source-side failure (open/read SRC); "copyx: " on stderr -// exit 3 — destination-side failure (open/write/close DST); same shape +// exit 0: success; "copied bytes" on stdout +// exit 2: source-side failure (open/read SRC); "copyx: " on stderr +// exit 3: destination-side failure (open/write/close DST); same shape // // EINTR on read(2)/write(2) is retried (the syscall was interrupted before // completing; reissuing it is the only correct policy), and short writes are @@ -46,8 +46,8 @@ func newOpError(op, path string, err error, exit int) *opError { return &opError{op: op, path: path, err: errno, exit: exit} } -// strerror renders an errno the way glibc's strerror(3) does — first letter -// capitalized — so all three implementations print identical reason text. +// strerror renders an errno the way glibc's strerror(3) does, first letter +// capitalized, so all three implementations print identical reason text. func strerror(errno unix.Errno) string { r := []rune(errno.Error()) if len(r) > 0 { diff --git a/examples/05-errors-three-ways/rust/src/main.rs b/examples/05-errors-three-ways/rust/src/main.rs index b1f9658..1b2f028 100644 --- a/examples/05-errors-three-ways/rust/src/main.rs +++ b/examples/05-errors-three-ways/rust/src/main.rs @@ -1,8 +1,8 @@ -//! copyx SRC DST — file copier that establishes the book's error taxonomy. +//! copyx SRC DST: file copier that establishes the book's error taxonomy. //! -//! exit 0 — success; "copied bytes" on stdout -//! exit 2 — source-side failure (open/read SRC); "copyx: " on stderr -//! exit 3 — destination-side failure (open/write/close DST); same shape +//! exit 0: success; "copied bytes" on stdout +//! exit 2: source-side failure (open/read SRC); "copyx: " on stderr +//! exit 3: destination-side failure (open/write/close DST); same shape //! //! EINTR on read(2)/write(2) is retried (the syscall was interrupted before //! completing; reissuing it is the only correct policy), and short writes are @@ -42,7 +42,7 @@ impl CopyError { /// Render an errno the way glibc's strerror(3) does, so all three /// implementations print identical reason text. `Errno`'s own Display goes -/// through `std::io::Error`, which appends " (os error N)" — strip it. +/// through `std::io::Error`, which appends " (os error N)"; strip it. fn strerror(errno: Errno) -> String { let s = std::io::Error::from(errno).to_string(); match s.rfind(" (os error ") { diff --git a/examples/06-concurrency-three-ways/README.md b/examples/06-concurrency-three-ways/README.md index d46bdd5..ad0b897 100644 --- a/examples/06-concurrency-three-ways/README.md +++ b/examples/06-concurrency-three-ways/README.md @@ -1,7 +1,7 @@ # 06-concurrency-three-ways -Chapter 6's example: `parhash`, a parallel checksummer implemented three times -— C++23, Go, and Rust — with each language's native concurrency toolkit, but +Chapter 6's example: `parhash`, a parallel checksummer implemented three times, +C++23, Go, and Rust, with each language's native concurrency toolkit, but one shared observable contract so a single `verify.lua` covers all three. ## The program @@ -11,7 +11,7 @@ parhash DIR ``` Walks every regular file under `DIR` (symlinks are never followed), hashes -each with FNV-1a 64 (implemented inline in all three sources — no hash +each with FNV-1a 64 (implemented inline in all three sources, no hash libraries), using a bounded pool of **4 workers**, then prints one line per file, **sorted by relative path**, followed by a summary: @@ -44,7 +44,7 @@ that completed, prints `parhash: interrupted` on stderr, and exits **130**. | **Go** | 4 worker goroutines receiving from an unbuffered `chan string` fed by a walker goroutine; results return on a second channel | `signal.NotifyContext` cancels a `context.Context`; workers `select` `ctx.Done()` ahead of the next path | | **Rust** | 4 scoped threads (`std::thread::scope`) sharing one `mpsc::Receiver` behind a `Mutex`; results return on a second `mpsc` channel | `signal-hook` sets a shared `AtomicBool` that walker and workers poll between files | -The Go build runs with `-race` (see `go/demo.sh`) — this is the concurrency +The Go build runs with `-race` (see `go/demo.sh`): this is the concurrency chapter, so the race detector referees every verification run. ## Layout @@ -62,15 +62,15 @@ chapter, so the race detector referees every verification run. Every language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (e.g. `./demo.sh run /etc`) +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (e.g. `./demo.sh run /etc`) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. All three `run` paths `exec` the binary, so a signal sent to the `demo.sh` -PID lands on `parhash` itself — which is exactly how `verify.lua` delivers +PID lands on `parhash` itself, which is exactly how `verify.lua` delivers its `SIGINT`. ## Verification @@ -79,7 +79,7 @@ its `SIGINT`. a time and asserts, per language: 1. a 5-file fixture (nested dirs, a binary file, an empty file) produces - stdout **byte-identical** (`diff -u`) to the precomputed expected output — + stdout **byte-identical** (`diff -u`) to the precomputed expected output: the same expected bytes for all three languages; 2. a missing argument prints the usage line and exits 2; 3. on a big sparse tree (~6 GiB logical, ~0 on disk, seconds of hashing), diff --git a/examples/06-concurrency-three-ways/cpp/src/main.cpp b/examples/06-concurrency-three-ways/cpp/src/main.cpp index 6d6e02e..517a073 100644 --- a/examples/06-concurrency-three-ways/cpp/src/main.cpp +++ b/examples/06-concurrency-three-ways/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// parhash: parallel FNV-1a 64 checksummer — C++23 flavor. +// parhash: parallel FNV-1a 64 checksummer: C++23 flavor. // // Concurrency shape: 4 std::jthread workers pull paths from a mutex-guarded // queue (condition_variable_any so waits are stop_token-aware). A dedicated diff --git a/examples/06-concurrency-three-ways/go/main.go b/examples/06-concurrency-three-ways/go/main.go index a978486..69109c6 100644 --- a/examples/06-concurrency-three-ways/go/main.go +++ b/examples/06-concurrency-three-ways/go/main.go @@ -1,4 +1,4 @@ -// parhash: parallel FNV-1a 64 checksummer — Go flavor. +// parhash: parallel FNV-1a 64 checksummer, Go flavor. // // Concurrency shape: 4 worker goroutines receive relative paths from an // unbuffered channel fed by a walker goroutine; results flow back on a second diff --git a/examples/06-concurrency-three-ways/rust/src/main.rs b/examples/06-concurrency-three-ways/rust/src/main.rs index bab4771..df1f62d 100644 --- a/examples/06-concurrency-three-ways/rust/src/main.rs +++ b/examples/06-concurrency-three-ways/rust/src/main.rs @@ -1,4 +1,4 @@ -// parhash: parallel FNV-1a 64 checksummer — Rust flavor. +// parhash: parallel FNV-1a 64 checksummer, Rust flavor. // // Concurrency shape: 4 scoped threads share one mpsc Receiver behind a Mutex // and pull relative paths from it; results flow back to the main thread on a diff --git a/examples/07-fds-raii-and-the-vfs/README.md b/examples/07-fds-raii-and-the-vfs/README.md index 6c97097..c6888c4 100644 --- a/examples/07-fds-raii-and-the-vfs/README.md +++ b/examples/07-fds-raii-and-the-vfs/README.md @@ -1,18 +1,18 @@ -# 07 — fds, RAII, and the VFS: `fwatch` v0 +# 07: fds, RAII, and the VFS, `fwatch` v0 The file-watcher artifact makes its first appearance. `fwatch` v0 has no -events and no timing — it is a purely deterministic snapshot/diff tool built +events and no timing: it is a purely deterministic snapshot/diff tool built on directory file descriptors, implemented three times (C++23, Go, Rust) with identical observable behavior. The point of the chapter: a directory is just another fd. Every implementation -walks the tree with *dirfd-relative* syscalls — `openat(2)` to descend, -`fstatat(2)` with `AT_SYMLINK_NOFOLLOW` to classify — and every fd lives +walks the tree with *dirfd-relative* syscalls (`openat(2)` to descend, +`fstatat(2)` with `AT_SYMLINK_NOFOLLOW` to classify), and every fd lives inside an owning wrapper so it is closed exactly once on every path, including errors: - **C++**: a move-only `Fd` class (RAII around `close(2)`) and a `DirStream` - class around `DIR*` — note that `fdopendir(3)` *takes ownership* of the fd, + class around `DIR*`. `fdopendir(3)` *takes ownership* of the fd, so `DirStream::adopt(Fd)` releases the fd into the stream and `closedir(3)` is the single cleanup. Fallible operations return `std::expected` (or a small `Failure` carrying the @@ -22,7 +22,7 @@ including errors: Errors are wrapped with `%w`; `errors.Is(err, unix.ENOENT)` skips entries that vanish mid-walk and `errors.As` digs the raw errno back out for the common message format. -- **Rust**: `rustix::fs::{open, openat, statat}` returning `OwnedFd` — drop is +- **Rust**: `rustix::fs::{open, openat, statat}` returning `OwnedFd`; drop is the close. `rustix::fs::Dir::read_from` dups the fd for its own cursor, so the `OwnedFd` stays usable as the anchor for the `*at()` calls. `anyhow` carries the error context; `?` does the plumbing. @@ -48,7 +48,7 @@ devices are ignored in v0. Entries that disappear between `readdir` and `fstatat` are silently skipped. **`fwatch diff DIR SNAPSHOT_FILE`** re-scans `DIR`, compares against the -saved snapshot, and prints — in sorted path order — +saved snapshot, and prints, in sorted path order, ``` + new file @@ -82,11 +82,11 @@ stderr). Every language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh` — build, then run a self-contained fixture demo: create a small +- `./demo.sh`: build, then run a self-contained fixture demo: create a small tree in a temp dir, `snapshot` it, mutate it (append / remove / create), and `diff` it against the saved snapshot. -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary with `args` +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary with `args` (e.g. `./demo.sh run snapshot /etc`) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of @@ -106,7 +106,7 @@ behavior end to end: `fwatch: 0 added, 0 removed, 0 modified`. 3. After adding `new.txt`, deleting `gone.txt`, and appending to + retouching `a.txt`, `diff` prints exactly `~ a.txt`, `- gone.txt`, `+ new.txt`, and - the `1 added, 1 removed, 1 modified` summary — nothing else. + the `1 added, 1 removed, 1 modified` summary; nothing else. 4. A fresh `snapshot` reflects the new size and mtime of `a.txt`. 5. Error paths: missing directory → exit 1 with `(errno 2)`; missing snapshot file → exit 1; missing arguments → exit 2 with the usage text. diff --git a/examples/07-fds-raii-and-the-vfs/cpp/src/main.cpp b/examples/07-fds-raii-and-the-vfs/cpp/src/main.cpp index 53bfa80..4cbe969 100644 --- a/examples/07-fds-raii-and-the-vfs/cpp/src/main.cpp +++ b/examples/07-fds-raii-and-the-vfs/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// fwatch v0 — snapshot/diff a directory tree with dirfd-relative syscalls. +// fwatch v0: snapshot/diff a directory tree with dirfd-relative syscalls. // // fwatch snapshot DIR one " " line per // regular file, sorted by path diff --git a/examples/07-fds-raii-and-the-vfs/go/main.go b/examples/07-fds-raii-and-the-vfs/go/main.go index 8040b5d..17d6cbf 100644 --- a/examples/07-fds-raii-and-the-vfs/go/main.go +++ b/examples/07-fds-raii-and-the-vfs/go/main.go @@ -1,4 +1,4 @@ -// fwatch v0 — snapshot/diff a directory tree with dirfd-relative syscalls. +// fwatch v0: snapshot/diff a directory tree with dirfd-relative syscalls. // // fwatch snapshot DIR one " " line per // regular file, sorted by path diff --git a/examples/07-fds-raii-and-the-vfs/rust/src/main.rs b/examples/07-fds-raii-and-the-vfs/rust/src/main.rs index e35edc1..4b791d5 100644 --- a/examples/07-fds-raii-and-the-vfs/rust/src/main.rs +++ b/examples/07-fds-raii-and-the-vfs/rust/src/main.rs @@ -1,4 +1,4 @@ -// fwatch v0 — snapshot/diff a directory tree with dirfd-relative syscalls. +// fwatch v0: snapshot/diff a directory tree with dirfd-relative syscalls. // // fwatch snapshot DIR one " " line per // regular file, sorted by path @@ -7,7 +7,7 @@ // summary // // The walk is deliberately explicit: rustix::fs::openat relative to the -// parent directory fd (an OwnedFd — closed exactly once, on drop), +// parent directory fd (an OwnedFd, closed exactly once, on drop), // rustix::fs::Dir to enumerate, and rustix::fs::statat with // SYMLINK_NOFOLLOW to classify entries. @@ -24,7 +24,7 @@ struct Info { mtime: i64, } -/// Sorted by path, bytewise — BTreeMap keeps snapshot output deterministic. +/// Sorted by path, bytewise: BTreeMap keeps snapshot output deterministic. type Tree = BTreeMap; const DIR_FLAGS: OFlags = OFlags::RDONLY diff --git a/examples/08-page-cache-and-durability/README.md b/examples/08-page-cache-and-durability/README.md index 8c7857d..98b0c43 100644 --- a/examples/08-page-cache-and-durability/README.md +++ b/examples/08-page-cache-and-durability/README.md @@ -1,8 +1,9 @@ -# 08-page-cache-and-durability — iobench +# 08-page-cache-and-durability: iobench -Chapter 8's example: the same small write benchmark implemented three times — -C++23, Go, and Rust — to show what the page cache gives you (throughput) and -what it defers (durability). One binary, three sync strategies: +This chapter's example: the same small write benchmark implemented three +times, in C++23, Go, and Rust, to show what the page cache gives you +(throughput) and what it defers (durability). One binary, three sync +strategies: ``` iobench --mode buffered|fsync-every|direct [--every N] [--size-mb M] FILE @@ -10,14 +11,14 @@ iobench --mode buffered|fsync-every|direct [--every N] [--size-mb M] FILE It writes M MiB (default 64) to FILE in 64 KiB blocks: -- `buffered` — plain `write(2)`s land in the page cache. The report line times +- `buffered`: plain `write(2)`s land in the page cache. The report line times the writes alone; a second line `fsync_ms=` times the closing - `fdatasync(2)` that actually makes the data durable. The gap between the two + `fdatasync(2)` that makes the data durable. The gap between the two is the chapter's point. -- `fsync-every` — `fdatasync(2)` every N blocks (default 8, plus a trailing +- `fsync-every`: `fdatasync(2)` every N blocks (default 8, plus a trailing sync if the block count is not a multiple of N), all inside the timed region. This is what "durable as you go" costs. -- `direct` — `O_DIRECT` with 4096-aligned buffers, bypassing the page cache. +- `direct`: `O_DIRECT` with 4096-aligned buffers, bypassing the page cache. If the filesystem rejects `O_DIRECT` at `open(2)` with `EINVAL`, the program prints `direct: unsupported on this filesystem` and exits 4. @@ -46,34 +47,34 @@ mode=direct bytes=67108864 ms=81 MiB/s=786.2 Buffered "throughput" is mostly the memcpy into the cache; the durable-as-you-go and cache-bypass modes pay the real device cost inside the timed region. On -tmpfs (`/tmp`), `fdatasync` is nearly free and the three modes converge — -worth showing in class, and why `verify.lua` asserts shapes, not timings. +tmpfs (`/tmp`), `fdatasync` is nearly free and the three modes converge, which +is why `verify.lua` asserts shapes, not timings. ## Implementations All three expose identical observable behavior (CLI, output, exit codes): -- `cpp/` — C++23: RAII `Fd` wrapper around the descriptor, +- `cpp/`: C++23: RAII `Fd` wrapper around the descriptor, `std::expected` for every fallible op, `std::println`, `std::aligned_alloc` under a `unique_ptr` for the O_DIRECT buffer. -- `go/` — Go 1.26: raw syscalls via `golang.org/x/sys/unix` +- `go/`: Go 1.26: raw syscalls via `golang.org/x/sys/unix` (`Open`/`Write`/`Fdatasync`), wrapped errors with `%w`, `errors.Is(err, unix.EINVAL)` for the O_DIRECT probe, manual 4096 alignment of the block. -- `rust/` — Rust 2024: `rustix` (`open`/`write`/`fdatasync` over `OwnedFd`), +- `rust/`: Rust 2024: `rustix` (`open`/`write`/`fdatasync` over `OwnedFd`), `anyhow` with context for error paths, `Errno::INVAL` match for the O_DIRECT probe, aligned block carved from a stable `Vec`. -Every write loop retries short writes and `EINTR` — a 64 KiB `write(2)` is not +Every write loop retries short writes and `EINTR`: a 64 KiB `write(2)` is not guaranteed to complete in one call. ## The demo contract Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -91,9 +92,9 @@ exit status: - fsync-every 4 MiB (`--every 4`) → single report line with `bytes=4194304`, no `fsync_ms` line, file exactly 4194304 bytes - direct 4 MiB → either the report line and correct size, or the documented - `EINVAL` message with exit 4 — both are valid outcomes per filesystem + `EINVAL` message with exit 4; both are valid outcomes per filesystem Timing fields are asserted for shape only (`ms=%d+`, `MiB/s=%d+%.%d`); asserting magnitudes would flake. An `strace -c` cross-check of `--mode fsync-every --every 4 --size-mb 4` shows exactly 64 data writes and -16 `fdatasync` calls — the syscall pattern the chapter describes. +16 `fdatasync` calls, the syscall pattern the chapter describes. diff --git a/examples/08-page-cache-and-durability/cpp/src/main.cpp b/examples/08-page-cache-and-durability/cpp/src/main.cpp index 38d16f9..366e6b6 100644 --- a/examples/08-page-cache-and-durability/cpp/src/main.cpp +++ b/examples/08-page-cache-and-durability/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// iobench: page cache vs durability — buffered writes, periodic fdatasync(2), +// iobench: page cache vs durability: buffered writes, periodic fdatasync(2), // and O_DIRECT, with identical CLI and output across the three languages. // // iobench --mode buffered|fsync-every|direct [--every N] [--size-mb M] FILE diff --git a/examples/08-page-cache-and-durability/go/main.go b/examples/08-page-cache-and-durability/go/main.go index 6697b2d..60ea894 100644 --- a/examples/08-page-cache-and-durability/go/main.go +++ b/examples/08-page-cache-and-durability/go/main.go @@ -1,4 +1,4 @@ -// iobench: page cache vs durability — buffered writes, periodic fdatasync(2), +// iobench: page cache vs durability: buffered writes, periodic fdatasync(2), // and O_DIRECT, with identical CLI and output across the three languages. // // iobench --mode buffered|fsync-every|direct [--every N] [--size-mb M] FILE diff --git a/examples/08-page-cache-and-durability/rust/src/main.rs b/examples/08-page-cache-and-durability/rust/src/main.rs index 5f10bb3..6c423ef 100644 --- a/examples/08-page-cache-and-durability/rust/src/main.rs +++ b/examples/08-page-cache-and-durability/rust/src/main.rs @@ -1,5 +1,6 @@ -// iobench: page cache vs durability — buffered writes, periodic fdatasync(2), -// and O_DIRECT, with identical CLI and output across the three languages. +// iobench: page cache vs durability, comparing buffered writes, periodic +// fdatasync(2), and O_DIRECT, with identical CLI and output across the three +// languages. // // iobench --mode buffered|fsync-every|direct [--every N] [--size-mb M] FILE // diff --git a/examples/08-page-cache-and-durability/verify.lua b/examples/08-page-cache-and-durability/verify.lua index b0389dd..2772434 100644 --- a/examples/08-page-cache-and-durability/verify.lua +++ b/examples/08-page-cache-and-durability/verify.lua @@ -1,5 +1,5 @@ -- Verify 08-page-cache-and-durability: run iobench in buffered, fsync-every, --- and direct modes and assert the observable behavior — report line shapes, +-- and direct modes and assert the observable behavior: report line shapes, -- exact byte counts, the on-disk file size (stat), and exit codes. Timing -- values are asserted for shape only (numbers flake). -- Invoked from the example directory with LSP_LANG (cpp|go|rust) and EXAMPLE_DIR set. diff --git a/examples/09-event-loops-epoll-inotify/README.md b/examples/09-event-loops-epoll-inotify/README.md index f437e55..9c3cd66 100644 --- a/examples/09-event-loops-epoll-inotify/README.md +++ b/examples/09-event-loops-epoll-inotify/README.md @@ -1,14 +1,14 @@ -# 09 — Event loops: epoll + inotify (`fwatch` v1) +# 09: Event loops: epoll + inotify (`fwatch` v1) -`fwatch` is the file watcher that grows across Part II. Chapter 07/08 gave it -polling-era subcommands (`snapshot`, `diff`); this chapter adds `watch`, a -real-time watcher — and the three implementations deliberately showcase the -two ways Linux programs consume readiness: +`fwatch` is the file watcher that grows across Part II. The previous +chapters gave it polling-era subcommands (`snapshot`, `diff`); this chapter +adds `watch`, a real-time watcher, and the three implementations showcase +the two ways Linux programs consume readiness: | Language | Architecture | |---|---| -| **C++23** | one explicit `epoll` loop over three fds — the inotify fd, a `timerfd` (drives both the 100 ms per-path debounce and the overall timeout), and a `signalfd` (SIGINT/SIGTERM as readable data). Single thread; every fd owned by an RAII wrapper; fallible setup returns `std::expected`. | -| **Rust (edition 2024)** | the same single-threaded epoll architecture via the `nix` crate (`Epoll`, `Inotify`, `TimerFd`, `SignalFd` — all `OwnedFd`-backed, closed on drop), with `anyhow` for error context. | +| **C++23** | one explicit `epoll` loop over three fds: the inotify fd, a `timerfd` (drives both the 100 ms per-path debounce and the overall timeout), and a `signalfd` (SIGINT/SIGTERM as readable data). Single thread; every fd owned by an RAII wrapper; fallible setup returns `std::expected`. | +| **Rust (edition 2024)** | the same single-threaded epoll architecture via the `nix` crate (`Epoll`, `Inotify`, `TimerFd`, `SignalFd`, all `OwnedFd`-backed, closed on drop), with `anyhow` for error context. | | **Go 1.26** | the idiomatic inversion: the runtime owns the poller. A goroutine reads the `O_NONBLOCK` inotify fd through `os.File` (parking on the netpoller, which is itself an epoll loop), and feeds a channel; `signal.Notify` and `time.After` are channels too; one `select` loop multiplexes all of it. | All three expose identical observable behavior, so one `verify.lua` covers @@ -23,12 +23,12 @@ usage: fwatch watch DIR [--timeout-ms T] watch DIR (default 2000 ms) until timeout or SIGINT/SIGTERM ``` -- `snapshot DIR` — top-level regular files only, sorted by name, printed to +- `snapshot DIR`: top-level regular files only, sorted by name, printed to stdout as `name\tsize\tmtime_ns`. -- `diff OLD NEW` — reads two snapshot files and prints one line per changed +- `diff OLD NEW`: reads two snapshot files and prints one line per changed name, in name order: `created `, `deleted `, `modified ` (size or mtime changed). Exit 0. -- `watch DIR` — prints `fwatch: watching DIR` to stderr, then one line per +- `watch DIR`: prints `fwatch: watching DIR` to stderr, then one line per filesystem event to stdout: ``` @@ -47,11 +47,11 @@ usage: fwatch Each language directory has the standard `demo.sh`: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (`app`); the local path +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (`app`); the local path `exec`s the binary so a backgrounded `run watch ...` can be signalled directly -- `./demo.sh` — build, then run a 1-second self-demo (`watch .`) +- `./demo.sh`: build, then run a 1-second self-demo (`watch .`) - With env `TARGET` set, `run` deploys to that lab VM via `scripts/lab/deploy-to-vm.sh` diff --git a/examples/09-event-loops-epoll-inotify/cpp/src/main.cpp b/examples/09-event-loops-epoll-inotify/cpp/src/main.cpp index f8ca463..6269083 100644 --- a/examples/09-event-loops-epoll-inotify/cpp/src/main.cpp +++ b/examples/09-event-loops-epoll-inotify/cpp/src/main.cpp @@ -1,11 +1,11 @@ -// fwatch v1 — a file watcher growing chapter by chapter. +// fwatch v1: a file watcher growing chapter by chapter. // // v0 subcommands (snapshot/diff) still work; v1 adds `watch`, built as ONE // epoll loop over three kernel file descriptors on a single thread: // -// inotify fd — filesystem events for the watched directory -// timerfd — the debounce (100 ms per path) AND the overall timeout -// signalfd — SIGINT/SIGTERM delivered as ordinary readable data +// inotify fd: filesystem events for the watched directory +// timerfd: the debounce (100 ms per path) AND the overall timeout +// signalfd: SIGINT/SIGTERM delivered as ordinary readable data // // The readiness model is explicit: nothing here blocks except epoll_wait(2). @@ -51,7 +51,7 @@ constexpr auto kDebounce = milliseconds{100}; } // --------------------------------------------------------------------------- -// RAII file descriptor — the only owner of every fd in this program. +// RAII file descriptor: the only owner of every fd in this program. // --------------------------------------------------------------------------- class Fd { @@ -193,7 +193,7 @@ int cmd_diff(const std::string& old_path, const std::string& new_path) { } // --------------------------------------------------------------------------- -// v1: watch — one epoll loop, three fds, zero threads beyond main. +// v1: watch: one epoll loop, three fds, zero threads beyond main. // --------------------------------------------------------------------------- enum class Kind : std::uint8_t { created, modified, deleted }; @@ -329,7 +329,7 @@ int cmd_watch(const std::string& dir, int timeout_ms) { for (;;) { // One timer covers both jobs: it is always armed to whichever comes - // first — the overall timeout or the earliest per-path debounce. + // first: the overall timeout or the earliest per-path debounce. auto next = deadline; for (const auto& [_, p] : pending) { next = std::min(next, p.due); diff --git a/examples/09-event-loops-epoll-inotify/go/main.go b/examples/09-event-loops-epoll-inotify/go/main.go index d87fa83..45a9715 100644 --- a/examples/09-event-loops-epoll-inotify/go/main.go +++ b/examples/09-event-loops-epoll-inotify/go/main.go @@ -1,4 +1,4 @@ -// fwatch v1 — a file watcher growing chapter by chapter. +// fwatch v1: a file watcher growing chapter by chapter. // // v0 subcommands (snapshot/diff) still work; v1 adds `watch`. Where the C++ // and Rust versions run one explicit epoll loop over inotify + timerfd + @@ -114,7 +114,7 @@ func cmdDiff(oldPath, newPath string) error { } // --------------------------------------------------------------------------- -// v1: watch — goroutine + channels; the runtime owns the epoll loop. +// v1 (watch): goroutine + channels; the runtime owns the epoll loop. // --------------------------------------------------------------------------- type fsEvent struct { diff --git a/examples/09-event-loops-epoll-inotify/rust/src/main.rs b/examples/09-event-loops-epoll-inotify/rust/src/main.rs index dd800d4..93d99e2 100644 --- a/examples/09-event-loops-epoll-inotify/rust/src/main.rs +++ b/examples/09-event-loops-epoll-inotify/rust/src/main.rs @@ -1,12 +1,12 @@ -// fwatch v1 — a file watcher growing chapter by chapter. +// fwatch v1: a file watcher growing chapter by chapter. // // v0 subcommands (snapshot/diff) still work; v1 adds `watch`, built exactly // like the C++ version: ONE epoll loop over three kernel file descriptors on -// a single thread — +// a single thread: // -// inotify fd — filesystem events for the watched directory -// timerfd — the debounce (100 ms per path) AND the overall timeout -// signalfd — SIGINT/SIGTERM delivered as ordinary readable data +// inotify fd : filesystem events for the watched directory +// timerfd : the debounce (100 ms per path) AND the overall timeout +// signalfd : SIGINT/SIGTERM delivered as ordinary readable data // // nix wraps each fd in an owning type (OwnedFd underneath), so every resource // closes on drop; the readiness model stays explicit. @@ -88,7 +88,7 @@ fn cmd_diff(old_path: &str, new_path: &str) -> Result<()> { } // --------------------------------------------------------------------------- -// v1: watch — one epoll loop, three fds, zero threads beyond main. +// v1 watch: one epoll loop, three fds, zero threads beyond main. // --------------------------------------------------------------------------- #[derive(Clone, Copy, PartialEq, Eq)] @@ -198,7 +198,7 @@ fn cmd_watch(dir: &str, timeout_ms: u64) -> Result<()> { loop { // One timer covers both jobs: it is always armed to whichever comes - // first — the overall timeout or the earliest per-path debounce. + // first: the overall timeout or the earliest per-path debounce. let next = pending.values().map(|p| p.due).min().map_or(deadline, |d| d.min(deadline)); arm_timer(&timer, next.saturating_duration_since(Instant::now()))?; diff --git a/examples/09-event-loops-epoll-inotify/verify.lua b/examples/09-event-loops-epoll-inotify/verify.lua index f183ffd..7fc039e 100644 --- a/examples/09-event-loops-epoll-inotify/verify.lua +++ b/examples/09-event-loops-epoll-inotify/verify.lua @@ -1,7 +1,7 @@ -- Verify fwatch v1 (09-event-loops-epoll-inotify) for LSP_LANG. -- -- Asserts observable behavior, identical across cpp/go/rust: --- 1. v0 regression — snapshot format and diff created/modified/deleted lines +-- 1. v0 regression: snapshot format and diff created/modified/deleted lines -- 2. watch: background run, mutate the directory after ~400 ms, expect one -- debounced event line per path and a clean "(timeout)" exit, exit 0 -- 3. watch: SIGTERM produces the "(signal)" exit line, exit 0 diff --git a/examples/10-io-uring/README.md b/examples/10-io-uring/README.md index 758c704..9bd5044 100644 --- a/examples/10-io-uring/README.md +++ b/examples/10-io-uring/README.md @@ -1,6 +1,6 @@ -# 10-io-uring — fwatch v2, the batched sync engine +# 10-io-uring: fwatch v2, the batched sync engine -Chapter 10 grows `fwatch` into a small directory sync tool with two copy +This chapter grows `fwatch` into a small directory sync tool with two copy engines, and uses the difference between them to teach io_uring: submission and completion rings shared with the kernel, batches of SQEs per syscall, and linked operations. @@ -27,10 +27,10 @@ fwatch sync SRCDIR DSTDIR [--engine rw|uring] # v2: batched copy (default rw) ## The two engines - `--engine rw` (all three languages): a plain `read(2)`/`write(2)` loop, - 128 KiB at a time — the baseline from earlier chapters. + 128 KiB at a time, the baseline from earlier chapters. - `--engine uring`: - **C++**: a self-contained educational io_uring layer built on raw - `syscall(2)` wrappers — `io_uring_setup(2)`, `io_uring_enter(2)`, and the + `syscall(2)` wrappers: `io_uring_setup(2)`, `io_uring_enter(2)`, and the `mmap(2)`'d SQ ring, CQ ring, and SQE array (no liburing; the host lacks `liburing-devel`, and seeing the rings by hand is the point). Each file is copied as batches of linked READ→WRITE SQE pairs (`IOSQE_IO_LINK`), one @@ -40,8 +40,8 @@ fwatch sync SRCDIR DSTDIR [--engine rw|uring] # v2: batched copy (default rw) - **Rust**: the same batched linked-pair design on the `io-uring` crate (0.7), which owns the ring setup/mmap details the C++ version spells out. - **Go**: prints `engine=uring: unsupported in Go (see chapter 10)` and - exits **64**. That is deliberate, not a gap: the Go runtime already - multiplexes goroutines over epoll in the netpoller, and io_uring support + exits **64** by design: the Go runtime already multiplexes goroutines + over epoll in the netpoller, and io_uring support (golang/go#31908) remains on hold. The chapter uses this asymmetry to discuss where each language's I/O story comes from. @@ -49,8 +49,8 @@ fwatch sync SRCDIR DSTDIR [--engine rw|uring] # v2: batched copy (default rw) Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (builds first if needed) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (builds first if needed) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -69,9 +69,9 @@ mkdir -p /tmp/fw/src && echo hi > /tmp/fw/src/a.txt ## Verification -`verify.lua` (run per language with `LSP_LANG` set) builds a fixture tree — -five files totalling 1,348,634 bytes, sized to straddle the 128 KiB chunk and -the 8-pair batch limit — then asserts: +`verify.lua` (run per language with `LSP_LANG` set) builds a fixture tree +(five files totalling 1,348,634 bytes, sized to straddle the 128 KiB chunk +and the 8-pair batch limit), then asserts: - exact `scanned`/`synced` counter lines, - `diff -r` byte-equality of the rw copy (all languages) and the uring copy diff --git a/examples/10-io-uring/cpp/src/main.cpp b/examples/10-io-uring/cpp/src/main.cpp index 61465e3..64ad6d8 100644 --- a/examples/10-io-uring/cpp/src/main.cpp +++ b/examples/10-io-uring/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// fwatch v2 — batched sync engine (chapter 10: io_uring). +// fwatch v2: batched sync engine (chapter 10: io_uring). // // Subcommands: // fwatch scan DIR (v0: polling snapshot) diff --git a/examples/10-io-uring/go/main.go b/examples/10-io-uring/go/main.go index 25bb54b..47277c4 100644 --- a/examples/10-io-uring/go/main.go +++ b/examples/10-io-uring/go/main.go @@ -1,4 +1,4 @@ -// fwatch v2 — batched sync engine (chapter 10: io_uring). +// fwatch v2: batched sync engine (chapter 10: io_uring). // // Subcommands: // @@ -9,7 +9,7 @@ // Go has no supported io_uring story: the runtime already multiplexes // goroutines over epoll in the netpoller, and the io_uring proposal // (golang/go#31908) has been on hold for years. So the uring engine is -// deliberately absent here — "fwatch sync ... --engine uring" reports it +// deliberately absent here: "fwatch sync ... --engine uring" reports it // unsupported and exits 64. That asymmetry IS the chapter's point. package main diff --git a/examples/10-io-uring/rust/src/main.rs b/examples/10-io-uring/rust/src/main.rs index c0a66f3..71e825c 100644 --- a/examples/10-io-uring/rust/src/main.rs +++ b/examples/10-io-uring/rust/src/main.rs @@ -1,4 +1,4 @@ -// fwatch v2 — batched sync engine (chapter 10: io_uring). +// fwatch v2: batched sync engine (chapter 10: io_uring). // // Subcommands: // fwatch scan DIR (v0: polling snapshot) @@ -102,8 +102,8 @@ fn copy_file_uring(ring: &mut IoUring, src: &Path, dst: &Path) -> Result { let len = CHUNK.min(size - off) as u32; lens[pairs] = len; let buf = bufs[pairs].as_mut_ptr(); - // READ this chunk into the pair's buffer, then — only if it - // fully succeeded (IO_LINK) — WRITE it at the same offset. + // READ this chunk into the pair's buffer, then, only if it + // fully succeeded (IO_LINK), WRITE it at the same offset. let read = opcode::Read::new(types::Fd(input.as_raw_fd()), buf, len) .offset(off) .build() diff --git a/examples/10-io-uring/verify.lua b/examples/10-io-uring/verify.lua index da9be6f..ee3f948 100644 --- a/examples/10-io-uring/verify.lua +++ b/examples/10-io-uring/verify.lua @@ -1,5 +1,5 @@ -- Verify fwatch v2: build a fixture tree, then assert observable behavior of --- every subcommand — exact scan/sync counters, byte-equal copies (diff -r), +-- every subcommand: exact scan/sync counters, byte-equal copies (diff -r), -- inotify event lines from watch, and the Go uring refusal (exit 64). -- Invoked from the example directory with LSP_LANG (cpp|go|rust) and EXAMPLE_DIR set. @@ -42,7 +42,7 @@ r = demo("scan " .. src) checks.expect_exit(r, 0, lang .. ": scan exits 0") checks.expect_match(r.out, "scanned 5 files 1348634 bytes", lang .. ": scan totals") --- v2: sync with the rw engine — counters plus a byte-identical tree. +-- v2: sync with the rw engine (counters plus a byte-identical tree). local dst_rw = ws .. "/dst-rw" r = demo("sync " .. src .. " " .. dst_rw .. " --engine rw") checks.expect_exit(r, 0, lang .. ": sync rw exits 0") diff --git a/examples/11-process-lifecycle/README.md b/examples/11-process-lifecycle/README.md index f540e9a..92b4c46 100644 --- a/examples/11-process-lifecycle/README.md +++ b/examples/11-process-lifecycle/README.md @@ -1,6 +1,6 @@ -# 11-process-lifecycle — pmon +# 11-process-lifecycle: pmon -A minimal process monitor, written three times — C++23, Go, and Rust — for +A minimal process monitor, written three times, in C++23, Go, and Rust, for the process lifecycle chapter: spawn a command, wait for it, and report the two things a supervisor always needs, *how the child died* and *what it cost*. @@ -39,7 +39,7 @@ $ echo $? Design decisions shared by all three: -- **The exit code mirrors the child** — the shell convention (`status`, or +- **The exit code mirrors the child**: the shell convention (`status`, or `128+signal`, or `127` for "could not exec"), so pmon composes with `&&`, `make`, and CI the same way the shell itself would. - **Signal names come from a hand-rolled table** (`15 -> TERM`, `9 -> KILL`, @@ -47,14 +47,14 @@ Design decisions shared by all three: locale-dependent prose ("Terminated") and differs across runtimes. - **`` is `strerror(3)`-shaped** (`No such file or directory`). Go's errno strings are lowercase, so the first letter is capitalized, and a PATH - miss (`exec.ErrNotFound`) is rendered as ENOENT — exactly what `execvp(3)` + miss (`exec.ErrNotFound`) is rendered as ENOENT, exactly what `execvp(3)` reports in the other two. - **Child stdio is inherited**, so the child's own output interleaves naturally and the report lines land after the wait. ### How each language spells the same lifecycle -- **C++23** (`cpp/`): the chapter's two-step model spelled out — +- **C++23** (`cpp/`): the chapter's two-step model spelled out: `fork(2)` + `execvp(3)` in the child, `waitpid(2)` + `getrusage(RUSAGE_CHILDREN)` in the parent (`posix_spawn(3)` is discussed in the chapter; fork is used here for teaching). Exec failure travels over @@ -62,7 +62,7 @@ Design decisions shared by all three: copy and the parent reads EOF, while a failed exec ships `errno` up before `_exit(127)`. A move-only RAII `Fd` owns each pipe end; fallible steps return `std::expected`; output is `std::println`. -- **Go** (`go/`): `os/exec` — whose `Start` performs the same +- **Go** (`go/`): `os/exec`, whose `Start` performs the same fork/exec-with-status-pipe dance internally, surfacing exec failure as an error before any child ran. The wait-side facts come from `os.ProcessState`: `Sys()` is the raw `syscall.WaitStatus`, `SysUsage()` @@ -77,8 +77,8 @@ Design decisions shared by all three: `strerror(3)` text. The wait wrapper is `io::Result` + `?`, with EINTR retried. -One semantic wrinkle worth noticing: C++ asks for `RUSAGE_CHILDREN` (all -waited-for children — equivalent here because there is exactly one), while Go +One semantic difference: C++ asks for `RUSAGE_CHILDREN` (all +waited-for children, equivalent here because there is exactly one), while Go and Rust read the per-child rusage that `wait4(2)` returns. Same numbers, two kernel interfaces. @@ -86,11 +86,11 @@ kernel interfaces. Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary with `args` +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary with `args` (with env `TARGET` set, deploys to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally) -- `./demo.sh` — build, then a local walkthrough of the four child fates: +- `./demo.sh`: build, then a local walkthrough of the four child fates: clean exit, `exit 42`, death by SIGTERM (exit 143), exec failure (127) The top-level `demo.sh` dispatches: `./demo.sh cpp run -- /bin/true`, @@ -103,7 +103,7 @@ behavior, not exit-0: - `/bin/true` / `/bin/false` / `sh -c 'exit 42'` report the status and mirror it in pmon's own exit code; -- the pid in the fate line is the *real* child pid — the child echoes `$$` +- the pid in the fate line is the *real* child pid: the child echoes `$$` and the two must agree; - `kill -TERM $$` reports `killed by signal 15 (TERM)` with exit 143, `kill -KILL $$` reports `signal 9 (KILL)` with exit 137; diff --git a/examples/11-process-lifecycle/cpp/src/main.cpp b/examples/11-process-lifecycle/cpp/src/main.cpp index e22b75e..08aa68d 100644 --- a/examples/11-process-lifecycle/cpp/src/main.cpp +++ b/examples/11-process-lifecycle/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// pmon run -- CMD [ARGS...] — minimal process monitor for the process +// pmon run -- CMD [ARGS...]: minimal process monitor for the process // lifecycle chapter: fork(2) + execvp(3) the command, waitpid(2) for it, // then report its fate and resource usage. // @@ -11,8 +11,8 @@ // exits 127 with no report lines (there was no child run to report on). // // The chapter discusses posix_spawn(3); fork/exec is used here because the -// two-step model — and the CLOEXEC self-pipe that reports exec failure back -// to the parent — is exactly what the chapter teaches. +// two-step model (and the CLOEXEC self-pipe that reports exec failure back +// to the parent) is exactly what the chapter teaches. #include #include diff --git a/examples/11-process-lifecycle/go/main.go b/examples/11-process-lifecycle/go/main.go index 034f672..5bd490f 100644 --- a/examples/11-process-lifecycle/go/main.go +++ b/examples/11-process-lifecycle/go/main.go @@ -1,4 +1,4 @@ -// pmon run -- CMD [ARGS...] — minimal process monitor for the process +// pmon run -- CMD [ARGS...]: minimal process monitor for the process // lifecycle chapter: spawn CMD, wait for it, report its fate and rusage. // // pmon: pid

exited status child exited normally @@ -10,7 +10,7 @@ // exits 127 with no report lines. // // Where C++ forks and execs by hand, Go's os/exec does the same dance -// internally (fork/exec with a CLOEXEC status pipe — see syscall.forkExec); +// internally (fork/exec with a CLOEXEC status pipe; see syscall.forkExec); // the exec failure that the C++ version ships over its self-pipe surfaces // here as an error from cmd.Start. The wait-side facts come from // os.ProcessState: Sys() is the raw wait status, SysUsage() the rusage that diff --git a/examples/11-process-lifecycle/rust/src/main.rs b/examples/11-process-lifecycle/rust/src/main.rs index 697abc8..e982926 100644 --- a/examples/11-process-lifecycle/rust/src/main.rs +++ b/examples/11-process-lifecycle/rust/src/main.rs @@ -1,4 +1,4 @@ -//! pmon run -- CMD [ARGS...] — minimal process monitor for the process +//! pmon run -- CMD [ARGS...]: minimal process monitor for the process //! lifecycle chapter: spawn CMD, wait for it, report its fate and rusage. //! //! ```text diff --git a/examples/11-process-lifecycle/verify.lua b/examples/11-process-lifecycle/verify.lua index 6ebc440..f840ae5 100644 --- a/examples/11-process-lifecycle/verify.lua +++ b/examples/11-process-lifecycle/verify.lua @@ -9,7 +9,7 @@ -- exact maxrss/user/sys/wall shape with plausible values (wall covers a real -- 200 ms sleep, maxrss is nonzero, user CPU registers for a busy loop); and -- an exec failure prints only "pmon: exec : " on stderr with --- exit 127 — no fate or rusage lines for a child that never ran. +-- exit 127: no fate or rusage lines for a child that never ran. local script_dir = arg[0]:match("(.*/)") or "./" local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") diff --git a/examples/12-signals-and-safe-handling/README.md b/examples/12-signals-and-safe-handling/README.md index 0aa6b00..b6dc356 100644 --- a/examples/12-signals-and-safe-handling/README.md +++ b/examples/12-signals-and-safe-handling/README.md @@ -1,8 +1,8 @@ -# 12-signals-and-safe-handling — pmon v1, the supervising loop +# 12-signals-and-safe-handling: pmon v1, the supervising loop -Chapter 12 grows `pmon` from "run one child and report" into a real -supervisor — and uses the restart loop as a vehicle for the chapter's actual -subject: how to consume signals *safely*. Each language deliberately uses a +This chapter grows `pmon` from "run one child and report" into a real +supervisor, using the restart loop as a vehicle for its actual +subject: how to consume signals *safely*. Each language uses a different mechanism, because the three mechanisms together are the lesson. ``` @@ -34,36 +34,36 @@ pmon supervise [--max-restarts N] [--backoff-ms B] -- CMD # v1: restart on fail ## Three signal mechanisms, one contract The restart loop is identical; how each language *hears* a signal is not. -None of them runs meaningful code in signal context — that is the point. +None of them runs meaningful code in signal context: that is the point. -- **C++ — signalfd(2) + poll(2), zero handlers.** SIGTERM/SIGINT/SIGHUP/ +- **C++: signalfd(2) + poll(2), zero handlers.** SIGTERM/SIGINT/SIGHUP/ SIGCHLD are blocked with `sigprocmask(2)` and read as `signalfd_siginfo` records from a file descriptor inside a `poll` loop (the backoff is just the poll timeout). Nothing ever executes asynchronously, so the - async-signal-safe function list stops mattering — except between `fork()` + async-signal-safe function list stops mattering, except between `fork()` and `execvp()`, where the child restores an empty mask (an inherited blocked SIGTERM survives exec and would make the child unkillable) using only async-signal-safe calls. -- **Go — signal.Notify + select.** The runtime owns the actual handlers and +- **Go: signal.Notify + select.** The runtime owns the actual handlers and turns delivery into channel sends. Supervision is one `select` over three channels: signals, child exits from a `cmd.Wait()` goroutine, and the backoff `time.Timer`. Child liveness and signal forwarding go through `golang.org/x/sys/unix`. -- **Rust — the self-pipe trick via signal-hook.** Each signal is registered +- **Rust: the self-pipe trick via signal-hook.** Each signal is registered twice: `signal_hook::flag::register` sets a per-signal `AtomicBool`, and `low_level::pipe::register_raw` (which takes its dup of the pipe's write - end as an `OwnedFd` and makes it non-blocking) writes a doorbell byte — + end as an `OwnedFd` and makes it non-blocking) writes a doorbell byte: `write(2)` is async-signal-safe. The main loop polls the doorbell's read end with `nix::poll`, drains it, checks the flags, and asks - `waitpid(WNOHANG)` what actually happened. SIGCHLD needs no flag: the + `waitpid(WNOHANG)` what happened. SIGCHLD needs no flag: the wait status is the source of truth. ## Demo contract Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (builds first if needed) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (builds first if needed) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -94,7 +94,7 @@ exit codes alone: - SIGTERM and SIGINT to a live supervisor produce the shutdown line, exit 0, and a reaped child (`kill -0` on the child pid fails afterwards); - SIGHUP produces the reload line and a replacement child whose pid - differs, with the old pid dead and the replacement alive — and a + differs, with the old pid dead and the replacement alive, and a follow-up SIGTERM still shuts everything down cleanly. Run it standalone from this directory: diff --git a/examples/12-signals-and-safe-handling/cpp/src/main.cpp b/examples/12-signals-and-safe-handling/cpp/src/main.cpp index 959e412..6ef314b 100644 --- a/examples/12-signals-and-safe-handling/cpp/src/main.cpp +++ b/examples/12-signals-and-safe-handling/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// pmon v1 — a mini process supervisor (chapter 12: signals and safe handling). +// pmon v1: a mini process supervisor (chapter 12: signals and safe handling). // // pmon run -- CMD [ARGS...] // pmon supervise [--max-restarts N] [--backoff-ms B] -- CMD [ARGS...] @@ -7,7 +7,7 @@ // all. SIGTERM/SIGINT/SIGHUP/SIGCHLD are blocked with sigprocmask(2) and // converted into ordinary file-descriptor reads via signalfd(2), consumed // from a plain poll(2) loop. No code ever runs in signal context, so nothing -// has to be async-signal-safe — except the tiny window in the forked child +// has to be async-signal-safe: except the tiny window in the forked child // between fork() and execvp(), which restores an empty signal mask (a // blocked, inherited SIGTERM would survive exec and make the supervised // child unkillable). diff --git a/examples/12-signals-and-safe-handling/go/main.go b/examples/12-signals-and-safe-handling/go/main.go index 29a0166..20e6a7b 100644 --- a/examples/12-signals-and-safe-handling/go/main.go +++ b/examples/12-signals-and-safe-handling/go/main.go @@ -1,4 +1,4 @@ -// pmon v1 — a mini process supervisor (chapter 12: signals and safe handling). +// pmon v1: a mini process supervisor (chapter 12: signals and safe handling). // // pmon run -- CMD [ARGS...] // pmon supervise [--max-restarts N] [--backoff-ms B] -- CMD [ARGS...] diff --git a/examples/12-signals-and-safe-handling/rust/src/main.rs b/examples/12-signals-and-safe-handling/rust/src/main.rs index e909fbb..c64078d 100644 --- a/examples/12-signals-and-safe-handling/rust/src/main.rs +++ b/examples/12-signals-and-safe-handling/rust/src/main.rs @@ -1,11 +1,11 @@ -// pmon v1 — a mini process supervisor (chapter 12: signals and safe handling). +// pmon v1: a mini process supervisor (chapter 12: signals and safe handling). // // pmon run -- CMD [ARGS...] // pmon supervise [--max-restarts N] [--backoff-ms B] -- CMD [ARGS...] // // The Rust take on async-signal-safe design: the classic self-pipe trick, // with the unsafe parts delegated to the signal-hook crate. Each watched -// signal is registered twice — flag::register sets a per-signal AtomicBool, +// signal is registered twice: flag::register sets a per-signal AtomicBool, // and low_level::pipe::register_raw writes one byte to a "doorbell" pipe // (write(2) is async-signal-safe; the write end is non-blocking so a full // pipe can never wedge the handler). The main loop is then ordinary code: diff --git a/examples/13-pidfd-modern-process-mgmt/README.md b/examples/13-pidfd-modern-process-mgmt/README.md index 3903628..3b48808 100644 --- a/examples/13-pidfd-modern-process-mgmt/README.md +++ b/examples/13-pidfd-modern-process-mgmt/README.md @@ -1,8 +1,8 @@ -# 13 — pidfd: modern process management +# 13: modern process management with pidfd -`pmon` v2, the chapter's process supervisor, implemented three times — C++23, -Go, and Rust — with identical observable behavior (CLI, output lines, exit -codes). This iteration adds a second supervision engine built on **pidfd**, +`pmon` v2, the chapter's process supervisor, implemented three times, in +C++23, Go, and Rust, with identical observable behavior (CLI, output lines, +exit codes). This iteration adds a second supervision engine built on **pidfd**, the modern child becomes-a-file-descriptor API, and makes it the default: ``` @@ -15,12 +15,12 @@ usage: pmon ## What each version added -- **v0 — `run`**: spawn a command, wait for it, report and mirror its exit +- **v0 (`run`)**: spawn a command, wait for it, report and mirror its exit (`status` → same exit code, fatal signal → `128+signo`). -- **v1 — `supervise --engine sigchld`**: restart the child on abnormal exit, +- **v1 (`supervise --engine sigchld`)**: restart the child on abnormal exit, driven the classic way: SIGCHLD arrives (via a signalfd in C++/Rust, `signal.Notify` in Go), then a `WNOHANG` wait reaps and classifies it. -- **v2 — `supervise --engine pidfd`** (default): no SIGCHLD anywhere. +- **v2 (`supervise --engine pidfd`, default)**: no SIGCHLD anywhere. `pidfd_open(2)` turns the child pid into a file descriptor; `poll(2)` on it reports the exit as ordinary readability; `waitid(P_PIDFD, ...)` reaps exactly the process behind the fd (a recycled pid can never be reaped or @@ -47,23 +47,23 @@ spawned exits 1 with `pmon: spawn: : ...`. ## Per-language mechanics -- **C++23** (`cpp/src/main.cpp`) — raw `syscall(SYS_pidfd_open)` and - `syscall(SYS_pidfd_send_signal)` on purpose (the libc wrappers are recent), +- **C++23** (`cpp/src/main.cpp`): raw `syscall(SYS_pidfd_open)` and + `syscall(SYS_pidfd_send_signal)` directly (the libc wrappers are recent), `poll(2)` over `{pidfd, signalfd}`, `waitid(P_PIDFD)` to reap. RAII `Fd` and `SpawnAttrs` owners, `std::expected` error paths, `std::println`, `posix_spawnp` with `POSIX_SPAWN_SETSIGMASK` so the child does not inherit the blocked signalfd mask. -- **Go 1.26** (`go/main.go`) — `unix.PidfdOpen` + a goroutine parked in +- **Go 1.26** (`go/main.go`): `unix.PidfdOpen` + a goroutine parked in `unix.Poll`, exit delivered over a channel into one `select` alongside `signal.Notify` and the deadline; a raw `waitid(P_PIDFD)` via `unix.Syscall6` with an explicit `siginfo_t` layout (x/sys keeps the union opaque); stop path via `unix.PidfdSendSignal`. Errors are wrapped with `%w`. -- **Rust edition 2024** (`rust/src/main.rs`) — `rustix::process::pidfd_open` +- **Rust edition 2024** (`rust/src/main.rs`): `rustix::process::pidfd_open` returning an `OwnedFd`, `rustix::event::poll` over the pidfd + a nix `SignalFd`, `rustix::process::waitid(WaitId::PidFd)`, stop path via `rustix::process::pidfd_send_signal`. `Result` + `?` throughout. Note: `std::process::Command` does **not** reset the child's signal mask, so - `pre_exec` clears it — otherwise a supervised `sleep` inherits a blocked + `pre_exec` clears it; otherwise a supervised `sleep` inherits a blocked SIGTERM and can never be stopped. ## Try it @@ -87,6 +87,6 @@ spawned exits 1 with `pmon: spawn: : ...`. anywhere in its output, and its restart lines are byte-identical to the pidfd engine's. - Both stop paths terminate the supervised child (`killed signal=15`), print - the `exiting (...)` line, exit 0 — and a `pgrep` process-tree check proves + the `exiting (...)` line, exit 0, and a `pgrep` process-tree check proves no supervised `sleep` outlives its supervisor. - Usage errors exit 2; an unspawnable command exits 1 and names the command. diff --git a/examples/13-pidfd-modern-process-mgmt/cpp/src/main.cpp b/examples/13-pidfd-modern-process-mgmt/cpp/src/main.cpp index c772d28..14a09f2 100644 --- a/examples/13-pidfd-modern-process-mgmt/cpp/src/main.cpp +++ b/examples/13-pidfd-modern-process-mgmt/cpp/src/main.cpp @@ -1,14 +1,14 @@ -// pmon v2 — a tiny process supervisor grown chapter by chapter. +// pmon v2: a tiny process supervisor grown chapter by chapter. // // v0 (`run`) spawns a command, waits, and mirrors its exit status. // v1 (`supervise --engine sigchld`) restarts a crashing child, driven by // SIGCHLD delivered as readable data through a signalfd. // v2 (`supervise --engine pidfd`, the DEFAULT) drops the SIGCHLD dependence // entirely: pidfd_open(2) turns the child into a pollable file -// descriptor (raw SYS_pidfd_open on purpose — the glibc wrapper is +// descriptor (raw SYS_pidfd_open on purpose: the glibc wrapper is // recent), poll(2) reports the exit as ordinary readability, // waitid(P_PIDFD) reaps exactly that child, and the stop path signals -// through the pidfd with pidfd_send_signal(2) — no pid-reuse race +// through the pidfd with pidfd_send_signal(2), no pid-reuse race // anywhere on the path. #include @@ -48,7 +48,7 @@ using std::chrono::steady_clock; } // --------------------------------------------------------------------------- -// RAII owners — every fd and every spawn attribute has exactly one. +// RAII owners: every fd and every spawn attribute has exactly one. // --------------------------------------------------------------------------- class Fd { @@ -166,7 +166,7 @@ make_signalfd(std::initializer_list sigs) { } // --------------------------------------------------------------------------- -// v0: run — spawn, wait, mirror. +// v0: run: spawn, wait, mirror. // --------------------------------------------------------------------------- int cmd_run(const std::vector& cmd) { @@ -187,7 +187,7 @@ int cmd_run(const std::vector& cmd) { } // --------------------------------------------------------------------------- -// v1/v2: supervise — two engines, one restart policy. +// v1/v2: supervise: two engines, one restart policy. // --------------------------------------------------------------------------- enum class Engine : std::uint8_t { pidfd, sigchld }; @@ -200,7 +200,7 @@ using Outcome = std::variant; return left < 0 ? 0 : static_cast(left); } -// waitid(P_PIDFD) reaps the process the fd refers to — never a recycled pid. +// waitid(P_PIDFD) reaps the process the fd refers to: never a recycled pid. [[nodiscard]] std::expected reap_pidfd(const Fd& pidfd) { siginfo_t si{}; while (::waitid(P_PIDFD, static_cast(pidfd.get()), &si, WEXITED) != 0) { diff --git a/examples/13-pidfd-modern-process-mgmt/go/main.go b/examples/13-pidfd-modern-process-mgmt/go/main.go index 87e9538..3c8986f 100644 --- a/examples/13-pidfd-modern-process-mgmt/go/main.go +++ b/examples/13-pidfd-modern-process-mgmt/go/main.go @@ -1,4 +1,4 @@ -// pmon v2 — a tiny process supervisor grown chapter by chapter. +// pmon v2: a tiny process supervisor grown chapter by chapter. // // v0 (`run`) spawns a command, waits, and mirrors its exit status. // v1 (`supervise --engine sigchld`) restarts a crashing child, driven by @@ -7,7 +7,7 @@ // entirely: unix.PidfdOpen turns the child into a pollable file descriptor, // a goroutine parked in unix.Poll reports the exit over a channel, a raw // waitid(P_PIDFD, ...) reaps exactly that child, and the stop path signals -// through the fd with unix.PidfdSendSignal — no pid-reuse race anywhere. +// through the fd with unix.PidfdSendSignal; no pid-reuse race anywhere. package main import ( @@ -57,7 +57,7 @@ func spawn(cmdline []string) (*exec.Cmd, error) { } // --------------------------------------------------------------------------- -// v0: run — spawn, wait, mirror. +// v0 (run): spawn, wait, mirror. // --------------------------------------------------------------------------- func cmdRun(cmdline []string) int { @@ -81,7 +81,7 @@ func cmdRun(cmdline []string) int { } // --------------------------------------------------------------------------- -// v2 plumbing: waitid(P_PIDFD) with an explicit siginfo_t layout — x/sys +// v2 plumbing: waitid(P_PIDFD) with an explicit siginfo_t layout; x/sys // keeps the union opaque, and the CLD_* fields are what we need. // --------------------------------------------------------------------------- @@ -99,7 +99,7 @@ type siginfoChld struct { const cldExited = 1 // si_code: child called _exit // reapPidfd blocks in waitid(P_PIDFD) until the process behind pidfd is -// reaped — it can never collect a recycled pid. +// reaped; it can never collect a recycled pid. func reapPidfd(pidfd int) (childStatus, error) { var si siginfoChld for { @@ -151,7 +151,7 @@ func blockingReap(pid int) (childStatus, error) { } // --------------------------------------------------------------------------- -// v1/v2: supervise — two engines, one restart policy. +// v1/v2 (supervise): two engines, one restart policy. // --------------------------------------------------------------------------- func cmdSupervise(engine string, maxRestarts int, timeout time.Duration, cmdline []string) int { diff --git a/examples/13-pidfd-modern-process-mgmt/rust/src/main.rs b/examples/13-pidfd-modern-process-mgmt/rust/src/main.rs index 56b2ce4..0f42821 100644 --- a/examples/13-pidfd-modern-process-mgmt/rust/src/main.rs +++ b/examples/13-pidfd-modern-process-mgmt/rust/src/main.rs @@ -1,4 +1,4 @@ -// pmon v2 — a tiny process supervisor grown chapter by chapter. +// pmon v2: a tiny process supervisor grown chapter by chapter. // // v0 (`run`) spawns a command, waits, and mirrors its exit status. // v1 (`supervise --engine sigchld`) restarts a crashing child, driven by @@ -7,7 +7,7 @@ // entirely: rustix::process::pidfd_open turns the child into a pollable // OwnedFd, rustix::event::poll reports the exit as ordinary readability, // waitid(WaitId::PidFd) reaps exactly that child, and the stop path -// signals through the fd with pidfd_send_signal — no pid-reuse race +// signals through the fd with pidfd_send_signal, with no pid-reuse race // anywhere on the path. use std::os::fd::{AsFd, AsRawFd}; @@ -50,7 +50,7 @@ fn spawn(cmdline: &[String]) -> Result { let mut cmd = Command::new(&cmdline[0]); cmd.args(&cmdline[1..]); // The supervisor blocks signals for its signalfd and the child inherits - // that mask across fork+exec (std does NOT reset it) — a supervised + // that mask across fork+exec (std does NOT reset it), so a supervised // `sleep` with SIGTERM blocked would never stop. Clear the mask in the // child; sigprocmask is async-signal-safe, so pre_exec may call it. unsafe { @@ -64,7 +64,7 @@ fn spawn(cmdline: &[String]) -> Result { } // --------------------------------------------------------------------------- -// v0: run — spawn, wait, mirror. +// v0 run: spawn, wait, mirror. // --------------------------------------------------------------------------- fn cmd_run(cmdline: &[String]) -> Result { @@ -80,7 +80,7 @@ fn cmd_run(cmdline: &[String]) -> Result { } // --------------------------------------------------------------------------- -// v1/v2: supervise — two engines, one restart policy. +// v1/v2 supervise: two engines, one restart policy. // --------------------------------------------------------------------------- #[derive(Clone, Copy, PartialEq)] @@ -126,7 +126,7 @@ fn status_from_waitid(w: rustix::process::WaitIdStatus) -> ChildStatus { } } -/// waitid(P_PIDFD) reaps the process the fd refers to — never a recycled pid. +/// waitid(P_PIDFD) reaps the process the fd refers to, never a recycled pid. fn reap_pidfd(pidfd: &impl AsFd) -> Result { let status = waitid(WaitId::PidFd(pidfd.as_fd()), WaitIdOptions::EXITED) .context("waitid(P_PIDFD)")? @@ -217,7 +217,10 @@ fn cmd_supervise( let (outcome, pidfd) = match engine { Engine::Pidfd => { - let pidfd = pidfd_open(Pid::from_raw(pid).unwrap(), PidfdFlags::empty()) + let pidfd = pidfd_open( + Pid::from_raw(pid).expect("child.id() is always a positive pid"), + PidfdFlags::empty(), + ) .context("pidfd_open")?; eprintln!("pmon: engine=pidfd child={pid} pidfd={}", pidfd.as_raw_fd()); (wait_pidfd(&pidfd, &sigfd, deadline)?, Some(pidfd)) diff --git a/examples/13-pidfd-modern-process-mgmt/verify.lua b/examples/13-pidfd-modern-process-mgmt/verify.lua index de4703e..276fdcf 100644 --- a/examples/13-pidfd-modern-process-mgmt/verify.lua +++ b/examples/13-pidfd-modern-process-mgmt/verify.lua @@ -1,16 +1,16 @@ -- Verify pmon v2 (13-pidfd-modern-process-mgmt) for LSP_LANG. -- -- Asserts observable behavior, identical across cpp/go/rust: --- 1. v0 regression — `run` mirrors the child's exit status and signal +-- 1. v0 regression: `run` mirrors the child's exit status and signal -- 2. pidfd engine (default) supervises a crashing child: per-spawn -- "engine=pidfd child= pidfd=" lines, restart lines in order, -- a giving-up line, exit 1 -- 3. sigchld engine still works and produces byte-identical restart lines -- 4. clean child exit ends supervision with exit 0 and no restart --- 5. stop paths — SIGTERM to pmon and the overall timeout both terminate +-- 5. stop paths: SIGTERM to pmon and the overall timeout both terminate -- the child (killed signal=15), print the exiting line, exit 0, and -- leave no orphaned child behind (process-tree check via pgrep) --- 6. error shapes — usage errors exit 2, spawn failure exits 1 +-- 6. error shapes: usage errors exit 2, spawn failure exits 1 -- -- Invoked from the example directory with LSP_LANG (cpp|go|rust) and -- EXAMPLE_DIR set; the interpreter may be lua or luajit. diff --git a/examples/14-identity-and-privilege/README.md b/examples/14-identity-and-privilege/README.md index 143fc9e..539f00a 100644 --- a/examples/14-identity-and-privilege/README.md +++ b/examples/14-identity-and-privilege/README.md @@ -1,21 +1,21 @@ -# 14 — Identity & privilege (`pmon` v3) +# 14: Identity & privilege (`pmon` v3) **VM example.** This chapter drops privilege the right way: a root process -hands a command to an unprivileged user while carrying exactly one capability — -`CAP_NET_BIND_SERVICE` — across the identity change and the following +hands a command to an unprivileged user while carrying exactly one capability, +`CAP_NET_BIND_SERVICE`, across the identity change and the following `execve(2)`. The whole lesson is the **order of operations**. Get the sequence wrong and either the capability evaporates (the command can't bind `:80`) or it never drops (the command runs as root, and you've proven nothing). `pmon drop` becomes user `` and then `exec`s `CMD`. `pmon bindprobe` -tries to bind a TCP port and reports its uid — the observable proof: **binding a +tries to bind a TCP port and reports its uid. The observable proof: **binding a port below 1024 as a non-root uid is impossible without the ambient cap.** | Language | How the drop is performed | |---|---| -| **C++23** | Raw `prctl(2)` / `capset(2)` / `setres[ug]id(2)` in-process, then `execvp`. Single-threaded on purpose (a privilege drop must cover every task, and `execve` only makes sense from one thread). RAII owns the socket; `std::expected` carries every syscall failure; no C-style `perror`/`exit`. | +| **C++23** | Raw `prctl(2)` / `capset(2)` / `setres[ug]id(2)` in-process, then `execvp`. Single-threaded by design (a privilege drop must cover every task, and `execve` only makes sense from one thread). RAII owns the socket; `std::expected` carries every syscall failure; no C-style `perror`/`exit`. | | **Rust (edition 2024)** | The same raw `libc` sequence, mirroring the C++ path; `OwnedFd` gives the socket RAII and errors flow through `Result`/`?`. Capability structs are `#[repr(C)]`; the two ABI constants libc omits (`_LINUX_CAPABILITY_VERSION_3`, `CAP_NET_BIND_SERVICE`) are defined locally. | -| **Go 1.26** | The idiomatic inversion. A Go program is already multithreaded and `setuid`/`capset` act on the calling thread only, so doing this in-process is a footgun. Instead the drop is arranged **pre-exec** via `syscall.SysProcAttr` — `Credential` (setgid/setgroups/setuid) plus `AmbientCaps` (which the runtime turns into the `PR_SET_KEEPCAPS → capset → PR_CAP_AMBIENT_RAISE` dance in the fork/exec child). Same observable result; the ordering lives in the standard library instead of in our code. | +| **Go 1.26** | The idiomatic inversion. A Go program is already multithreaded and `setuid`/`capset` act on the calling thread only, so doing this in-process is a correctness hazard. Instead the drop is arranged **pre-exec** via `syscall.SysProcAttr`: `Credential` (setgid/setgroups/setuid) plus `AmbientCaps` (which the runtime turns into the `PR_SET_KEEPCAPS → capset → PR_CAP_AMBIENT_RAISE` dance in the fork/exec child). Same observable result; the ordering lives in the standard library instead of in our code. | All three produce byte-for-byte identical output and exit codes, so one `verify.lua` covers them. @@ -47,11 +47,11 @@ usage: pmon bindprobe [--port 80] ``` -- `drop --user ` — must run as root; becomes `` and `exec`s `CMD`. +- `drop --user `: must run as root; becomes `` and `exec`s `CMD`. With `--keep-cap net_bind_service` it carries `CAP_NET_BIND_SERVICE` across the drop; without it, `CMD` runs with no capabilities. `CMD`'s own exit status is propagated. -- `bindprobe [--port 80]` — binds a TCP socket on the port. On success prints +- `bindprobe [--port 80]`: binds a TCP socket on the port. On success prints `bindprobe: uid= bound :` to stdout and exits `0`. On failure prints `bindprobe: bind :: ` to stderr and exits `3`. - Exit codes: `0` success, `1` runtime error (not root, unknown user, exec @@ -61,8 +61,8 @@ usage: Each language directory has the standard `demo.sh`: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (`app`) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (`app`) - With env `TARGET` set, `run` deploys to that lab VM via `scripts/lab/deploy-to-vm.sh`; set `SUDO=1` for the `drop` path, which needs root on the guest @@ -91,16 +91,16 @@ SUDO=1 ./demo.sh cpp run drop --user nobody -- /tmp/pmon-verify bindprobe --port ## Verification `verify.lua` (run per language under the runner's **vm mode**, -`TARGET=systems-target` with `SUDO=1`) asserts the paired behavior that makes -this example real, not merely exit-0: +`TARGET=systems-target` with `SUDO=1`) asserts the paired behavior, not +merely exit-0: -1. **Positive** — `drop --user nobody --keep-cap net_bind_service` running +1. **Positive**: `drop --user nobody --keep-cap net_bind_service` running `bindprobe --port 80` prints `bindprobe: uid=65534 bound :80` and exits `0`. The uid is captured and asserted **non-zero**: an unprivileged process bound a privileged port, which is only possible because the ambient cap survived. -2. **Negative control** — the *same* drop **without** `--keep-cap` prints +2. **Negative control**: the *same* drop **without** `--keep-cap` prints `bindprobe: bind :80: Permission denied` and exits `3`. Same uid, cap - withheld, port denied — this is what proves assertion 1 came from the + withheld, port denied: this is what proves assertion 1 came from the capability and not from a stray root or a lowered `net.ipv4.ip_unprivileged_port_start`. @@ -112,7 +112,7 @@ LIBVIRT_DEFAULT_URI=qemu:///system \ ``` **Why observe the effect, not a trace:** you cannot `strace` across a -`setuid`+`execve` boundary to confirm the cap moved — `ptrace` is dropped at +`setuid`+`execve` boundary to confirm the cap moved: `ptrace` is dropped at the privilege transition. The port bind *is* the observation: `:80` bound at `uid=65534` is a fact the kernel only permits when `CAP_NET_BIND_SERVICE` is in the effective set. The negative control closes the loop by showing the same diff --git a/examples/14-identity-and-privilege/cpp/src/main.cpp b/examples/14-identity-and-privilege/cpp/src/main.cpp index a5053aa..4cf6768 100644 --- a/examples/14-identity-and-privilege/cpp/src/main.cpp +++ b/examples/14-identity-and-privilege/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// pmon v3 — identity & privilege (chapter 14). +// pmon v3: identity & privilege (chapter 14). // // Subcommands: // pmon drop --user [--keep-cap net_bind_service] -- CMD [args...] @@ -9,19 +9,16 @@ // change and the following execve(2) via the ambient-capability mechanism. // The whole lesson is the ORDER of operations: // -// 1. prctl(PR_SET_KEEPCAPS, 1) — before the uid change, so the permitted -// set survives setresuid(2) instead of -// being scrubbed when uid goes 0 -> !0. -// 2. setgroups / setresgid / setresuid — become the target user. -// 3. capset(2) — pin CAP_NET_BIND_SERVICE into the -// permitted AND inheritable sets (ambient -// requires both), dropping everything else. -// 4. prctl(PR_CAP_AMBIENT_RAISE) — raise it into the ambient set, the only -// set that a non-root, non-file-cap execve -// preserves. -// 5. execvp(2) — CMD now runs as yet keeps :80. +// 1. prctl(PR_SET_KEEPCAPS, 1): before the uid change, so the permitted set +// survives setresuid(2) instead of being scrubbed when uid goes 0 -> !0. +// 2. setgroups / setresgid / setresuid: become the target user. +// 3. capset(2): pin CAP_NET_BIND_SERVICE into the permitted AND inheritable +// sets (ambient requires both), dropping everything else. +// 4. prctl(PR_CAP_AMBIENT_RAISE): raise it into the ambient set, the only +// set that a non-root, non-file-cap execve preserves. +// 5. execvp(2): CMD now runs as yet keeps :80. // -// Raw prctl(2) / capset(2) / setres*id(2) on purpose — no libcap. RAII owns +// Raw prctl(2) / capset(2) / setres*id(2) on purpose, no libcap. RAII owns // the socket; std::expected carries syscall failures; nothing is handled the // C way (no scattered perror/exit). Deliberately single-threaded: a privilege // drop must touch every task, and execve() only makes sense from one thread. @@ -64,7 +61,7 @@ std::expected checked(long rc, std::string_view what) { return {}; } -// RAII owner for a file descriptor — closed exactly once on scope exit. +// RAII owner for a file descriptor: closed exactly once on scope exit. class Fd { public: explicit Fd(int fd) noexcept : fd_(fd) {} @@ -129,14 +126,14 @@ int cmd_bindprobe(std::span args) { std::expected arrange_and_exec(const passwd& pw, bool keep_cap, std::span cmd) { // (1) Retain the permitted set across the coming uid change. Must precede - // setresuid — after it, it is too late. + // setresuid: after it, it is too late. if (keep_cap) { if (auto r = checked(::prctl(PR_SET_KEEPCAPS, 1, 0, 0, 0), "PR_SET_KEEPCAPS"); !r) return r; } - // (2) Become the target user: groups first, gid before uid — once uid drops + // (2) Become the target user: groups first, gid before uid: once uid drops // we can no longer change gids. gid_t gid = pw.pw_gid; if (auto r = checked(::setgroups(1, &gid), "setgroups"); !r) return r; @@ -159,7 +156,7 @@ std::expected arrange_and_exec(const passwd& pw, bool keep_cap, if (auto r = checked(::syscall(SYS_capset, &hdr, data.data()), "capset"); !r) return r; - // (4) Raise it into the ambient set — the only set an unprivileged execve + // (4) Raise it into the ambient set: the only set an unprivileged execve // of a file without file-caps carries forward. if (auto r = checked(::prctl(PR_CAP_AMBIENT, PR_CAP_AMBIENT_RAISE, CAP_NET_BIND_SERVICE, 0, 0), diff --git a/examples/14-identity-and-privilege/go/main.go b/examples/14-identity-and-privilege/go/main.go index 29fbd56..0d8f151 100644 --- a/examples/14-identity-and-privilege/go/main.go +++ b/examples/14-identity-and-privilege/go/main.go @@ -1,11 +1,11 @@ -// pmon v3 — identity & privilege (chapter 14). +// pmon v3: identity & privilege (chapter 14). // // Subcommands: // // pmon drop --user [--keep-cap net_bind_service] -- CMD [args...] // pmon bindprobe [--port 80] // -// Go note — why this differs from the C++/Rust in-process drop: +// Go note: why this differs from the C++/Rust in-process drop. // // A Go program is multithreaded before main() runs, and setresuid(2) / // capset(2) act on the CALLING thread only. There is no portable way to make @@ -15,7 +15,7 @@ // Credential (the setgid/setgroups/setresuid) and AmbientCaps (the // PR_SET_KEEPCAPS -> capset -> PR_CAP_AMBIENT_RAISE sequence) which the fork/exec // child applies in the correct order for us. Same observable result as the raw -// C++/Rust path — the ordering lives in the standard library instead of here. +// C++/Rust path; the ordering lives in the standard library instead of here. package main import ( diff --git a/examples/14-identity-and-privilege/rust/src/main.rs b/examples/14-identity-and-privilege/rust/src/main.rs index b11d46b..3e31fda 100644 --- a/examples/14-identity-and-privilege/rust/src/main.rs +++ b/examples/14-identity-and-privilege/rust/src/main.rs @@ -1,4 +1,4 @@ -//! pmon v3 — identity & privilege (chapter 14). +//! pmon v3: identity & privilege (chapter 14). //! //! Subcommands: //! pmon drop --user [--keep-cap net_bind_service] -- CMD [args...] @@ -14,7 +14,7 @@ //! 2. setgroups / setresgid / setresuid become the target user. //! 3. capset(2) pins CAP_NET_BIND_SERVICE into permitted + inheritable //! (ambient requires both). -//! 4. prctl(PR_CAP_AMBIENT_RAISE) raises it into the ambient set — the only +//! 4. prctl(PR_CAP_AMBIENT_RAISE) raises it into the ambient set, the only //! set a non-root, no-file-cap execve keeps. //! 5. execvp(3): CMD runs as , keeping :80 iff the cap was raised. //! diff --git a/examples/14-identity-and-privilege/verify.lua b/examples/14-identity-and-privilege/verify.lua index e1113ac..21a0627 100644 --- a/examples/14-identity-and-privilege/verify.lua +++ b/examples/14-identity-and-privilege/verify.lua @@ -1,11 +1,11 @@ --- Verify pmon v3 (identity & privilege) — a VM example. Behavioral assertions: +-- Verify pmon v3 (identity & privilege), a VM example. Behavioral assertions: -- -- 1. `drop --user nobody --keep-cap net_bind_service -- CMD bindprobe :80` -- must run CMD as an UNPRIVILEGED uid (65534, not 0) yet still bind :80. -- That pairing is impossible without the ambient CAP_NET_BIND_SERVICE --- surviving the setuid + execve — so it is the proof the cap was carried. +-- surviving the setuid + execve, which is the proof the cap was carried. -- --- 2. NEGATIVE CONTROL — the same drop WITHOUT --keep-cap must fail the bind +-- 2. NEGATIVE CONTROL: the same drop WITHOUT --keep-cap must fail the bind -- with "Permission denied" and exit 3. Same uid, cap withheld, port -- denied: this is what makes assertion 1 meaningful rather than trivially -- true (a root process, or a lowered ip_unprivileged_port_start, would @@ -77,7 +77,7 @@ else lang .. ": uid must be non-root but was " .. tostring(uid)) end --- Negative control: same drop, NO cap — bind :80 must be denied, exit 3. +-- Negative control: same drop, NO cap; bind :80 must be denied, exit 3. local neg = checks.run(string.format( "SUDO=1 ./demo.sh %s run drop --user nobody -- %s bindprobe --port 80", lang, staged)) diff --git a/examples/15-virtual-memory/README.md b/examples/15-virtual-memory/README.md index 891186c..cff4fdd 100644 --- a/examples/15-virtual-memory/README.md +++ b/examples/15-virtual-memory/README.md @@ -1,4 +1,4 @@ -# 15-virtual-memory — memmap +# 15-virtual-memory: memmap A self-inspecting virtual-memory tool, implemented three times (C++23, Go, Rust) with identical CLI, output shapes, and exit codes. `memmap` performs one @@ -43,7 +43,7 @@ memmap: faults minor= major= print `memmap: error: …` on stderr and exit 1; the exact message text is each language's native errno rendering. -Example (heap, defaults — 64 MiB touched means ~64 MiB of RSS growth and one +Example (heap, defaults: 64 MiB touched means ~64 MiB of RSS growth and one minor fault per touched 4 KiB page): ``` @@ -60,14 +60,14 @@ memmap: faults minor=16385 major=0 pays for physical pages when they are touched: the RSS delta and the minor fault delta line up with the touched page count, not the allocation call. - **Fault-around.** In `fault-walk` and `mmap-file`, 512 touched file pages - produce ~32 minor faults per step, not 512 — the kernel maps a batch of + produce ~32 minor faults per step, not 512: the kernel maps a batch of surrounding page-cache pages per fault. Anonymous writes fault one page at a time, so `heap`/`mmap-anon` report ~1 fault per page. - **Minor vs major.** The walk file is written immediately before it is mapped, so every page is in the page cache and `major=0`; a major fault would mean a real disk read. - **Where allocations live.** A 64 MiB `malloc`/`Vec`/`make` does not extend - `[heap]` — the allocator satisfies it with a private anonymous mapping, and + `[heap]`: the allocator satisfies it with a private anonymous mapping, and the excerpt shows exactly that region. ## Per-language notes diff --git a/examples/15-virtual-memory/cpp/src/main.cpp b/examples/15-virtual-memory/cpp/src/main.cpp index 2a07c16..8ac4174 100644 --- a/examples/15-virtual-memory/cpp/src/main.cpp +++ b/examples/15-virtual-memory/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// memmap: self-inspecting virtual-memory tool — perform one kind of +// memmap: self-inspecting virtual-memory tool: perform one kind of // allocation, touch its pages, then report what the kernel now shows for // this very process: the /proc/self/maps region backing the allocation, // the VmRSS growth, and the getrusage(2) page-fault deltas. diff --git a/examples/15-virtual-memory/go/main.go b/examples/15-virtual-memory/go/main.go index 32b284f..9127972 100644 --- a/examples/15-virtual-memory/go/main.go +++ b/examples/15-virtual-memory/go/main.go @@ -1,4 +1,4 @@ -// memmap: self-inspecting virtual-memory tool — perform one kind of +// memmap: self-inspecting virtual-memory tool. Perform one kind of // allocation, touch its pages, then report what the kernel now shows for // this very process: the /proc/self/maps region backing the allocation, // the VmRSS growth, and the getrusage(2) page-fault deltas. diff --git a/examples/15-virtual-memory/rust/src/main.rs b/examples/15-virtual-memory/rust/src/main.rs index adc9de3..b9e353b 100644 --- a/examples/15-virtual-memory/rust/src/main.rs +++ b/examples/15-virtual-memory/rust/src/main.rs @@ -1,4 +1,4 @@ -// memmap: self-inspecting virtual-memory tool — perform one kind of +// memmap: self-inspecting virtual-memory tool. Perform one kind of // allocation, touch its pages, then report what the kernel now shows for // this very process: the /proc/self/maps region backing the allocation, // the VmRSS growth, and the getrusage(2) page-fault deltas. @@ -194,21 +194,14 @@ fn report(addr: usize, len: usize, mode: &str, base: &Baseline) -> Result<()> { // --------------------------------------------------------------------------- fn touch_writable(mem: &mut [u8], page: usize) { - let mut i = 0; - while i < mem.len() { + for i in (0..mem.len()).step_by(page) { mem[i] = 1; - i += page; } black_box(mem); } fn touch_readable(mem: &[u8], page: usize) { - let mut sum = 0u64; - let mut i = 0; - while i < mem.len() { - sum += u64::from(mem[i]); - i += page; - } + let sum: u64 = (0..mem.len()).step_by(page).map(|i| u64::from(mem[i])).sum(); black_box(sum); } @@ -417,7 +410,12 @@ fn main() { run_mmap_anon(bytes, page) } // Prints its own header line (size comes from the file). - Mode::MmapFile => run_mmap_file(opts.file.as_deref().unwrap(), page), + Mode::MmapFile => run_mmap_file( + opts.file + .as_deref() + .expect("parse_args guarantees FILE is set for mmap-file mode"), + page, + ), Mode::FaultWalk => { let bytes = opts.mb * MIB; println!( diff --git a/examples/15-virtual-memory/verify.lua b/examples/15-virtual-memory/verify.lua index 0337a9b..ac9082f 100644 --- a/examples/15-virtual-memory/verify.lua +++ b/examples/15-virtual-memory/verify.lua @@ -1,5 +1,5 @@ -- Verify 15-virtual-memory: run memmap in every mode and assert the --- observable behavior — header shapes, the annotated /proc/self/maps +-- observable behavior: header shapes, the annotated /proc/self/maps -- excerpt (region range, permissions, backing path), real VmRSS growth -- (heap/mmap-anon must gain at least half the touched allocation), nonzero -- minor-fault deltas, the per-step fault-walk table shape, exit codes for diff --git a/examples/16-mmap-and-shared-mappings/README.md b/examples/16-mmap-and-shared-mappings/README.md index 5d5c589..dd9c229 100644 --- a/examples/16-mmap-and-shared-mappings/README.md +++ b/examples/16-mmap-and-shared-mappings/README.md @@ -1,7 +1,7 @@ -# 16-mmap-and-shared-mappings — shmkv v0 +# 16-mmap-and-shared-mappings: shmkv v0 A tiny key/value store whose entire database is one `mmap`'d file, implemented -three times — C++23, Go, and Rust — against **one on-disk format**. That format +three times, in C++23, Go, and Rust, against **one on-disk format**. That format is the whole point: the Rust binary can create a store, the Go binary can write into it, and the C++ binary can read it back, because all three agree on every byte. `verify.lua` proves this with `cmp(1)`, not by trusting exit codes. @@ -20,11 +20,11 @@ offset 10 slot_count slots, 256 bytes each: [ 64..256) value, NUL-padded (max 191 bytes) ``` -File size is always exactly `10 + slot_count * 256` bytes — `create --slots 8` +File size is always exactly `10 + slot_count * 256` bytes: `create --slots 8` produces a 2058-byte file, and any file whose size disagrees with its header is rejected as `not a shmkv v0 store`. `create` truncates to zero and extends with -`ftruncate`, so every slot starts as guaranteed-zero bytes: "empty" falls out -of the file format for free. +`ftruncate`, so every slot starts as guaranteed-zero bytes: the file format +alone makes "empty" the default, with no extra code required. ## CLI @@ -50,22 +50,22 @@ Exit codes, identical in all three languages: ## What each language shows -- **C++** (`cpp/src/main.cpp`) — RAII throughout: an `Fd` class owns the +- **C++** (`cpp/src/main.cpp`): RAII throughout: an `Fd` class owns the descriptor, a `Mapping` class owns the `mmap`/`munmap` pair and exposes `msync` as `sync()`; every fallible path returns `std::expected`, and output goes through `std::println`. -- **Go** (`go/main.go`) — `unix.Mmap` returns the mapping as a `[]byte`, and the +- **Go** (`go/main.go`): `unix.Mmap` returns the mapping as a `[]byte`, and the discipline is never touching that slice after `unix.Munmap`; errors wrap causes with `%w` while a `cliError` type pins the exact stderr line and exit code; `errors.Is`/`errors.As` route them in `main`. -- **Rust** (`rust/src/main.rs`) — the `memmap2` crate, with the program's single +- **Rust** (`rust/src/main.rs`): the `memmap2` crate, with the program's single `unsafe` block confined to one function (`map_shared`) behind an explicit boundary comment spelling out why the call is sound; the `File` owns the fd as an `OwnedFd`, and drop order (map before file) mirrors the C++ RAII types. The dump ordering is bytewise key sort in all three, and `set` clears the whole 256-byte slot before writing, so a shorter value never leaves a longer one's -tail behind — both details matter for the byte-identical guarantee. +tail behind; both details matter for the byte-identical guarantee. ## Run it @@ -92,4 +92,4 @@ binary, and with `TARGET` set `run` deploys to that lab VM instead. - **cross-language interop**: Rust creates and writes a store, Go writes into it, C++ and Go read it back; the three `dump` outputs are compared with `cmp`; then C++ creates a second store replaying the same operations and - Rust reads it — and `cmp` shows the two store files are byte-identical. + Rust reads it; `cmp` shows the two store files are byte-identical. diff --git a/examples/16-mmap-and-shared-mappings/cpp/src/main.cpp b/examples/16-mmap-and-shared-mappings/cpp/src/main.cpp index edd293d..7ff76d3 100644 --- a/examples/16-mmap-and-shared-mappings/cpp/src/main.cpp +++ b/examples/16-mmap-and-shared-mappings/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// shmkv v0 — a tiny mmap-backed key/value store (C++23). +// shmkv v0: a tiny mmap-backed key/value store (C++23). // // On-disk format, byte-exact and shared with the Go and Rust implementations: // @@ -235,7 +235,7 @@ class Store { }; [[nodiscard]] Result parse_slots(std::string_view text) { - // Digits only, in range [1, u32 max] — identical rules in all three + // Digits only, in range [1, u32 max]: identical rules in all three // languages so the CLIs reject exactly the same inputs. if (text.empty() || text.size() > 10 || !std::ranges::all_of(text, [](char c) { return c >= '0' && c <= '9'; })) { @@ -296,7 +296,7 @@ class Store { std::ranges::sort(pairs); // bytewise by key, matching Go/Rust for (const auto& [k, v] : pairs) std::println("{}={}", k, v); // Flush before the stderr summary: stdout is fully buffered on a pipe, - // and Go/Rust emit the pairs first — keep the interleaving identical. + // and Go/Rust emit the pairs first: keep the interleaving identical. std::fflush(stdout); std::println(stderr, "shmkv: {}/{} slots used", pairs.size(), store->slot_count()); return {}; diff --git a/examples/16-mmap-and-shared-mappings/go/main.go b/examples/16-mmap-and-shared-mappings/go/main.go index 7fad70b..16dccea 100644 --- a/examples/16-mmap-and-shared-mappings/go/main.go +++ b/examples/16-mmap-and-shared-mappings/go/main.go @@ -1,4 +1,4 @@ -// shmkv v0 — a tiny mmap-backed key/value store (Go). +// shmkv v0: a tiny mmap-backed key/value store (Go). // // On-disk format, byte-exact and shared with the C++ and Rust implementations: // @@ -166,7 +166,7 @@ func (s *store) slot(i uint32) []byte { func (s *store) slotKey(i uint32) string { return field(s.slot(i)[:keyField]) } func (s *store) slotValue(i uint32) string { return field(s.slot(i)[keyField:]) } -// set: linear probe — overwrite the slot holding key, else claim the first +// set: linear probe; overwrite the slot holding key, else claim the first // empty slot; exit-5 failure when every slot is taken. func (s *store) set(key, value string) error { target := s.slots @@ -201,7 +201,7 @@ func (s *store) set(key, value string) error { return nil } -// parseSlots: digits only, in range [1, u32 max] — identical rules in all +// parseSlots: digits only, in range [1, u32 max]; identical rules in all // three languages so the CLIs reject exactly the same inputs. func parseSlots(text string) (uint32, error) { if text == "" || len(text) > 10 { diff --git a/examples/16-mmap-and-shared-mappings/rust/src/main.rs b/examples/16-mmap-and-shared-mappings/rust/src/main.rs index 2a75c22..bd54f87 100644 --- a/examples/16-mmap-and-shared-mappings/rust/src/main.rs +++ b/examples/16-mmap-and-shared-mappings/rust/src/main.rs @@ -1,4 +1,4 @@ -// shmkv v0 — a tiny mmap-backed key/value store (Rust, edition 2024). +// shmkv v0: a tiny mmap-backed key/value store (Rust, edition 2024). // // On-disk format, byte-exact and shared with the C++ and Go implementations: // @@ -44,7 +44,7 @@ impl Fail { type Result = std::result::Result; /// An open store. `file` is a std::fs::File, which owns the descriptor as an -/// OwnedFd — dropping the Store unmaps (MmapMut's Drop runs munmap) and then +/// OwnedFd; dropping the Store unmaps (MmapMut's Drop runs munmap) and then /// closes the fd, in that order (fields drop in declaration order). struct Store { map: MmapMut, @@ -65,7 +65,7 @@ struct Store { /// * we size the mapping from ftruncate/fstat in the same process and /// no shmkv command ever shrinks an existing store file; /// * the demo runs shmkv commands sequentially, so cross-process writes -/// are ordered by process exit (and each writer msyncs before exit) — +/// are ordered by process exit (and each writer msyncs before exit); /// the concurrent-writer race the type system cannot rule out is /// excluded by the tool's usage model, not by the compiler. /// Everything outside this function stays in safe Rust. @@ -114,7 +114,11 @@ impl Store { if map[..MAGIC.len()] != MAGIC { return Err(bad()); } - let slots = u32::from_le_bytes(map[MAGIC.len()..HEADER_SIZE].try_into().unwrap()); + let slots = u32::from_le_bytes( + map[MAGIC.len()..HEADER_SIZE] + .try_into() + .expect("slice is exactly 4 bytes (HEADER_SIZE - MAGIC.len())"), + ); if len != HEADER_SIZE + slots as usize * SLOT_SIZE { return Err(bad()); } @@ -173,7 +177,7 @@ fn field(bytes: &[u8]) -> &[u8] { } } -/// Digits only, in range [1, u32 max] — identical rules in all three +/// Digits only, in range [1, u32 max]: identical rules in all three /// languages so the CLIs reject exactly the same inputs. fn parse_slots(text: &str) -> Result { if text.is_empty() || text.len() > 10 || !text.bytes().all(|b| b.is_ascii_digit()) { diff --git a/examples/16-mmap-and-shared-mappings/verify.lua b/examples/16-mmap-and-shared-mappings/verify.lua index 558b9e4..d7acaeb 100644 --- a/examples/16-mmap-and-shared-mappings/verify.lua +++ b/examples/16-mmap-and-shared-mappings/verify.lua @@ -1,7 +1,7 @@ -- Verify 16-mmap-and-shared-mappings: shmkv v0, an mmap-backed KV store with -- ONE on-disk format shared by all three languages. Asserts behavior: exact -- header bytes (od), file sizes (stat), CLI stdout/stderr shapes and exit --- codes — and, the point of the example, CROSS-LANGUAGE INTEROP: a store +-- codes, and, the point of the example, CROSS-LANGUAGE INTEROP: a store -- created and written by the Rust binary is read by the Go and C++ binaries -- (byte-identical dumps), then the reverse (created by C++, read by Rust), -- with cmp(1) proving the store files themselves are byte-identical. diff --git a/examples/17-allocators-and-gc-runtimes/README.md b/examples/17-allocators-and-gc-runtimes/README.md index 68de019..542ff25 100644 --- a/examples/17-allocators-and-gc-runtimes/README.md +++ b/examples/17-allocators-and-gc-runtimes/README.md @@ -1,6 +1,6 @@ -# 17-allocators-and-gc-runtimes — allocbench +# 17-allocators-and-gc-runtimes: allocbench -Chapter 17 puts the three languages' memory stories side by side with one +This chapter puts the three languages' memory stories side by side with one allocation-heavy workload: build a string→string index (1000 distinct keys, overwritten round-robin) out of `N` iterations of short-lived intermediate strings, then query every key back. @@ -9,12 +9,12 @@ strings, then query every key back. allocbench [--allocs N] [--variant default|arena] ``` -- `--allocs N` — iterations of churn (default 200000; must be a positive +- `--allocs N`: iterations of churn (default 200000; must be a positive integer). -- `--variant default` — every intermediate goes through the language's +- `--variant default`: every intermediate goes through the language's general-purpose allocation path (glibc malloc, the Go heap + GC, the Rust global allocator). -- `--variant arena` — the same logical work, but the short-lived +- `--variant arena`: the same logical work, but the short-lived intermediates are batch-allocated and reclaimed wholesale (or recycled) instead of being freed one by one. @@ -27,13 +27,13 @@ allocbench [--allocs N] [--variant default|arena] - **Go only** appends ` gc_cycles=`: the delta of `runtime.MemStats.NumGC` across the workload, read via `runtime.ReadMemStats`. C++ and Rust have no collector, so the field would - be a lie there — its absence is part of the lesson. + be a lie there; its absence is part of the lesson. - Bad usage (unknown flag, non-positive `--allocs`, unknown variant) prints a diagnostic plus `usage: allocbench [--allocs N] [--variant default|arena]` on stderr and exits 2. Runtime failures print `allocbench: ...` on stderr and exit 1. - The query phase is a real check: if any of the distinct keys is missing - from the index or the summed value bytes are zero, the run fails — the + from the index or the summed value bytes are zero, the run fails: the benchmark cannot "pass" by optimizing the work away. ## The two variants, per language @@ -41,21 +41,21 @@ allocbench [--allocs N] [--variant default|arena] | | default | arena | |---|---|---| | **C++** | `std::string` intermediates from the global heap, freed one by one when each batch's scratch `std::vector` is destroyed | per-batch `std::pmr::monotonic_buffer_resource` over one reusable 256 KiB slab; intermediates are `std::pmr::string`s held in a `std::pmr::vector`, released wholesale when the resource is destroyed | -| **Go** | fresh strings every iteration (`strconv.Itoa`, `+` concatenation) — all of it garbage for the collector | `sync.Pool` of scratch objects (three `bytes.Buffer`s plus a `strconv.AppendInt` byte slice): churn buffers are recycled, not collected | +| **Go** | fresh strings every iteration (`strconv.Itoa`, `+` concatenation): all of it garbage for the collector | `sync.Pool` of scratch objects (three `bytes.Buffer`s plus a `strconv.AppendInt` byte slice): churn buffers are recycled, not collected | | **Rust** | `String`/`format!` intermediates from the global allocator, dropped individually | `bumpalo::Bump` arena: `bumpalo::collections::String`/`Vec` intermediates, reclaimed in O(1) by `bump.reset()` after each 1000-iteration batch | The long-lived index itself is an ordinary hash map in all six builds; only the final key/value pairs are copied into it. That is the arena discipline in one sentence: bulk-free the churn, copy out what survives. -Go deliberately is not "arena" in the C++/Rust sense — the language offers +Go is not "arena" in the C++/Rust sense: the language offers no stable arena API (the `arena` experiment stalled), so the idiomatic answer to allocation churn is object reuse via `sync.Pool`, which is what the variant demonstrates. ## Knobs worth playing with (Go) -The Go runtime exposes its memory policy as environment variables — no +The Go runtime exposes its memory policy as environment variables, no rebuild required: ```console @@ -71,9 +71,9 @@ allocbench: variant=default allocs=200000 peak_rss=37292KB ms=51 gc_cycles=0 ``` `GOMEMLIMIT` (Go 1.19+) is a soft cap on the runtime's total memory: as the -heap approaches it, the collector runs more aggressively to stay under — -and if the limit is never approached (last run above), the collector simply -never fires and the garbage piles up as RSS. Watch `gc_cycles` and +heap approaches it, the collector runs more aggressively to stay under; if +the limit is never approached (last run above), the collector never fires +and the garbage piles up as RSS. Watch `gc_cycles` and `peak_rss` move in opposite directions as you trade one for the other. `GOGC` scales how much new garbage may accumulate between cycles (default 100 = the heap may double). @@ -93,9 +93,9 @@ $ ./demo.sh rust run --variant arena allocbench: variant=arena allocs=200000 peak_rss=2576KB ms=31 ``` -(Numbers vary by host and run; only the line shape is stable.) Note what -the C++ pair shows: at this workload size the arena's peak RSS is -*comparable*, not lower — glibc malloc recycles the churn through the same +(Numbers vary by host and run; only the line shape is stable.) The C++ pair +shows that at this workload size the arena's peak RSS is +*comparable*, not lower: glibc malloc recycles the churn through the same pages, while the arena's always-touched 256 KiB slab is a fixed cost. The arena's win here is deterministic batch reclamation (no per-object frees), not a smaller high-water mark; crank `--allocs` or the value sizes and the @@ -105,8 +105,8 @@ curves separate. Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (builds first if needed) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (builds first if needed) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -118,9 +118,9 @@ The top-level `demo.sh` dispatches: `./demo.sh go run --variant arena` execs `verify.lua` (run per language with `LSP_LANG` set) asserts behavior, not just exit codes: bad usage exits 2 with the usage line; both variants emit the exact single-line report; `variant`/`allocs` echo the request; -`peak_rss`/`ms` (and `gc_cycles` for Go — which must be absent for C++ and +`peak_rss`/`ms` (and `gc_cycles` for Go, which must be absent for C++ and Rust) parse to sane numbers; the bare run defaults to `variant=default allocs=200000`; and a 7-alloc run still answers its queries. The default-vs-arena RSS delta is printed to the log but not -asserted — ordering can flake on a loaded host, and the chapter text, not +asserted: ordering can flake on a loaded host, and the chapter text, not CI, owns that claim. diff --git a/examples/17-allocators-and-gc-runtimes/cpp/src/main.cpp b/examples/17-allocators-and-gc-runtimes/cpp/src/main.cpp index 7e8321a..d835a20 100644 --- a/examples/17-allocators-and-gc-runtimes/cpp/src/main.cpp +++ b/examples/17-allocators-and-gc-runtimes/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// allocbench — allocation-heavy index build/query (chapter 17: allocators +// allocbench: allocation-heavy index build/query (chapter 17: allocators // and GC runtimes). // // allocbench [--allocs N] [--variant default|arena] @@ -8,7 +8,7 @@ // strings, then query every distinct key back. What differs is where the // intermediates live: // -// default: every key/frag/value is a plain std::string — each a trip to +// default: every key/frag/value is a plain std::string: each a trip to // the global heap (malloc), each freed individually when its // batch scratch vector is destroyed. // arena: per batch of 1000 iterations, a std::pmr::monotonic_buffer_- @@ -20,7 +20,7 @@ // long-lived index. // // Reports: "allocbench: variant= allocs= peak_rss=KB ms=" -// (peak_rss is getrusage(2) ru_maxrss — the VmHWM high-water mark — in KB). +// (peak_rss is getrusage(2) ru_maxrss, the VmHWM high-water mark, in KB). #include #include diff --git a/examples/17-allocators-and-gc-runtimes/go/main.go b/examples/17-allocators-and-gc-runtimes/go/main.go index d423b59..3e4536c 100644 --- a/examples/17-allocators-and-gc-runtimes/go/main.go +++ b/examples/17-allocators-and-gc-runtimes/go/main.go @@ -1,4 +1,4 @@ -// allocbench — allocation-heavy index build/query (chapter 17: allocators +// allocbench: allocation-heavy index build/query (chapter 17: allocators // and GC runtimes). // // allocbench [--allocs N] [--variant default|arena] @@ -9,7 +9,7 @@ // the intermediates: // // - default: every key/frag/value is a freshly allocated string. All of -// that garbage is the GC's problem — watch gc_cycles climb with N. +// that garbage is the GC's problem; watch gc_cycles climb with N. // - arena: Go has no std::pmr / bumpalo equivalent, so the idiom is // sync.Pool: each iteration borrows a *scratch (three bytes.Buffers and // a number-formatting []byte) from the pool, builds the strings in @@ -19,7 +19,7 @@ // // Reports: "allocbench: variant= allocs= peak_rss=KB ms= gc_cycles=" // where gc_cycles is the delta of runtime.MemStats.NumGC across the -// build+query phase (the Go-only field — C++ and Rust have no collector), +// build+query phase (the Go-only field; C++ and Rust have no collector), // and peak_rss comes from unix.Getrusage (ru_maxrss, KB). // // The GC's appetite is tunable without touching this code: GOGC scales how diff --git a/examples/17-allocators-and-gc-runtimes/rust/src/main.rs b/examples/17-allocators-and-gc-runtimes/rust/src/main.rs index 3af6036..14f9b41 100644 --- a/examples/17-allocators-and-gc-runtimes/rust/src/main.rs +++ b/examples/17-allocators-and-gc-runtimes/rust/src/main.rs @@ -1,4 +1,4 @@ -// allocbench — allocation-heavy index build/query (chapter 17: allocators +// allocbench: allocation-heavy index build/query (chapter 17: allocators // and GC runtimes). // // allocbench [--allocs N] [--variant default|arena] @@ -8,7 +8,7 @@ // strings, then query every distinct key back. What differs is where the // intermediates live: // -// default: every key/frag/value is a plain String — each a trip to the +// default: every key/frag/value is a plain String: each a trip to the // global allocator, each freed individually when its batch // scratch Vec is dropped. // arena: a bumpalo::Bump arena. Per batch of 1000 iterations the @@ -20,7 +20,7 @@ // the long-lived index. // // Reports: "allocbench: variant= allocs= peak_rss=KB ms=" -// (peak_rss is getrusage(2) ru_maxrss — the VmHWM high-water mark — in KB). +// (peak_rss is getrusage(2) ru_maxrss, the VmHWM high-water mark, in KB). use std::collections::HashMap; use std::fmt::Write as _; @@ -81,8 +81,7 @@ fn parse_args() -> Result { /// default variant: plain heap strings, freed one by one at batch end. fn build_default(allocs: usize) -> HashMap { let mut index = HashMap::with_capacity(KEYSPACE); - let mut start = 0; - while start < allocs { + for start in (0..allocs).step_by(BATCH) { let end = (start + BATCH).min(allocs); let mut scratch: Vec = Vec::with_capacity(end - start); for i in start..end { @@ -96,7 +95,6 @@ fn build_default(allocs: usize) -> HashMap { index.insert(key, value); scratch.push(frag); // keeps the churn alive per batch } - start = end; } // drop(scratch): one deallocation per surviving string index } @@ -106,8 +104,7 @@ fn build_default(allocs: usize) -> HashMap { fn build_arena(allocs: usize) -> HashMap { let mut bump = Bump::with_capacity(SLAB_BYTES); let mut index = HashMap::with_capacity(KEYSPACE); - let mut start = 0; - while start < allocs { + for start in (0..allocs).step_by(BATCH) { let end = (start + BATCH).min(allocs); { let mut scratch: BumpVec = BumpVec::with_capacity_in(end - start, &bump); @@ -126,7 +123,6 @@ fn build_arena(allocs: usize) -> HashMap { } } // borrows on the bump end here... bump.reset(); // ...so the whole batch vanishes in O(1) - start = end; } index } diff --git a/examples/17-allocators-and-gc-runtimes/verify.lua b/examples/17-allocators-and-gc-runtimes/verify.lua index d5f3652..c30778b 100644 --- a/examples/17-allocators-and-gc-runtimes/verify.lua +++ b/examples/17-allocators-and-gc-runtimes/verify.lua @@ -3,7 +3,7 @@ -- --allocs must be 200000. Go alone must additionally report gc_cycles= -- (its collector is the chapter's point; C++/Rust must NOT print it). -- Timing/RSS ordering between variants is measured and printed for the log --- but deliberately not asserted — it can flake on a loaded host. +-- but deliberately not asserted: it can flake on a loaded host. -- Invoked from the example directory with LSP_LANG (cpp|go|rust) and EXAMPLE_DIR set. local script_dir = arg[0]:match("(.*/)") or "./" diff --git a/examples/18-pipes-fifos-splice/README.md b/examples/18-pipes-fifos-splice/README.md index 185c2d3..e37dd19 100644 --- a/examples/18-pipes-fifos-splice/README.md +++ b/examples/18-pipes-fifos-splice/README.md @@ -1,30 +1,30 @@ -# 18 — Pipes, FIFOs, splice (`pmon` v4) +# 18: Pipes, FIFOs, splice (`pmon` v4) `pmon` is the process supervisor that grows across the book: v0 `run`, -v1 the sigchld supervise engine, v2 the pidfd engine (chapter 13), v3 the +v1 the sigchld supervise engine, v2 the pidfd engine, v3 the privilege-drop chapter. v4 keeps both supervise engines exactly as they were and turns the supervisor into a log pipeline built from kernel pipe machinery: -- **`supervise` captures the child** — the child's stdout and stderr each +- **`supervise` captures the child**: the child's stdout and stderr each flow through their own `pipe(2)` back into the supervisor, which relays them line-buffered into a log file as `[out] ...` / `[err] ...` lines. - The two pipe read ends simply join the fd set each engine already + The two pipe read ends join the fd set each engine already multiplexes; nothing about the v1/v2 restart/timeout/stop behavior changes. -- **`tail` republishes the log** — it creates a FIFO (`mkfifo(3)`), follows +- **`tail` republishes the log**: it creates a FIFO (`mkfifo(3)`), follows the log file, and relays appended bytes into whatever reader opens the FIFO. C++ and Rust move the bytes kernel-side with `splice(2)` (file → pipe, no userspace copy; automatic `read`/`write` fallback on filesystems without splice support). A reader that disappears is not fatal: SIGPIPE is ignored, the resulting `EPIPE` prints - `pmon: tail reader detached`, and the relay waits for the next reader — - losing nothing, because the log offset only advances for bytes that - actually reached the pipe. + `pmon: tail reader detached`, and the relay waits for the next reader, + losing nothing because the log offset only advances for bytes that + reached the pipe. | Language | Architecture | |---|---| -| **C++23** | one `poll(2)` loop per child over four fds — the two capture pipes, the signalfd, and (pidfd engine) the pidfd from raw `SYS_pidfd_open`; `waitid(P_PIDFD)` reaps race-free. RAII `Fd`/`SpawnAttrs`/`FileActions` owners, `std::expected` error paths, `posix_spawnp` wiring the pipes via `adddup2` and resetting the child's signal mask. `tail` uses the `splice(2)` fast path with an offset-preserving `read`/`write` fallback. | +| **C++23** | one `poll(2)` loop per child over four fds: the two capture pipes, the signalfd, and (pidfd engine) the pidfd from raw `SYS_pidfd_open`; `waitid(P_PIDFD)` reaps race-free. RAII `Fd`/`SpawnAttrs`/`FileActions` owners, `std::expected` error paths, `posix_spawnp` wiring the pipes via `adddup2` and resetting the child's signal mask. `tail` uses the `splice(2)` fast path with an offset-preserving `read`/`write` fallback. | | **Rust (edition 2024)** | the same poll set via `rustix::event::poll` + a nix `SignalFd`, `pidfd_open` returning an `OwnedFd`, `waitid(WaitId::PidFd)`; `rustix::pipe::splice` for the tail fast path; `anyhow` + `?` throughout, every fd `OwnedFd`-backed. | | **Go 1.26** | the idiomatic inversion: one goroutine per capture pipe feeds a channel of prefixed lines into the same `select` that already multiplexes the pidfd poller (or `signal.Notify(SIGCHLD)`), the stop signals, and the deadline. `tail` relays with `io.Copy` per reader session, rewinding the log to `start+written` on `EPIPE`; `mkfifo`/nonblocking FIFO open via `golang.org/x/sys/unix`. | @@ -60,7 +60,7 @@ New in v4: `[out] line` / `[err] line`; a trailing partial line is flushed with a newline when the stream closes, and the log is fully drained before the exit observation is printed. -- `tail --log FILE --fifo PATH` — creates PATH as a FIFO (an existing FIFO +- `tail --log FILE --fifo PATH`: creates PATH as a FIFO (an existing FIFO is reused; anything else is an error), prints `pmon: tail ready (fifo PATH)` to stderr, then relays FILE from the beginning into each successive FIFO reader, polling for appended bytes. @@ -73,11 +73,11 @@ New in v4: Each language directory has the standard `demo.sh`: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (`app`); the local path +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (`app`); the local path `exec`s the binary so a backgrounded `run tail ...` can be signalled directly -- `./demo.sh` — build, then a one-shot self-demo (`run -- sh -c ...`) +- `./demo.sh`: build, then a one-shot self-demo (`run -- sh -c ...`) - With env `TARGET` set, `run` deploys to that lab VM via `scripts/lab/deploy-to-vm.sh` diff --git a/examples/18-pipes-fifos-splice/cpp/src/main.cpp b/examples/18-pipes-fifos-splice/cpp/src/main.cpp index 624e85f..8bf78ed 100644 --- a/examples/18-pipes-fifos-splice/cpp/src/main.cpp +++ b/examples/18-pipes-fifos-splice/cpp/src/main.cpp @@ -1,16 +1,16 @@ -// pmon v4 — the process supervisor becomes a log pipeline. +// pmon v4: the process supervisor becomes a log pipeline. // // v0 (`run`) spawns a command, waits, and mirrors its exit status. // v1 (`supervise --engine sigchld`) restarts a crashing child via SIGCHLD // delivered as readable data through a signalfd. // v2 (`supervise --engine pidfd`, the DEFAULT) supervises through -// pidfd_open(2)/poll(2)/waitid(P_PIDFD) — no pid-reuse race. +// pidfd_open(2)/poll(2)/waitid(P_PIDFD), no pid-reuse race. // v4 (this chapter) keeps both engines and adds pipes: // * supervise captures the child's stdout AND stderr through two -// pipe(2)s into a log file as "[out] ..."/"[err] ..." lines — the two +// pipe(2)s into a log file as "[out] ..."/"[err] ..." lines: the two // pipe read ends simply join the fd set each engine already polls; // * `tail --log F --fifo PATH` creates a FIFO and relays log bytes into -// whatever reader attaches — splice(2) fast path (file -> pipe, +// whatever reader attaches: splice(2) fast path (file -> pipe, // kernel-side), read/write fallback; SIGPIPE ignored, EPIPE reported // as "pmon: tail reader detached" and survived, nothing lost. @@ -58,7 +58,7 @@ constexpr int kPollMs = 50; } // --------------------------------------------------------------------------- -// RAII owners — every fd and every spawn attribute has exactly one. +// RAII owners: every fd and every spawn attribute has exactly one. // --------------------------------------------------------------------------- class Fd { @@ -198,7 +198,7 @@ make_signalfd(std::initializer_list sigs) { } // --------------------------------------------------------------------------- -// v0: run — spawn, wait, mirror (stdio inherited, no capture). +// v0: run: spawn, wait, mirror (stdio inherited, no capture). // --------------------------------------------------------------------------- int cmd_run(const std::vector& cmd) { @@ -341,7 +341,7 @@ using Outcome = std::variant; return left < 0 ? 0 : static_cast(left); } -// waitid(P_PIDFD) reaps the process the fd refers to — never a recycled pid. +// waitid(P_PIDFD) reaps the process the fd refers to: never a recycled pid. [[nodiscard]] std::expected reap_pidfd(const Fd& pidfd) { siginfo_t si{}; while (::waitid(P_PIDFD, static_cast(pidfd.get()), &si, WEXITED) != 0) { @@ -520,7 +520,7 @@ int cmd_supervise(Engine engine, int max_restarts, int timeout_ms, } // --------------------------------------------------------------------------- -// v4: tail — follow the log, relay into a FIFO, survive reader churn. +// v4: tail: follow the log, relay into a FIFO, survive reader churn. // --------------------------------------------------------------------------- volatile sig_atomic_t g_stop = 0; @@ -559,7 +559,7 @@ enum class Relay : std::uint8_t { moved, idle, detached, no_splice }; // splice(2) fast path: move log bytes kernel-side into the pipe. The file // offset only advances on success, so a detached reader (EPIPE) loses -// nothing — the same bytes are respliced for the next reader. +// nothing: the same bytes are respliced for the next reader. [[nodiscard]] std::expected relay_splice(const Fd& log, const Fd& w) { const ssize_t n = ::splice(log.get(), nullptr, w.get(), nullptr, kChunk, 0); diff --git a/examples/18-pipes-fifos-splice/go/main.go b/examples/18-pipes-fifos-splice/go/main.go index 3e4b40a..03376ee 100644 --- a/examples/18-pipes-fifos-splice/go/main.go +++ b/examples/18-pipes-fifos-splice/go/main.go @@ -1,10 +1,10 @@ -// pmon v4 — the process supervisor becomes a log pipeline. +// pmon v4: the process supervisor becomes a log pipeline. // // v0 (`run`) spawns a command, waits, and mirrors its exit status. // v1 (`supervise --engine sigchld`) restarts a crashing child via SIGCHLD // through signal.Notify, reaped with wait4(WNOHANG). // v2 (`supervise --engine pidfd`, the DEFAULT) supervises through -// unix.PidfdOpen + unix.Poll + waitid(P_PIDFD) — no pid-reuse race. +// unix.PidfdOpen + unix.Poll + waitid(P_PIDFD); no pid-reuse race. // v4 (this chapter) keeps both engines and adds pipes: // - supervise captures the child's stdout AND stderr through two pipes // into a log file as "[out] ..."/"[err] ..." lines. Where C++/Rust add @@ -12,7 +12,7 @@ // goroutine per pipe feeds a channel of prefixed lines into the same // select loop that already multiplexes exit, signals, and the deadline; // - `tail --log F --fifo PATH` creates a FIFO and relays log bytes into -// whatever reader attaches — io.Copy per reader session, rewinding the +// whatever reader attaches: io.Copy per reader session, rewinding the // log offset to start+written on EPIPE so nothing is lost; SIGPIPE // ignored, the detach reported as "pmon: tail reader detached". package main @@ -64,7 +64,7 @@ func fromWaitStatus(ws syscall.WaitStatus) childStatus { } // --------------------------------------------------------------------------- -// v0: run — spawn, wait, mirror (stdio inherited, no capture). +// v0 (run): spawn, wait, mirror (stdio inherited, no capture). // --------------------------------------------------------------------------- func cmdRun(cmdline []string) int { @@ -89,7 +89,7 @@ func cmdRun(cmdline []string) int { } // --------------------------------------------------------------------------- -// v2 plumbing: waitid(P_PIDFD) with an explicit siginfo_t layout — x/sys +// v2 plumbing: waitid(P_PIDFD) with an explicit siginfo_t layout; x/sys // keeps the union opaque, and the CLD_* fields are what we need. // --------------------------------------------------------------------------- @@ -107,7 +107,7 @@ type siginfoChld struct { const cldExited = 1 // si_code: child called _exit // reapPidfd blocks in waitid(P_PIDFD) until the process behind pidfd is -// reaped — it can never collect a recycled pid. +// reaped; it can never collect a recycled pid. func reapPidfd(pidfd int) (childStatus, error) { var si siginfoChld for { @@ -187,7 +187,7 @@ func writeLine(logw *os.File, l logLine) error { } // capture wires both stdio streams of an un-started command into the log. -// lines closes when both pipes hit EOF — i.e. the capture is complete. +// lines closes when both pipes hit EOF, meaning the capture is complete. func capture(c *exec.Cmd) (<-chan logLine, error) { outp, err := c.StdoutPipe() if err != nil { @@ -379,7 +379,7 @@ func cmdSupervise(engine string, maxRestarts int, timeout time.Duration, } // --------------------------------------------------------------------------- -// v4: tail — follow the log, relay into a FIFO, survive reader churn. +// v4 (tail): follow the log, relay into a FIFO, survive reader churn. // --------------------------------------------------------------------------- // ensureFifo creates PATH as a FIFO, tolerating an existing FIFO at PATH. @@ -463,7 +463,7 @@ reader: } // One copy session: on EPIPE (reader detached) io.Copy reports // how many bytes actually reached the pipe, so the log offset is - // rewound to the first unwritten byte — the next reader loses + // rewound to the first unwritten byte; the next reader loses // nothing. start, serr := logf.Seek(0, io.SeekCurrent) if serr != nil { diff --git a/examples/18-pipes-fifos-splice/rust/src/main.rs b/examples/18-pipes-fifos-splice/rust/src/main.rs index 9393b56..d0c2af7 100644 --- a/examples/18-pipes-fifos-splice/rust/src/main.rs +++ b/examples/18-pipes-fifos-splice/rust/src/main.rs @@ -1,17 +1,17 @@ -// pmon v4 — the process supervisor becomes a log pipeline. +// pmon v4: the process supervisor becomes a log pipeline. // // v0 (`run`) spawns a command, waits, and mirrors its exit status. // v1 (`supervise --engine sigchld`) restarts a crashing child via SIGCHLD // delivered as readable data through a signalfd (nix). // v2 (`supervise --engine pidfd`, the DEFAULT) supervises through -// rustix::process::pidfd_open + poll + waitid(WaitId::PidFd) — no +// rustix::process::pidfd_open + poll + waitid(WaitId::PidFd): no // pid-reuse race. // v4 (this chapter) keeps both engines and adds pipes: // * supervise captures the child's stdout AND stderr through two pipes -// into a log file as "[out] ..."/"[err] ..." lines — the two pipe read +// into a log file as "[out] ..."/"[err] ..." lines. The two pipe read // ends simply join the fd set each engine already polls; // * `tail --log F --fifo PATH` creates a FIFO and relays log bytes into -// whatever reader attaches — rustix::pipe::splice fast path +// whatever reader attaches: rustix::pipe::splice fast path // (file -> pipe, kernel-side), read/write fallback; SIGPIPE ignored, // EPIPE reported as "pmon: tail reader detached" and survived, // nothing lost. @@ -65,7 +65,7 @@ fn report_exit(pid: i32, st: ChildStatus) -> i32 { } } -/// Spawn with an empty signal mask in the child — the supervisor blocks +/// Spawn with an empty signal mask in the child: the supervisor blocks /// signals for its signalfd and std does NOT reset the mask across /// fork+exec (a supervised `sleep` with SIGTERM blocked would never stop). /// sigprocmask is async-signal-safe, so pre_exec may call it. @@ -86,7 +86,7 @@ fn spawn(cmdline: &[String], piped: bool) -> Result { } // --------------------------------------------------------------------------- -// v0: run — spawn, wait, mirror (stdio inherited, no capture). +// v0 (run): spawn, wait, mirror (stdio inherited, no capture). // --------------------------------------------------------------------------- fn cmd_run(cmdline: &[String]) -> Result { @@ -257,7 +257,7 @@ fn status_from_waitid(w: rustix::process::WaitIdStatus) -> ChildStatus { } } -/// waitid(P_PIDFD) reaps the process the fd refers to — never a recycled pid. +/// waitid(P_PIDFD) reaps the process the fd refers to; never a recycled pid. fn reap_pidfd(pidfd: &impl AsFd) -> Result { let status = waitid(WaitId::PidFd(pidfd.as_fd()), WaitIdOptions::EXITED) .context("waitid(P_PIDFD)")? @@ -373,8 +373,11 @@ fn cmd_supervise( let pidfd = match engine { Engine::Pidfd => { - let pidfd = pidfd_open(Pid::from_raw(pid).unwrap(), PidfdFlags::empty()) - .context("pidfd_open")?; + let pidfd = pidfd_open( + Pid::from_raw(pid).expect("child.id() is always a positive pid"), + PidfdFlags::empty(), + ) + .context("pidfd_open")?; eprintln!("pmon: engine=pidfd child={pid} pidfd={}", pidfd.as_raw_fd()); Some(pidfd) } @@ -435,7 +438,7 @@ fn cmd_supervise( } // --------------------------------------------------------------------------- -// v4: tail — follow the log, relay into a FIFO, survive reader churn. +// v4 (tail): follow the log, relay into a FIFO, survive reader churn. // --------------------------------------------------------------------------- static STOP: AtomicBool = AtomicBool::new(false); @@ -481,7 +484,7 @@ enum Relay { /// splice(2) fast path: move log bytes kernel-side into the pipe. The file /// offset only advances on success, so a detached reader (EPIPE) loses -/// nothing — the same bytes are respliced for the next reader. +/// nothing; the same bytes are respliced for the next reader. fn relay_splice(log: &File, w: &OwnedFd) -> Result { match splice(log, None, w, None, CHUNK, SpliceFlags::empty()) { Ok(0) => Ok(Relay::Idle), diff --git a/examples/18-pipes-fifos-splice/verify.lua b/examples/18-pipes-fifos-splice/verify.lua index d42e6b1..f580478 100644 --- a/examples/18-pipes-fifos-splice/verify.lua +++ b/examples/18-pipes-fifos-splice/verify.lua @@ -2,11 +2,11 @@ -- -- Asserts observable behavior, identical across cpp/go/rust: -- 1. usage errors exit 2 with the usage text --- 2. v0 `run` still mirrors exits — status 3 stays 3, SIGKILL becomes 137 +-- 2. v0 `run` still mirrors exits: status 3 stays 3, SIGKILL becomes 137 -- with "killed signal=9" -- 3. v2 pidfd engine + v4 capture: a child that fails once and then -- succeeds yields two "engine=pidfd child=... pidfd=..." spawn lines, --- "restart 1/3", exit 0 — and the log holds the child's stdout AND +-- "restart 1/3", exit 0, and the log holds the child's stdout AND -- stderr as "[out] ..."/"[err] ..." lines from both runs -- 4. v1 sigchld engine still works with the capture attached: same restart -- shape, no "pidfd=" anywhere, "giving up" exit 1, log filled diff --git a/examples/19-unix-sockets-fd-passing/README.md b/examples/19-unix-sockets-fd-passing/README.md index 0737cbf..e5ef308 100644 --- a/examples/19-unix-sockets-fd-passing/README.md +++ b/examples/19-unix-sockets-fd-passing/README.md @@ -1,8 +1,8 @@ -# 19-unix-sockets-fd-passing — pmon v5 +# 19-unix-sockets-fd-passing: pmon v5 pmon grows a control plane. `pmon supervise` runs a child command under -supervision — own process group, stdout+stderr appended to a log file, -automatic restart 300 ms after every exit — and listens on a UNIX stream +supervision (own process group, stdout+stderr appended to a log file, +automatic restart 300 ms after every exit) and listens on a UNIX stream socket. `pmon pmctl` (same binary, different subcommand) talks to it: ``` @@ -25,9 +25,9 @@ $ ./demo.sh cpp run pmctl --ctl /tmp/p/ctl.sock stop stopping ``` -## What the chapter is really about +## The point of this chapter -`logfd` is the headline. The supervisor does not send the log *path* — it +`logfd` is the headline. The supervisor does not send the log *path*; it sends its **open log file descriptor** through the socket in an `SCM_RIGHTS` control message, and pmctl reads the last 3 lines directly off the received fd (positional reads from offset 0, so the supervisor's `O_APPEND` writer is @@ -38,11 +38,11 @@ by diffing the `via-fd:` output against `tail -3` of the file, byte for byte. Two supporting ideas: -- **SO_PEERCRED** — before trusting a single command byte, the supervisor +- **SO_PEERCRED**: before trusting a single command byte, the supervisor asks the kernel who is on the other end of the connection and logs `pmon: ctl connect uid= pid=

`. Credentials from the kernel, not from the client's own claims. -- **The log is opened O_RDWR|O_APPEND** — a write-only fd would arrive +- **The log is opened O_RDWR|O_APPEND**: a write-only fd would arrive write-only in pmctl (the *file description* crosses the socket, flags and all), and the read side would fail with EBADF. @@ -58,13 +58,13 @@ plus cmsg) out. | **Go 1.26** | `unix.UnixRights` / `unix.ParseSocketControlMessage` / `unix.ParseUnixRights` over `net.UnixConn.WriteMsgUnix`/`ReadMsgUnix` | reaper goroutine; restart backoff vs. shutdown is a `select` on a timer and a stop channel; wrapped errors end at `pmon: error:` | | **Rust (2024)** | `nix::sys::socket` `sendmsg`/`recvmsg` with `ControlMessage::ScmRights` and `cmsg_space!` | reaper thread, `Arc<(Mutex, Condvar)>`; the received fd goes straight into an `OwnedFd` so it closes on drop | -## Gotchas worth remembering +## Points worth remembering - **`sun_path` is 108 bytes.** Bind/connect fail with `EINVAL`-flavored errors when the socket path is longer (deep CI workspace paths hit this immediately). Keep control sockets under short directories. - Passing an fd duplicates the *open file description*, so offset and status - flags travel with it — hence O_RDWR above, and hence pmctl using `pread` + flags travel with it: hence O_RDWR above, and hence pmctl using `pread` instead of seeking, which would move the shared offset. - The child gets its own process group (`setpgid`), so `stop` can SIGTERM the whole tree, including anything the child forked. diff --git a/examples/19-unix-sockets-fd-passing/cpp/src/main.cpp b/examples/19-unix-sockets-fd-passing/cpp/src/main.cpp index c4e3729..9253a63 100644 --- a/examples/19-unix-sockets-fd-passing/cpp/src/main.cpp +++ b/examples/19-unix-sockets-fd-passing/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// pmon v5 — a process supervisor with a UNIX-socket control plane. +// pmon v5: a process supervisor with a UNIX-socket control plane. // // `pmon supervise` forks/execs a child command (its own process group, // stdout+stderr appended to a log file), restarts it 300 ms after every @@ -15,7 +15,7 @@ // // Every accepted connection is identified with SO_PEERCRED before a single // command byte is trusted. The cmsg plumbing here is done by hand with -// sendmsg(2)/recvmsg(2) — CMSG_FIRSTHDR and friends, no wrapper library. +// sendmsg(2)/recvmsg(2): CMSG_FIRSTHDR and friends, no wrapper library. #include #include @@ -59,7 +59,7 @@ template using Result = std::expected; // --------------------------------------------------------------------------- -// RAII file descriptor — the only owner of every fd in this program. +// RAII file descriptor: the only owner of every fd in this program. // --------------------------------------------------------------------------- class Fd { diff --git a/examples/19-unix-sockets-fd-passing/go/main.go b/examples/19-unix-sockets-fd-passing/go/main.go index f44bbdd..a028a8d 100644 --- a/examples/19-unix-sockets-fd-passing/go/main.go +++ b/examples/19-unix-sockets-fd-passing/go/main.go @@ -1,4 +1,4 @@ -// pmon v5 — a process supervisor with a UNIX-socket control plane. +// pmon v5: a process supervisor with a UNIX-socket control plane. // // `pmon supervise` spawns a child command (its own process group, // stdout+stderr appended to a log file), restarts it 300 ms after every @@ -56,7 +56,7 @@ type state struct { } // spawnChild starts argv in its own process group with stdout+stderr going to -// the (O_APPEND) log file — the fd is inherited directly, no pipe goroutine. +// the (O_APPEND) log file; the fd is inherited directly, no pipe goroutine. func spawnChild(argv []string, logFile *os.File) (*exec.Cmd, error) { cmd := exec.Command(argv[0], argv[1:]...) cmd.Stdout = logFile diff --git a/examples/19-unix-sockets-fd-passing/rust/src/main.rs b/examples/19-unix-sockets-fd-passing/rust/src/main.rs index 9b81124..d7c2c5a 100644 --- a/examples/19-unix-sockets-fd-passing/rust/src/main.rs +++ b/examples/19-unix-sockets-fd-passing/rust/src/main.rs @@ -1,4 +1,4 @@ -// pmon v5 — a process supervisor with a UNIX-socket control plane. +// pmon v5: a process supervisor with a UNIX-socket control plane. // // `pmon supervise` spawns a child command (its own process group, // stdout+stderr appended to a log file), restarts it 300 ms after every @@ -13,7 +13,7 @@ // the path. // // The cmsg plumbing is nix::sys::socket sendmsg/recvmsg with -// ControlMessage::ScmRights — explicit, not a convenience crate — and the +// ControlMessage::ScmRights (explicit, not a convenience crate), and the // received fd is wrapped in an OwnedFd immediately so it closes on drop. use std::fs::File; @@ -58,7 +58,7 @@ struct State { type Shared = Arc<(Mutex, Condvar)>; /// Spawn argv in its own process group with stdout+stderr appended to the -/// log file — the fd is inherited directly, no pipe thread. +/// log file: the fd is inherited directly, no pipe thread. fn spawn_child(argv: &[String], log: &File) -> Result { let mut cmd = Command::new(&argv[0]); cmd.args(&argv[1..]) diff --git a/examples/19-unix-sockets-fd-passing/verify.lua b/examples/19-unix-sockets-fd-passing/verify.lua index 9db65c9..0211f4b 100644 --- a/examples/19-unix-sockets-fd-passing/verify.lua +++ b/examples/19-unix-sockets-fd-passing/verify.lua @@ -7,7 +7,7 @@ -- command (process-tree check via /proc) -- 3. SO_PEERCRED: the supervisor logs the CALLING uid for every ctl connect -- 4. logfd: pmctl's "via-fd:" output equals `tail -3` of the log file byte --- for byte — proof the fd crossed the socket and was read directly +-- for byte: proof the fd crossed the socket and was read directly -- 5. restarts: a short-lived child drives restarts=N>0 in status, restart -- lines on stderr, and one "pmon: start child" log line per (re)start -- 6. stop: reply is exactly "stopping"; supervisor exits, removes the @@ -29,7 +29,7 @@ end local demo = "./demo.sh " .. lang .. " run" -- Scratch area. NB: sun_path caps UNIX socket paths at ~107 bytes, so this --- must stay short — mktemp -d under /tmp is fine, deep CI workspaces are not. +-- must stay short: mktemp -d under /tmp is fine, deep CI workspaces are not. local wd = checks.run("mktemp -d").out:gsub("%s+$", "") if wd == "" or not wd:match("^/") then checks.skip("mktemp -d did not yield a directory") diff --git a/examples/20-shared-memory-coordination/README.md b/examples/20-shared-memory-coordination/README.md index 2d25537..40b9f56 100644 --- a/examples/20-shared-memory-coordination/README.md +++ b/examples/20-shared-memory-coordination/README.md @@ -1,9 +1,9 @@ -# 20-shared-memory-coordination — shmkv v1 +# 20-shared-memory-coordination: shmkv v1 shmkv grows from a passive shared-memory key-value file (v0) into a live coordination fabric: one process publishes updates, other processes learn about them **without polling**, through a seqlock for consistency and a futex -for wakeups — the same pair of primitives glibc builds mutexes and condition +for wakeups: the same pair of primitives glibc builds mutexes and condition variables from. Implemented three times (C++23, Go 1.26, Rust 2024) with identical observable behavior. @@ -40,7 +40,7 @@ shmkv bench FILE [--rounds N] [--channel futex|mq|poll] `observed update : =` for every new id, exiting 0 after N events (or 1 after a 30 s self-termination deadline). - **bench**: producer/consumer threads over the same mapping measure one-way - notification latency for three channels — `futex` (wake/wait), `mq` (a + notification latency for three channels: `futex` (wake/wait), `mq` (a POSIX message queue carrying the update id), `poll` (a 1 ms sleep loop re-reading the counter; it sleeps *before* looking, so every observation costs at least one full nap). Rounds are strictly serialized by an ack word @@ -60,15 +60,15 @@ read: s1 = seq (acquire) ; if s1 odd retry s2 = seq (relaxed) ; consistent iff s1 == s2, else retry ``` -The data copy races the writer on purpose; a torn copy is detected by the -seq re-check and discarded. That trade-off — writers never block, readers -retry — is exactly the kernel's seqlock. The retry loop is bounded (10^6 +The data copy races the writer by design; a torn copy is detected by the +seq re-check and discarded. That trade-off (writers never block, readers +retry) is exactly the kernel's seqlock. The retry loop is bounded (10^6 attempts) so a corrupt file cannot livelock a reader forever. The futex is **shared** (no `FUTEX_PRIVATE_FLAG`): the kernel keys waiters by the file's inode, which is what lets two unrelated processes rendezvous through the same mapped page. `EAGAIN`, `ETIMEDOUT`, and `EINTR` from -`FUTEX_WAIT` are all normal — the waiter re-reads shared state and decides +`FUTEX_WAIT` are all normal: the waiter re-reads shared state and decides again. ## Language notes diff --git a/examples/20-shared-memory-coordination/cpp/src/main.cpp b/examples/20-shared-memory-coordination/cpp/src/main.cpp index 21b1ff7..035544b 100644 --- a/examples/20-shared-memory-coordination/cpp/src/main.cpp +++ b/examples/20-shared-memory-coordination/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// shmkv v1 — cross-process coordination over a shared-memory key-value file. +// shmkv v1: cross-process coordination over a shared-memory key-value file. // // The v0 file format (fixed slots in an mmap'd file) grows a v1 header with // SHKV2 magic: @@ -12,9 +12,9 @@ // 64 512 8 slots x 64 bytes: key[24] NUL-padded, value[40] NUL-padded // // Three coordination channels are demonstrated: -// serve — writer: seqlock-protected publish + FUTEX_WAKE after each update -// watch — reader: FUTEX_WAIT loop + seqlock-consistent snapshot reads -// bench — latency shoot-out: futex wake vs POSIX message queue vs 1 ms +// serve (writer): seqlock-protected publish + FUTEX_WAKE after each update +// watch (reader): FUTEX_WAIT loop + seqlock-consistent snapshot reads +// bench: latency shoot-out: futex wake vs POSIX message queue vs 1 ms // sleep-polling of the counter // // Seqlock protocol (single writer): @@ -128,7 +128,7 @@ class Mapping { std::size_t len_ = 0; }; -// mq_close + mq_unlink on scope exit — no queue name survives a bench run. +// mq_close + mq_unlink on scope exit, no queue name survives a bench run. class MessageQueue { public: MessageQueue(mqd_t d, std::string name) : d_(d), name_(std::move(name)) {} @@ -390,7 +390,7 @@ int cmd_bench(const std::string& path, std::uint64_t rounds, Channel channel, break; case Channel::kPoll: // Sleep FIRST, then look: by construction every observation costs - // at least one full 1 ms nap — the cost futex wakeups avoid. + // at least one full 1 ms nap: the cost futex wakeups avoid. for (;;) { std::this_thread::sleep_for(1ms); if (shm->counter().load(std::memory_order_acquire) >= i) break; diff --git a/examples/20-shared-memory-coordination/go/main.go b/examples/20-shared-memory-coordination/go/main.go index aa1a82b..026845b 100644 --- a/examples/20-shared-memory-coordination/go/main.go +++ b/examples/20-shared-memory-coordination/go/main.go @@ -1,4 +1,4 @@ -// shmkv v1 — cross-process coordination over a shared-memory key-value file. +// shmkv v1: cross-process coordination over a shared-memory key-value file. // // Same SHKV2 layout as the C++ and Rust versions: // @@ -11,7 +11,7 @@ // 64 512 8 slots x 64 bytes: key[24] NUL-padded, value[40] NUL-padded // // Go futex caveat: FUTEX_WAIT is a blocking syscall, so it parks the calling -// OS thread inside the kernel — the goroutine scheduler cannot preempt it. +// OS thread inside the kernel; the goroutine scheduler cannot preempt it. // The runtime notices the thread is gone (sysmon) and hands its P to another // thread, so the rest of the program keeps running, but every concurrent // waiter pins one kernel thread for the duration. Channels and the netpoller @@ -371,7 +371,7 @@ func cmdBench(path string, rounds uint64, channel string) error { } case "poll": // Sleep FIRST, then look: by construction every observation - // costs at least one full 1 ms nap — what futex wakes avoid. + // costs at least one full 1 ms nap, which futex wakes avoid. for { time.Sleep(time.Millisecond) if atomic.LoadUint64(s.counter()) >= i { diff --git a/examples/20-shared-memory-coordination/rust/src/main.rs b/examples/20-shared-memory-coordination/rust/src/main.rs index c7a0ada..56903cb 100644 --- a/examples/20-shared-memory-coordination/rust/src/main.rs +++ b/examples/20-shared-memory-coordination/rust/src/main.rs @@ -1,4 +1,4 @@ -// shmkv v1 — cross-process coordination over a shared-memory key-value file. +// shmkv v1: cross-process coordination over a shared-memory key-value file. // // Same SHKV2 layout as the C++ and Go versions: // @@ -10,13 +10,13 @@ // 24 40 reserved // 64 512 8 slots x 64 bytes: key[24] NUL-padded, value[40] NUL-padded // -// The file is opened through rustix (OwnedFd — the descriptor closes on +// The file is opened through rustix (OwnedFd, the descriptor closes on // drop), the mapping is a small RAII type that munmaps on drop, and the -// futex is the raw syscall via libc::syscall — no wrapper crate, so the +// futex is the raw syscall via libc::syscall: no wrapper crate, so the // FUTEX_WAIT/FUTEX_WAKE contract stays visible. The header words are // accessed as &AtomicU32/&AtomicU64 references into the mapping; the slot // bytes are copied non-atomically under the seqlock, and a torn copy is -// detected by the seq re-check and retried — the standard seqlock trade-off. +// detected by the seq re-check and retried: the standard seqlock trade-off. use std::os::fd::{AsFd, OwnedFd}; use std::process::ExitCode; @@ -195,7 +195,7 @@ fn futex_wake_all(word: &AtomicU32) { // --- POSIX message queue (libc wrappers) ----------------------------------- -/// mq_close + mq_unlink on drop — no queue name survives a bench run. +/// mq_close + mq_unlink on drop: no queue name survives a bench run. struct MessageQueue { mqd: libc::mqd_t, name: std::ffi::CString, @@ -388,7 +388,10 @@ fn cmd_bench(path: &str, rounds: u64, channel: &str) -> Result<()> { } }, _ => { - mq_receive_id(mq.as_ref().unwrap())?; + mq_receive_id( + mq.as_ref() + .expect("mq channel guarantees a MessageQueue"), + )?; } } recv[i as usize - 1] = Instant::now(); @@ -409,7 +412,10 @@ fn cmd_bench(path: &str, rounds: u64, channel: &str) -> Result<()> { } send_ts[i as usize - 1] = Instant::now(); if channel == "mq" { - mq_send_id(mq.as_ref().unwrap(), i)?; + mq_send_id( + mq.as_ref().expect("mq channel guarantees a MessageQueue"), + i, + )?; } else { seqlock_write(&shm, || shm.counter().store(i, Ordering::Relaxed)); shm.futex_word().store(i as u32, Ordering::Release); diff --git a/examples/21-tcp-sockets/README.md b/examples/21-tcp-sockets/README.md index fcc1ee3..8e0a1df 100644 --- a/examples/21-tcp-sockets/README.md +++ b/examples/21-tcp-sockets/README.md @@ -1,4 +1,4 @@ -# 21-tcp-sockets — chatterd v0 +# 21-tcp-sockets: chatterd v0 `chatterd` is the book's peer-to-peer chat program. This first cut is a thread-per-connection TCP **chat room**: one binary is both the daemon and the @@ -32,10 +32,10 @@ that arrives as `: `. ## The wire protocol (fixed for the whole book) -This chapter introduces **the** chatterd chat frame: chapters 22, 24, and 27 +This chapter introduces **the** chatterd chat frame: later chatterd versions reuse this exact header and only ever *add* frame types, never change it. A frame that a `chatctl` from this chapter reads or writes is byte-identical to -the one a v1 (ch22), v3 (ch24), or v4 (ch27) build would produce. +the one a later chatterd build would produce. ``` byte: 0 1 2 3 4 5 6 6+L-1 @@ -45,12 +45,12 @@ the one a v1 (ch22), v3 (ch24), or v4 (ch27) build would produce. +----+----+------+------+----+----+---------------------------+ ``` -- **magic** — the two bytes `0x43 0x48` (ASCII `"CH"`). Any other value means +- **magic**: the two bytes `0x43 0x48` (ASCII `"CH"`). Any other value means "not a chatterd frame"; the reader closes the connection. -- **version** — one byte, `0x01`. This chapter is version 0 of the *program* +- **version**: one byte, `0x01`. This chapter is version 0 of the *program* but the wire's version byte has always been `1`; only the frame *header* would ever bump it, and it never has. -- **type** — one byte selecting the payload's shape. This chapter uses three +- **type**: one byte selecting the payload's shape. This chapter uses three of the five book-wide types: | Type | Value | Payload | Who sends it | Meaning | @@ -59,13 +59,14 @@ the one a v1 (ch22), v3 (ch24), or v4 (ch27) build would produce. | **MSG** | `2` | `text` | client, per stdin line | "I say `text`" | | **DELIVER** | `3` | `name` `00` `text` | server, to every client | a broadcast | - `WELCOME` (`4`) and `PING` (`5`) are reserved for chapters 22 and 24; this - program never sends either, and a `chatctl` reader here skips any frame type - it does not recognize instead of erroring out — which is exactly what lets a - ch21 client stay connected to a newer server that does send them. -- **length** — a 2-byte unsigned payload length in **big-endian** order, + `WELCOME` (`4`) and `PING` (`5`) are reserved for later chatterd versions; + this program never sends either, and a `chatctl` reader here skips any + frame type it does not recognize instead of erroring out, which is exactly + what lets a ch21 client stay connected to a newer server that does send + them. +- **length**: a 2-byte unsigned payload length in **big-endian** order, `0..65535`. It counts the payload only, never the 6 header bytes. -- **payload** — `length` bytes of UTF-8, shaped per the `type` above. A +- **payload**: `length` bytes of UTF-8, shaped per the `type` above. A `DELIVER` payload's `name` and `text` are separated by a single `0x00` NUL byte; neither field may itself contain a NUL. @@ -79,8 +80,8 @@ the reserved string `server`: So a client that reads a `DELIVER` payload `server\0alice joined` prints `server: alice joined`, and one that reads `alice\0hello bob` prints -`alice: hello bob`. That is the whole protocol for this chapter; chapters -22–24 and 27 add `WELCOME`/`PING` and new engines around it, never a new +`alice: hello bob`. That is the whole protocol for this chapter; later +chapters add `WELCOME`/`PING` and new engines around it, never a new header. ### Worked bytes @@ -129,7 +130,7 @@ second JOIN). ## The three implementations -Identical CLI, identical stdout/stderr shapes, identical exit codes — one +Identical CLI, identical stdout/stderr shapes, identical exit codes: one `verify.lua` covers all three. | | connection model | listener shutdown | fd ownership | @@ -138,10 +139,10 @@ Identical CLI, identical stdout/stderr shapes, identical exit codes — one | **Go 1.26** | one goroutine per client + a per-client writer goroutine draining a channel; a `select` over send/`done`/`ctx` never wedges a broadcaster | a `context` cancelled on SIGINT; `ln.Close()` unblocks `Accept` | `net.Conn` / `net.Listener` | | **Rust 2024** | one `std::thread` per client; broadcasts write to `try_clone`d `TcpStream`s under a `Mutex` | `AtomicBool` set by the SIGINT handler, polled by a non-blocking `accept` loop | listener built via `nix` and taken as an `OwnedFd`; `TcpStream` owns each connection | -No async runtime here on purpose — this chapter is about blocking sockets and -threads. Chapter 27 is where `chatterd` meets tokio. +This chapter covers blocking sockets and threads; an async runtime comes +later, when `chatterd` meets tokio. -## Gotchas worth remembering +## Details worth remembering - **Length prefix, not newline.** TCP is a byte stream with no message boundaries; a single `read` may return half a frame or two frames. Every @@ -150,7 +151,7 @@ threads. Chapter 27 is where `chatterd` meets tokio. do. - **Broadcast excludes the sender; join notices do not.** A speaker already sees what they typed, so their own message is not echoed back. A join, - though, goes to every member — which is why both clients in a two-party room + though, goes to every member, which is why both clients in a two-party room each see a join line. - **SIGINT and a blocking accept.** A flag set in the handler is invisible to a thread parked in `accept`. C++ interrupts `poll` with the signal; Go closes @@ -173,7 +174,7 @@ cpp/ go/ rust/ # one chatterd each; binary always ./…/app; demo.sh contract id `python3 scripts/test-all-examples.py --only 21-tcp-sockets` builds all three and runs `verify.lua` per language. The checks are behavioral: two real `chatctl` clients are driven over loopback through scripted stdin fifos, and the -test asserts the **actually delivered lines** — bob receives `alice: hello`, +test asserts the **delivered lines**: bob receives `alice: hello`, alice sees no echo of her own message, both clients see a join notice from the `server` sender, the daemon logs each join, and SIGINT leaves the listener closed with exit `0`. diff --git a/examples/21-tcp-sockets/cpp/src/main.cpp b/examples/21-tcp-sockets/cpp/src/main.cpp index 4ffab88..999b579 100644 --- a/examples/21-tcp-sockets/cpp/src/main.cpp +++ b/examples/21-tcp-sockets/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// chatterd v0 — a thread-per-connection TCP chat room. +// chatterd v0: a thread-per-connection TCP chat room. // // One binary, two subcommands: // @@ -91,7 +91,7 @@ template using Result = std::expected; // --------------------------------------------------------------------------- -// RAII Socket — the sole owner of every descriptor in this program. +// RAII Socket: the sole owner of every descriptor in this program. // --------------------------------------------------------------------------- class Socket { diff --git a/examples/21-tcp-sockets/go/main.go b/examples/21-tcp-sockets/go/main.go index f40b636..d2da479 100644 --- a/examples/21-tcp-sockets/go/main.go +++ b/examples/21-tcp-sockets/go/main.go @@ -1,4 +1,4 @@ -// chatterd v0 — a thread-per-connection TCP chat room. +// chatterd v0: a thread-per-connection TCP chat room. // // One binary, two subcommands: // @@ -270,7 +270,7 @@ func (h *hub) serveClient(ctx context.Context, conn net.Conn, id int) { // --------------------------------------------------------------------------- func cmdServe(host string, port int) error { - addr := fmt.Sprintf("%s:%d", host, port) + addr := net.JoinHostPort(host, strconv.Itoa(port)) ln, err := net.Listen("tcp", addr) // Go sets SO_REUSEADDR on TCP listeners if err != nil { return fmt.Errorf("listen %s: %w", addr, err) @@ -309,7 +309,7 @@ func cmdServe(host string, port int) error { // --------------------------------------------------------------------------- func cmdChatctl(host string, port int, name string) error { - addr := fmt.Sprintf("%s:%d", host, port) + addr := net.JoinHostPort(host, strconv.Itoa(port)) conn, err := net.Dial("tcp", addr) if err != nil { return fmt.Errorf("connect %s: %w", addr, err) diff --git a/examples/21-tcp-sockets/rust/src/main.rs b/examples/21-tcp-sockets/rust/src/main.rs index 749443c..9a4cff7 100644 --- a/examples/21-tcp-sockets/rust/src/main.rs +++ b/examples/21-tcp-sockets/rust/src/main.rs @@ -1,4 +1,4 @@ -// chatterd v0 — a thread-per-connection TCP chat room. +// chatterd v0: a thread-per-connection TCP chat room. // // One binary, two subcommands: // @@ -258,7 +258,7 @@ fn listen_tcp(host: &str, port: u16) -> Result { .context("socket")?; setsockopt(&fd, ReuseAddr, &true).context("SO_REUSEADDR")?; bind(fd.as_raw_fd(), &SockaddrIn::from(addr)).with_context(|| format!("bind {host}:{port}"))?; - listen(&fd, Backlog::new(128).unwrap()).context("listen")?; + listen(&fd, Backlog::new(128).expect("128 is a valid backlog size")).context("listen")?; Ok(TcpListener::from(fd)) } diff --git a/examples/21-tcp-sockets/verify.lua b/examples/21-tcp-sockets/verify.lua index 94a0083..bc96b8f 100644 --- a/examples/21-tcp-sockets/verify.lua +++ b/examples/21-tcp-sockets/verify.lua @@ -5,7 +5,7 @@ -- the failure on stderr. -- 2. A real two-client session over loopback TCP: start `serve` in the -- background, connect two chatctl clients (bob, then alice) through --- scripted stdin fifos, and assert the actually delivered lines — +-- scripted stdin fifos, and assert the actually delivered lines: -- * alice's message reaches bob as "alice: hello" -- * alice never sees her own message echoed back (messages go to -- OTHER clients only) @@ -85,7 +85,7 @@ for _ in $(seq 1 200); do ss -ltn 2>/dev/null | grep -q ":@PORT@ " && break; sle mkfifo @WD@/afifo @WD@/bfifo -# bob joins first, then alice — so bob is present to witness alice's join. +# bob joins first, then alice, so bob is present to witness alice's join. @DEMO@ chatctl --port @PORT@ --name bob < @WD@/bfifo > @WD@/b.out 2>&1 & B=$! exec 4> @WD@/bfifo diff --git a/examples/22-scaling-the-server/README.md b/examples/22-scaling-the-server/README.md index f67c380..2c0df64 100644 --- a/examples/22-scaling-the-server/README.md +++ b/examples/22-scaling-the-server/README.md @@ -1,11 +1,11 @@ -# 22-scaling-the-server — chatterd v1 +# 22-scaling-the-server: chatterd v1 -chatterd is a peer-to-peer chat daemon we grow across chapters 21–24 and 27. -ch21 introduced the length-prefixed frame protocol and a thread-per-connection -server. ch22 is about **how a server holds many connections at once**: it keeps -the threaded engine and adds a single-threaded **epoll** engine — one thread, -nonblocking accept, and a per-connection read/write state machine with an -outbound queue and real backpressure. +chatterd is a peer-to-peer chat daemon that grows across several chapters in +this book. The previous chapter introduced the length-prefixed frame protocol +and a thread-per-connection server. Holding many connections at once is the +focus here: the daemon keeps the threaded engine and adds a single-threaded +**epoll** engine: one thread, nonblocking accept, and a per-connection +read/write state machine with an outbound queue and real backpressure. The daemon and its client are the same binary (`app`): `serve` is the daemon, `send` / `listen` / `flood` are the client (`chatctl`) modes. @@ -31,15 +31,15 @@ chatterd: stopped engine=epoll messages=2 peak_conns=50 Every MSG is delivered to **every** connected client, including the sender, so a client confirms its message was accepted by seeing its own line echoed. -## What the chapter is really about +## Threaded vs. epoll Two ways to service N connections from one process: -- **`--engine threaded`** — one OS thread per connection blocks in `recv()`; a +- **`--engine threaded`**: one OS thread per connection blocks in `recv()`; a shared registry (mutex + per-socket write mutex) fans each message out. Simple to reason about, one kernel thread per client. The accept loop `poll(2)`s the listening socket alongside a `signalfd`, so a signal breaks it out cleanly. -- **`--engine epoll`** — one thread owns an `epoll` set over the listener, a +- **`--engine epoll`**: one thread owns an `epoll` set over the listener, a `signalfd`, and every client socket. Accept is nonblocking (`accept4` in a loop until `EAGAIN`); each connection is a state machine with an **inbound buffer** (frames are reassembled across reads) and an **outbound queue**. @@ -58,7 +58,7 @@ that the threaded engine does the same, and that both shut down cleanly with a Go exposes the same `--engine threaded|epoll` flag, but **both** engines are the same idiomatic code: one goroutine per connection reading, one writing, and -a hub goroutine fanning out over channels. There is no hand-rolled epoll — the +a hub goroutine fanning out over channels. There is no hand-rolled epoll: the Go runtime's **network poller** already multiplexes every socket through `epoll_wait(2)` and parks a *goroutine* (not an OS thread) on each blocked Read/Write. The epoll the other two languages write by hand, Go's netpoller runs @@ -67,8 +67,9 @@ identical across all three. ## Wire format -One frame protocol, introduced in ch21 and reused unchanged by every later -chatterd version and every language. All multi-byte fields are **big-endian**. +One frame protocol, introduced earlier in the series and reused unchanged by +every later chatterd version and every language. All multi-byte fields are +**big-endian**. ``` byte: 0 1 2 3 4 5 6 ............. 6+LEN-1 @@ -92,22 +93,22 @@ Header is always 6 bytes; total frame length is `6 + LENGTH`. | 0x02 | MSG | client → server | message text, UTF-8 | | 0x03 | DELIVER | server → client | `nick + 0x00 (NUL) + text` | | 0x04 | WELCOME | server → client | empty (sent once, on accept) | -| 0x05 | PING | — | reserved (introduced in ch24; unused here) | +| 0x05 | PING | n/a | reserved (introduced later; unused here) | -This is the same frame every chatterd chapter and language speaks: a v0 -(ch21) client can chat through a v1 (this chapter's) or v3/v4 server and back -because JOIN/MSG/DELIVER never change shape — later chapters only add types -(WELCOME here, PING in ch24), never touch the 6-byte header or repurpose an -existing type. +This is the same frame every chatterd version and language speaks: a v0 +client can chat through a v1 (this chapter's) or v3/v4 server and back +because JOIN/MSG/DELIVER never change shape; later versions only add types +(WELCOME here, PING added later), never touch the 6-byte header or +repurpose an existing type. `DELIVER` carries the originating nickname, then a single NUL (`0x00`) byte, -then the message text — split the payload on its first NUL to recover the two +then the message text: split the payload on its first NUL to recover the two fields (a nickname is never allowed to contain a NUL). WELCOME is sent the -instant the server accepts a connection — receiving it is proof the connection +instant the server accepts a connection; receiving it is proof the connection is registered in the server's fan-out set, which is how `flood` knows all N peers are present before it triggers the broadcast. -Example — `alice` sends `hi` (`msg`), and the server delivers it: +Example: `alice` sends `hi` (`msg`), and the server delivers it: ``` client -> server 43 48 01 01 00 05 61 6c 69 63 65 JOIN "alice" @@ -144,7 +145,7 @@ Same CLI, stdout/stderr shapes, exit codes, and wire protocol, so one | | epoll engine | threaded engine | shutdown | |---|---|---|---| | **C++23** | hand-rolled `epoll_create1`/`epoll_ctl`/`epoll_wait`; per-conn `EConn` with `inbuf`/`outbuf`, `EPOLLOUT` toggled on backpressure; RAII `Fd` owns every socket, `std::expected` for setup | `std::jthread` per connection, shared `Registry` (mutex + per-socket write mutex), `std::atomic` counters | `signalfd` in the loop / poll set; `SHUT_RDWR` wakes reader threads, jthreads join | -| **Go 1.26** | netpoller (see note) — goroutine per conn + hub goroutine, channel fan-out with blocking sends for backpressure | *same code* as epoll; the flag only changes the printed label | `signal.Notify`, close listener, `quit`/`done` channels, atomics read after the hub drains | +| **Go 1.26** | netpoller (see note): goroutine per conn + hub goroutine, channel fan-out with blocking sends for backpressure | *same code* as epoll; the flag only changes the printed label | `signal.Notify`, close listener, `quit`/`done` channels, atomics read after the hub drains | | **Rust 2024** | `mio` `Poll`/`Token`/`Interest`; per-conn `EConn` with `inbuf`/`outbuf`, reregister `READABLE\|WRITABLE` on backpressure; `signalfd` registered via `SourceFd`; `OwnedFd`-backed `mio::net` sockets | `std::thread` per connection, `Arc` (mutex map + per-socket `Mutex` write clone), atomics | `nix::poll` over listener + `signalfd`; `TcpStream::shutdown` wakes readers, threads join | Build/run: `./demo.sh [cpp|go|rust] build` and `./demo.sh run `. diff --git a/examples/22-scaling-the-server/cpp/src/main.cpp b/examples/22-scaling-the-server/cpp/src/main.cpp index e6aac8b..e2a4a7d 100644 --- a/examples/22-scaling-the-server/cpp/src/main.cpp +++ b/examples/22-scaling-the-server/cpp/src/main.cpp @@ -1,7 +1,7 @@ -// chatterd v1 — a peer-to-peer chat daemon growing chapter by chapter. +// chatterd v1: a peer-to-peer chat daemon growing chapter by chapter. // // ch21 introduced the length-prefixed frame protocol and a threaded server -// (one thread per connection). ch22 — "scaling the server" — keeps that +// (one thread per connection). ch22 ("scaling the server") keeps that // threaded engine working and adds a single-threaded epoll engine: one thread, // many connections, nonblocking accept plus a per-connection read/write state // machine with an outbound queue and real backpressure (partial writes are @@ -60,7 +60,7 @@ namespace { } // --------------------------------------------------------------------------- -// RAII file descriptor — the only owner of every fd/socket in this program. +// RAII file descriptor: the only owner of every fd/socket in this program. // --------------------------------------------------------------------------- class Fd { @@ -210,7 +210,7 @@ make_listener(std::uint16_t port, bool nonblock) { } // --------------------------------------------------------------------------- -// Threaded engine — one thread per connection, a shared broadcast registry. +// Threaded engine: one thread per connection, a shared broadcast registry. // --------------------------------------------------------------------------- struct Conn { @@ -348,7 +348,7 @@ int serve_threaded(std::uint16_t port) { } // --------------------------------------------------------------------------- -// Epoll engine — one thread, nonblocking, per-connection state machines. +// Epoll engine: one thread, nonblocking, per-connection state machines. // --------------------------------------------------------------------------- struct EConn { @@ -698,7 +698,7 @@ int cmd_flood(std::uint16_t port, int n, const std::string& text) { if (k == 0) { (void)write_all(fd->get(), encode(Type::msg, text)); } - // Exactly one broadcast is expected — conn0's message reaches all. + // Exactly one broadcast is expected: conn0's message reaches all. for (;;) { auto fr = recv_frame(fd->get(), inbuf); if (!fr) { diff --git a/examples/22-scaling-the-server/go/main.go b/examples/22-scaling-the-server/go/main.go index c19b2ac..bb031ad 100644 --- a/examples/22-scaling-the-server/go/main.go +++ b/examples/22-scaling-the-server/go/main.go @@ -1,7 +1,7 @@ -// chatterd v1 — a peer-to-peer chat daemon growing chapter by chapter. +// chatterd v1: a peer-to-peer chat daemon growing chapter by chapter. // // ch21 introduced the length-prefixed frame protocol and a threaded server; -// ch22 — "scaling the server" — is about how a server holds thousands of +// ch22, "scaling the server", is about how a server holds thousands of // connections. The C++ and Rust versions offer two engines: a thread-per- // connection "threaded" engine and a single-thread "epoll" engine with // explicit nonblocking read/write state machines. @@ -9,7 +9,7 @@ // Go exposes the SAME "--engine threaded|epoll" flag, but BOTH engines are the // same code: one goroutine per connection reading, one writing, and a central // hub goroutine that fans messages out over channels. There is no hand-rolled -// epoll here on purpose — the Go runtime's network poller already multiplexes +// epoll here on purpose: the Go runtime's network poller already multiplexes // every socket through epoll_wait(2) under the hood and parks a goroutine // (not an OS thread) on each blocked Read/Write. The "epoll" the other // languages write by hand, Go's netpoller runs for us; the engine flag only @@ -130,7 +130,7 @@ func writeAll(conn net.Conn, data []byte) error { } // --------------------------------------------------------------------------- -// Server — one hub goroutine fans out to one goroutine-pair per connection. +// Server: one hub goroutine fans out to one goroutine-pair per connection. // --------------------------------------------------------------------------- type client struct { @@ -343,7 +343,7 @@ func cmdListen(port int, nick string, count int) error { return nil } -// errQuiet flags a nonzero exit without printing a "chatctl: ..." line — the +// errQuiet flags a nonzero exit without printing a "chatctl: ..." line; the // flood counts are already on stdout for the caller to inspect. var errQuiet = errors.New("flood incomplete") @@ -383,7 +383,7 @@ func cmdFlood(port, n int, text string) error { return } } - // Exactly one broadcast is expected — conn0's message reaches all. + // Exactly one broadcast is expected: conn0's message reaches all. for { typ, _, err := fr.next() if err != nil { diff --git a/examples/22-scaling-the-server/rust/src/main.rs b/examples/22-scaling-the-server/rust/src/main.rs index 0490ad0..a9f41a9 100644 --- a/examples/22-scaling-the-server/rust/src/main.rs +++ b/examples/22-scaling-the-server/rust/src/main.rs @@ -1,7 +1,7 @@ -// chatterd v1 — a peer-to-peer chat daemon growing chapter by chapter. +// chatterd v1: a peer-to-peer chat daemon growing chapter by chapter. // // ch21 introduced the length-prefixed frame protocol and a threaded server -// (one thread per connection). ch22 — "scaling the server" — keeps that +// (one thread per connection). ch22 ("scaling the server") keeps that // threaded engine working and adds a single-threaded epoll engine, built with // mio: one thread, many connections, nonblocking accept plus a per-connection // read/write state machine with an outbound queue and real backpressure @@ -105,7 +105,7 @@ fn take_frame(buf: &mut Vec) -> Option<(u8, Vec)> { } // --------------------------------------------------------------------------- -// Signals — SIGINT/SIGTERM blocked process-wide, delivered as readable data. +// Signals: SIGINT/SIGTERM blocked process-wide, delivered as readable data. // --------------------------------------------------------------------------- fn block_signals() -> Result { @@ -118,7 +118,7 @@ fn block_signals() -> Result { } // --------------------------------------------------------------------------- -// Threaded engine — one std thread per connection, a shared broadcast registry. +// Threaded engine: one std thread per connection, a shared broadcast registry. // --------------------------------------------------------------------------- struct ConnHandle { @@ -251,7 +251,7 @@ fn serve_threaded(port: u16) -> Result<()> { } // --------------------------------------------------------------------------- -// Epoll engine — one thread, mio-driven, per-connection state machines. +// Epoll engine: one thread, mio-driven, per-connection state machines. // --------------------------------------------------------------------------- struct EConn { @@ -590,7 +590,7 @@ fn cmd_flood(port: u16, n: usize, text: &str) -> Result { if k == 0 { let _ = w.write_all(&encode(TYPE_MSG, text.as_bytes())); } - // Exactly one broadcast is expected — conn0's message reaches all. + // Exactly one broadcast is expected: conn0's message reaches all. loop { match fs.next() { Ok(Some((typ, _))) if typ == TYPE_DELIVER => { diff --git a/examples/22-scaling-the-server/verify.lua b/examples/22-scaling-the-server/verify.lua index 1bb0f97..7ce551a 100644 --- a/examples/22-scaling-the-server/verify.lua +++ b/examples/22-scaling-the-server/verify.lua @@ -1,15 +1,15 @@ -- Verify chatterd v1 (22-scaling-the-server) for LSP_LANG. -- -- Asserts observable behavior, identical across cpp/go/rust: --- 1. epoll engine, two-party transcript — a listener receives the exact +-- 1. epoll engine, two-party transcript: a listener receives the exact -- "alice: hello world" line the sender broadcast; server stop summary -- reports messages=1 peak_conns=2; clean exit --- 2. epoll engine, load — `flood 50` opens 50 concurrent connections and one +-- 2. epoll engine, load: `flood 50` opens 50 concurrent connections and one -- broadcast is DELIVERED to all 50 (delivered==connected==50, no dropped -- frames), exit 0; server stop summary reports messages=1 peak_conns=50 --- 3. threaded engine — the same two-party transcript and 50-client flood +-- 3. threaded engine: the same two-party transcript and 50-client flood -- still work and shut down cleanly --- 4. error shapes — no subcommand and bad flags exit 2; a client against a +-- 4. error shapes: no subcommand and bad flags exit 2; a client against a -- dead port exits 1 -- -- Each scenario runs its own fresh daemon so the message and peak counts are @@ -81,7 +81,7 @@ cat @WD@/flood.out ]] -- --------------------------------------------------------------------------- --- 1. epoll engine — two-party transcript +-- 1. epoll engine: two-party transcript -- --------------------------------------------------------------------------- local e1 = scenario("epoll", two_party) @@ -98,7 +98,7 @@ checks.expect_match(e1.out, "chatterd: stopped engine=epoll messages=1 peak_conn lang .. ": epoll stop summary reports 1 message across 2 peak connections") -- --------------------------------------------------------------------------- --- 2. epoll engine — 50-connection flood, broadcast delivered to all +-- 2. epoll engine: 50-connection flood, broadcast delivered to all -- --------------------------------------------------------------------------- local e2 = scenario("epoll", flood) @@ -112,7 +112,7 @@ checks.expect_match(e2.out, "chatterd: stopped engine=epoll messages=1 peak_conn lang .. ": epoll stop summary reports 1 message across 50 peak connections") -- --------------------------------------------------------------------------- --- 3. threaded engine — same transcript and flood still work +-- 3. threaded engine: same transcript and flood still work -- --------------------------------------------------------------------------- local t1 = scenario("threaded", two_party) diff --git a/examples/23-udp-and-peer-discovery/README.md b/examples/23-udp-and-peer-discovery/README.md index f9e35b4..106a9b6 100644 --- a/examples/23-udp-and-peer-discovery/README.md +++ b/examples/23-udp-and-peer-discovery/README.md @@ -1,12 +1,13 @@ -# 23-udp-and-peer-discovery — chatterd v2 +# 23-udp-and-peer-discovery: chatterd v2 -chatterd grows peer discovery. Where chapters 21–22 built the TCP side (the -canonical chatterd chat frame between two known endpoints), `chatterd -discover` adds the missing half: finding peers you were never told about. It -sends periodic **UDP multicast beacons**, listens for everyone else's, and the -first time it hears a new peer it opens a TCP connection to that peer's -advertised port and exchanges one chat frame each way — the same canonical -wire frame every chatterd version speaks, reused here for JOIN/DELIVER. +chatterd grows peer discovery. The TCP side built earlier in the book +established the canonical chatterd chat frame between two known endpoints; +`chatterd discover` adds the missing half: finding peers you were never told +about. It sends periodic **UDP multicast beacons**, listens for everyone +else's, and the first time it hears a new peer it opens a TCP connection to +that peer's advertised port and exchanges one chat frame each way, the same +canonical wire frame every chatterd version speaks, reused here for +JOIN/DELIVER. ``` $ ./demo.sh go run discover --group 239.23.7.1 --port 51823 \ @@ -28,24 +29,24 @@ Each process announces on stderr (`chatterd: announcing as on `--announce-ms` apart, then exits 0 after an 800 ms grace window so in-flight exchanges finish. -## What the chapter is really about +## What the chapter is about UDP multicast is the one-to-many primitive. A beacon sent once to a group address (`239.0.0.0/8`, the administratively-scoped block) reaches every member on the interface without the sender knowing who or how many they are. Discovery is: **join the group, announce yourself, and learn from what you -hear.** The three implementations exercise the same setsockopt dance — +hear.** The three implementations exercise the same setsockopt dance: - **`IP_ADD_MEMBERSHIP`** (`ip_mreq { imr_multiaddr = group, imr_interface = - iface }`) — join the group on a specific interface. Without this the kernel + iface }`): join the group on a specific interface. Without this the kernel drops group traffic before it reaches the socket. -- **`IP_MULTICAST_IF`** — pin *outgoing* multicast to the same interface, so a +- **`IP_MULTICAST_IF`**: pin *outgoing* multicast to the same interface, so a multi-homed host sends the beacon where its peers are listening. -- **`IP_MULTICAST_LOOP`** — deliver our own multicast to other sockets **on +- **`IP_MULTICAST_LOOP`**: deliver our own multicast to other sockets **on this host**. This is what makes the local two-process verify work over the loopback interface; across two machines it is irrelevant (the packet is on the wire either way). -- **`IP_MULTICAST_TTL = 1`** — keep beacons on the local link; do not let a +- **`IP_MULTICAST_TTL = 1`**: keep beacons on the local link; do not let a router forward them. The receive side dedupes by peer name, so "discovered peer …" prints exactly @@ -58,11 +59,11 @@ loops group traffic **when the multicast interface is set to `127.0.0.1`**. That is why the local verify passes `--iface 127.0.0.1`: both processes join and send on the loopback address and the kernel loops each beacon to the other. On the VMs the interface is the guest's `enp1s0` address and the -beacons cross the `virbr0` bridge for real. +beacons cross the `virbr0` bridge between the two guests. ## Wire formats (byte-exact, identical across all three languages) -### 1. Discovery beacon — UDP datagram to `:` +### 1. Discovery beacon: UDP datagram to `:` An ASCII text datagram, no framing (UDP preserves message boundaries): @@ -70,16 +71,16 @@ An ASCII text datagram, no framing (UDP preserves message boundaries): CHATTERD1 ``` -- `CHATTERD1` — literal magic / version tag (9 bytes). +- `CHATTERD1`: literal magic / version tag (9 bytes). - single ASCII spaces separate the four fields; `` contains no space. -- `` — decimal, the TCP port this peer accepts chat frames on. -- `` — dotted-quad the peer wants to be dialed at (its `--iface`). +- ``: decimal, the TCP port this peer accepts chat frames on. +- ``: dotted-quad the peer wants to be dialed at (its `--iface`). A receiver splits on whitespace, requires exactly 4 fields with `tok[0] == "CHATTERD1"`, and ignores a beacon whose `` equals its own. Captured on the peer guest's `enp1s0` during the two-host run (`tcpdump -X`), -a beacon from systems-target — 35 payload bytes: +a beacon from systems-target, 35 payload bytes: ``` IP 192.168.124.7.40516 > 239.23.7.1.51824: UDP, length 35 @@ -89,12 +90,12 @@ IP 192.168.124.7.40516 > 239.23.7.1.51824: UDP, length 35 = "CHATTERD1 target 9241 192.168.124.7" ``` -### 2. Chat frame — TCP, the canonical chatterd frame (reused, not this chapter's invention) +### 2. Chat frame: TCP, the canonical chatterd frame (reused, not this chapter's invention) -This is **not** a ch23-specific format — it is the one wire frame every -chatterd version (ch21's thread-per-connection server, ch22's epoll engine, -ch24's heartbeat client, ch27's async transport) reads and writes. ch23 just -dials it peer-to-peer instead of client-to-server. The UDP beacon above is a +This wire frame is shared across every chatterd version (the +thread-per-connection server, the epoll engine, the heartbeat client, and the +async transport covered elsewhere in the book); ch23 dials it peer-to-peer +instead of client-to-server. The UDP beacon above is a completely separate object; this frame only ever travels over the TCP connection opened *after* discovery. @@ -117,8 +118,8 @@ header. ch23's post-discovery exchange uses two of the five types, one frame each way: 1. the discoverer dials `:` and sends a **JOIN** frame - whose payload is its own name (``) — the same "announce on connect" - JOIN chapters 21/22/24/27 use; + whose payload is its own name (``), the same "announce on connect" + JOIN pattern used by every other chatterd version; 2. the listener reads that frame, replies with a **DELIVER** frame whose payload is `\0hello from ` (name, then a NUL byte, then the greeting text), and closes; @@ -126,10 +127,10 @@ each way: byte, and prints `peer says: `. Both peers discover each other independently, so each dials the other once and -each prints one `peer … says:` line — the message crossed the network. Because -the header is shared, a ch23 peer's JOIN/DELIVER exchange is wire-compatible -with a ch21/ch22/ch24/ch27 chatterd speaking the same frame; the newer-only -types (WELCOME, PING) simply never appear in this chapter's exchange. +each prints one `peer … says:` line, confirming the message crossed the +network. Because the header is shared, a ch23 peer's JOIN/DELIVER exchange is +wire-compatible with any other chatterd version speaking the same frame; the +newer-only types (WELCOME, PING) never appear in this chapter's exchange. ## The three implementations @@ -159,7 +160,7 @@ Flags: `--group ` and `--port ` and `--name ` are required; ## Verification `verify.lua` (mode `vm-peer`) runs the **local** two-process assertion on -loopback multicast — it clears `TARGET` so `demo.sh run` executes on the host, +loopback multicast: it clears `TARGET` so `demo.sh run` executes on the host, starts `alice` and `bob` with distinct names and TCP ports, and asserts each discovers the other exactly once, exchanges the chat frame, and exits 0, plus the usage/exit-code shapes. Distinct ports per language keep back-to-back diff --git a/examples/23-udp-and-peer-discovery/cpp/src/main.cpp b/examples/23-udp-and-peer-discovery/cpp/src/main.cpp index 04731c3..cb5af38 100644 --- a/examples/23-udp-and-peer-discovery/cpp/src/main.cpp +++ b/examples/23-udp-and-peer-discovery/cpp/src/main.cpp @@ -1,9 +1,9 @@ -// chatterd v2 (C++23) — UDP multicast peer discovery + a TCP chat exchange. +// chatterd v2 (C++23): UDP multicast peer discovery + a TCP chat exchange. // Grows the same chatterd program introduced in chapter 21: the TCP frame is // the canonical chatterd chat frame (2-byte magic "CH" + 1-byte version + // 1-byte type + 2-byte big-endian length + UTF-8 payload) shared by every // chatterd version (ch21-ch27). The UDP discovery beacon below is a -// completely separate wire object — a plain ASCII datagram, no relation to +// completely separate wire object: a plain ASCII datagram, no relation to // the chat frame's magic/version/type/length header. // // chatterd discover --group 239.7.7.7 --port 51888 --name alice \ @@ -43,7 +43,7 @@ constexpr std::string_view kBeaconMagic = "CHATTERD1"; // UDP beacon tag only // 0..65535), payload (UTF-8, type-specific). This header is byte-identical // across every chatterd version; ch23's post-discovery greeting uses only // JOIN (announce the dialer's name) and DELIVER (name + NUL + text, the -// listener's reply) — the same types chapters 21/22/24/27 share. +// listener's reply): the same types chapters 21/22/24/27 share. constexpr std::uint8_t kFrameMagic0 = 0x43; // 'C' constexpr std::uint8_t kFrameMagic1 = 0x48; // 'H' constexpr std::uint8_t kFrameVersion = 0x01; diff --git a/examples/23-udp-and-peer-discovery/go/main.go b/examples/23-udp-and-peer-discovery/go/main.go index ae27e27..e763a8f 100644 --- a/examples/23-udp-and-peer-discovery/go/main.go +++ b/examples/23-udp-and-peer-discovery/go/main.go @@ -1,9 +1,9 @@ -// chatterd v2 (Go) — UDP multicast peer discovery + a TCP chat exchange. +// chatterd v2 (Go): UDP multicast peer discovery + a TCP chat exchange. // Grows the same chatterd program introduced in chapter 21: the TCP frame is // the canonical chatterd chat frame (2-byte magic "CH" + 1-byte version + // 1-byte type + 2-byte big-endian length + UTF-8 payload) shared by every // chatterd version (ch21-ch27). The UDP discovery beacon below is a -// completely separate wire object — a plain ASCII datagram, no relation to +// completely separate wire object: a plain ASCII datagram, no relation to // the chat frame's magic/version/type/length header. // // chatterd discover --group 239.7.7.7 --port 51888 --name alice \ @@ -36,7 +36,7 @@ const beaconMagic = "CHATTERD1" // UDP beacon tag only // 0..65535), payload (UTF-8, type-specific). This header is byte-identical // across every chatterd version; ch23's post-discovery greeting uses only // JOIN (announce the dialer's name) and DELIVER (name + NUL + text, the -// listener's reply) — the same types chapters 21/22/24/27 share. +// listener's reply), the same types chapters 21/22/24/27 share. const ( frameMagic0 = 0x43 // 'C' frameMagic1 = 0x48 // 'H' diff --git a/examples/23-udp-and-peer-discovery/rust/src/main.rs b/examples/23-udp-and-peer-discovery/rust/src/main.rs index 239bade..2533eeb 100644 --- a/examples/23-udp-and-peer-discovery/rust/src/main.rs +++ b/examples/23-udp-and-peer-discovery/rust/src/main.rs @@ -1,9 +1,9 @@ -// chatterd v2 (Rust 2024) — UDP multicast peer discovery + a TCP chat +// chatterd v2 (Rust 2024): UDP multicast peer discovery + a TCP chat // exchange. Grows the same chatterd program introduced in chapter 21: the TCP // frame is the canonical chatterd chat frame (2-byte magic "CH" + 1-byte // version + 1-byte type + 2-byte big-endian length + UTF-8 payload) shared by // every chatterd version (ch21-ch27). The UDP discovery beacon below is a -// completely separate wire object — a plain ASCII datagram, no relation to +// completely separate wire object: a plain ASCII datagram, no relation to // the chat frame's magic/version/type/length header. // // chatterd discover --group 239.7.7.7 --port 51888 --name alice \ @@ -32,7 +32,7 @@ const BEACON_MAGIC: &str = "CHATTERD1"; // UDP beacon tag only // 0..65535), payload (UTF-8, type-specific). This header is byte-identical // across every chatterd version; ch23's post-discovery greeting uses only // JOIN (announce the dialer's name) and DELIVER (name + NUL + text, the -// listener's reply) — the same types chapters 21/22/24/27 share. +// listener's reply), the same types chapters 21/22/24/27 share. const FRAME_MAGIC: [u8; 2] = [0x43, 0x48]; // "CH" const FRAME_VERSION: u8 = 0x01; const FRAME_JOIN: u8 = 1; diff --git a/examples/23-udp-and-peer-discovery/verify.lua b/examples/23-udp-and-peer-discovery/verify.lua index a2ceefc..593ee59 100644 --- a/examples/23-udp-and-peer-discovery/verify.lua +++ b/examples/23-udp-and-peer-discovery/verify.lua @@ -4,7 +4,7 @@ -- mode:vm-peer, but the verify itself is the LOCAL two-process assertion: two -- chatterd instances with distinct names run on loopback multicast, discover -- each other, and exchange one canonical chatterd TCP chat frame (JOIN then --- DELIVER — the same frame ch21/22/24/27 share). (The real two-host demo +-- DELIVER, the same frame ch21/22/24/27 share). (The real two-host demo -- across systems-target/systems-peer is driven separately by the chapter's -- demo; see README.) demo.sh run is forced local here by clearing TARGET, so -- the assertion never touches a VM. diff --git a/examples/24-time-timers-deadlines/README.md b/examples/24-time-timers-deadlines/README.md index 221566f..a4d92e3 100644 --- a/examples/24-time-timers-deadlines/README.md +++ b/examples/24-time-timers-deadlines/README.md @@ -1,7 +1,7 @@ -# 24-time-timers-deadlines — chatterd v3, heartbeats + deadlines +# 24-time-timers-deadlines: chatterd v3, heartbeats + deadlines -Chapter 24 grows `chatterd` — the peer-to-peer chat daemon introduced in -chapter 21 — into a version that survives a peer going away. The lesson is +This chapter grows `chatterd` (the peer-to-peer chat daemon introduced in +the TCP sockets chapter) into a version that survives a peer going away. The lesson is *time*: how to keep a connection alive with a periodic heartbeat, how to declare a peer dead by measuring **silence on a monotonic clock**, and how to come back with a **jittered exponential backoff** instead of a hot reconnect @@ -32,8 +32,8 @@ opts: --heartbeat-ms H (default 1000) PING interval `res` is `clock_getres(CLOCK_MONOTONIC)`; the two `actual` fields are the measured wall-time of a 1 ms `nanosleep(2)` and a 1 ms one-shot `timerfd`, each timed on a **monotonic/steady clock** (never `CLOCK_REALTIME`). On a - typical host both land near 1050–1100 µs — the overshoot is the scheduler's - wake-up latency, which is exactly the point. + typical host both land near 1050–1100 µs: the overshoot is the scheduler's + wake-up latency, which is the point. - **`listen`** binds, prints `chatterd: listening on :` (a `:0` port is reported as the actual ephemeral port), and accepts a peer. @@ -49,14 +49,14 @@ opts: --heartbeat-ms H (default 1000) PING interval received frame (JOIN/MSG/PING) is "traffic" and resets a `--timeout-ms` deadline. When the deadline fires with no traffic the peer is dropped: `chatterd: peer timed out`. A closed socket (peer - killed) is handled the same way — reads stop, but it is the *deadline*, on + killed) is handled the same way: reads stop, but it is the *deadline*, on the monotonic clock, that declares death. - After a drop the **connector** reconnects with equal-jitter exponential - backoff — `delay = B/2 + rand(0..B/2)`, with `B` doubling each failed + backoff (`delay = B/2 + rand(0..B/2)`, with `B` doubling each failed attempt up to `--max-backoff-ms` and reset to the base on a fresh - connection — printing `chatterd: reconnecting to in ms` before - each wait. The **listener** simply returns to `accept()` to welcome a + connection), printing `chatterd: reconnecting to in ms` before + each wait. The **listener** returns to `accept()` to welcome a reconnecting peer. - **SIGINT / SIGTERM** at any point (linked, mid-backoff, or waiting to @@ -66,11 +66,13 @@ opts: --heartbeat-ms H (default 1000) PING interval ## Wire format (identical across languages and every chatterd version) -This is **the** chatterd chat frame: introduced in chapter 21, and extended -in later chapters only by adding new `TYPE` values — the header itself never -changes, so every chatterd version (thread-per-connection in ch21, epoll in -ch22, the UDP-discovered peers of ch23, this chapter's heartbeats, the async -runtimes of ch27) reads and writes exactly these bytes. All multi-byte +This is **the** chatterd chat frame: introduced earlier in the series, and +extended in later chapters only by adding new `TYPE` values. The header +itself never changes, so every chatterd version (thread-per-connection in +the TCP sockets chapter, epoll in the server-scaling chapter, the +UDP-discovered peers of the peer-discovery chapter, this chapter's +heartbeats, the async runtimes of a later chapter) reads and writes exactly +these bytes. All multi-byte integers are big-endian. The header is a fixed 6 bytes: ``` @@ -96,13 +98,14 @@ Framing rules, byte-exact and version-stable: - `MAGIC`/`VERSION` mismatch is a protocol error and drops the connection. - `PING` is the frame type added in **v3** (this chapter) for heartbeating. `DELIVER` and `WELCOME` belong to the server-broadcast chatterd versions - (ch22, ch27) — this chapter's peer-to-peer `listen`/`connect` model never - emits them, but since the header is shared, a peer that ever receives one - (or any `0x06..0xFF` value) treats it as generic liveness traffic and skips - its `LENGTH` bytes rather than erroring. `JOIN` (the peer-name handshake) - and `MSG` (chat text) are unchanged since ch21, so a v3 peer here still + (the server-scaling and async-runtimes chapters); this chapter's + peer-to-peer `listen`/`connect` model never emits them, but since the + header is shared, a peer that ever receives one (or any `0x06..0xFF` + value) treats it as generic liveness traffic and skips its `LENGTH` bytes + rather than erroring. `JOIN` (the peer-name handshake) and `MSG` (chat + text) are unchanged since the earliest version, so a v3 peer here still links with older or newer chatterd peers for everything except - heartbeating — dropping a silent peer is the only thing PING adds. + heartbeating: dropping a silent peer is the only thing PING adds. Example: a JOIN announcing the name `A` is the 7 bytes `43 48 01 01 00 01 41`; a PING is the 6 bytes `43 48 01 05 00 00`. @@ -112,24 +115,24 @@ Example: a JOIN announcing the name `A` is the 7 bytes The protocol and the observable lines are identical; how each language builds the heartbeat/deadline loop is the lesson. -- **C++ — `timerfd_create(CLOCK_MONOTONIC)` + `poll(2)`.** A periodic timerfd +- **C++: `timerfd_create(CLOCK_MONOTONIC)` + `poll(2)`.** A periodic timerfd is the heartbeat; a `std::chrono::steady_clock` deadline is peer liveness (the poll timeout is the time left until it). The peer socket, the heartbeat timerfd, and a `signalfd(2)` shutdown doorbell share one `poll` loop. Every fd is owned by a RAII `UniqueFd`; sends use `MSG_NOSIGNAL`, and errors flow through `std::expected`. -- **Go — `time.Ticker` + `time.Timer` + `context`.** A `Ticker` fires the +- **Go: `time.Ticker` + `time.Timer` + `context`.** A `Ticker` fires the heartbeat, a `Timer` is the deadline (`Reset` on every frame), and the context from `signal.NotifyContext` is shutdown. A per-connection reader goroutine turns the socket into a channel of frames, so the session is one `select` over ticker / deadline / frames / `ctx.Done()`. `time.Now` carries a monotonic reading, so `time.Since` never sees a wall-clock step. -- **Rust — `nix` timerfd + `std::time::Instant` + `poll`.** A periodic +- **Rust: `nix` timerfd + `std::time::Instant` + `poll`.** A periodic `nix::sys::timerfd::TimerFd` is the heartbeat, an `Instant` deadline is peer - liveness, and a `nix` `SignalFd` is shutdown — all three in one + liveness, and a `nix` `SignalFd` is shutdown: all three in one `nix::poll::poll` loop over `std` `TcpStream`/`TcpListener` fds (`OwnedFd` - ownership throughout). Errors flow through `Result`/`?`. No tokio here — - async is chapter 27. + ownership throughout). Errors flow through `Result`/`?`. No tokio here; + async is covered in a later chapter. ## Layout @@ -144,7 +147,7 @@ the heartbeat/deadline loop is the lesson. Each language directory has a `demo.sh` with the identical interface used book-wide: `./demo.sh build`, `./demo.sh run [args]`, and a bare `./demo.sh` -that builds then runs. With `TARGET` set, `run` deploys to that lab VM — but +that builds then runs. With `TARGET` set, `run` deploys to that lab VM, but this chapter runs purely on the host (loopback), so `TARGET` is unused here. Try it by hand: diff --git a/examples/24-time-timers-deadlines/cpp/src/main.cpp b/examples/24-time-timers-deadlines/cpp/src/main.cpp index 209a5a0..7ab0560 100644 --- a/examples/24-time-timers-deadlines/cpp/src/main.cpp +++ b/examples/24-time-timers-deadlines/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// chatterd v3 — heartbeats + deadlines (chapter 24: time, timers, deadlines). +// chatterd v3: heartbeats + deadlines (chapter 24: time, timers, deadlines). // // chatterd clockprobe // chatterd listen --name N --addr HOST:PORT [opts] @@ -11,7 +11,7 @@ // clock: a peer is declared dead by *silence* measured on a monotonic clock, // never by a wall-clock timestamp that a settimeofday(2) could rewind. // -// Wire format — this is THE canonical chatterd chat frame, unchanged since it +// Wire format: this is THE canonical chatterd chat frame, unchanged since it // was introduced in chapter 21 and extended only by adding TYPE values in // later chapters; the header never changes. All integers big-endian: // @@ -23,7 +23,7 @@ // byte 6.. PAYLOAD N bytes: JOIN/MSG/WELCOME/PING are UTF-8 text (PING // empty); DELIVER is name + 0x00 (NUL) + text. // -// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING — DELIVER and +// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING: DELIVER and // WELCOME belong to the server-broadcast versions (ch22/ch27) and, since the // header is shared, are simply treated as liveness traffic here if ever seen. // PING is the v3 addition over v0/v1. Any received frame is "traffic" and @@ -60,7 +60,7 @@ namespace { using namespace std::chrono; // --------------------------------------------------------------------------- -// RAII around a file descriptor — never a naked close(2). +// RAII around a file descriptor: never a naked close(2). // --------------------------------------------------------------------------- class UniqueFd { public: diff --git a/examples/24-time-timers-deadlines/go/main.go b/examples/24-time-timers-deadlines/go/main.go index f0cbb98..9b57897 100644 --- a/examples/24-time-timers-deadlines/go/main.go +++ b/examples/24-time-timers-deadlines/go/main.go @@ -1,4 +1,4 @@ -// chatterd v3 — heartbeats + deadlines (chapter 24: time, timers, deadlines). +// chatterd v3: heartbeats + deadlines (chapter 24: time, timers, deadlines). // // chatterd clockprobe // chatterd listen --name N --addr HOST:PORT [opts] @@ -12,7 +12,7 @@ // ctx.Done(). time.Now carries a monotonic reading, so "silence" is measured // on the monotonic clock, never the wall clock. // -// Wire format — this is THE canonical chatterd chat frame, unchanged since it +// Wire format: this is THE canonical chatterd chat frame, unchanged since it // was introduced in chapter 21 and extended only by adding TYPE values in // later chapters; the header never changes. All integers big-endian: // @@ -24,7 +24,7 @@ // byte 6.. PAYLOAD N bytes: JOIN/MSG/WELCOME/PING are UTF-8 text (PING // empty); DELIVER is name + 0x00 (NUL) + text. // -// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING — DELIVER and +// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING; DELIVER and // WELCOME belong to the server-broadcast versions (ch22/ch27) and, since the // header is shared, are simply treated as liveness traffic here if ever seen. // PING is the v3 addition over v0/v1. Any received frame is "traffic" and diff --git a/examples/24-time-timers-deadlines/rust/src/main.rs b/examples/24-time-timers-deadlines/rust/src/main.rs index 3893c12..b130923 100644 --- a/examples/24-time-timers-deadlines/rust/src/main.rs +++ b/examples/24-time-timers-deadlines/rust/src/main.rs @@ -1,4 +1,4 @@ -// chatterd v3 — heartbeats + deadlines (chapter 24: time, timers, deadlines). +// chatterd v3: heartbeats + deadlines (chapter 24: time, timers, deadlines). // // chatterd clockprobe // chatterd listen --name N --addr HOST:PORT [opts] @@ -9,9 +9,9 @@ // liveness, and a signalfd doorbell drives clean shutdown. All three are fed // to one poll(2) loop alongside the peer socket. Instant is a monotonic // clock, so a peer is declared dead by silence, never by a wall-clock reading -// that a clock step could rewind. No tokio here — that is chapter 27. +// that a clock step could rewind. No tokio here; that is chapter 27. // -// Wire format — this is THE canonical chatterd chat frame, unchanged since it +// Wire format: this is THE canonical chatterd chat frame, unchanged since it // was introduced in chapter 21 and extended only by adding TYPE values in // later chapters; the header never changes. All integers big-endian: // @@ -23,7 +23,7 @@ // byte 6.. PAYLOAD N bytes: JOIN/MSG/WELCOME/PING are UTF-8 text (PING // empty); DELIVER is name + 0x00 (NUL) + text. // -// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING — DELIVER and +// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING; DELIVER and // WELCOME belong to the server-broadcast versions (ch22/ch27) and, since the // header is shared, are simply treated as liveness traffic here if ever seen. // PING is the v3 addition over v0/v1. Any received frame is "traffic" and @@ -242,7 +242,7 @@ fn run_session(o: &Opts, mut stream: TcpStream, sfd: &SignalFd, mut peer: String Err(e) => panic!("poll: {e}"), } let ev = |idx: usize, flags: PollFlags| { - pfds[idx].revents().map(|r| r.intersects(flags)).unwrap_or(false) + pfds[idx].revents().is_some_and(|r| r.intersects(flags)) }; let sk = if sock_dead { false @@ -364,7 +364,7 @@ fn run_listen(o: &Opts, sfd: &SignalFd) -> i32 { Err(Errno::EINTR) => continue, Err(e) => panic!("poll: {e}"), } - let ev = |idx: usize| pfds[idx].revents().map(|r| r.intersects(PollFlags::POLLIN)).unwrap_or(false); + let ev = |idx: usize| pfds[idx].revents().is_some_and(|r| r.intersects(PollFlags::POLLIN)); (ev(0), ev(1)) }; if ready.0 { diff --git a/examples/25-shared-state-and-the-futex/README.md b/examples/25-shared-state-and-the-futex/README.md index 5eb607d..e644cb6 100644 --- a/examples/25-shared-state-and-the-futex/README.md +++ b/examples/25-shared-state-and-the-futex/README.md @@ -1,7 +1,7 @@ # 25-shared-state-and-the-futex Chapter 25's example: `workq`, a bounded **MPMC job queue + worker pool** -implemented three times — C++23, Go, and Rust — with each language's native +implemented three times, C++23, Go, and Rust, with each language's native shared-state toolkit, but one observable contract so a single `verify.lua` covers all three. The queue's blocking is exactly the mutex/condvar dance the kernel implements with **futexes**; `strace`-ing any of these binaries shows @@ -24,7 +24,7 @@ workq: produced= consumed= checksum= ms= - `produced` / `consumed` are exact counts (both equal `N` on the correct path). - `checksum` is a 16-digit zero-padded lowercase hex `u64`. -- `ms` is wall-clock milliseconds (timing — the only non-deterministic field). +- `ms` is wall-clock milliseconds (timing, the only non-deterministic field). Defaults: `--seed 0x0123456789ABCDEF`, `--cap 256`. `--producers`, `--consumers`, and `--items` are required. @@ -37,7 +37,7 @@ a flag with no value → `workq: --producers needs a value`; a non-integer value ## The value contract (identical across languages and runs) Each item index `i ∈ [0, N)` has a payload that is a **pure function** of -`(seed, i)` — a splitmix64 finalizer over `seed + (i+1)·GOLDEN`, all in wrapping +`(seed, i)`: a splitmix64 finalizer over `seed + (i+1)·GOLDEN`, all in wrapping `u64` arithmetic: ``` @@ -55,7 +55,7 @@ Producer `p` owns the indices `{ i : i mod P == p }`, so the **set** of items is `{0..N-1}` for any `P`. The checksum folds payloads with **XOR**, which is commutative and associative, so the result is independent of how items are split among producers or the order consumers drain them. For a fixed -`(seed, N)` the checksum is therefore one fixed constant — byte-identical in +`(seed, N)` the checksum is therefore one fixed constant, byte-identical in C++, Go, and Rust: | seed | N | checksum | @@ -82,16 +82,16 @@ workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms=62 |---|---|---| | **C++23** | `std::mutex` + two `std::condition_variable`s (`not_full` / `not_empty`) guarding a `std::deque`; `std::jthread` producers/consumers, `std::atomic` produced counter | per-consumer local fold, XOR-combined after join | | **Go** | a buffered `chan uint64` (capacity `K`) as the blocking queue; a closer goroutine `sync.WaitGroup.Wait()`s the producers then `close()`s the channel, ending every consumer's `range` | per-consumer local fold, XOR-combined after the consumer `WaitGroup` | -| **Rust** | `Mutex>` + two `Condvar`s inside `std::thread::scope`; no `Arc` — the scope lets the threads borrow the queue directly | per-consumer `(count, sum)` returned from `ScopedJoinHandle::join`, XOR-combined | +| **Rust** | `Mutex>` + two `Condvar`s inside `std::thread::scope`; no `Arc`, the scope lets the threads borrow the queue directly | per-consumer `(count, sum)` returned from `ScopedJoinHandle::join`, XOR-combined | -None of these needs an atomic on the hot path except the produced counter — the +None of these needs an atomic on the hot path except the produced counter: the mutex/condvar (or channel) already establishes the happens-before edges. -## The `--buggy` demo (a real data race — chapter material, not verified) +## The `--buggy` demo (a data race, chapter material, not verified) `--buggy` keeps the same correct, synchronized queue but routes **every** consumer into a single **unsynchronized** shared counter + checksum. That is a -genuine data race; `verify.lua` never runs it, but the chapter does: +data race; `verify.lua` never runs it, but the chapter does: - **Go** builds with `-race` (see `go/demo.sh`), so `--buggy` prints `WARNING: DATA RACE` and ends with `Found N data race(s)` and a nonzero exit. @@ -99,7 +99,7 @@ genuine data race; `verify.lua` never runs it, but the chapter does: - **C++** has a ThreadSanitizer preset: `cmake --preset tsan && cmake --build --preset tsan`, then `./build/tsan/app ... --buggy` reports `WARNING: ThreadSanitizer: data race` at the two shared-write lines. -- **Rust** won't let you write this in safe code at all — the buggy path needs +- **Rust** won't let you write this in safe code at all: the buggy path needs an `unsafe` `*mut` shared across threads. Run it and the borrow checker's point is made empirically: `consumed` and `checksum` come out different every time (`consumed=97282`, `95773`, …), never the correct `100000` / @@ -120,9 +120,9 @@ genuine data race; `verify.lua` never runs it, but the chapter does: Every language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. @@ -135,18 +135,18 @@ run …`; bare `./demo.sh` (or `all`) builds and runs all three. `verify.lua` runs from this directory with `LSP_LANG` set to one language at a time and asserts, per language (25 checks each): -1. the error contract — no args / unknown flag / missing required flag / bad +1. the error contract: no args / unknown flag / missing required flag / bad integer / value-less flag all print the usage line and exit 2; 2. a fixed run (`P=4 C=4 N=100000`, default seed) reports `produced=100000 consumed=100000` and the shared checksum constant `42b3746c6cee7465`; -3. determinism — three more shapes with different `P`, `C`, and `--cap` over the +3. determinism: three more shapes with different `P`, `C`, and `--cap` over the same `(N, seed)` all yield that same constant with `produced==consumed==N`; 4. seed `42` (`N=50000`) yields its own shared constant `fceb5608a4b73dc0`, and `N=0` drains to `produced=0 consumed=0 checksum=0000000000000000`. -The `--buggy` race is a sanitizer/`-race` chapter demo and is deliberately -outside the passing verify. +The `--buggy` race is a sanitizer/`-race` chapter demo, kept outside the +passing verify. Verified on the Fedora 44 reference host (kernel 7.1, GCC 16.1.1, go1.26.5, rustc 1.97.1): all three languages PASS 25/25, Go under `-race`. diff --git a/examples/25-shared-state-and-the-futex/cpp/src/main.cpp b/examples/25-shared-state-and-the-futex/cpp/src/main.cpp index 8d1ee77..3d9c926 100644 --- a/examples/25-shared-state-and-the-futex/cpp/src/main.cpp +++ b/examples/25-shared-state-and-the-futex/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// workq — a bounded MPMC job queue and worker pool (C++23). +// workq: a bounded MPMC job queue and worker pool (C++23). // // P producers push N items total into a bounded blocking queue; C consumers // pop items and fold each item's payload into a running checksum. The correct @@ -8,8 +8,8 @@ // ThreadSanitizer (build/tsan) has a real data race to report. // // The item set {0..N-1} is fixed for a given (seed, N), each item's payload is -// a pure function of its index, and the checksum folds them with XOR — which is -// commutative — so the correct run's checksum is deterministic regardless of P, +// a pure function of its index, and the checksum folds them with XOR, which is +// commutative, so the correct run's checksum is deterministic regardless of P, // C, or the order consumers happen to drain the queue. #include diff --git a/examples/25-shared-state-and-the-futex/go/main.go b/examples/25-shared-state-and-the-futex/go/main.go index 60396ca..8fd3c01 100644 --- a/examples/25-shared-state-and-the-futex/go/main.go +++ b/examples/25-shared-state-and-the-futex/go/main.go @@ -1,7 +1,7 @@ -// workq — a bounded MPMC job queue and worker pool (Go 1.26). +// workq: a bounded MPMC job queue and worker pool (Go 1.26). // // P producers push N items total into a bounded blocking queue (a buffered -// channel — the CSP-flavored blocking queue), C consumers pop and fold each +// channel, the CSP-flavored blocking queue), C consumers pop and fold each // item's payload into a checksum. A closer goroutine waits on the producers' // WaitGroup and closes the channel, which ends every consumer's range loop. // diff --git a/examples/25-shared-state-and-the-futex/rust/src/main.rs b/examples/25-shared-state-and-the-futex/rust/src/main.rs index 1063b58..7c0b77a 100644 --- a/examples/25-shared-state-and-the-futex/rust/src/main.rs +++ b/examples/25-shared-state-and-the-futex/rust/src/main.rs @@ -1,19 +1,19 @@ -// workq — a bounded MPMC job queue and worker pool (Rust 2024). +// workq: a bounded MPMC job queue and worker pool (Rust 2024). // // P producers push N items total into a bounded blocking queue built from a -// Mutex plus two Condvars — the futex-backed pair this chapter is +// Mutex plus two Condvars: the futex-backed pair this chapter is // about. C consumers pop items and fold each payload into a checksum. // // The correct path gives every consumer its own accumulator and combines them // after the scoped threads join; nothing shared is written without the lock. // --buggy is the deliberate counterexample: safe Rust will NOT let two threads // write one plain counter, so the buggy path reaches for `unsafe` and a raw -// pointer to force the data race — and then prints the corrupted result that +// pointer to force the data race, and then prints the corrupted result that // the borrow checker exists to prevent. (No sanitizer needed to see it fail; // run it a few times and consumed/checksum wander.) // // Payloads are a pure function of index folded with XOR, so the correct -// checksum is deterministic for a given (seed, N) regardless of P and C — and +// checksum is deterministic for a given (seed, N) regardless of P and C, and // byte-identical to the C++ and Go builds. use std::collections::VecDeque; @@ -178,11 +178,9 @@ unsafe impl Send for Racy {} unsafe impl Sync for Racy {} fn produce(queue: &BoundedQueue, produced: &AtomicI64, cfg: &Config, p: i64) { - let mut i = p; - while i < cfg.items { + for i in (p..cfg.items).step_by(cfg.producers as usize) { queue.push(payload(cfg.seed, i as u64)); produced.fetch_add(1, Ordering::Relaxed); - i += cfg.producers; } } diff --git a/examples/25-shared-state-and-the-futex/verify.lua b/examples/25-shared-state-and-the-futex/verify.lua index 06d30df..f6cd966 100644 --- a/examples/25-shared-state-and-the-futex/verify.lua +++ b/examples/25-shared-state-and-the-futex/verify.lua @@ -8,7 +8,7 @@ -- 2. a fixed run (P=4 C=4 N=100000, default seed) prints -- workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms= -- the checksum is the SAME 64-bit constant for all three languages --- 3. determinism: the checksum does not depend on P, C, or the queue cap — +-- 3. determinism: the checksum does not depend on P, C, or the queue cap: -- four different shapes over the same (N, seed) all yield that constant, -- and every run reports produced==consumed==N -- 4. a different seed yields a different-but-shared constant, and N=0 yields diff --git a/examples/26-atomics-and-lock-free/README.md b/examples/26-atomics-and-lock-free/README.md index 0d3ce37..518c245 100644 --- a/examples/26-atomics-and-lock-free/README.md +++ b/examples/26-atomics-and-lock-free/README.md @@ -1,7 +1,7 @@ # 26-atomics-and-lock-free -Chapter 26's example: `spscring`, a single-producer/single-consumer lock-free -ring buffer implemented three times — C++23, Go, and Rust — each with that +This chapter's example: `spscring`, a single-producer/single-consumer lock-free +ring buffer implemented three times, in C++23, Go, and Rust, each with that language's atomics, but one shared observable contract so a single `verify.lua` covers all three. @@ -15,7 +15,7 @@ A **producer** thread pushes `N` `u64` values (`0, 1, ..., N-1`) through a bounded ring of `K` slots; the **consumer** (the main thread) pops each value and sums it. There is no mutex and no lock: the ring is coordinated by two `u64` atomic counters, `head` (advanced by the consumer) and `tail` (advanced -by the producer), using acquire/release ordering — Lamport's classic SPSC +by the producer), using acquire/release ordering: Lamport's classic SPSC queue. When the producer has finished all `N` pushes and the consumer all `N` pops, the program prints exactly one line: @@ -23,15 +23,15 @@ pops, the program prints exactly one line: spscring: items=1000000 sum=499999500000 throughput_mops=37.29 pad=off ``` -- `items=` — echoes the requested count. -- `sum=` — the consumer's running total. Because the values are `0..N-1`, +- `items=`: echoes the requested count. +- `sum=`: the consumer's running total. Because the values are `0..N-1`, a correct run **always** yields `s = N·(N-1)/2`. Any dropped, duplicated, or torn hand-off changes this number, so the sum is a concurrency oracle. -- `throughput_mops=` — millions of pushes per second, `N / wall_seconds / 1e6`, +- `throughput_mops=`: millions of pushes per second, `N / wall_seconds / 1e6`, printed with two decimals. Its magnitude is machine- and run-dependent; the book uses it only to *narrate* the false-sharing effect, and `verify.lua` only asserts that it parses as a positive number. -- `pad=` — echoes the `--pad` mode. +- `pad=`: echoes the `--pad` mode. The three binaries produce the same line shape and the same `items`/`sum`/`pad` fields for the same arguments. @@ -39,16 +39,16 @@ fields for the same arguments. **`--pad on|off`** toggles cache-line padding between `head` and `tail` (default `off`): -- `off` — the two counters sit adjacent in one 64-byte cache line. The +- `off`: the two counters sit adjacent in one 64-byte cache line. The producer's `store(tail)` and the consumer's `store(head)` then invalidate - each other's copy of the shared line on every hand-off — **false sharing** — + each other's copy of the shared line on every hand-off (**false sharing**), even though neither thread touches the other's counter. -- `on` — each counter is `alignas(64)` / `#[repr(align(64))]` / padded onto its +- `on`: each counter is `alignas(64)` / `#[repr(align(64))]` / padded onto its own cache line, so the two cores stop fighting over one line. Both modes compute the same, correct sum; the throughput difference between them is the chapter's whole point, not a hard assertion (on a lightly loaded -box a single 1M-item run is noisy — the book measures the gap with repeats). +box a single 1M-item run is noisy; the book measures the gap with repeats). ## The ring protocol (identical across languages) @@ -81,13 +81,13 @@ push(v): pop() -> v: | | index atomics | memory ordering | padding knob | |---|---|---|---| | **C++23** | `std::atomic` head/tail in a `Control` struct; producer is a `std::jthread`, consumer is `main` | explicit `memory_order_acquire` on the loads, `memory_order_release` on the stores; the plain `std::vector` slots ride the release/acquire edge | `Control` gives head and tail each `alignas(64)`; `Control` packs them adjacent | -| **Go** | `atomic.Uint64` head/tail; producer goroutine, consumer on `main`, joined over a `chan struct{}` | **no per-op order argument** — Go's `sync/atomic` is sequentially consistent, and the memory model's "an atomic write is synchronized-before the read that observes it" *is* the acquire/release edge, just not relaxable | `ctrlPadded` inserts `[56]byte` after each counter; `ctrlPacked` leaves them adjacent | -| **Rust** | `AtomicU64` head/tail shared across a `std::thread::scope`; slots are `Vec` too | explicit `Ordering::{Acquire, Release}` on head/tail, `Ordering::Relaxed` on the slots — so the data race is ruled out by the type system, no `unsafe` | `CtrlPadded` wraps each counter in `#[repr(align(64))]`; `CtrlPacked` leaves them adjacent | +| **Go** | `atomic.Uint64` head/tail; producer goroutine, consumer on `main`, joined over a `chan struct{}` | **no per-op order argument**: Go's `sync/atomic` is sequentially consistent, and the memory model's "an atomic write is synchronized-before the read that observes it" *is* the acquire/release edge, just not relaxable | `ctrlPadded` inserts `[56]byte` after each counter; `ctrlPacked` leaves them adjacent | +| **Rust** | `AtomicU64` head/tail shared across a `std::thread::scope`; slots are `Vec` too | explicit `Ordering::{Acquire, Release}` on head/tail, `Ordering::Relaxed` on the slots, so the data race is ruled out by the type system, no `unsafe` | `CtrlPadded` wraps each counter in `#[repr(align(64))]`; `CtrlPacked` leaves them adjacent | **What Go gives you (and doesn't):** C++ and Rust let you name the exact ordering per operation (`acquire`, `release`, `relaxed`) and could, in principle, relax the slot accesses further. Go's `sync/atomic` exposes only -sequentially-consistent `Load`/`Store`/`Add` — you cannot ask for a weaker +sequentially-consistent `Load`/`Store`/`Add`: you cannot ask for a weaker order, but you also cannot get it wrong: every atomic access already carries the happens-before edge this queue needs. The Go build runs under `-race` (see `go/demo.sh`); the detector understands the atomic edge and stays silent @@ -109,14 +109,14 @@ race-free rather than merely lucky. Every language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (e.g. `./demo.sh run --capacity 1024 --items 1000000 --pad on`) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. -This is a host-only example — pure threads and atomics, no sockets, no VM. +This is a host-only example: pure threads and atomics, no sockets, no VM. ## Verification @@ -124,11 +124,11 @@ This is a host-only example — pure threads and atomics, no sockets, no VM. time and asserts, per language: 1. `--capacity 1024 --items 1000000 --pad off` exits 0 and prints - `items=1000000 sum=499999500000 throughput_mops= pad=off` — the sum + `items=1000000 sum=499999500000 throughput_mops= pad=off`; the sum is the exact triangular number `N·(N-1)/2`, so every one of the million hand-offs delivered its value once; 2. the same with `--pad on` yields the identical sum with `pad=on`; -3. `--capacity 1 --items 1000` (a one-slot ring — maximum head/tail +3. `--capacity 1 --items 1000` (a one-slot ring, maximum head/tail contention) still sums to exactly `499500`; 4. the `throughput_mops` field parses as a positive number; 5. no arguments prints `usage: spscring --capacity K --items N [--pad on|off]` diff --git a/examples/26-atomics-and-lock-free/cpp/src/main.cpp b/examples/26-atomics-and-lock-free/cpp/src/main.cpp index 009ccc8..7ef6bbf 100644 --- a/examples/26-atomics-and-lock-free/cpp/src/main.cpp +++ b/examples/26-atomics-and-lock-free/cpp/src/main.cpp @@ -5,7 +5,7 @@ // A producer thread pushes N u64 values (0, 1, ..., N-1) through a bounded // ring of K slots; the consumer (this thread) pops and sums them. The head // and tail indices are std::atomic monotonic counters synchronised -// with explicit acquire/release ordering (Lamport's SPSC queue) — no mutex, +// with explicit acquire/release ordering (Lamport's SPSC queue), no mutex, // no lock. --pad places head and tail on separate cache lines to remove the // false sharing that otherwise ping-pongs the line between the two cores. // diff --git a/examples/26-atomics-and-lock-free/go/main.go b/examples/26-atomics-and-lock-free/go/main.go index c487bf9..026c5b4 100644 --- a/examples/26-atomics-and-lock-free/go/main.go +++ b/examples/26-atomics-and-lock-free/go/main.go @@ -4,12 +4,12 @@ // // A producer goroutine pushes N u64 values (0..N-1) through a bounded ring of // K slots; the consumer (main goroutine) pops and sums them. head and tail are -// atomic.Uint64 monotonic counters (Lamport's SPSC queue) — no mutex. +// atomic.Uint64 monotonic counters (Lamport's SPSC queue); no mutex. // // What Go's sync/atomic gives you: unlike C++ or Rust there is NO per-operation // memory-order argument. Every Load/Store/Add is sequentially consistent, and // the Go memory model promises that an atomic write is "synchronized before" -// the atomic read that observes it — that happens-before edge is exactly the +// the atomic read that observes it; that happens-before edge is exactly the // acquire/release pairing we need, just without the ability to relax it. So the // plain buf[] writes before tail.Store are visible to the consumer that reads // tail, and the -race detector (see demo.sh) understands that edge and stays diff --git a/examples/26-atomics-and-lock-free/rust/src/main.rs b/examples/26-atomics-and-lock-free/rust/src/main.rs index d345277..32d8a13 100644 --- a/examples/26-atomics-and-lock-free/rust/src/main.rs +++ b/examples/26-atomics-and-lock-free/rust/src/main.rs @@ -7,7 +7,7 @@ //! A producer thread pushes N u64 values (0..N-1) through a bounded ring of K //! slots; the consumer (main thread) pops and sums them. head and tail are //! `AtomicU64` monotonic counters synchronised with explicit -//! `Ordering::{Acquire, Release}` (Lamport's SPSC queue) — no mutex, no lock. +//! `Ordering::{Acquire, Release}` (Lamport's SPSC queue): no mutex, no lock. //! The ring slots are themselves `AtomicU64` accessed `Relaxed`; the head/tail //! acquire/release pair is what publishes each slot to the other thread, so the //! data race is ruled out at the type level rather than by `unsafe`. diff --git a/examples/27-async-runtimes-and-coroutines/README.md b/examples/27-async-runtimes-and-coroutines/README.md index fa53e44..f9dc039 100644 --- a/examples/27-async-runtimes-and-coroutines/README.md +++ b/examples/27-async-runtimes-and-coroutines/README.md @@ -1,16 +1,16 @@ -# 27 — async runtimes and coroutines (chatterd v4) +# 27: async runtimes and coroutines (chatterd v4) `chatterd` is the book's recurring peer-to-peer chat daemon. This chapter is the **async rewrite**: the *same* server, with its connection handling rebuilt on -each language's async model. The observable behaviour is unchanged — a message -from one client is broadcast to every other connected client — so a single +each language's async model. The observable behaviour is unchanged: a message +from one client is broadcast to every other connected client, so a single `verify.lua` covers all three languages. | language | `--engine async` is… | |----------|------------------------------------------------------------------| | **C++** | hand-rolled **C++20 coroutines** over an epoll reactor: each connection is a coroutine that `co_await`s socket readability. The `task`/awaitable machinery lives in `cpp/src/main.cpp`; no external async library is linked. | | **Rust** | **tokio**: an `async fn` per connection on a `tokio::net::TcpListener`, spawned as tasks, fanning out over per-client `mpsc` channels. | -| **Go** | **unchanged from v1** — goroutines already *are* the async model. `--engine async` selects the same goroutine-per-connection hub. That is the lesson: Go has no separate coroutine layer to add. | +| **Go** | **unchanged from v1**: goroutines already *are* the async model. `--engine async` selects the same goroutine-per-connection hub. That is the lesson: Go has no separate coroutine layer to add. | Every version also keeps the prior **`--engine thread`** engine working (one OS thread per client), so you can compare the two side by side and confirm @@ -18,10 +18,10 @@ they are byte-for-byte identical on the wire. ## The wire format (canonical chatterd chat frame) -This is the **same frame every chatterd chapter uses** — introduced in ch21 -and extended only additively (by adding new `type` values) ever since; the -header never changes shape. Every message on a connection, in both -directions, is a single frame: +This is the **same frame every chatterd version uses**: introduced earlier +in the series and extended only additively (by adding new `type` values) +ever since; the header never changes shape. Every message on a connection, +in both directions, is a single frame: ``` byte: 0 1 2 3 4 5 6 6+L-1 @@ -31,28 +31,29 @@ directions, is a single frame: +----+----+----+----+----+----+=========================+ ``` -* `magic` — 2 bytes, always `0x43 0x48` (ASCII `"CH"`). -* `version` — 1 byte, `0x01`. -* `type` — 1 byte: `JOIN=1`, `MSG=2`, `DELIVER=3`, `WELCOME=4`, `PING=5`. -* `length` (`L`) — a 16-bit **big-endian** unsigned integer: the number of +* `magic`: 2 bytes, always `0x43 0x48` (ASCII `"CH"`). +* `version`: 1 byte, `0x01`. +* `type`: 1 byte: `JOIN=1`, `MSG=2`, `DELIVER=3`, `WELCOME=4`, `PING=5`. +* `length` (`L`): a 16-bit **big-endian** unsigned integer: the number of bytes in `payload` (0‥65535). -* `payload` — `L` bytes, UTF-8, type-specific (see below). +* `payload`: `L` bytes, UTF-8, type-specific (see below). This chapter's engines (thread and async, in all three languages) use only -three of the five types — the same three ch21 introduced: +three of the five types: the same three introduced earlier in the series: -* **JOIN** — the **first** frame a client sends after connecting; payload is +* **JOIN**: the **first** frame a client sends after connecting; payload is the client's nick (e.g. `alice`). -* **MSG** — every **later** frame the client sends; payload is message text. -* **DELIVER** — for a MSG with payload `T` from the client whose nick is `N`, +* **MSG**: every **later** frame the client sends; payload is message text. +* **DELIVER**: for a MSG with payload `T` from the client whose nick is `N`, the server sends every **other** connected client a DELIVER frame whose payload is exactly `N + 0x00 (NUL) + T`. The sender never receives its own message. -`WELCOME` and `PING` are other chapters' additions to this same frame (ch22's -welcome/roster message, ch24's heartbeat) — this chapter never sends them, and -because the header is unchanged, a ch21-era client that doesn't know those -types still exchanges JOIN/MSG/DELIVER correctly with this chapter's server. +`WELCOME` and `PING` are other chapters' additions to this same frame (one +chapter's welcome/roster message, another's heartbeat): this chapter never +sends them, and because the header is unchanged, an earlier-version client +that doesn't know those types still exchanges JOIN/MSG/DELIVER correctly +with this chapter's server. ### Byte-exact examples @@ -63,7 +64,7 @@ DELIVER "alice\0hi" 43 48 01 03 00 08 61 6c 69 63 65 00 68 69 ``` That last line is a real capture: with `alice` and `bob` both connected, -`alice` sending `hi` puts these fourteen bytes on `bob`'s socket — the 6-byte +`alice` sending `hi` puts these fourteen bytes on `bob`'s socket: the 6-byte header (`43 48` magic, `01` version, `03` type DELIVER, `00 08` length) then the 8-byte payload `alice\0hi` (nick, NUL, text). The framing and the `N NUL T` payload are identical across C++, Go, and Rust, and across the @@ -116,9 +117,9 @@ background, waits for the port, runs the 20-client `loadtest`, and asserts the observable behaviour for **both** engines: * `chatterd: listening on 127.0.0.1:PORT engine=` announced, -* `loadtest: delivered 20/20` — the broadcast reached all 20 clients, +* `loadtest: delivered 20/20`: the broadcast reached all 20 clients, * the loadtest exits `0`, -* `SIGTERM` produces `chatterd: shutdown` and a `0` exit — a clean shutdown, +* `SIGTERM` produces `chatterd: shutdown` and a `0` exit: a clean shutdown, plus the CLI contract (missing subcommand → exit 2, unknown engine → exit 2). The same assertions pass identically for C++, Go, and Rust. @@ -139,12 +140,12 @@ The same assertions pass identically for C++, Go, and Rust. `cpp/src/main.cpp` implements the minimum coroutine runtime the async engine needs, and nothing more: -* **`Detached`** — a fire-and-forget coroutine (eager start, self-destroying at +* **`Detached`**: a fire-and-forget coroutine (eager start, self-destroying at `final_suspend`). The acceptor and each per-connection handler are `Detached`. -* **`Task`** — a lazy, awaitable coroutine that returns a value and uses +* **`Task`**: a lazy, awaitable coroutine that returns a value and uses symmetric transfer to resume its awaiter on completion. `fill()` (one non-blocking `recv`, suspending on `EAGAIN`) is a `Task`. -* **`Reactor`** — an epoll loop mapping a waited-on fd to the +* **`Reactor`**: an epoll loop mapping a waited-on fd to the `std::coroutine_handle` parked on its readability (`EPOLLONESHOT`, re-armed on each `co_await`). A `co_await ReadAwaitable{reactor, fd}` suspends the current coroutine until `fd` is readable. diff --git a/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp b/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp index 35ee1d2..fa37608 100644 --- a/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp +++ b/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp @@ -1,8 +1,8 @@ -// chatterd v4 (C++23) — a peer-to-peer chat daemon. +// chatterd v4 (C++23): a peer-to-peer chat daemon. // // The connection-handling engine is selectable: // * thread : one std::jthread per client (the ch21 baseline). -// * async : a single-threaded C++20 coroutine reactor over epoll — each +// * async : a single-threaded C++20 coroutine reactor over epoll: each // connection is a coroutine that `co_await`s socket readability. // The task/awaitable machinery is hand-rolled here; no external // async library is used. @@ -21,7 +21,7 @@ // frame is MSG (payload = message text). For a MSG with text T from the // client whose nick is N, the server sends every OTHER client a DELIVER // frame whose payload is N + 0x00 (NUL) + T. This chapter's engines use -// only JOIN/MSG/DELIVER — WELCOME and PING are other versions' additions to +// only JOIN/MSG/DELIVER: WELCOME and PING are other versions' additions to // the same frame and are simply never sent here. #include @@ -85,7 +85,7 @@ void install_signals() { } // -------------------------------------------------------------------------- -// RAII owner for a file descriptor — closes on scope exit, move-only. +// RAII owner for a file descriptor: closes on scope exit, move-only. // -------------------------------------------------------------------------- class Fd { public: @@ -117,7 +117,7 @@ class Fd { }; // -------------------------------------------------------------------------- -// Frame helpers — canonical chatterd chat frame (see file header). +// Frame helpers: canonical chatterd chat frame (see file header). // -------------------------------------------------------------------------- constexpr int kMaxBody = 65535; constexpr std::size_t kHeaderLen = 6; @@ -287,7 +287,7 @@ std::expected connect_to(const std::string& host, std::uint16_t } // ========================================================================== -// thread engine — one jthread per client. +// thread engine: one jthread per client. // ========================================================================== void serve_client_blocking(Server& server, Fd conn) { int fd = conn.get(); @@ -360,7 +360,7 @@ std::expected run_thread(const std::string& host, std::uint16 } // ========================================================================== -// async engine — hand-rolled coroutine reactor over epoll. +// async engine: hand-rolled coroutine reactor over epoll. // ========================================================================== // A fire-and-forget coroutine: starts eagerly, destroys its own frame when it @@ -607,7 +607,7 @@ std::expected run_async(const std::string& host, std::uint16_ } // ========================================================================== -// loadtest client — drives the broadcast assertion. +// loadtest client: drives the broadcast assertion. // ========================================================================== int run_loadtest(const std::string& host, std::uint16_t port, int nclients) { std::vector receivers; diff --git a/examples/27-async-runtimes-and-coroutines/go/main.go b/examples/27-async-runtimes-and-coroutines/go/main.go index 101c7fb..05b86a9 100644 --- a/examples/27-async-runtimes-and-coroutines/go/main.go +++ b/examples/27-async-runtimes-and-coroutines/go/main.go @@ -1,4 +1,4 @@ -// chatterd v4 (Go) — a peer-to-peer chat daemon. +// chatterd v4 (Go): a peer-to-peer chat daemon. // // This is the async chapter, and Go's answer is unchanged from v1: goroutines // already ARE the async model. `serve --engine async` and `--engine thread` @@ -15,7 +15,7 @@ // frame is MSG (payload = message text). For a MSG with text T from the // client whose nick is N, the server sends every OTHER client a DELIVER // frame whose payload is N + 0x00 (NUL) + T. This chapter's engines use -// only JOIN/MSG/DELIVER — WELCOME and PING are other versions' additions to +// only JOIN/MSG/DELIVER; WELCOME and PING are other versions' additions to // the same frame and are simply never sent here. // // Usage: @@ -42,7 +42,7 @@ import ( ) // --------------------------------------------------------------------------- -// Frame helpers — canonical chatterd chat frame (see file header). +// Frame helpers: canonical chatterd chat frame (see file header). // --------------------------------------------------------------------------- const ( @@ -232,7 +232,7 @@ func runServe(engine, host string, port int) int { } // --------------------------------------------------------------------------- -// loadtest client — drives the broadcast assertion. +// loadtest client: drives the broadcast assertion. // --------------------------------------------------------------------------- func runLoadtest(host string, port, nclients int) int { diff --git a/examples/27-async-runtimes-and-coroutines/rust/src/main.rs b/examples/27-async-runtimes-and-coroutines/rust/src/main.rs index 132b8b8..0476166 100644 --- a/examples/27-async-runtimes-and-coroutines/rust/src/main.rs +++ b/examples/27-async-runtimes-and-coroutines/rust/src/main.rs @@ -1,9 +1,9 @@ -// chatterd v4 (Rust) — a peer-to-peer chat daemon. +// chatterd v4 (Rust): a peer-to-peer chat daemon. // // The connection-handling engine is selectable: // * thread : one std thread per client (the ch21 baseline; std::net + a // signal-hook stop flag). -// * async : tokio — an async fn per connection on a tokio::net::TcpListener, +// * async : tokio, an async fn per connection on a tokio::net::TcpListener, // spawned as tasks, broadcasting over per-client mpsc channels. // // Both engines speak the SAME length-prefixed frame protocol and are @@ -18,7 +18,7 @@ // frame is MSG (payload = message text). For a MSG with text T from the // client whose nick is N, the server sends every OTHER client a DELIVER // frame whose payload is N + 0x00 (NUL) + T. This chapter's engines use -// only JOIN/MSG/DELIVER — WELCOME and PING are other versions' additions to +// only JOIN/MSG/DELIVER; WELCOME and PING are other versions' additions to // the same frame and are simply never sent here. // // Usage: @@ -35,7 +35,7 @@ use std::thread; use std::time::Duration; // --------------------------------------------------------------------------- -// Frame helpers — canonical chatterd chat frame (see file header). +// Frame helpers: canonical chatterd chat frame (see file header). // --------------------------------------------------------------------------- const HEADER_LEN: usize = 6; @@ -69,7 +69,7 @@ fn read_frame_conn(s: &mut TcpStream) -> io::Result<(u8, String)> { } // ========================================================================== -// thread engine — one std thread per client. +// thread engine: one std thread per client. // ========================================================================== type ThreadClients = Arc>>>>; @@ -191,7 +191,7 @@ fn run_thread(host: &str, port: u16) -> io::Result<()> { } // ========================================================================== -// async engine — tokio. +// async engine: tokio. // ========================================================================== mod async_engine { @@ -307,7 +307,7 @@ mod async_engine { } // ========================================================================== -// loadtest client — drives the broadcast assertion. +// loadtest client: drives the broadcast assertion. // ========================================================================== fn run_loadtest(host: &str, port: u16, nclients: usize) -> io::Result { diff --git a/examples/28-gdb-and-remote-debugging/README.md b/examples/28-gdb-and-remote-debugging/README.md index 63a3f59..901849e 100644 --- a/examples/28-gdb-and-remote-debugging/README.md +++ b/examples/28-gdb-and-remote-debugging/README.md @@ -1,11 +1,11 @@ -# 28-gdb-and-remote-debugging — bugfarm scenario 1: a crashing pmon +# 28-gdb-and-remote-debugging: bugfarm scenario 1, a crashing pmon -Chapter 28 is the first of the "bugfarm" examples: `app` is deliberately -dual-use. `app run` is a normal, well-behaved program. `app crash` is a -debugging TARGET — it walks the identical three/four-frame call chain but +This chapter is the first of the "bugfarm" examples: `app` is dual-use by +design. `app run` is a normal, well-behaved program. `app crash` is a +debugging TARGET: it walks the identical three/four-frame call chain but feeds it a value that was never filled in, and goes straight into the bug -class this chapter is about: reading through a pointer/reference/`Option` -that a concurrent-looking event handler assumed — wrongly — was already +class central to this chapter: reading through a pointer/reference/`Option` +that a concurrent-looking event handler assumed (wrongly) was already populated. ``` @@ -24,10 +24,10 @@ main -> run_supervisor -> handle_child -> deref `run_supervisor` fakes a supervisor that just reaped child pid 4242. In `run` mode the reaper filled in the child's exit-status slot before `handle_child` looks at it. In `crash` mode, `handle_child` runs as if it -fired *before* the reaper got there — the slot was never populated — which -is exactly the shape of race a real signal-driven supervisor (see chapter -12's `pmon`) can hit if a `SIGCHLD` notification and the code that consumes -it get out of step. +fired *before* the reaper got there (the slot was never populated), which +is exactly the shape of race a real signal-driven supervisor (see the +signal-handling chapter's `pmon`) can hit if a `SIGCHLD` notification and +the code that consumes it get out of step. - `app run` prints: ``` @@ -37,7 +37,7 @@ it get out of step. ``` and exits 0. - `app crash` prints only `pmon: supervisor starting`, then hits the bug. - **What happens next is different in every language — that difference is + **What happens next is different in every language; that difference is the point of this chapter:** | | mechanism | exit | core? | name the frame with | @@ -52,25 +52,25 @@ it get out of step. ### Why C++ gets a core and Go/Rust don't C++ has no runtime standing between your bug and the kernel: a null-pointer -read simply generates a hardware page fault, the kernel delivers `SIGSEGV`, +read generates a hardware page fault, the kernel delivers `SIGSEGV`, and the default disposition for an unhandled `SIGSEGV` is "dump core, then terminate." Go and Rust both interpose a runtime between your code and the raw fault. Go's runtime installs its own `SIGSEGV` handler, recognizes the fault came from ordinary code (not from inside the runtime itself), and -converts it into a normal — if unrecovered — panic; there is no +converts it into a normal (if unrecovered) panic; there is no `sigaction(SA_RESETHAND)` back to the default disposition, so no core. Rust's `.unwrap()` calls `panic!` directly: no signal is even involved, just the ordinary unwind-and-abort (here, unwind-and-`exit(101)`) machinery. `gdb` still attaches and runs both binaries fine, but there is no signal to -catch — asking gdb for a backtrace after the fact gets you nothing useful, +catch: asking gdb for a backtrace after the fact gets you nothing useful, which is why the Go/Rust checks below read the runtime's own trace instead. -One more thing worth noticing once you're inside gdb on the C++ build: the -`ShutdownGuard` RAII type in `cpp/src/main.cpp`, whose destructor prints the -"exiting cleanly" line, never fires on the crash path. RAII buys you -cleanup on every C++-level exit (`return`, an exception, `std::exit` via a -`terminate` handler) — it buys you nothing against a hardware fault, because -a signal doesn't unwind the stack. Frame `#0` in the release backtrace below +Inside gdb on the C++ build, the `ShutdownGuard` RAII type in +`cpp/src/main.cpp`, whose destructor prints the "exiting cleanly" line, +never fires on the crash path. RAII buys you cleanup on every C++-level +exit (`return`, an exception, `std::exit` via a `terminate` handler); it +buys you nothing against a hardware fault, because a signal doesn't unwind +the stack. Frame `#0` in the release backtrace below even lands a level *inside* `std::println`'s own formatting machinery, rather than exactly on `*child.exit_slot`: at `-O2`, forming the reference argument doesn't fault, only reading through it during formatting does. @@ -79,8 +79,8 @@ argument doesn't fault, only reading through it during formatting does. Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (builds first if needed) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (builds first if needed) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -89,7 +89,7 @@ All three builds keep debug info in an otherwise optimized binary: the C++ emits DWARF, and Rust's `Cargo.toml` sets `[profile.release] debug = true`. Every frame function (`deref`, `handle_child`, `run_supervisor`) is marked noinline (`[[gnu::noinline]]` / `//go:noinline` / `#[inline(never)]`) so -optimization can't fold the call chain away — Rust's `handle_child` also +optimization can't fold the call chain away: Rust's `handle_child` also needs a `std::hint::black_box(())` after the call, or the optimizer turns it into a tail call and erases its own frame from the backtrace. @@ -155,17 +155,17 @@ RUST_BACKTRACE=1 ./target/release/app crash `verify.lua` (run per language with `LSP_LANG` set) asserts: - `run` is identical across languages: exit 0, the three status lines. -- bare / unknown subcommand: exit 2, usage block — identical across languages. +- bare / unknown subcommand: exit 2, usage block, identical across languages. - `crash`, per language: - - **cpp** — exit 139 (SIGSEGV); the RAII cleanup line never appears; under + - **cpp**: exit 139 (SIGSEGV); the RAII cleanup line never appears; under `ulimit -c unlimited` in a scratch temp dir, `coredumpctl` finds and extracts a core for the crashed pid; a scripted `gdb --batch -ex run -ex bt` names `deref`, `handle_child`, and `run_supervisor` in the backtrace. - - **go** — exit 2 (not 139, no core); the panic message names the + - **go**: exit 2 (not 139, no core); the panic message names the standard nil-pointer-dereference text; the goroutine trace names `main.deref`, `main.handleChild`, `main.runSupervisor`. - - **rust** — exit 101 (not 139, no core) either way; with + - **rust**: exit 101 (not 139, no core) either way; with `RUST_BACKTRACE=1`, the backtrace names `app::deref`, `app::handle_child`, `app::run_supervisor`; without it, those names are absent (the backtrace is opt-in) but the exit code is unchanged. diff --git a/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp b/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp index 220f470..e0f1221 100644 --- a/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp +++ b/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp @@ -1,8 +1,8 @@ -// app — bugfarm scenario 1: a crashing pmon-like supervisor +// app: bugfarm scenario 1: a crashing pmon-like supervisor // (chapter 28: gdb and remote debugging). // -// app run -- normal operation: reap a child, report its exit, exit 0 -// app crash -- the SAME call path, fed a child record whose exit-status +// app run: normal operation, reap a child, report its exit, exit 0 +// app crash: the SAME call path, fed a child record whose exit-status // slot was never populated (as if handle_child() ran before // the reaper filled it in) -> a null pointer dereference // three frames down: @@ -17,7 +17,7 @@ // // One more thing to notice once you're inside gdb: ShutdownGuard below never // gets to print its message on the crash path. SIGSEGV is not a C++ -// exception -- the kernel delivers a signal, the default disposition +// exception: the kernel delivers a signal, the default disposition // terminates the process immediately, and nothing unwinds the stack. RAII // buys you cleanup on every *C++-level* exit path (return, throw, exit via // exceptions); it buys you nothing against a hardware fault. @@ -48,7 +48,7 @@ struct ChildRecord { [[gnu::noinline]] void deref(const ChildRecord& child) { // BUG: no null check before the read. In "crash" mode exit_slot is // nullptr because handle_child ran before the reaper recorded the exit - // status -- exactly the kind of race a real supervisor can hit. + // status: exactly the kind of race a real supervisor can hit. std::println("pmon: child {} exited status={}", child.pid, *child.exit_slot); } diff --git a/examples/28-gdb-and-remote-debugging/go/main.go b/examples/28-gdb-and-remote-debugging/go/main.go index 6e75820..70358df 100644 --- a/examples/28-gdb-and-remote-debugging/go/main.go +++ b/examples/28-gdb-and-remote-debugging/go/main.go @@ -1,8 +1,8 @@ -// app -- bugfarm scenario 1: a crashing pmon-like supervisor +// app, bugfarm scenario 1: a crashing pmon-like supervisor // (chapter 28: gdb and remote debugging). // -// app run -- normal operation: reap a child, report its exit, exit 0 -// app crash -- the SAME call path, fed a child record whose exit-status +// app run: normal operation: reap a child, report its exit, exit 0 +// app crash: the SAME call path, fed a child record whose exit-status // pointer was never populated (as if handleChild ran before // the reaper filled it in) -> a nil pointer dereference // three frames down: @@ -17,7 +17,7 @@ // panics. Left unrecovered, that panic unwinds to main, prints // "panic: runtime error: invalid memory address or nil pointer dereference" // plus a full goroutine trace naming every frame below, and the process -// exits with status 2 -- not 139, and there is no core file for gdb to load. +// exits with status 2, not 139, and there is no core file for gdb to load. // A goroutine trace (or dlv) is the tool for this crash, not gdb + core. package main @@ -37,7 +37,7 @@ type childRecord struct { //go:noinline func deref(child childRecord) { // BUG: no nil check before the read. In "crash" mode exitSlot is nil - // because handleChild ran before the reaper recorded the exit status -- + // because handleChild ran before the reaper recorded the exit status; // exactly the kind of race a real supervisor can hit. fmt.Printf("pmon: child %d exited status=%d\n", child.pid, *child.exitSlot) } diff --git a/examples/28-gdb-and-remote-debugging/rust/src/main.rs b/examples/28-gdb-and-remote-debugging/rust/src/main.rs index 63adac0..02fec7d 100644 --- a/examples/28-gdb-and-remote-debugging/rust/src/main.rs +++ b/examples/28-gdb-and-remote-debugging/rust/src/main.rs @@ -1,8 +1,8 @@ -//! app -- bugfarm scenario 1: a crashing pmon-like supervisor +//! app, bugfarm scenario 1: a crashing pmon-like supervisor //! (chapter 28: gdb and remote debugging). //! -//! app run -- normal operation: reap a child, report its exit, exit 0 -//! app crash -- the SAME call path, fed a child record whose exit-status +//! app run: normal operation: reap a child, report its exit, exit 0 +//! app crash: the SAME call path, fed a child record whose exit-status //! slot was never populated (as if handle_child ran before //! the reaper filled it in) -> Option::unwrap() on a None //! three frames down: @@ -15,7 +15,7 @@ //! Rust's answer to the C++ null-pointer bug class: there is no raw pointer //! to leave dangling, so the equivalent defect is an `Option` that should //! have been filled in by now and was not. `.unwrap()` on that `None` panics -//! rather than segfaulting -- with `panic = "unwind"` (this crate's default) +//! rather than segfaulting; with `panic = "unwind"` (this crate's default) //! gdb sees the unwinder's machinery, not a clean SIGSEGV, so //! `RUST_BACKTRACE=1` is the idiomatic tool here, printing every frame below //! by name. Each frame function below is `#[inline(never)]` so release @@ -30,7 +30,7 @@ struct ChildRecord { #[inline(never)] fn deref(child: &ChildRecord) { // BUG: unwrap() with no check. In "crash" mode exit_slot is None because - // handle_child ran before the reaper recorded the exit status -- exactly + // handle_child ran before the reaper recorded the exit status: exactly // the kind of race a real supervisor can hit. println!( "pmon: child {} exited status={}", diff --git a/examples/28-gdb-and-remote-debugging/verify.lua b/examples/28-gdb-and-remote-debugging/verify.lua index a75ec78..bb663b2 100644 --- a/examples/28-gdb-and-remote-debugging/verify.lua +++ b/examples/28-gdb-and-remote-debugging/verify.lua @@ -1,7 +1,7 @@ -- Verify bugfarm scenario 1: `app run` is a normal, dual-use program; -- `app crash` is a debugging TARGET whose call chain is identical across -- languages (main -> run_supervisor -> handle_child -> deref) but whose --- observable crash mechanism is NOT -- that difference is the chapter: +-- observable crash mechanism is NOT; that difference is the chapter: -- * C++: a real SIGSEGV, exit 139, a systemd-coredump core recoverable -- with coredumpctl, and a scripted `gdb --batch -ex run -ex bt` backtrace -- naming every frame. @@ -62,7 +62,7 @@ checks.expect_match(r.out, "pmon: child 4242 exited status=0", checks.expect_match(r.out, "pmon: supervisor exiting cleanly", lang .. ": run reaches its normal cleanup line") --- 3. "crash" -- each language's real crash story. +-- 3. "crash": each language's real crash story. if lang == "cpp" then -- 3a. A real SIGSEGV: exit 139, and the RAII ShutdownGuard's destructor -- never runs (a signal is not stack unwinding). @@ -77,7 +77,7 @@ if lang == "cpp" then -- confirm a core was actually dropped. This host's core_pattern pipes -- through systemd-coredump (`cat /proc/sys/kernel/core_pattern` -- starts with '|'), so "drops a core file" means coredumpctl can find - -- and extract one into the temp dir -- not a bare `core` file + -- and extract one into the temp dir, not a bare `core` file -- appearing there on its own. local ws = checks.run("mktemp -d").out:gsub("%s+$", "") assert(ws:match("^/"), "mktemp -d must yield an absolute path") @@ -156,7 +156,7 @@ elseif lang == "rust" then lang .. ": no code past the panic point runs") -- Without RUST_BACKTRACE, the same panic and exit code happen, but no - -- frame names are printed -- documents that the backtrace is opt-in. + -- frame names are printed, which documents that the backtrace is opt-in. r = checks.run("./" .. bin .. " crash") checks.expect_exit(r, 101, lang .. ": panic exit code is unchanged without a backtrace") diff --git a/examples/29-valgrind-sanitizers-miri/README.md b/examples/29-valgrind-sanitizers-miri/README.md index 2290d1a..9fa3468 100644 --- a/examples/29-valgrind-sanitizers-miri/README.md +++ b/examples/29-valgrind-sanitizers-miri/README.md @@ -1,6 +1,6 @@ -# 29-valgrind-sanitizers-miri — bugfarm +# 29-valgrind-sanitizers-miri: bugfarm -Chapter 29 is a menu of five seeded defects, one program each in C++, Go, +This chapter is a menu of five seeded defects, one program each in C++, Go, and Rust: ``` @@ -8,11 +8,11 @@ app ``` Each subcommand runs a small routine that exhibits exactly one memory or -concurrency bug — *where the language lets it*. This is not a program with +concurrency bug, where the language lets it. This is not a program with a normal happy path: the point of the chapter is that a plain run of most of these subcommands looks fine (or crashes somewhere unhelpful, far from the real bug) and it takes the right tool, pointed at the right binary, to -actually name the defect. +name the defect. ## What's expressible in each language @@ -24,23 +24,23 @@ bugs before the program can exist at all. | bug | C++ | Go | Rust | |---|---|---|---| | `leak` | heap alloc, never freed | goroutine blocked forever (leaked) | `Box::leak` (safe API, opts out of reclamation) | -| `uaf` | use-after-free | — (GC won't free a live reference) | — (borrow checker) | -| `uninit` | read of indeterminate value | — (every value is zeroed) | — (no way to name an uninitialised value in safe code) | -| `overflow` | heap buffer overflow, one past the end | — (slice/array access panics, checked) | — (bounds-checked, or rejected at compile time for a fixed index) | -| `race` | data race on a shared int | data race on a shared int (allowed; GC keeps it memory-safe) | — (`Send`/`Sync` rules reject the unsynchronized sharing) | +| `uaf` | use-after-free | n/a (GC won't free a live reference) | n/a (borrow checker) | +| `uninit` | read of indeterminate value | n/a (every value is zeroed) | n/a (no way to name an uninitialised value in safe code) | +| `overflow` | heap buffer overflow, one past the end | n/a (slice/array access panics, checked) | n/a (bounds-checked, or rejected at compile time for a fixed index) | +| `race` | data race on a shared int | data race on a shared int (allowed; GC keeps it memory-safe) | n/a (`Send`/`Sync` rules reject the unsynchronized sharing) | Where a bug isn't expressible, the subcommand still exists (so the CLI -shape is identical across all three) — it just prints a fixed note and +shape is identical across all three); it just prints a fixed note and exits **64** (`EX_USAGE`) instead of attempting anything: - Go: `go: prevented by the runtime — see chapter` - Rust: `rust: prevented at compile time — see chapter` -That message *is* the finding for those cells, not a cop-out: it's the -same "no bug to reproduce" result you'd get chasing the equivalent C++ bug -in that language, just arrived at differently (a runtime check vs. a -rejected compilation) — and it's reproducible non-interactively, which is -this chapter's whole bar. +That message is the finding for those cells: the same "no bug to +reproduce" result you'd get chasing the equivalent C++ bug in that +language, just arrived at differently (a runtime check vs. a rejected +compilation), and it's reproducible non-interactively, which is this +chapter's whole bar. Bad usage (no subcommand, or an unrecognized one) prints `usage: app ` on stderr and exits 2, same @@ -49,8 +49,7 @@ shape as every other chapter's `app`. ## Catching each C++ bug: the tool table C++ is the only language that expresses all five, so it's the one where -"run it under the matching tool" actually means something for every -subcommand: +"run it under the matching tool" means something for every subcommand: | bug | tool | build | signature this chapter asserts | exit | |---|---|---|---|---| @@ -62,7 +61,7 @@ subcommand: The `asan` preset was already in this book's CMake template (`-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined`); this chapter adds -a matching `tsan` preset. Neither is part of the `demo.sh` contract — +a matching `tsan` preset. Neither is part of the `demo.sh` contract: build them directly: ```console @@ -75,28 +74,27 @@ $ ./build/tsan/app race WARNING: ThreadSanitizer: data race (pid=...) ``` -For Rust, valgrind confirms the one expressible bug (`leak`), and — where -available — miri adds a second, independent confirmation (see below). +For Rust, valgrind confirms the one expressible bug (`leak`); where +available, miri adds a second, independent confirmation (see below). -## A gotcha worth knowing: the optimizer can eat the bug +## Optimizer effects: the compiler can eliminate the bug Building `leak`/`uaf`/`uninit`/`overflow` at `-O2` with no barrier, GCC proved on this host that the allocation was never otherwise observed and -deleted the whole thing — `malloc`, the write, and all — before valgrind +deleted the whole thing (`malloc`, the write, and all) before valgrind ever saw an allocation to lose. Same story in Rust release builds: a `Box::leak`'d `Vec` whose only observable use is `.len()` (a compile-time-known constant) can be fully constant-folded away, allocation included. Every `do_*` function in `cpp/src/main.cpp` routes its pointer -through a `touch()` helper (`asm volatile("" : : "r,m"(value) : "memory")` -— the same trick as benchmark libraries' `DoNotOptimize`), and the Rust +through a `touch()` helper (`asm volatile("" : : "r,m"(value) : "memory")`, +the same trick as benchmark libraries' `DoNotOptimize`), and the Rust `leak()` does the same with `std::hint::black_box`. Pull either barrier out -and re-run valgrind: the leak silently disappears. This is itself a small -lesson about why "it didn't crash" is not evidence of "there's no bug" — -the compiler is allowed to make dead code disappear, and an unobserved bug -is dead code. +and re-run valgrind: the leak disappears. This is a small lesson about why +"it didn't crash" is not evidence of "there's no bug": the compiler is +allowed to make dead code disappear, and an unobserved bug is dead code. Separately: a plain (uninstrumented) run of C++ `overflow` can abort with -a glibc `Fatal glibc error: malloc.c:... assertion failed` — the +a glibc `Fatal glibc error: malloc.c:... assertion failed`: the one-past-the-end write corrupts the next heap chunk's bookkeeping, and `delete[]`'s consistency check catches the corruption far from where it happened. That crash, when it happens, *is* the natural (if unhelpful) @@ -106,14 +104,14 @@ consequence of the bug; it's why the ASan build exists. `rust-toolchain.toml` pins `1.97.1`, and `rustup component add miri --toolchain 1.97.1` fails on this host (`component 'miri' ... is -unavailable for download for channel '1.97.1'`) — miri tracks nightly, not +unavailable for download for channel '1.97.1'`): miri tracks nightly, not stable. Where a `nightly` toolchain with the `miri` component happens to be installed, `verify.lua` runs it as a **bonus, informational-only** check (`cargo +nightly miri run -- leak`, expecting `memory leaked` in its output) and prints what it found; where it isn't installed, it prints why -it's skipping and moves on. Neither path can fail the example — the -valgrind check above is what's actually asserted for Rust's `leak`. This -is exactly the "degrade gracefully" case: miri is a nice extra sample of +it's skipping and moves on. Neither path can fail the example: the +valgrind check above is what's asserted for Rust's `leak`. This +is exactly the "degrade gracefully" case: miri is a useful extra sample of "a completely different technique (symbolic interpretation instead of runtime instrumentation) finds the same bug," not a required tool for this chapter. @@ -149,8 +147,8 @@ strings and exit codes are what's stable and what `verify.lua` checks.) Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only (the plain, uninstrumented binary) -- `./demo.sh run [args]` — run the built binary +- `./demo.sh build`: build only (the plain, uninstrumented binary) +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -158,18 +156,17 @@ The top-level `demo.sh` dispatches: `./demo.sh cpp run leak` execs `cpp/demo.sh run leak`. The binary is always named `app`. Because this chapter's whole point is tool-assisted detection, the *instrumented* builds (C++ ASan/TSan presets, the Go `-race` binary) live outside that -contract — see the tables above for how to build and run them directly. +contract: see the tables above for how to build and run them directly. There's no sensible default action for a bare `./demo.sh` (no subcommand) -here — unlike most chapters, this one has no "just run it" happy path — so -running with no subcommand prints the usage line and exits 2. That's -intentional, not an oversight. +here: unlike most chapters, this one has no "just run it" happy path, so +running with no subcommand prints the usage line and exits 2. ## Verification `verify.lua` (run per language with `LSP_LANG` set) builds the plain -binary, checks the shared bad-usage/unknown-bug behavior, then — per -language — builds whatever instrumented variant that language's real bugs +binary, checks the shared bad-usage/unknown-bug behavior, then, per +language, builds whatever instrumented variant that language's real bugs need and asserts the detecting tool exits nonzero (66 for both race detectors' default halt code, 1 otherwise) with the expected signature string in its output, and that every "prevented" subcommand exits 64 with diff --git a/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp b/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp index b79f9c8..e451b3f 100644 --- a/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp +++ b/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// bugfarm — chapter 29: valgrind, sanitizers, and miri. +// bugfarm: chapter 29: valgrind, sanitizers, and miri. // // app // @@ -15,7 +15,7 @@ // // Plain (uninstrumented) runs are deliberately left un-hardened: leak and // uninit stay silent, uaf usually "works" by accident, race just produces -// a wrong count -- and overflow can abort with a glibc heap-corruption +// a wrong count, and overflow can abort with a glibc heap-corruption // message when its write clobbers the next chunk's bookkeeping. That // unpredictability *is* the lesson: these bugs hide from a normal run and // need a tool that watches memory, not eyeballs on stdout. @@ -115,7 +115,7 @@ int do_overflow() { int do_race() { { - std::jthread t1(bump); // RAII joins both on scope exit -- lifetime + std::jthread t1(bump); // RAII joins both on scope exit: lifetime std::jthread t2(bump); // is safe; the *data* race on `counter` is not } std::println("bugfarm: race: counter={} (expected 400000; a wrong value is the benign symptom)", diff --git a/examples/29-valgrind-sanitizers-miri/go/main.go b/examples/29-valgrind-sanitizers-miri/go/main.go index 595873a..459c4bc 100644 --- a/examples/29-valgrind-sanitizers-miri/go/main.go +++ b/examples/29-valgrind-sanitizers-miri/go/main.go @@ -1,4 +1,4 @@ -// bugfarm — chapter 29: valgrind, sanitizers, and miri. +// bugfarm: chapter 29 (valgrind, sanitizers, and miri). // // app // @@ -6,17 +6,17 @@ // after free (the GC never reclaims memory a live reference still points // at), no read of uninitialised memory (every value starts zeroed), and no // out-of-bounds write that silently corrupts a neighbor (a slice/array -// access past its bound panics immediately -- a runtime check, not +// access past its bound panics immediately, a runtime check, not // something a background tool has to catch after the fact). So uaf, // uninit, and overflow print a "prevented by the runtime" note and exit // 64: that absence of a bug *is* the language's story here. // // What Go can still get wrong: // -// leak -- a goroutine that blocks forever. There's no valgrind +// leak: a goroutine that blocks forever. There's no valgrind // equivalent watching the Go heap for this; the way you notice // is runtime.NumGoroutine() not going back down. -// race -- unsynchronized access to a shared variable from goroutines. +// race: unsynchronized access to a shared variable from goroutines. // Build with `go build -race` and the runtime's race detector // prints "WARNING: DATA RACE" and the process exits 66. package main diff --git a/examples/29-valgrind-sanitizers-miri/rust/src/main.rs b/examples/29-valgrind-sanitizers-miri/rust/src/main.rs index 92f63ce..b7ac789 100644 --- a/examples/29-valgrind-sanitizers-miri/rust/src/main.rs +++ b/examples/29-valgrind-sanitizers-miri/rust/src/main.rs @@ -1,4 +1,4 @@ -//! bugfarm — chapter 29: valgrind, sanitizers, and miri. +//! bugfarm, chapter 29: valgrind, sanitizers, and miri. //! //! app //! @@ -7,13 +7,13 @@ //! data race compile, and there's no way to name an uninitialised value or //! index past a slice's bound without either a panic (bounds check) or an //! explicit `unsafe` block. So `uaf`, `uninit`, `overflow`, and `race` each -//! print a fixed note and exit 64 -- the compile-time rejection *is* the +//! print a fixed note and exit 64: the compile-time rejection *is* the //! finding, reproduced here as a documented outcome rather than a crash. //! //! What safe Rust can still do is leak memory on purpose: `Box::leak` //! consumes a `Box` and hands back a `'static` reference, deliberately //! opting the allocation out of Rust's normal drop-based reclamation. -//! Nothing frees it -- valgrind's memcheck reports the same "definitely +//! Nothing frees it: valgrind's memcheck reports the same "definitely //! lost" block it would for an equivalent C/C++ leak. use std::env; @@ -44,13 +44,13 @@ fn parse_bug(arg: &str) -> Result { } // leak: Box::leak is a safe API whose entire purpose is to opt an -// allocation out of automatic reclamation -- the program exits normally, +// allocation out of automatic reclamation: the program exits normally, // but the 64 KiB Vec (and the Box that briefly held it) are never freed. // // black_box on the pointer is load-bearing, not decoration: without it, // LLVM can see that nothing observable depends on the buffer's *contents* // (only its statically-known length gets printed) and optimizes the -// entire allocation away, leak and all -- confirmed on this host by +// entire allocation away, leak and all: confirmed on this host by // valgrind reporting zero bytes lost until the barrier was added. fn leak() { let leaked: &'static mut Vec = Box::leak(Box::new(vec![0xAAu8; 65536])); diff --git a/examples/29-valgrind-sanitizers-miri/verify.lua b/examples/29-valgrind-sanitizers-miri/verify.lua index 6abb734..c9db70a 100644 --- a/examples/29-valgrind-sanitizers-miri/verify.lua +++ b/examples/29-valgrind-sanitizers-miri/verify.lua @@ -1,23 +1,23 @@ -- Verify bugfarm (chapter 29: valgrind, sanitizers, and miri). -- -- "app " runs one seeded defect. This isn't --- a normal program with an observable happy-path output -- the thing to +-- a normal program with an observable happy-path output: the thing to -- observe is a DETECTING TOOL catching the defect, so most of this script -- builds an instrumented variant and runs the tool directly against the -- binary (not through demo.sh, which only knows the plain build). -- -- Per language, what's actually expressible differs on purpose: --- cpp -- all five: valgrind (leak, uninit), ASan (uaf, overflow), +-- cpp : all five: valgrind (leak, uninit), ASan (uaf, overflow), -- TSan (race). --- go -- leak (self-detected via runtime.NumGoroutine(), no external +-- go : leak (self-detected via runtime.NumGoroutine(), no external -- tool watches Go goroutine leaks the way valgrind watches -- malloc) and race (-race build, "DATA RACE", exit 66). uaf/ -- uninit/overflow are not expressible in safe Go: bounds/nil -- checks and the GC rule them out at runtime, so those exit 64 -- with a fixed note. --- rust -- leak only (Box::leak, a real unfreed allocation valgrind +-- rust : leak only (Box::leak, a real unfreed allocation valgrind -- reports as "definitely lost"; miri additionally confirms it --- when available -- informational, since the pinned 1.97.1 +-- when available; informational, since the pinned 1.97.1 -- toolchain does not ship miri). uaf/uninit/overflow/race are -- rejected by the borrow checker / bounds checks at compile -- time or by construction, so those exit 64 with a fixed note. @@ -65,7 +65,7 @@ if lang == "cpp" then -- memory or threads, not a plain run, to produce its finding. -- ASan+UBSan and TSan builds aren't part of the plain demo.sh contract - -- (build/run/TARGET deploy) -- build them directly via the CMake + -- (build/run/TARGET deploy); build them directly via the CMake -- presets that ship alongside the release preset. r = checks.run("cd cpp && cmake --preset asan >/dev/null && cmake --build --preset asan >/dev/null") checks.expect_exit(r, 0, "cpp: ASan+UBSan build configures and compiles") @@ -114,7 +114,7 @@ elseif lang == "go" then -- race: needs a -race build, which is not what the plain demo.sh build -- produces for this chapter (unlike ch06, this chapter is not - -- exclusively about concurrency) -- build the instrumented binary + -- exclusively about concurrency); build the instrumented binary -- directly. r = checks.run("cd go && go build -race -o bin/app-race .") checks.expect_exit(r, 0, "go: race-instrumented build compiles") @@ -145,7 +145,7 @@ else -- rust -- miri: informational only. The pinned toolchain (rust-toolchain.toml, -- 1.97.1) does not ship a miri component; a nightly toolchain with miri - -- installed is a bonus double-check, never a requirement -- absence + -- installed is a bonus double-check, never a requirement; absence -- must degrade gracefully, not fail the example. local probe = checks.run("rustup which --toolchain nightly miri") if probe.exit == 0 then diff --git a/examples/30-ebpf-observation-toolkit/README.md b/examples/30-ebpf-observation-toolkit/README.md index 8236052..7bd1881 100644 --- a/examples/30-ebpf-observation-toolkit/README.md +++ b/examples/30-ebpf-observation-toolkit/README.md @@ -1,11 +1,11 @@ -# 30 — eBPF observation toolkit +# 30: eBPF observation toolkit -**VM example.** This chapter is not about writing eBPF — it's about pointing +**VM example.** This chapter is not about writing eBPF: it's about pointing the standard eBPF *observation* tools (bcc-tools, bpftrace) at an ordinary userspace program and reading what they say. `app work --seconds N` runs a small, labeled workload; a driver script (`observe.sh`, running as root on the lab VM) starts it in the background and then runs five separate tools against -it in turn, each proving it really saw an event our program produced: +it in turn, each proving it saw an event our program produced: | Tool | What it watches | Bait in `app work` | |---|---|---| @@ -15,10 +15,10 @@ it in turn, each proving it really saw an event our program produced: | `offcputime` | off-CPU time by stack | a ~230ms sleep between iterations | | `bpftrace` | a one-line uprobe program | the same `busy_hash()`, counted with `@calls` | -`busy_hash()` is deliberately kept unmangled and unstripped, and marked -noinline, in all three builds — cpp and rust emit the plain symbol -`busy_hash`, Go emits `main.busy_hash` — so `observe.sh` can target every -language with the *same* pattern: `:*busy_hash*`. +`busy_hash()` is kept unmangled and unstripped, and marked noinline, in all +three builds (cpp and rust emit the plain symbol `busy_hash`, Go emits +`main.busy_hash`), so `observe.sh` can target every language with the +*same* pattern: `:*busy_hash*`. All three languages produce byte-for-byte identical output, including the final accumulated hash value (same seed, same algorithm, same iteration @@ -36,20 +36,20 @@ work: done seconds=60 iters=260 opens=260 execs=65 busy_calls=260 busy_hash=1286 Each iteration (~230ms): open/write/close the bait file; every 4th iteration, fork+exec `true` and wait for it; call `busy_hash()` once; sleep ~230ms. None -of this needs root — only the *observation* side does. +of this needs root; only the *observation* side does. | Language | How the loop runs | |---|---| | **C++23** | The loop body runs inside a `std::jthread` (takes a `std::stop_token`, ready for cooperative cancellation); `main()` joins it explicitly so the summary line always sees final counters. An RAII `Fd` class owns the bait file's descriptor. Every fallible syscall returns `std::expected`. Output goes through `std::println`. | -| **Go 1.26** | The loop runs in a goroutine; `main` blocks on a buffered result channel — the goroutine's "join". Every fallible call is wrapped with `fmt.Errorf("...: %w", err)`. | -| **Rust (edition 2024)** | Straight-line `Result`/`?` through the whole loop; the bait file's descriptor is a `rustix::fs` `OwnedFd`, closed by `Drop` — no explicit `close(2)`. | +| **Go 1.26** | The loop runs in a goroutine; `main` blocks on a buffered result channel (the goroutine's "join"). Every fallible call is wrapped with `fmt.Errorf("...: %w", err)`. | +| **Rust (edition 2024)** | Straight-line `Result`/`?` through the whole loop; the bait file's descriptor is a `rustix::fs` `OwnedFd`, closed by `Drop`, no explicit `close(2)`. | -### A real optimizer gotcha found while building this +### An optimizer effect found while building this example The first version of this workload printed `busy_calls` but never the accumulated hash value itself. At `-O2`, GCC's interprocedural analysis proved `busy_hash()` had no observable side effects and that its final result was -never used anywhere — and deleted the *entire* call chain, `noinline` or not +never used anywhere, and deleted the *entire* call chain, `noinline` or not (`noinline` only forbids inlining; it says nothing about dead-code elimination of a provably-unused, provably-pure call). `funccount` and the `bpftrace` uprobe both reported zero calls even though the binary clearly @@ -57,9 +57,9 @@ still exported the `busy_hash` symbol and a plain `nm`/`objdump` looked completely normal. The fix: print the final hash in the `done` summary line, in all three languages. The moment the result is observable, the whole chain is provably necessary again, and the uprobe fires. `busy_hash` is part of the -printed summary for that reason — it isn't an incidental extra field. +printed summary to keep the call observable to the uprobe. -## `observe.sh` — the actual demo +## `observe.sh`: the demo `observe.sh [seconds]` runs **on the guest, as root** (bcc-tools and bpftrace need `CAP_BPF`/`CAP_PERFMON`, which on this lab image @@ -72,18 +72,18 @@ means root). It: 3. prints a `=== summary ===` block deriving a hit count per tool, 4. lets the workload finish (or stops it) before exiting. -It deliberately does **not** use `set -e` — a bcc tool killed by its own -`timeout` wrapper (several of these tools don't reliably self-terminate on -their internal `-d`/duration flag; see below) is expected, not a driver -failure. Each step's output is checked on its own, at the end. +It does **not** use `set -e`: a bcc tool killed by its own `timeout` wrapper +(several of these tools don't reliably self-terminate on their internal +`-d`/duration flag; see below) is expected, not a driver failure. Each +step's output is checked on its own, at the end. -### A real bcc-tools quirk this driver works around +### bcc-tools' duration-flag behavior, handled here `opensnoop`'s own `-d SECONDS` flag did not reliably make it exit on its own -in testing — it consistently ran until an external `timeout` killed it, +in testing: it consistently ran until an external `timeout` killed it, regardless of the requested duration. Every invocation in `observe.sh` is therefore wrapped in an outer `timeout`, and a `124` exit from that wrapper is -treated as expected, not a failure — only the captured *content* is asserted +treated as expected, not a failure; only the captured *content* is asserted on, never the tool's own exit code. ## CLI @@ -99,14 +99,14 @@ usage line to stderr and exits `2`. Each language directory has the standard `demo.sh`: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (`app`) locally +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (`app`) locally - With env `TARGET` set, `run` deploys to that lab VM via `scripts/lab/deploy-to-vm.sh` and runs there instead (no root needed for the workload itself) The top-level `./demo.sh [cpp|go|rust|all|build]` dispatches per language. -`observe.sh` is deliberately separate from this dispatcher — it's a custom +`observe.sh` is kept separate from this dispatcher: it's a custom root-run driver script (starts the workload *and* orchestrates five different tools against it), not a single binary+args invocation, so it doesn't fit the `deploy-to-vm.sh` shape and is staged/run directly instead. @@ -164,13 +164,13 @@ app_summary: work: done seconds=60 iters=260 opens=260 execs=65 busy_calls=260 b `verify.lua` (run per language under the runner's **vm mode**, `TARGET=systems-target`) stages the built binary under `/tmp/lsp30-/app` -— so its `/proc` comm is exactly `app` regardless of which language built -it — plus `observe.sh`, then runs the driver as root and asserts, from its -captured output, that each tool really observed our program: +(so its `/proc` comm is exactly `app` regardless of which language built +it) plus `observe.sh`, then runs the driver as root and asserts, from its +captured output, that each tool observed our program: 1. **opensnoop** shows comm `app` opening the bait file's exact path. 2. **execsnoop** shows a `true` child whose **PPID equals the workload's own - pid** (parsed from `observe.sh`'s own `workload pid=` line) — not just + pid** (parsed from `observe.sh`'s own `workload pid=` line), not just that *some* exec happened somewhere on the box. 3. **funccount** reports a **nonzero** call count for `busy_hash`. 4. **offcputime** shows a stack naming comm `app` (the ~230ms sleep between @@ -178,7 +178,7 @@ captured output, that each tool really observed our program: 5. **bpftrace**'s uprobe one-liner reports `@calls` **nonzero**. Each is a positive count/match on a *specific* signal our program produced, -not merely "the tool ran and exited 0" — a tool that loads and exits cleanly +not merely "the tool ran and exited 0." A tool that loads and exits cleanly while counting nothing would fail every one of these. Run it: @@ -188,10 +188,10 @@ LIBVIRT_DEFAULT_URI=qemu:///system \ python3 scripts/test-all-examples.py --only 30-ebpf-observation-toolkit --mode vm ``` -**Why this is real observation, not an inference:** every assertion is tied -to something our own program did — the exact bait path, the workload's own -pid as the exec's parent, a call count on the exact function we wrote. This -is the standard eBPF-based observability toolkit doing what it's designed to -do: watch an ordinary, unmodified userspace binary from the outside. We write -no kernel-side eBPF program here — bcc-tools and bpftrace already ship -theirs; this chapter is about aiming them correctly and reading the answer. +Every assertion is tied to something our own program did: the exact bait +path, the workload's own pid as the exec's parent, a call count on the +exact function we wrote. This is the standard eBPF-based observability +toolkit doing what it's designed to do: watch an ordinary, unmodified +userspace binary from the outside. We write no kernel-side eBPF program +here; bcc-tools and bpftrace already ship theirs. Aiming them correctly and +reading the answer is the whole exercise. diff --git a/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp b/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp index 857ff34..57c9688 100644 --- a/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp +++ b/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// app work --seconds N — a small, observable workload for the eBPF +// app work --seconds N: a small, observable workload for the eBPF // observation toolkit (chapter 30). It does four things in a loop, each one // deliberate bait for a specific bcc-tools/bpftrace probe: // @@ -78,7 +78,7 @@ class Fd { // open(2) on the same path. [[nodiscard]] std::expected open_bait(long iter) { // 0666: the bait file may get created by a root-run observe.sh one time - // and an unprivileged demo.sh run the next — world-writable avoids a + // and an unprivileged demo.sh run the next: world-writable avoids a // permission mismatch between those two ownership scenarios. Fd fd{::open(kBaitPath, O_CREAT | O_WRONLY | O_TRUNC, 0666)}; if (!fd.valid()) { @@ -115,7 +115,7 @@ struct Counters { std::atomic opens{0}; std::atomic execs{0}; std::atomic busy_calls{0}; - // The accumulated busy_hash() result. Printed in the summary line below — + // The accumulated busy_hash() result. Printed in the summary line below: // that's what keeps the optimizer from proving the whole busy_hash call // chain's result is unused and deleting it outright (noinline only stops // inlining, not interprocedural dead-code elimination of a provably-pure, @@ -158,7 +158,7 @@ int cmd_work(int seconds) { Counters counters; // jthread: takes a stop_token as its first parameter (so it's ready for // cooperative cancellation, e.g. from a signal handler) and joins on - // request/destruction like a plain thread — join explicitly here so the + // request/destruction like a plain thread: join explicitly here so the // summary line below always sees the final counters. std::jthread worker(run_workload, seconds, std::ref(counters)); worker.join(); diff --git a/examples/30-ebpf-observation-toolkit/go/main.go b/examples/30-ebpf-observation-toolkit/go/main.go index e9deb7e..9369c5a 100644 --- a/examples/30-ebpf-observation-toolkit/go/main.go +++ b/examples/30-ebpf-observation-toolkit/go/main.go @@ -1,4 +1,4 @@ -// app work --seconds N — a small, observable workload for the eBPF +// app work --seconds N: a small, observable workload for the eBPF // observation toolkit (chapter 30). It does four things in a loop, each one // deliberate bait for a specific bcc-tools/bpftrace probe: // @@ -50,7 +50,7 @@ func busy_hash(x uint64) uint64 { // Every call is a fresh open(2) on the same path. func openBait(iter int) error { // 0666: the bait file may get created by a root-run observe.sh one time - // and an unprivileged demo.sh run the next — world-writable avoids a + // and an unprivileged demo.sh run the next; world-writable avoids a // permission mismatch between those two ownership scenarios. f, err := os.OpenFile(baitPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o666) if err != nil { @@ -116,7 +116,7 @@ func runWorkload(seconds int, done chan<- result) { func cmdWork(seconds int) int { fmt.Printf("work: start seconds=%d pid=%d bait=%s\n", seconds, os.Getpid(), baitPath) - // The goroutine is the worker; the channel is its "join" — main blocks + // The goroutine is the worker; the channel is its "join"; main blocks // on the result rather than sharing counters behind a mutex. done := make(chan result, 1) go runWorkload(seconds, done) diff --git a/examples/30-ebpf-observation-toolkit/rust/src/main.rs b/examples/30-ebpf-observation-toolkit/rust/src/main.rs index ec64b04..6b7c8c3 100644 --- a/examples/30-ebpf-observation-toolkit/rust/src/main.rs +++ b/examples/30-ebpf-observation-toolkit/rust/src/main.rs @@ -1,4 +1,4 @@ -//! app work --seconds N — a small, observable workload for the eBPF +//! app work --seconds N: a small, observable workload for the eBPF //! observation toolkit (chapter 30). It does four things in a loop, each one //! deliberate bait for a specific bcc-tools/bpftrace probe: //! @@ -13,7 +13,7 @@ //! //! Rust idioms: every fallible call flows through Result and `?`; the bait //! file's descriptor is a rustix OwnedFd, closed by its Drop impl the moment -//! it goes out of scope — no explicit close(2). +//! it goes out of scope: no explicit close(2). use std::time::{Duration, Instant}; @@ -45,7 +45,7 @@ pub extern "C" fn busy_hash(mut x: u64) -> u64 { /// fresh open(2) on the same path; the OwnedFd's Drop does the close(2). fn open_bait(iter: u64) -> Result<()> { // 0666: the bait file may get created by a root-run observe.sh one time - // and an unprivileged demo.sh run the next — world-writable avoids a + // and an unprivileged demo.sh run the next: world-writable avoids a // permission mismatch between those two ownership scenarios. let fd = open( BAIT_PATH, @@ -76,7 +76,7 @@ struct Counters { // The accumulated busy_hash() result, printed in the summary line. LLVM // can prove a #[no_mangle]/#[inline(never)] function is still pure and // dead-code-eliminate a whole call chain whose final result is never - // observed — keeping this value alive here is what guarantees every call + // observed: keeping this value alive here is what guarantees every call // really executed, not just that the symbol exists in the binary. busy_hash: u64, } diff --git a/examples/30-ebpf-observation-toolkit/verify.lua b/examples/30-ebpf-observation-toolkit/verify.lua index b291898..61626fe 100644 --- a/examples/30-ebpf-observation-toolkit/verify.lua +++ b/examples/30-ebpf-observation-toolkit/verify.lua @@ -1,4 +1,4 @@ --- Verify the eBPF observation toolkit (chapter 30) — a VM example. This is +-- Verify the eBPF observation toolkit (chapter 30), a VM example. This is -- TOOLING (bcc-tools + bpftrace) observing our userspace binary via uprobes -- and tracepoints; we write no kernel-side eBPF program of our own. -- @@ -8,12 +8,12 @@ -- assertion below is the OBSERVABLE proof that a specific tool really saw a -- real event our program produced, not just that the tool ran without error: -- --- 1. opensnoop — shows our bait file's path opened by comm "app". --- 2. execsnoop — shows a "true" child with PPID == our workload's pid. --- 3. funccount — a nonzero per-call count for busy_hash() via uprobe. --- 4. offcputime — a stack naming comm "app" (the sleep between iterations +-- 1. opensnoop : shows our bait file's path opened by comm "app". +-- 2. execsnoop : shows a "true" child with PPID == our workload's pid. +-- 3. funccount : a nonzero per-call count for busy_hash() via uprobe. +-- 4. offcputime : a stack naming comm "app" (the sleep between iterations -- put it off-CPU during the trace window). --- 5. bpftrace — a one-liner uprobe on busy_hash() reports @calls > 0. +-- 5. bpftrace : a one-liner uprobe on busy_hash() reports @calls > 0. -- -- funccount/bpftrace both target the pattern ":*busy_hash*", which -- matches the plain symbol "busy_hash" (cpp/rust) and the Go runtime's @@ -21,7 +21,7 @@ -- -- Runs under the runner's vm mode (TARGET=systems-target). Staging goes -- straight through ssh/scp (not deploy-to-vm.sh, which only shells out a --- single binary+args — this needs a custom root-run driver script too). +-- single binary+args, this needs a custom root-run driver script too). local script_dir = arg[0]:match("(.*/)") or "./" local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") @@ -62,7 +62,7 @@ local ssh_opts = "-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=" .. home .. "/.ssh/known_hosts" -- Stage the binary under a per-language directory so the deployed file's --- basename — and therefore its /proc comm — is exactly "app" regardless of +-- basename (and therefore its /proc comm) is exactly "app" regardless of -- which language built it, matching the book-wide "binary stays app" rule. local remote_dir = "/tmp/lsp30-" .. lang local remote_bin = remote_dir .. "/app" diff --git a/examples/31-per-language-toolbelts/README.md b/examples/31-per-language-toolbelts/README.md index b22f3aa..d5a70b2 100644 --- a/examples/31-per-language-toolbelts/README.md +++ b/examples/31-per-language-toolbelts/README.md @@ -1,22 +1,22 @@ -# 31-per-language-toolbelts — toolbelt +# 31-per-language-toolbelts: toolbelt -Chapter 31 closes the debugging arc by putting each language's *native* +This chapter closes the debugging arc by putting each language's *native* profiling toolbelt to work on the same program. There is nothing wrong with -this code — the point is not to find a defect, but to point a real profiler -at a real binary and watch it name the function that is actually costing the +this code; the point is not to find a defect, but to point a real profiler +at a real binary and watch it name the function that is costing the cycles or the bytes. ``` app [--n N] ``` -- `hot` — counts primes in `[2, n)` by unoptimized trial division (every +- `hot`: counts primes in `[2, n)` by unoptimized trial division (every candidate divisor up to `sqrt(i)`). One function, `spin_hot` / - `hotSpin`, does all the work — it is the whole point of a CPU profile. -- `alloc` — builds a 1000-key string index from `n` iterations of + `hotSpin`, does all the work; it is the whole point of a CPU profile. +- `alloc`: builds a 1000-key string index from `n` iterations of round-robin overwrite, each iteration allocating a fresh key and value string. One function, `alloc_churn` / `allocChurn`, is the only - allocation site — it is the whole point of an allocation profile. + allocation site; it is the whole point of an allocation profile. - `--n N` overrides the default size (`hot`: 3,000,000; `alloc`: 200,000); must be a positive integer. @@ -32,13 +32,13 @@ app [--n N] - Bad usage (missing/unknown mode, non-positive `--n`, unknown flag) prints a diagnostic plus `usage: app [--n N]` on stderr and exits 2. - Go's binary additionally accepts `--cpuprofile FILE`, `--memprofile FILE`, - and `--trace FILE` — see "Why Go's CLI has extra flags" below. Its usage + and `--trace FILE`; see "Why Go's CLI has extra flags" below. Its usage banner lists them; C++ and Rust do not accept them (unknown-argument errors bad-usage the same as any other unrecognized flag). ### Why Go's CLI has extra flags -`perf` attaches to *any* ELF binary from the outside — nothing in the C++ or +`perf` attaches to *any* ELF binary from the outside: nothing in the C++ or Rust source needs to know it is being profiled. Go's CPU and heap profilers are different: `runtime/pprof` and `runtime/trace` are libraries the program calls into, so writing a profile is part of the binary's own behavior. That @@ -61,12 +61,12 @@ app: unknown mode: bogus usage: app [--n N] ``` -(Timings vary by host and load; `result` does not — same algorithm, same +(Timings vary by host and load; `result` does not: same algorithm, same input, same answer, in all three languages.) -## The toolbelts, run for real +## The toolbelts in practice -Every command below was actually run against the binaries in this directory +Every command below ran against the binaries in this directory (`cpp/build/release/app`, `go/bin/app`, `rust/target/release/app`) on this host (Fedora 44, GCC 16.1.1, clang 22.1.8, go1.26.5, rustc 1.97.1, perf 7.1.3, gdb 17.2). Numbers are host- and load-dependent; the shapes and the @@ -74,7 +74,7 @@ named functions are the point. ### C++: perf record/report, clang-tidy, -fanalyzer -`spin_hot` has C linkage on purpose (`extern "C"`), so perf and gdb print it +`spin_hot` has C linkage by design (`extern "C"`), so perf and gdb print it as the plain symbol `spin_hot` instead of an Itanium-mangled name. ```console @@ -108,8 +108,8 @@ $ perf report -i cpp.perf.data --stdio `flamegraph.pl` scripts are not installed on this host (no `FlameGraph/` tree on `PATH` or under any locatable directory), so per the fallback this chapter documents, the C++ step stops at `perf report --stdio` above rather than -producing an SVG. (Compare the Rust section, where `cargo flamegraph` — a -Rust-native reimplementation of the same collapse+render pipeline — *is* +producing an SVG. (Compare the Rust section, where `cargo flamegraph`, a +Rust-native reimplementation of the same collapse+render pipeline, *is* installable and produces a real SVG.) `clang-tidy` (bugprone-*, modernize-*, performance-*, readability-*): @@ -124,14 +124,14 @@ main.cpp:143:5: warning: an exception may be thrown in function 'main' which One warning, and it is a known shape with `std::format`/`std::println`: `bugprone-exception-escape` traces a theoretical `std::format_error` path -that only fires on a malformed format string — ours are literals, so the +that only fires on a malformed format string; ours are literals, so the path is unreachable in practice, but the checker cannot prove that statically. Nothing else fires. GCC's `-fanalyzer` (its own preset: `cmake --build --preset analyze`) is -clean — zero diagnostics on this source. +clean: zero diagnostics on this source. -### Go: pprof (CPU + heap), go tool trace, delve +### Go: pprof (CPU + heap), go tool trace, `dlv` ```console $ ./go/bin/app hot --cpuprofile cpu.prof @@ -157,12 +157,12 @@ $ go tool pprof --top --cum --alloc_space --nodecount=6 ./go/bin/app mem.prof `pprof.WriteHeapProfile` runs, the churned map from the just-finished `allocChurn` call is already garbage (it went out of scope when the function returned), so the default *in-use* view would show almost nothing. -`alloc_space` is pprof's cumulative-since-start counter — it remembers +`alloc_space` is pprof's cumulative-since-start counter: it remembers everything `mallocgc` ever handed out, freed or not, which is the view that -actually answers "where did the allocations happen". +answers "where did the allocations happen". `go tool trace` opens an interactive, browser-based UI by default, which -does not script well — but `-d=parsed` dumps the same event stream as text, +does not script well; `-d=parsed` dumps the same event stream as text, which is enough to confirm the tracer captured `hotSpin`'s execution: ```console @@ -173,7 +173,8 @@ $ go tool trace -d=parsed trace.out | grep -B2 hotSpin | head -3 main.hotSpin @ 0x4d272c ``` -Delve, scripted non-interactively (no TUI) with a piped command script: +The Go debugger, `dlv`, scripted non-interactively (no TUI) with a piped +command script: ```console $ echo -e "break main.hotSpin\ncontinue\nprint n\nbt\ncontinue\nquit" \ @@ -187,11 +188,11 @@ $ echo -e "break main.hotSpin\ncontinue\nprint n\nbt\ncontinue\nquit" \ at ./go/main.go:175 ``` -(`dlv` is not preinstalled on this host; `go install -github.com/go-delve/delve/cmd/dlv@latest` pulls it from the Go module proxy -into `$(go env GOPATH)/bin`. Where that install is unavailable — no module -proxy reachable — the pprof and trace steps above still stand on their own; -delve is demonstrated here, not gated on by `verify.lua`.) +(`dlv` is not preinstalled on this host; `go install github.com/go-delve/delve/cmd/dlv@latest` +pulls it from the Go module proxy +into `$(go env GOPATH)/bin`. Where that install is unavailable (no module +proxy reachable), the pprof and trace steps above still stand on their own; +`dlv` is demonstrated here, not gated on by `verify.lua`.) ### Rust: perf record/report, cargo flamegraph, clippy @@ -206,7 +207,7 @@ $ perf report -i rust.perf.data --stdio spin_hot (inlined) ``` -`cargo-flamegraph` is not preinstalled, but *is* installable — +`cargo-flamegraph` is not preinstalled, but *is* installable: `cargo install flamegraph --locked` pulls it (and the `inferno` crate it wraps, a Rust reimplementation of the classic Perl collapse+render scripts) from crates.io in well under a minute: @@ -222,14 +223,14 @@ $ grep -c spin_hot rust-hot.svg The resulting SVG contains a `spin_hot` frame, confirming the same story `perf report` tells above, rendered instead of tabulated. Where network access to crates.io is unavailable, the `perf record`/`perf report --stdio` -pair above is the fallback path — same data, no plot. +pair above is the fallback path: same data, no plot. -`tokio-console`: not applicable here — this program has no async runtime +`tokio-console` does not apply here: this program has no async runtime (no `tokio`, no `.await`), so there are no tasks for `tokio-console` to -show. It belongs to the async chapter (27), not this one. +show. It belongs to the async runtimes and coroutines chapter, not this one. `cargo clippy --release --all-targets` and `cargo fmt --check` are both -clean — no findings. +clean: no findings. ## Layout @@ -243,15 +244,15 @@ clean — no findings. ``` The C++ `analyze` preset (`cmake --build --preset analyze`) builds with -`-fanalyzer` instead of running it as a separate step — it is just another +`-fanalyzer` instead of running it as a separate step: it is just another `CMakeLists.txt`-driven build, same as `asan`. ## Demo contract Each language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (builds first if needed) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (builds first if needed) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. @@ -275,7 +276,7 @@ against the real built binary: value the same algorithm produces in every language (`216816` / `5000` at the defaults; `4` / `7` at the small sizes). - `perf stat -e cycles,instructions` on `app hot` exits 0 and its output - parses two numeric counters — this step is identical for all three + parses two numeric counters: this step is identical for all three languages, since perf attaches to any ELF binary. - **C++ and Rust**: `perf record` then `perf report --stdio` exits 0 and the report names `spin_hot`. @@ -285,7 +286,7 @@ against the real built binary: `go tool trace`, `delve`, `cargo flamegraph`, `clang-tidy`, and `-fanalyzer` are demonstrated above with real captured output but are not -asserted by `verify.lua` — they either open an interactive UI +asserted by `verify.lua`: they either open an interactive UI (`go tool trace`'s default mode), depend on a piped TTY session (`delve`), or depend on a tool this session had to install over the network (`dlv`, `cargo-flamegraph`) that may not be present in every @@ -299,15 +300,15 @@ flaky one; the chapter text, not `verify.lua`, carries that evidence. | Build | `cmake --preset release` + ninja | `go build` | `cargo build --release` | | CPU profile | `perf record` / `perf report --stdio` | `runtime/pprof` + `go tool pprof --top` | `perf record` / `perf report --stdio`, or `cargo flamegraph` | | Allocation profile | *(not asserted; `alloc` mode exits 0)* | `runtime/pprof` heap + `go tool pprof --top --cum --alloc_space` | *(not asserted; `alloc` mode exits 0)* | -| Execution trace | — | `go tool trace -d=parsed` | — | -| Interactive debugger | `gdb` (chapter 28) | `dlv` | `gdb`/`rust-gdb` (chapter 28) | +| Execution trace | n/a | `go tool trace -d=parsed` | n/a | +| Interactive debugger | `gdb` (the gdb chapter) | `dlv` | `gdb`/`rust-gdb` (the gdb chapter) | | Static analysis | `clang-tidy`, `-fanalyzer` | `go vet` | `cargo clippy` | | Result asserted by `verify.lua` | `spin_hot` named in `perf report --stdio` | `main.hotSpin`/`main.allocChurn` named in `go tool pprof --top` | `spin_hot` named in `perf report --stdio` | ## Mapping to a chapter -This example backs chapter 31 (per-language toolbelts), the closing chapter -of the debugging arc that opened with chapter 28 (gdb and remote debugging) -and chapter 29 (valgrind, sanitizers, miri). Where those chapters aim a +This example backs the per-language toolbelts chapter, the closing chapter +of the debugging arc that opened with the gdb and remote debugging chapter +and the valgrind/sanitizers/miri chapter. Where those chapters aim a generic tool at a specific defect, this one aims each language's own profiler at a workload built to have exactly one obvious answer. diff --git a/examples/31-per-language-toolbelts/cpp/src/main.cpp b/examples/31-per-language-toolbelts/cpp/src/main.cpp index 8a14d79..f9f5c30 100644 --- a/examples/31-per-language-toolbelts/cpp/src/main.cpp +++ b/examples/31-per-language-toolbelts/cpp/src/main.cpp @@ -1,14 +1,14 @@ -// toolbelt — three focused profiling targets, one binary per language, each +// toolbelt: three focused profiling targets, one binary per language, each // exercised by that language's native profiler (chapter 31: per-language // toolbelts). // // app [--n N] // // hot: counts primes in [2, n) by trial division. spin_hot is the only -// function that does real work — it dominates a CPU profile. +// function that does real work: it dominates a CPU profile. // alloc: builds a 1000-key string index by round-robin overwrite from n // iterations of freshly allocated strings. alloc_churn is the only -// allocation site — it dominates an allocation profile. +// allocation site: it dominates an allocation profile. // // Output: "app: mode= n= result= ms=" on success. // spin_hot has C linkage on purpose: perf/gdb show it as the plain symbol diff --git a/examples/31-per-language-toolbelts/go/main.go b/examples/31-per-language-toolbelts/go/main.go index 7c97a48..2defa41 100644 --- a/examples/31-per-language-toolbelts/go/main.go +++ b/examples/31-per-language-toolbelts/go/main.go @@ -1,4 +1,4 @@ -// toolbelt — three focused profiling targets, one binary per language, each +// toolbelt: three focused profiling targets, one binary per language, each // exercised by that language's native profiler (chapter 31: per-language // toolbelts). // @@ -6,20 +6,20 @@ // // hot: counts primes in [2, n) by trial division. hotSpin is the only // -// function that does real work — it dominates a CPU profile. +// function that does real work; it dominates a CPU profile. // // alloc: builds a 1000-key string index by round-robin overwrite from n // // iterations of freshly allocated strings. allocChurn is the only -// allocation site — it dominates an allocation profile. +// allocation site; it dominates an allocation profile. // // Output: "app: mode= n= result= ms=" on success. // // --cpuprofile/--memprofile/--trace are Go-only: unlike perf (which attaches // to any binary from the outside), Go's runtime/pprof and runtime/trace are // libraries the program calls into, so writing a profile or a trace is part -// of this binary's own behavior. C++ and Rust have no equivalent flags — -// their profiling is done externally with perf — which is why the usage +// of this binary's own behavior. C++ and Rust have no equivalent flags; +// their profiling is done externally with perf, which is why the usage // banner differs per language even though the core [--n N] // contract is identical. package main diff --git a/examples/31-per-language-toolbelts/rust/src/main.rs b/examples/31-per-language-toolbelts/rust/src/main.rs index 4ed0f80..2658878 100644 --- a/examples/31-per-language-toolbelts/rust/src/main.rs +++ b/examples/31-per-language-toolbelts/rust/src/main.rs @@ -1,15 +1,15 @@ -//! toolbelt — three focused profiling targets, one binary per language, each +//! toolbelt: three focused profiling targets, one binary per language, each //! exercised by that language's native profiler (chapter 31: per-language //! toolbelts). //! //! app [--n N] //! //! hot: counts primes in [2, n) by trial division. `spin_hot` is the only -//! function that does real work — it dominates a CPU profile (perf +//! function that does real work: it dominates a CPU profile (perf //! record/report, or `cargo flamegraph` where installable). //! alloc: builds a 1000-key string index by round-robin overwrite from n //! iterations of freshly allocated strings. `alloc_churn` is the -//! only allocation site — it dominates an allocation profile. +//! only allocation site: it dominates an allocation profile. //! //! Output: "app: mode= n= result= ms=" on success. diff --git a/examples/31-per-language-toolbelts/verify.lua b/examples/31-per-language-toolbelts/verify.lua index f2a788c..58b47dd 100644 --- a/examples/31-per-language-toolbelts/verify.lua +++ b/examples/31-per-language-toolbelts/verify.lua @@ -83,12 +83,12 @@ local function check_run(args, want_mode, want_n, want_result) end -- Defaults: identical across all three languages, including the numeric --- result — same algorithm, same input, same answer. +-- result: same algorithm, same input, same answer. check_run("hot", "hot", "3000000", "216816") check_run("alloc", "alloc", "200000", "5000") -- Small sizes still work (n below the 1000-entry keyspace, and a tiny hot --- count) — the arithmetic isn't hardcoded to the defaults. +-- count), the arithmetic isn't hardcoded to the defaults. check_run("hot --n 10", "hot", "10", "4") check_run("alloc --n 7", "alloc", "7", "7") diff --git a/examples/32-namespaces-and-cgroups/README.md b/examples/32-namespaces-and-cgroups/README.md index 57ea525..0bf9002 100644 --- a/examples/32-namespaces-and-cgroups/README.md +++ b/examples/32-namespaces-and-cgroups/README.md @@ -1,4 +1,4 @@ -# 32 — Namespaces & cgroups (`pmon` v6) +# 32: Namespaces & cgroups (`pmon` v6) **VM example.** This chapter builds the two primitives every container runtime is made of, by hand, with no runtime in between: **namespaces** to @@ -14,8 +14,8 @@ pmon containerize [--hostname NAME] [--mem-max BYTES|max] It `unshare`s `CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET|CLONE_NEWPID`, writes `memory.max`/`cpu.max` (and disables swap) on a dedicated cgroup *before* CMD -ever runs, forks the pid-1 child into that setup, and — as the child, before -handing off to CMD — reports the two things that are otherwise invisible from +ever runs, forks the pid-1 child into that setup, and (as the child, before +handing off to CMD) reports the two things that are otherwise invisible from the outside: ``` @@ -33,15 +33,15 @@ pmon: child exited status= # or: pmon: child killed signal= (` — `flags=0`, `stack=0` gives fork(2) semantics, `exit_signal=SIGCHLD` makes it `waitpid`-able. The child prints its proof lines, flushes, and `execvp`s CMD. `std::expected` carries every syscall failure; RAII (`Fd`) owns every descriptor; single-threaded throughout, on purpose — see the Go entry for what goes wrong when a runtime isn't. | +| **C++23** | Raw `unshare(2)` (mount/uts/net namespaces land on the caller immediately; the pid namespace is deferred to the next child), then raw `clone3(2)` via ``: `flags=0`, `stack=0` gives fork(2) semantics, `exit_signal=SIGCHLD` makes it `waitpid`-able. The child prints its proof lines, flushes, and `execvp`s CMD. `std::expected` carries every syscall failure; RAII (`Fd`) owns every descriptor; single-threaded throughout, by design; see the Go entry for what goes wrong when a runtime isn't. | | **Rust (edition 2024)** | `nix::sched::unshare` + `nix::unistd::fork` (the same "no code runs between clone and exec except what we write" shape as C++'s `clone3(flags=0)`), `nix::mount::mount`, `nix::unistd::{sethostname, execvp}`. Errors flow through `Result`/`?`; cgroup file I/O is plain `std::fs`. | -| **Go 1.26** | The idiomatic inversion, one step further than chapter 14's: Go can't safely `fork()` and keep running Go code in the child (the runtime assumes a live multi-threaded process on both sides), so `unshare(2)` happens on a `runtime.LockOSThread()`-pinned goroutine, and the **fork+exec is a re-exec of pmon itself** (`/proc/self/exe __ns_init__ CMD...`) via `os/exec`, whose fork always runs on the calling goroutine's *current* OS thread — the one we just unshared and pinned. The freshly exec'd pmon is born inside the new namespaces (PID 1 included), prints the proof lines, and `syscall.Exec`s into the real CMD. `LockOSThread` is load-bearing: without it, the goroutine that called `Unshare` could be rescheduled onto a different, un-unshared OS thread before the fork, silently discarding every namespace change. | +| **Go 1.26** | The idiomatic inversion, one step further than the identity-and-privilege chapter's: Go can't safely `fork()` and keep running Go code in the child (the runtime assumes a live multi-threaded process on both sides), so `unshare(2)` happens on a `runtime.LockOSThread()`-pinned goroutine, and the **fork+exec is a re-exec of pmon itself** (`/proc/self/exe __ns_init__ CMD...`) via `os/exec`, whose fork always runs on the calling goroutine's *current* OS thread: the one we just unshared and pinned. The freshly exec'd pmon is born inside the new namespaces (PID 1 included), prints the proof lines, and `syscall.Exec`s into the real CMD. `LockOSThread` is essential here: without it, the goroutine that called `Unshare` could be rescheduled onto a different, un-unshared OS thread before the fork, silently discarding every namespace change. | All three: identical CLI, output lines, and exit codes, so one `verify.lua` covers them. @@ -77,20 +77,20 @@ covers them. - **The root cgroup is exempt from the "no internal process" rule.** A non-root cgroup can't hold processes directly *and* delegate a controller - to children at the same time — but the root cgroup can, which is exactly + to children at the same time, but the root cgroup can, which is exactly how `containerize` enables `+memory +cpu` in `/sys/fs/cgroup/cgroup.subtree_control` without first relocating itself into a leaf. - **Cgroup membership is inherited across fork, not reset by exec.** `containerize` writes its *own* pid into the new cgroup before unsharing - or forking, so the pid-1 child — and CMD after it execs — are under the + or forking, so the pid-1 child (and CMD after it execs) are under the limits from their very first instruction. - **`memory.swap.max=0` makes the limit mean something.** Without it, a cgroup that hits `memory.max` may just swap and slow down instead of - triggering the OOM killer — on a host with swap enabled, the negative + triggering the OOM killer: on a host with swap enabled, the negative control would flake. `containerize` sets it (best-effort) alongside `memory.max`. - **PSI (`memory.pressure`) is a per-cgroup file**, not a single systemwide - number — reading `/memory.pressure` after the child exits shows the + number: reading `/memory.pressure` after the child exits shows the stall this specific cgroup experienced, independent of pressure anywhere else on the box. @@ -98,10 +98,10 @@ covers them. Each language directory has the standard `demo.sh`: -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary (`app`) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (`app`) - With env `TARGET` set, `run` deploys to that lab VM via - `scripts/lab/deploy-to-vm.sh`; set `SUDO=1` — `containerize` needs root for + `scripts/lab/deploy-to-vm.sh`; set `SUDO=1`: `containerize` needs root for `unshare(2)` and cgroup writes. The top-level `./demo.sh [cpp|go|rust|all|build]` dispatches per language. @@ -140,12 +140,12 @@ Cross-check from a second guest shell while a `containerize` run is live: `nsenter -t -p -m -u -n ps -ef` shows the contained process tree with its own pid 1 and hostname, from the host's side of the fence. -## A deploy-to-vm.sh gotcha this example works around +## Keeping CMD tokens simple for deploy-to-vm.sh `deploy-to-vm.sh` rebuilds the remote command line with a bare `$*`, which flattens argv and loses any quoting a single argument relied on. A `CMD` like -`bash -c 'a=$(head -c N /dev/zero | tr ...); sleep 2'` — one argument full of -spaces and shell metacharacters — arrives on the guest corrupted. Both +`bash -c 'a=$(head -c N /dev/zero | tr ...); sleep 2'` (one argument full of +spaces and shell metacharacters) arrives on the guest corrupted. Both `verify.lua` and the walkthrough above route around it the same way: keep every `CMD` token free of embedded spaces (`/usr/bin/echo hi`, `sh /tmp/hog.sh`), staging anything that needs real shell logic as a file on the @@ -161,14 +161,14 @@ guest first. `pmon: cgroup mem.pressure some=` line is present, and the run exits 0 with `pmon: child exited status=0`. 2. The guest's **real** hostname is fetched independently over `ssh` and - asserted **different** from `pmon-vm-check` — the fact pmon is even + asserted **different** from `pmon-vm-check`: the fact pmon is even allowed to compare the two is only interesting because they're both real, independently-obtained values. -3. **Negative control** — a real memory hog (`head -c 200000000 /dev/zero | +3. **Negative control**: a real memory hog (`head -c 200000000 /dev/zero | tr '\0' x` captured into a shell variable, staged as a guest script) run under `containerize --mem-max 67108864` (64 MiB, far below the ~190 MiB it tries to hold) exits **137**, prints `pmon: child killed signal=9 (KILL)`, - and never reaches its own trailing `echo should-not-print` — proof the + and never reaches its own trailing `echo should-not-print`: proof the cgroup limit terminated it mid-allocation rather than merely slowing it down. @@ -180,9 +180,9 @@ LIBVIRT_DEFAULT_URI=qemu:///system \ ``` **Why observe the effect, not a trace:** there is no syscall you can -`strace` from *outside* a process to confirm it believes it's pid 1 — that +`strace` from *outside* a process to confirm it believes it's pid 1: that belief is only visible in what the process itself reports, or in what happens to it. Pairing "reports pid 1" with an independently-fetched host -hostname, and pairing "cgroup configured" with "hog actually dies", is what +hostname, and pairing "cgroup configured" with "hog dies", is what turns three plausible-looking print statements into a demonstration that the namespaces and the cgroup limit are real. diff --git a/examples/32-namespaces-and-cgroups/cpp/src/main.cpp b/examples/32-namespaces-and-cgroups/cpp/src/main.cpp index 7b46ca6..b55f855 100644 --- a/examples/32-namespaces-and-cgroups/cpp/src/main.cpp +++ b/examples/32-namespaces-and-cgroups/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// pmon v6 — namespaces & cgroups (chapter 32). +// pmon v6: namespaces & cgroups (chapter 32). // // Subcommand: // pmon containerize [--hostname NAME] [--mem-max BYTES|max] @@ -8,13 +8,13 @@ // `containerize` builds the two primitives every real container runtime is // made of, by hand: // -// 1. NAMESPACES — unshare(2) with CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET| +// 1. NAMESPACES: unshare(2) with CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET| // CLONE_NEWPID puts the calling process into a fresh mount, UTS and // network namespace immediately, and arranges for the NEXT clone3(2) // child to become PID 1 of a fresh PID namespace (unshare(2) never -// moves the caller itself into a new PID namespace — only its future +// moves the caller itself into a new PID namespace: only its future // children). -// 2. CGROUP v2 LIMITS — a dedicated cgroup gets memory.max and cpu.max +// 2. CGROUP v2 LIMITS: a dedicated cgroup gets memory.max and cpu.max // written before the child is ever created, so the limits are already // live the instant CMD execs. // @@ -22,7 +22,7 @@ // no fork()/vfork() distinction to reason about, and it is the syscall the // chapter is actually teaching). RAII closes every fd; std::expected carries // every syscall failure; single-threaded throughout (mixing threads with -// unshare()/clone3() is its own footgun — see the Go build's comment for +// unshare()/clone3() is its own footgun: see the Go build's comment for // what goes wrong when a runtime does that under you). #include @@ -63,7 +63,7 @@ std::expected checked(long rc, std::string_view what) { return {}; } -// RAII owner for a file descriptor — closed exactly once on scope exit. +// RAII owner for a file descriptor: closed exactly once on scope exit. class Fd { public: explicit Fd(int fd) noexcept : fd_(fd) {} @@ -83,7 +83,7 @@ class Fd { }; // clone3(2)'s argument struct. We deliberately declare only the -// CLONE_ARGS_SIZE_VER0 prefix (through `tls`) — redefines +// CLONE_ARGS_SIZE_VER0 prefix (through `tls`): redefines // the CLONE_NEW* macros already pulled in by , so we avoid including // it and hand-roll the layout the syscall actually reads. struct clone_args_v0 { @@ -118,7 +118,7 @@ std::expected read_file(const std::string& path) { } // Whether `tokens` (a whitespace-separated list, as cgroupfs writes it) -// contains `word` as a whole token — "cpu" must not match inside "cpuset". +// contains `word` as a whole token: "cpu" must not match inside "cpuset". bool has_token(std::string_view tokens, std::string_view word) { std::size_t pos = 0; while (pos < tokens.size()) { @@ -132,7 +132,7 @@ bool has_token(std::string_view tokens, std::string_view word) { // The root cgroup is exempt from the "no internal process" constraint, so we // can enable controllers there even though it holds every process on the -// box — this is exactly how a fresh sibling cgroup gets memory/cpu control +// box: this is exactly how a fresh sibling cgroup gets memory/cpu control // without first relocating ourselves into a leaf. std::expected ensure_root_controllers() { auto have = read_file("/sys/fs/cgroup/cgroup.subtree_control"); diff --git a/examples/32-namespaces-and-cgroups/go/main.go b/examples/32-namespaces-and-cgroups/go/main.go index cd81b7b..e401404 100644 --- a/examples/32-namespaces-and-cgroups/go/main.go +++ b/examples/32-namespaces-and-cgroups/go/main.go @@ -1,4 +1,4 @@ -// pmon v6 — namespaces & cgroups (chapter 32). +// pmon v6: namespaces & cgroups (chapter 32). // // Subcommand: // @@ -6,31 +6,31 @@ // [--cpu-max "QUOTA PERIOD"|max] [--cgroup NAME] // -- CMD [ARGS...] // -// Go note — why this can't be the straight-line fork(2) the C++/Rust builds -// use: +// Go note: why this can't be the straight-line fork(2) the C++/Rust builds +// use. // // A Go program is multithreaded before main() runs, and unshare(2) changes -// only the CALLING THREAD's namespaces — the first attempt below called +// only the CALLING THREAD's namespaces; the first attempt below called // unix.Unshare directly on a runtime.LockOSThread()-pinned goroutine, then // re-exec'd the binary through os/exec so the fork would land on that same, // already-unshared thread. It built, and the pid/hostname proof lines were // real, but the memory-hog negative control failed on real hardware with -// "fork/exec: cannot allocate memory" (see the chapter's Errors section) — +// "fork/exec: cannot allocate memory" (see the chapter's Errors section); // a genuine kernel restriction: unshare(CLONE_NEWPID) lets the caller's // process create exactly ONE child in the fresh PID namespace before further // forks there return ENOMEM, and os/exec's own vfork/pidfd fast path // performs an internal probe fork on first use that silently consumes that // one shot. // -// The fix — and the one actually shipped here — is the same move Chapter +// The fix (and the one actually shipped here) is the same move Chapter // 14 made for capabilities: stop calling the namespace syscall directly in // live Go code, and instead describe it declaratively on SysProcAttr, so the -// runtime's own fork/exec trampoline (which never has this problem — it +// runtime's own fork/exec trampoline (which never has this problem; it // creates the namespaces AND the real child in the same clone(2)) applies // it. `Cloneflags` on the re-exec's SysProcAttr creates the child already // inside fresh mount/uts/net/pid namespaces, in one step, with no prior -// unshare() and no LockOSThread needed at all. The re-exec target — pmon -// itself, invoked with the hidden "__ns_init__" marker — is therefore where +// unshare() and no LockOSThread needed at all. The re-exec target (pmon +// itself, invoked with the hidden "__ns_init__" marker) is therefore where // the mount-private remount and sethostname now run: it is already living // inside the new namespaces the instant its code starts, since they were // established at clone(2) time and carried across its own execve. @@ -48,7 +48,7 @@ import ( "golang.org/x/sys/unix" ) -// Hidden re-exec marker — not a public subcommand, so it stays out of usage(). +// Hidden re-exec marker: not a public subcommand, so it stays out of usage(). const reexecMarker = "__ns_init__" // Names as printed in "killed signal= ()". A hand-rolled table @@ -72,7 +72,7 @@ func signalName(sig int) string { // ---- cgroup helpers -------------------------------------------------- // hasToken reports whether tokens (a whitespace-separated list, as -// cgroupfs writes it) contains word as a whole token — "cpu" must not +// cgroupfs writes it) contains word as a whole token; "cpu" must not // match inside "cpuset". func hasToken(tokens, word string) bool { for _, t := range strings.Fields(tokens) { @@ -85,7 +85,7 @@ func hasToken(tokens, word string) bool { // The root cgroup is exempt from the "no internal process" constraint, so // controllers can be enabled there even though it holds every process on -// the box — this is how a fresh sibling cgroup gets memory/cpu control +// the box; this is how a fresh sibling cgroup gets memory/cpu control // without first relocating ourselves into a leaf. func ensureRootControllers() error { data, err := os.ReadFile("/sys/fs/cgroup/cgroup.subtree_control") @@ -234,7 +234,7 @@ func cmdContainerize(args []string) int { c.Stdin, c.Stdout, c.Stderr = os.Stdin, os.Stdout, os.Stderr // Cloneflags asks the runtime's own fork/exec trampoline to create the // child already inside fresh mount/uts/net/pid namespaces, atomically, - // in the SAME clone(2) that creates the process — no separate unshare(2) + // in the SAME clone(2) that creates the process, no separate unshare(2) // on our side, and so none of the "one child per fresh PID namespace" // restriction that broke the direct-unshare attempt. c.SysProcAttr = &syscall.SysProcAttr{ @@ -282,10 +282,10 @@ func cmdContainerize(args []string) int { // // args: -- CMD [ARGS...]. Already running inside the fresh // mount/uts/net/pid namespaces (Cloneflags put THIS process there at -// clone(2) time, before this code's own execve even happened) — but the +// clone(2) time, before this code's own execve even happened), but the // mount-private remount and sethostname are still ours to do, since the // parent process never entered these namespaces itself. Then prints the -// proof lines and execs into the real CMD, replacing itself — CMD keeps +// proof lines and execs into the real CMD, replacing itself; CMD keeps // PID 1. func cmdNsInit(args []string) int { if len(args) < 2 || args[1] != "--" { diff --git a/examples/32-namespaces-and-cgroups/rust/src/main.rs b/examples/32-namespaces-and-cgroups/rust/src/main.rs index 67f1e82..7590b87 100644 --- a/examples/32-namespaces-and-cgroups/rust/src/main.rs +++ b/examples/32-namespaces-and-cgroups/rust/src/main.rs @@ -1,4 +1,4 @@ -//! pmon v6 — namespaces & cgroups (chapter 32). +//! pmon v6: namespaces & cgroups (chapter 32). //! //! Subcommand: //! pmon containerize [--hostname NAME] [--mem-max BYTES|max] @@ -8,20 +8,20 @@ //! Same two primitives as the C++ build, through `nix` instead of raw libc //! calls: //! -//! 1. NAMESPACES — unshare(2) with CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET| +//! 1. NAMESPACES: unshare(2) with CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWNET| //! CLONE_NEWPID puts the caller into a fresh mount, UTS and network //! namespace immediately, and arranges for the next fork(2) child to //! become PID 1 of a fresh PID namespace (unshare(2) never moves the -//! caller itself into a new PID namespace — only its future children). -//! 2. CGROUP v2 LIMITS — a dedicated cgroup gets memory.max and cpu.max +//! caller itself into a new PID namespace, only its future children). +//! 2. CGROUP v2 LIMITS: a dedicated cgroup gets memory.max and cpu.max //! written before the child is ever created, so the limits are already //! live the instant CMD execs. //! //! `fork(2)` (not clone3) here: nix's `fork` gives the same "no code runs //! between clone and exec except what we ourselves write" shape as the //! C++ build's `clone3(flags=0)`. Its safety contract is the same as libc -//! fork's — the child may only call async-signal-safe operations until it -//! execs — which is exactly what the child below does: read its own pid, +//! fork's: the child may only call async-signal-safe operations until it +//! execs, which is exactly what the child below does: read its own pid, //! read the hostname, print, exec. use std::ffi::CString; @@ -54,14 +54,14 @@ fn signal_name(sig: i32) -> String { // ---- cgroup helpers -------------------------------------------------- /// Whether `tokens` (a whitespace-separated list, as cgroupfs writes it) -/// contains `word` as a whole token — "cpu" must not match inside "cpuset". +/// contains `word` as a whole token: "cpu" must not match inside "cpuset". fn has_token(tokens: &str, word: &str) -> bool { tokens.split_whitespace().any(|t| t == word) } /// The root cgroup is exempt from the "no internal process" constraint, so /// controllers can be enabled there even though it holds every process on -/// the box — this is how a fresh sibling cgroup gets memory/cpu control +/// the box: this is how a fresh sibling cgroup gets memory/cpu control /// without first relocating ourselves into a leaf. fn ensure_root_controllers() -> Result<(), String> { let have = fs::read_to_string("/sys/fs/cgroup/cgroup.subtree_control") @@ -230,7 +230,7 @@ fn cmd_containerize(args: &[String]) -> i32 { // SAFETY: the child performs only async-signal-safe operations // (getpid, gethostname, writes to already-open stdio fds, execvp) - // before it either execs or _exits — it never returns up through this + // before it either execs or _exits: it never returns up through this // stack frame into arbitrary Rust code, satisfying fork(2)'s contract. match unsafe { fork() } { Ok(ForkResult::Child) => { diff --git a/examples/32-namespaces-and-cgroups/verify.lua b/examples/32-namespaces-and-cgroups/verify.lua index ce5aaba..67fb5a2 100644 --- a/examples/32-namespaces-and-cgroups/verify.lua +++ b/examples/32-namespaces-and-cgroups/verify.lua @@ -1,13 +1,13 @@ --- Verify pmon v6 (namespaces & cgroups) — a VM example. Behavioral assertions: +-- Verify pmon v6 (namespaces & cgroups), a VM example. Behavioral assertions: -- --- 1. PID NAMESPACE — `containerize -- /usr/bin/echo cmd-ran` must print +-- 1. PID NAMESPACE: `containerize -- /usr/bin/echo cmd-ran` must print -- "pmon: child sees pid 1": the child only sees itself as pid 1 because -- it is the first (and only) task in a fresh PID namespace. --- 2. UTS NAMESPACE — the same run must print "pmon: hostname=" with +-- 2. UTS NAMESPACE: the same run must print "pmon: hostname=" with -- the hostname we asked for, and that name must be DIFFERENT from the --- guest's real hostname (fetched independently over ssh) — proof the +-- guest's real hostname (fetched independently over ssh): proof the -- change landed in a private UTS namespace, not the host's. --- 3. CGROUP MEMORY LIMIT (negative control) — a real memory hog started +-- 3. CGROUP MEMORY LIMIT (negative control): a real memory hog started -- under `--mem-max` far below what it tries to allocate must be -- OOM-killed (SIGKILL, exit 137), not merely slowed down. Without the -- cgroup limit doing real work, the same hog would run to completion; @@ -20,7 +20,7 @@ -- transit. We route around that: the pid/hostname check uses a CMD of -- plain, space-free tokens (`/usr/bin/echo cmd-ran`), and the memory hog is -- staged as a real script file on the guest first, then invoked as --- `sh /tmp/pmon-hog.sh` — two more plain tokens. +-- `sh /tmp/pmon-hog.sh`, two more plain tokens. local script_dir = arg[0]:match("(.*/)") or "./" local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") @@ -86,10 +86,10 @@ else end -- --------------------------------------------------------------------------- --- 3: cgroup memory.max negative control — a real hog gets OOM-killed. +-- 3: cgroup memory.max negative control, a real hog gets OOM-killed. -- --------------------------------------------------------------------------- --- Stage the hog as a file (not a `bash -c '...'` argument — see file header) +-- Stage the hog as a file (not a `bash -c '...'` argument, see file header) -- so deploy-to-vm.sh's flattening never sees its internal spaces/pipe/`;`. local hog_path = "/tmp/pmon-hog.sh" local stage = checks.run(string.format( @@ -104,7 +104,7 @@ checks.expect_exit(stage, 0, lang .. ": stage memory-hog script on guest") -- 64 MiB ceiling, no swap (containerize disables memory.swap.max itself): -- comfortably above any of the three runtimes' own baseline RSS, but far --- below the ~190 MiB the hog tries to hold, so the hog — not pmon itself — +-- below the ~190 MiB the hog tries to hold, so the hog (not pmon itself) -- is what the cgroup OOM killer picks. local neg = checks.run(string.format( "SUDO=1 ./demo.sh %s run containerize --mem-max 67108864 --cgroup pmon-hog-%s -- sh %s", diff --git a/examples/33-seccomp-and-landlock/README.md b/examples/33-seccomp-and-landlock/README.md index 545e187..5c745f1 100644 --- a/examples/33-seccomp-and-landlock/README.md +++ b/examples/33-seccomp-and-landlock/README.md @@ -1,15 +1,15 @@ -# 33 — seccomp + Landlock (`fwatch` v3) +# 33: seccomp + Landlock (`fwatch` v3) `fwatch` gets sandboxed. `watch --sandbox DIR` applies two independent kernel access-control layers before the epoll/inotify watch loop from -[chapter 09](../09-event-loops-epoll-inotify) ever runs: +[the event-loop chapter](../09-event-loops-epoll-inotify) ever runs: | Layer | What it restricts | Mechanism | |---|---|---| -| **Landlock** | filesystem **reads**, to `DIR` only | an unprivileged, self-imposed ruleset the kernel enforces against this process and every descendant — it cannot be lifted short of exec-ing something that never asked for it | +| **Landlock** | filesystem **reads**, to `DIR` only | an unprivileged, self-imposed ruleset the kernel enforces against this process and every descendant; it cannot be lifted short of exec-ing something that never asked for it | | **seccomp** | which **syscalls** may run at all | a syscall allowlist covering only what the watch loop needs; anything else returns `EPERM` | -Two `probe` subcommands are negative controls that apply ONE layer in +Two `probe` subcommands are negative controls that apply *one* layer in isolation and prove it denies exactly what it claims to: ``` @@ -32,13 +32,13 @@ the sandbox. The Go implementation avoids this with `runtime.LockOSThread()` (pins the sandbox-installing goroutine to a stable OS thread) plus the `LANDLOCK_RESTRICT_SELF_TSYNC` and `SECCOMP_FILTER_FLAG_TSYNC` flags, which apply the new state to **every** thread already running in the process, not -just the caller's — confirmed locally: a goroutine deliberately scheduled -after the sandbox is installed still gets EACCES/EPERM. +just the caller's. Confirmed locally: a goroutine scheduled after the +sandbox is installed still gets EACCES/EPERM. The Landlock ABI has no libc/x/sys wrapper in any of the three languages; every version probes it at runtime with `landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)` rather -than assuming a version — the printed `ABI=` is a live kernel query, not a +than assuming a version: the printed `ABI=` is a live kernel query, not a constant. ## CLI @@ -53,15 +53,16 @@ usage: fwatch probe --forbidden-syscall negative control: socket(2) under a seccomp allowlist that omits it ``` -- `snapshot` / `diff` / unsandboxed `watch` are unchanged from chapter 09. -- `watch --sandbox DIR` — applies Landlock (read+watch access restricted to +- `snapshot` / `diff` / unsandboxed `watch` are unchanged from the event-loop + chapter. +- `watch --sandbox DIR`: applies Landlock (read+watch access restricted to `DIR`) then the seccomp allowlist, prints both status lines to stderr, then runs the identical watch loop. Events, debounce, and the - `(timeout)`/`(signal)` exit lines are exactly as in chapter 09 — the - sandbox is invisible to the loop's own logic. -- `probe --sandbox DIR --outside PATH` — restricts Landlock to `DIR`, then + `(timeout)`/`(signal)` exit lines are exactly as in the event-loop chapter; + the sandbox is invisible to the loop's own logic. +- `probe --sandbox DIR --outside PATH`: restricts Landlock to `DIR`, then attempts to open `PATH` (expected to be outside `DIR`) for reading. -- `probe --forbidden-syscall` — installs the same seccomp allowlist (which +- `probe --forbidden-syscall`: installs the same seccomp allowlist (which never includes `socket(2)`), then calls `socket(AF_INET, SOCK_STREAM, 0)`. ### Exit codes @@ -71,8 +72,8 @@ usage: fwatch | `0` | success (snapshot/diff/watch, including a sandboxed watch's clean timeout/signal exit) | | `1` | runtime error, or a probe hit something other than the expected denial | | `2` | usage error | -| `20` | **probe confirmed denial** — the sandbox blocked what it should (the passing case for a negative control) | -| `21` | **probe was NOT denied** — the sandbox failed to block what it should (a real sandbox bug) | +| `20` | **probe confirmed denial**: the sandbox blocked what it should (the passing case for a negative control) | +| `21` | **probe was *not* denied**: the sandbox failed to block what it should (a real sandbox bug) | ## Try it @@ -88,7 +89,7 @@ wait # landlock/seccomp status lines, then the same event stream as ch09 ./cpp/build/release/app probe --sandbox /tmp/fw-sandboxed --outside /etc/hostname; echo "exit=$?" # EACCES, exit 20 ``` -Landlock and seccomp are both **unprivileged** self-sandboxing mechanisms — +Landlock and seccomp are both **unprivileged** self-sandboxing mechanisms: no root is needed to run any of the above on a Linux 5.13+ kernel with Landlock enabled. @@ -100,10 +101,10 @@ runs the driver as root (the book's convention for every namespace/cgroup/seccomp/Landlock demo, even though these two mechanisms don't themselves require it). The driver exercises, in one ssh round trip: -1. `probe --forbidden-syscall` — asserts `EPERM` and exit `20`. -2. `probe --sandbox ... --outside ...` — asserts `EACCES` and exit `20`. +1. `probe --forbidden-syscall`: asserts `EPERM` and exit `20`. +2. `probe --sandbox ... --outside ...`: asserts `EACCES` and exit `20`. 3. `watch --sandbox ...` with a file created/modified/deleted inside the - tree while it runs — asserts an `event: ... inside.txt` line, the + tree while it runs: asserts an `event: ... inside.txt` line, the `(timeout)` exit line, and exit `0` (the positive control: sandboxing doesn't break the watcher). diff --git a/examples/33-seccomp-and-landlock/cpp/src/main.cpp b/examples/33-seccomp-and-landlock/cpp/src/main.cpp index 9431a0f..d67b20f 100644 --- a/examples/33-seccomp-and-landlock/cpp/src/main.cpp +++ b/examples/33-seccomp-and-landlock/cpp/src/main.cpp @@ -1,14 +1,14 @@ -// fwatch v3 — the ch09 watcher, now sandboxed. +// fwatch v3: the ch09 watcher, now sandboxed. // // `watch --sandbox DIR` applies two independent kernel access-control layers // before the epoll loop ever runs: // -// Landlock — a ruleset that restricts filesystem READS to DIR and nothing +// Landlock: a ruleset that restricts filesystem READS to DIR and nothing // else. It is an unprivileged, self-imposed sandbox: the kernel // enforces it against this process and every descendant, and it // cannot be lifted short of exec-ing something that never asked // for it in the first place. -// seccomp — a syscall allowlist (libseccomp) covering only the syscalls +// seccomp: a syscall allowlist (libseccomp) covering only the syscalls // the watch loop actually issues (epoll/inotify/timerfd/ // signalfd plumbing, read/write/close, the allocator). Anything // else returns EPERM instead of running. @@ -21,7 +21,7 @@ // Landlock ABI has no libc wrapper; every call here goes through the raw // syscall(2) numbers in / . The ABI is // probed at runtime (LANDLOCK_CREATE_RULESET_VERSION) rather than assumed, in -// line with "don't hardcode the ABI" — a program built against a newer +// line with "don't hardcode the ABI": a program built against a newer // kernel's headers must still degrade sanely on an older one. #include @@ -72,7 +72,7 @@ constexpr auto kDebounce = milliseconds{100}; } // --------------------------------------------------------------------------- -// RAII file descriptor — the only owner of every fd in this program. +// RAII file descriptor: the only owner of every fd in this program. // --------------------------------------------------------------------------- class Fd { @@ -111,7 +111,7 @@ class Fd { } // --------------------------------------------------------------------------- -// Landlock — raw syscalls, no libc wrapper exists. +// Landlock: raw syscalls, no libc wrapper exists. // --------------------------------------------------------------------------- [[nodiscard]] long sys_landlock_create_ruleset(const landlock_ruleset_attr* attr, std::size_t size, @@ -138,8 +138,8 @@ class Fd { // Create a ruleset that handles READ_FILE + READ_DIR (available since ABI // v1), add one path-beneath rule granting exactly those rights under `dir`, -// then restrict_self. On success the calling process — and everything it -// execs from here on — can only read inside `dir`. +// then restrict_self. On success the calling process, and everything it +// execs from here on, can only read inside `dir`. [[nodiscard]] std::expected apply_landlock(const std::string& dir) { constexpr std::uint64_t access = LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR; landlock_ruleset_attr attr{.handled_access_fs = access, .handled_access_net = 0, .scoped = 0}; @@ -171,7 +171,7 @@ class Fd { } // --------------------------------------------------------------------------- -// seccomp — libseccomp allowlist covering exactly what the watch loop needs. +// seccomp: libseccomp allowlist covering exactly what the watch loop needs. // --------------------------------------------------------------------------- // Syscalls the epoll/inotify/timerfd/signalfd watch loop issues, empirically @@ -315,7 +315,7 @@ int cmd_diff(const std::string& old_path, const std::string& new_path) { } // --------------------------------------------------------------------------- -// v1: watch — one epoll loop, three fds, zero threads beyond main. +// v1: watch: one epoll loop, three fds, zero threads beyond main. // --------------------------------------------------------------------------- enum class Kind : std::uint8_t { created, modified, deleted }; @@ -551,7 +551,7 @@ int cmd_watch(const std::string& dir, int timeout_ms, bool sandbox) { } // --------------------------------------------------------------------------- -// probes — negative controls, one Landlock, one seccomp, exercised alone. +// probes: negative controls, one Landlock, one seccomp, exercised alone. // --------------------------------------------------------------------------- // Exit codes for the probes are distinct from the ordinary 0/1/2 contract so @@ -559,7 +559,7 @@ int cmd_watch(const std::string& dir, int timeout_ms, bool sandbox) { // control) apart from "ran into some other error" or "was NOT denied" (a // sandbox bug). constexpr int kProbeDenied = 20; // Landlock/seccomp did deny it, as expected -constexpr int kProbeNotDenied = 21; // sandbox failed to deny it — a real bug +constexpr int kProbeNotDenied = 21; // sandbox failed to deny it: a real bug int cmd_probe_outside(const std::string& sandbox_dir, const std::string& outside_path) { const int abi = landlock_abi(); diff --git a/examples/33-seccomp-and-landlock/go/main.go b/examples/33-seccomp-and-landlock/go/main.go index 597e932..b70f2c1 100644 --- a/examples/33-seccomp-and-landlock/go/main.go +++ b/examples/33-seccomp-and-landlock/go/main.go @@ -1,19 +1,19 @@ -// fwatch v3 — the ch09 watcher, now sandboxed. +// fwatch v3: the ch09 watcher, now sandboxed. // // `watch --sandbox DIR` applies two independent kernel access-control layers // before the watch loop starts: // -// Landlock — a ruleset restricting filesystem READS to DIR. Unprivileged, -// self-imposed, enforced by the kernel against this process and -// every descendant. -// seccomp — a syscall allowlist (a hand-assembled classic-BPF program, -// installed via the raw seccomp(2) syscall) covering only what -// the watch loop needs. Everything else returns EPERM. +// Landlock: a ruleset restricting filesystem READS to DIR. Unprivileged, +// self-imposed, enforced by the kernel against this process and +// every descendant. +// seccomp: a syscall allowlist (a hand-assembled classic-BPF program, +// installed via the raw seccomp(2) syscall) covering only what +// the watch loop needs. Everything else returns EPERM. // // GO RUNTIME CAVEAT: both mechanisms are, by default, per-THREAD kernel // state. The Go runtime schedules goroutines across OS threads it creates // and destroys on its own schedule (GC workers, sysmon, threads parked in -// blocking syscalls) — applying either sandbox to only the calling thread +// blocking syscalls), applying either sandbox to only the calling thread // would leave those other threads unrestricted, and a goroutine the runtime // later migrates onto one of them would silently run outside the sandbox. // This program avoids that two ways: @@ -23,7 +23,7 @@ // changes out from under it. // - Both landlock_restrict_self and the seccomp(2) syscall are called with // their respective TSYNC flag, which applies the new state to every -// thread in the process atomically — not just the calling one — so +// thread in the process atomically, not just the calling one, so // threads that already exist (and the ones any of them later clone) are // covered too. // @@ -53,7 +53,7 @@ import ( const debounce = 100 * time.Millisecond // --------------------------------------------------------------------------- -// Landlock — raw syscalls (x/sys/unix has the numbers, not wrappers). +// Landlock: raw syscalls (x/sys/unix has the numbers, not wrappers). // --------------------------------------------------------------------------- const ( @@ -138,7 +138,7 @@ func applyLandlock(dir string) error { } // --------------------------------------------------------------------------- -// seccomp — a hand-assembled classic-BPF allowlist, installed via seccomp(2). +// seccomp: a hand-assembled classic-BPF allowlist, installed via seccomp(2). // --------------------------------------------------------------------------- const ( @@ -216,7 +216,7 @@ func installSeccomp(allowed []uint32, denyErrno uint32) (int, error) { // Syscalls the watch loop issues, empirically confirmed with `strace -f` // across a full watch session (goroutine reading the inotify fd through the -// netpoller, signal.Notify, time.After/time.Timer) — plus the small, +// netpoller, signal.Notify, time.After/time.Timer), plus the small, // well-known set the Go runtime itself needs for scheduling, GC, and stack // management on any goroutine (futex, mmap/munmap/mprotect, clone, sigaltstack, // rt_sigaction). Anything else returns EPERM once the filter loads. @@ -320,7 +320,7 @@ func cmdDiff(oldPath, newPath string) error { } // --------------------------------------------------------------------------- -// v1: watch — goroutine + channels; the runtime owns the epoll loop. +// v1 (watch): goroutine + channels; the runtime owns the epoll loop. // --------------------------------------------------------------------------- type fsEvent struct { @@ -494,7 +494,7 @@ func cmdWatch(dir string, timeoutMs int, sandbox bool) error { } // --------------------------------------------------------------------------- -// probes — negative controls, one Landlock, one seccomp, exercised alone. +// probes: negative controls, one Landlock, one seccomp, exercised alone. // --------------------------------------------------------------------------- // Exit codes distinct from the ordinary 0/1/2 contract: a verifier can tell diff --git a/examples/33-seccomp-and-landlock/rust/src/main.rs b/examples/33-seccomp-and-landlock/rust/src/main.rs index 8a5e083..34c46e9 100644 --- a/examples/33-seccomp-and-landlock/rust/src/main.rs +++ b/examples/33-seccomp-and-landlock/rust/src/main.rs @@ -1,21 +1,21 @@ -// fwatch v3 — the ch09 watcher, now sandboxed. +// fwatch v3: the ch09 watcher, now sandboxed. // // `watch --sandbox DIR` applies two independent kernel access-control layers // before the epoll loop ever runs: // -// Landlock — a ruleset (the `landlock` crate) restricting filesystem READS -// to DIR. Unprivileged, self-imposed, enforced by the kernel -// against this process and every descendant. -// seccomp — a syscall allowlist (the `seccompiler` crate — Firecracker's -// BPF compiler) covering only what the watch loop needs. -// Everything else returns EPERM. +// Landlock: a ruleset (the `landlock` crate) restricting filesystem READS +// to DIR. Unprivileged, self-imposed, enforced by the kernel +// against this process and every descendant. +// seccomp: a syscall allowlist (the `seccompiler` crate, Firecracker's +// BPF compiler) covering only what the watch loop needs. +// Everything else returns EPERM. // // Landlock's own syscall numbers have no libc wrapper; probing the running // kernel's ABI (`landlock_abi()` below) goes through the raw syscall(2) // numbers directly, mirroring the C++ and Go versions of this example. Those // numbers are part of the kernel's frozen syscall table (once assigned they // never change, unlike a struct field offset), so hardcoding them per -// architecture is the same thing glibc's own does — but the +// architecture is the same thing glibc's own does, but the // ABI *version* returned by the call is always probed at runtime, never // assumed. @@ -38,7 +38,7 @@ use seccompiler::{BpfProgram, SeccompAction, SeccompFilter, TargetArch}; const DEBOUNCE: Duration = Duration::from_millis(100); // --------------------------------------------------------------------------- -// Landlock ABI probe — raw syscall, no libc/crate wrapper exists for this. +// Landlock ABI probe: raw syscall, no libc/crate wrapper exists for this. // --------------------------------------------------------------------------- #[cfg(target_arch = "x86_64")] @@ -78,7 +78,7 @@ fn apply_landlock(dir: &str) -> Result<()> { } // --------------------------------------------------------------------------- -// seccomp — a `seccompiler` allowlist covering exactly what the watch loop +// seccomp: a `seccompiler` allowlist covering exactly what the watch loop // needs (empirically confirmed with `strace -f` across a full watch // session: create/modify/delete + debounce flush + timeout/signal exit). // Anything else returns EPERM once the filter loads. @@ -176,7 +176,7 @@ fn cmd_diff(old_path: &str, new_path: &str) -> Result<()> { } // --------------------------------------------------------------------------- -// v1: watch — one epoll loop, three fds, zero threads beyond main. +// v1 (watch): one epoll loop, three fds, zero threads beyond main. // --------------------------------------------------------------------------- #[derive(Clone, Copy, PartialEq, Eq)] @@ -289,7 +289,7 @@ fn run_watch_loop(dir: &str, timeout_ms: u64) -> Result<()> { loop { // One timer covers both jobs: it is always armed to whichever comes - // first — the overall timeout or the earliest per-path debounce. + // first: the overall timeout or the earliest per-path debounce. let next = pending.values().map(|p| p.due).min().map_or(deadline, |d| d.min(deadline)); arm_timer(&timer, next.saturating_duration_since(Instant::now()))?; @@ -361,7 +361,7 @@ fn cmd_watch(dir: &str, timeout_ms: u64, sandbox: bool) -> Result<()> { } // --------------------------------------------------------------------------- -// probes — negative controls, one Landlock, one seccomp, exercised alone. +// probes: negative controls, one Landlock, one seccomp, exercised alone. // --------------------------------------------------------------------------- // Exit codes distinct from the ordinary 0/1/2 contract: a verifier can tell diff --git a/examples/33-seccomp-and-landlock/verify.lua b/examples/33-seccomp-and-landlock/verify.lua index 530f7fa..8ab1e53 100644 --- a/examples/33-seccomp-and-landlock/verify.lua +++ b/examples/33-seccomp-and-landlock/verify.lua @@ -1,24 +1,24 @@ --- Verify fwatch v3 (seccomp + Landlock) — a VM example. Behavioral +-- Verify fwatch v3 (seccomp + Landlock), a VM example. Behavioral -- assertions, driven by run-sandbox-checks.sh staged and run as root -- (SUDO=1) on the lab guest: -- --- 1. NEGATIVE CONTROL — `probe --forbidden-syscall` installs a seccomp +-- 1. NEGATIVE CONTROL: `probe --forbidden-syscall` installs a seccomp -- allowlist that omits socket(2), then calls it. The bare exit code -- alone would prove nothing (a program that always exits nonzero would -- "pass"); the assertion is that socket(2) specifically returned EPERM, -- confirmed both by the process's own exit code (20, this program's -- "confirmed denied" convention) AND the printed errno text. --- 2. NEGATIVE CONTROL — `probe --sandbox DIR --outside PATH` restricts +-- 2. NEGATIVE CONTROL: `probe --sandbox DIR --outside PATH` restricts -- Landlock to DIR, then opens a file under a DIFFERENT directory. -- Same shape: exit 20 plus an explicit EACCES line. --- 3. POSITIVE CONTROL — `watch --sandbox DIR` with both layers applied +-- 3. POSITIVE CONTROL: `watch --sandbox DIR` with both layers applied -- still sees create/modify/delete events for a file created INSIDE -- DIR and exits 0 on timeout. This is what makes 1 and 2 meaningful: -- the sandbox denies specific things without breaking the watcher's -- actual job. -- -- Runs under the runner's vm mode (TARGET=systems-target, deploy via a --- custom root-run driver script — deploy-to-vm.sh only shells out a single +-- custom root-run driver script: deploy-to-vm.sh only shells out a single -- binary+args, and this needs mkdir/rm scaffolding around three invocations -- of the same binary in one ssh round trip, same shape as example 30's -- observe.sh). diff --git a/examples/34-our-programs-in-containers/README.md b/examples/34-our-programs-in-containers/README.md index 1668207..1d7bafc 100644 --- a/examples/34-our-programs-in-containers/README.md +++ b/examples/34-our-programs-in-containers/README.md @@ -1,7 +1,7 @@ -# 34 — our programs in containers +# 34: our programs in containers -`app`, a tiny PID-1 container entrypoint, implemented three times — C++23, -Go, and Rust — each built into its own multi-stage **Containerfile** (UBI10 +`app`, a tiny PID-1 container entrypoint, implemented three times, in C++23, +Go, and Rust, each built into its own multi-stage **Containerfile** (UBI10 builder → UBI10-micro runtime) and run under **rootless podman**. It demonstrates three things a program running as a container's entrypoint has to get right that a normal program never has to think about: @@ -11,17 +11,17 @@ usage: app serve|naive # C++, Go, Rust usage: app serve|naive|worker|job # Go, Rust (self-reexec children; see below) ``` -1. **pid-1 signal duties** — `serve` reaps every child (so short-lived jobs +1. **pid-1 signal duties**: `serve` reaps every child (so short-lived jobs never pile up as zombies) and forwards `SIGTERM` to the worker it supervises before exiting. `naive` installs no signal handling at all, the mistake this example exists to make visible. -2. **container-aware resource detection** — `serve`/`naive` read their own +2. **container-aware resource detection**: `serve`/`naive` read their own cgroup v2 `cpu.max`/`memory.max` and print `container: cpu.max= effective_parallelism= mem.max=`. Go's `GOMAXPROCS` and Rust's `available_parallelism()` are cgroup-aware; C++'s - `hardware_concurrency()` is not — it reports the **host's** cpu count + `hardware_concurrency()` is not: it reports the **host's** cpu count however small `--cpus` is set. That mismatch is the trap. -3. **debugging across namespaces** — `podman exec` and a rootless +3. **debugging across namespaces**: `podman exec` and a rootless `podman unshare nsenter` from the host both reach the same PID 1 inside the running container. @@ -64,18 +64,18 @@ resulting binary needs only `libc`/`libgcc_s`, both already in | stdout | `app: naive heartbeat tick=` | `naive`, every 1s, forever | | stdout | `app: shutting down (SIGTERM\|SIGINT)` | `serve` only, right before it exits 0 | -`naive` never prints a shutdown line — that absence is itself the evidence +`naive` never prints a shutdown line: that absence is itself the evidence the example verifies. `cpu.max` is printed as `max` (unconstrained) or `/` (derived from the cgroup file's raw `" "` content); `mem.max` is printed exactly as the cgroup reports it (`max` or a byte count). -## The trap, reproduced for real +## The trap, confirmed Run with `podman run --cpus=2 --memory=128m`, all three correctly report the container's `cpu.max=200000/100000` and `mem.max=134217728` (not the host's -`max`/`max`) — but `effective_parallelism` tells a different story per +`max`/`max`), but `effective_parallelism` tells a different story per language, captured from a real run on a 16-cpu host: ``` @@ -88,43 +88,44 @@ C++'s `std::thread::hardware_concurrency()` is `sysconf(_SC_NPROCESSORS_ONLN)` under the hood: it has no idea a cgroup exists. Go's `GOMAXPROCS` (default container-aware as of the `go 1.26` language version) and Rust's `available_parallelism()` both divide the cgroup's cpu quota by its period -and report `2` — the number that is actually true. +and report `2`, the number that matches the actual constraint. ## Per-language mechanics -- **C++23** (`cpp/src/main.cpp`) — raw `fork(2)` for both the worker and each +- **C++23** (`cpp/src/main.cpp`): raw `fork(2)` for both the worker and each job (no exec: they are just different functions in the same binary). This - is the one place a subtle, very real bug lived during development: the - forked worker **inherits the parent's blocked signal mask** + is the one place where signal-mask inheritance needs explicit handling: + the forked worker **inherits the parent's blocked signal mask** (`sigprocmask(SIG_BLOCK, ...)` for `SIGTERM`/`SIGINT`/`SIGCHLD`, set up so - `serve` can receive them via `signalfd(2)`). Left alone, a forwarded + `serve` can receive them via `signalfd(2)`). Left unmasked, a forwarded `SIGTERM` to the worker would just sit *pending* forever instead of - running its default (terminate) action — the worker would be unkillable. - The fix is one `sigprocmask(SIG_SETMASK, &empty, ...)` in the child right - after `fork()`, the same principle as pmon's (ch. 12) reset-before-`execvp`, - just with no `exec` to do it for you. `reap_all()` drains - `waitpid(-1, WNOHANG)` from the ordinary `poll(2)` loop (never from signal - context) whenever the `signalfd` reports `SIGCHLD`. A `std::jthread` (with - a `stop_token`, RAII-joined) drives the periodic job spawns. -- **Go 1.26** (`go/main.go`) — no `fork(2)`-without-exec (Go's runtime is not + running its default (terminate) action, leaving the worker unkillable. + One `sigprocmask(SIG_SETMASK, &empty, ...)` in the child right + after `fork()` resets it, the same principle as the signal-handling + chapter's `pmon` reset-before-`execvp`, just with no `exec` to do it for + you. `reap_all()` drains `waitpid(-1, WNOHANG)` from the ordinary + `poll(2)` loop (never from signal context) whenever the `signalfd` + reports `SIGCHLD`. A `std::jthread` (with a `stop_token`, RAII-joined) + drives the periodic job spawns. +- **Go 1.26** (`go/main.go`): no `fork(2)`-without-exec (Go's runtime is not fork-safe once other goroutines/threads exist), so the worker and each job are separate processes started by **re-executing the same binary** (`os.Executable()` + `exec.Command(self, "worker"|"job", ...)`). No `SIGCHLD`/`signalfd` anywhere: `Cmd.Wait()` calls `wait4(2)` for you, so one - goroutine parked in `Wait()` per spawned child *is* the reap loop. A real, - verified surprise from building this: a Go binary that never calls - `signal.Notify` still **dies immediately** on `SIGTERM`, even as PID 1 — + goroutine parked in `Wait()` per spawned child *is* the reap loop. A + notable behavior from building this: a Go binary that never calls + `signal.Notify` still **dies immediately** on `SIGTERM`, even as PID 1: Go's runtime installs its own handler for fatal signals like `SIGTERM` at startup regardless of user code (see `runtime`'s internal signal table), so it never has an unclaimed `SIG_DFL` disposition for the kernel's PID-1 special case to mask out. `naive` in Go is therefore not "unresponsive" the - way C++'s/Rust's are — it is abrupt instead: no reaping, no forwarding, no + way C++'s/Rust's are; it is abrupt instead: no reaping, no forwarding, no shutdown message, just an instant, uncontrolled death. -- **Rust edition 2024** (`rust/src/main.rs`) — same self-reexec approach as +- **Rust edition 2024** (`rust/src/main.rs`): same self-reexec approach as Go, via `std::env::current_exe()` + `std::process::Command`, for the same reason (no fork-without-exec once threads exist). `signal_hook::iterator:: Signals::new([SIGTERM, SIGINT])` turns both into a blocking iterator on the - main thread — no polling, no manual mask bookkeeping, and (unlike the + main thread: no polling, no manual mask bookkeeping, and (unlike the `sigprocmask`-based approach) nothing to reset before `exec`, since `signal_hook` only installs handlers and never blocks the signals it watches. Reaping is `std::thread::spawn` + `Child::wait()` per job, exactly @@ -158,39 +159,39 @@ podman rm -f demo34; podman rmi -f lsp34-go ``` `nsenter` directly (without `podman unshare`) fails rootless with -`Operation not permitted` — the container's namespaces live inside podman's +`Operation not permitted`: the container's namespaces live inside podman's own unprivileged user namespace, and `podman unshare` is what puts you inside it first. ## Verification (behavioral) `verify.lua` requires podman on the host (`skip()`, exit 77, if it is -missing) and, for every language, actually builds the image and drives real +missing) and, for every language, builds the image and drives real containers rather than trusting exit codes: -- **Resource detection under real constraints** — `podman run --cpus=2 +- **Resource detection under real constraints**: `podman run --cpus=2 --memory=128m`, then the container's own logs must show the *constrained* - `cpu.max=200000/100000` and `mem.max=134217728` — never the host's `max`. + `cpu.max=200000/100000` and `mem.max=134217728`, never the host's `max`. C++'s `effective_parallelism` must equal the **host's** live `nproc` (the - trap, reproduced); Go's and Rust's must equal `2` (the cgroup quota). The + trap, confirmed); Go's and Rust's must equal `2` (the cgroup quota). The host must report `nproc >= 3` or the test skips, since otherwise C++'s "wrong" number would coincidentally equal the "right" one. -- **PID-1 identity and reaping** — `app: pid=1 ppid=0` in the logs; several +- **PID-1 identity and reaping**: `app: pid=1 ppid=0` in the logs; several `app: reaped pid=... status=0` lines with no pile-up; a `podman exec` zombie scan of `/proc/[0-9]*/stat` reports `zombies=0` while jobs are being spawned and reaped. -- **Cross-namespace debugging** — `podman exec ... cat /proc/1/status` names +- **Cross-namespace debugging**: `podman exec ... cat /proc/1/status` names the process `app`; a rootless `podman unshare nsenter --target --pid --mount -- readlink /proc/1/exe` from the host resolves to the same `/usr/local/bin/app`. -- **`serve` vs `naive`, timed** — `podman stop -t 3` on `serve` completes in +- **`serve` vs `naive`, timed**: `podman stop -t 3` on `serve` completes in well under a second (real runs: ~90-220ms) with a `shutting down (SIGTERM)` line in the logs. `podman stop -t 2` on `naive`: for C++/Rust this takes the **full grace period** (~2.1s) and podman's own log shows it fell back to `SIGKILL`; for Go it stops fast anyway (~90ms) for the runtime reason above, but in every language the shutdown line **never appears** in `naive`'s logs, proving no graceful path ran regardless of how the process - actually died. + died. - Every container and the built image are removed at the end, whether the checks passed or not. diff --git a/examples/34-our-programs-in-containers/cpp/src/main.cpp b/examples/34-our-programs-in-containers/cpp/src/main.cpp index 1ad9ab6..2d36d02 100644 --- a/examples/34-our-programs-in-containers/cpp/src/main.cpp +++ b/examples/34-our-programs-in-containers/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// app — a tiny PID-1 container entrypoint (chapter 34: our programs in +// app: a tiny PID-1 container entrypoint (chapter 34: our programs in // containers). // // app serve run as the container's PID 1: reap zombies, forward SIGTERM @@ -10,11 +10,11 @@ // instead of running that signal's default action. Only // SIGKILL (and SIGSTOP) still work. `naive` never calls // sigaction for anything, so `podman stop` cannot end it -// gracefully -- the engine has to wait out the stop timeout +// gracefully: the engine has to wait out the stop timeout // and fall back to SIGKILL. // // Container-aware resource detection: hardware_concurrency() is -// sysconf(_SC_NPROCESSORS_ONLN) under the hood -- the HOST's cpu count. It +// sysconf(_SC_NPROCESSORS_ONLN) under the hood: the HOST's cpu count. It // never reads a cgroup, so under `podman run --cpus=2` this still prints the // host's full core count. That mismatch against Go's GOMAXPROCS and Rust's // available_parallelism() (both cgroup-aware) is the whole point of this @@ -139,7 +139,7 @@ void write_line(std::string_view s) { } // A short-lived "job": forks, prints one line, exits immediately. Left -// unreaped it would sit as a zombie -- reap_all() in the serve loop is what +// unreaped it would sit as a zombie: reap_all() in the serve loop is what // keeps these from piling up. void spawn_job(int seq) { const pid_t pid = ::fork(); @@ -157,7 +157,7 @@ void spawn_job(int seq) { } // The persistent "worker": the long-running child that stands in for a -// real service. It installs no handlers of its own -- an ordinary +// real service. It installs no handlers of its own: an ordinary // (non-PID-1) process, so SIGTERM's default action (terminate) applies the // moment `serve` forwards the signal to it. [[nodiscard]] std::expected spawn_worker() { @@ -169,7 +169,7 @@ void spawn_job(int seq) { // Undo the parent's SIG_BLOCK: without this the child inherits // TERM/INT/CHLD as *blocked*, so a forwarded SIGTERM would just sit // pending forever instead of running its default (terminate) - // action -- the worker would be unkillable. Same fix pmon (ch. 12) + // action: the worker would be unkillable. Same fix pmon (ch. 12) // applies before execvp; here there is no exec, so it happens // directly. sigset_t empty; @@ -270,7 +270,7 @@ int serve() { // SIGTERM or SIGINT: this is the pid-1 duty naive skips. Stop // spawning new jobs, forward the signal to the worker, reap it, and - // only then exit -- so nothing is left behind when the container + // only then exit, so nothing is left behind when the container // stops. job_spawner.request_stop(); if (::kill(worker_pid, SIGTERM) == 0 || errno == ESRCH) { @@ -286,7 +286,7 @@ int serve() { // THE OTHER HALF OF THE TRAP: naive installs no signal handling whatsoever. // As this container's PID 1, the kernel will not run SIGTERM's default -// action (terminate) for it -- there is no handler, so the signal is simply +// action (terminate) for it: there is no handler, so the signal is simply // dropped. `podman stop` has no way to ask it to leave; only SIGKILL ends it. int naive() { print_container_line(); diff --git a/examples/34-our-programs-in-containers/go/main.go b/examples/34-our-programs-in-containers/go/main.go index edcc1a6..a3d50d5 100644 --- a/examples/34-our-programs-in-containers/go/main.go +++ b/examples/34-our-programs-in-containers/go/main.go @@ -1,4 +1,4 @@ -// app — a tiny PID-1 container entrypoint (chapter 34: our programs in +// app: a tiny PID-1 container entrypoint (chapter 34: our programs in // containers). // // app serve run as the container's PID 1: reap children, forward SIGTERM @@ -9,7 +9,7 @@ // which it has installed no explicit handler is dropped // instead of running that signal's default action. Only // SIGKILL (and SIGSTOP) still work. `naive` never calls -// signal.Notify, so `podman stop` cannot end it gracefully -- +// signal.Notify, so `podman stop` cannot end it gracefully; // the engine has to wait out the stop timeout and fall back to // SIGKILL. // app worker the long-running child `serve` supervises (self-reexec). @@ -22,7 +22,7 @@ // Container-aware resource detection: runtime.GOMAXPROCS(0), as of the // go.mod "go 1.26" language version, defaults to the average cgroup CPU // throughput limit (cpu.max quota/period) when it is lower than the host's -// cpu count -- this is the one language of the three that gets this right +// cpu count; this is the one language of the three that gets this right // automatically. Compare this program's number against the C++ version's // (which reports the host's cpu count no matter what `--cpus` says). package main @@ -51,7 +51,7 @@ func die(err error) { } // readTrim reads a whole file, trailing whitespace trimmed. Empty string -// (not an error) if the file cannot be opened -- cgroup files may be absent +// (not an error) if the file cannot be opened; cgroup files may be absent // on unusual hosts, and callers fall back to "unknown". func readTrim(path string) string { b, err := os.ReadFile(path) @@ -133,7 +133,7 @@ func jobRun(seq string) int { } // selfPath resolves the running binary's own path so serve can re-exec it as -// "worker"/"job" -- there is no fork(2)-without-exec in Go, so this is the +// "worker"/"job"; there is no fork(2)-without-exec in Go, so this is the // idiomatic way to spawn "more of this program" as a real child process. func selfPath() string { p, err := os.Executable() @@ -190,7 +190,7 @@ func serve() int { case s := <-sigCh: // This is the pid-1 duty naive skips: stop spawning new jobs, // forward the signal to the worker, wait for it, and only then - // exit -- so nothing is left behind when the container stops. + // exit, so nothing is left behind when the container stops. ticker.Stop() name := "SIGINT" if s == syscall.SIGTERM { @@ -209,7 +209,7 @@ func serve() int { // THE OTHER HALF OF THE TRAP: naive installs no signal handling whatsoever. // As this container's PID 1, the kernel will not run SIGTERM's default -// action (terminate) for it -- there is no handler, so the signal is simply +// action (terminate) for it; there is no handler, so the signal is simply // dropped. `podman stop` has no way to ask it to leave; only SIGKILL ends it. func naiveRun() int { printContainerLine() diff --git a/examples/34-our-programs-in-containers/rust/src/main.rs b/examples/34-our-programs-in-containers/rust/src/main.rs index 71520a6..703da4b 100644 --- a/examples/34-our-programs-in-containers/rust/src/main.rs +++ b/examples/34-our-programs-in-containers/rust/src/main.rs @@ -1,4 +1,4 @@ -// app — a tiny PID-1 container entrypoint (chapter 34: our programs in +// app: a tiny PID-1 container entrypoint (chapter 34: our programs in // containers). // // app serve run as the container's PID 1: reap children, forward SIGTERM @@ -9,7 +9,7 @@ // which it has installed no explicit handler is dropped // instead of running that signal's default action. Only // SIGKILL (and SIGSTOP) still work. `naive` never registers a -// handler, so `podman stop` cannot end it gracefully -- the +// handler, so `podman stop` cannot end it gracefully: the // engine has to wait out the stop timeout and fall back to // SIGKILL. // app worker the long-running child `serve` supervises (self-reexec). @@ -19,12 +19,12 @@ // is no signalfd/SIGCHLD anywhere here. std::process::Child::wait() calls // waitpid(2) for you; parking one std::thread per spawned child in wait() is // the reap loop. signal_hook::iterator::Signals turns SIGTERM/SIGINT into a -// blocking iterator on the main thread -- ordinary code, nothing async- +// blocking iterator on the main thread: ordinary code, nothing async- // signal-unsafe runs in a handler. // // Container-aware resource detection: std::thread::available_parallelism() // on Linux divides the cgroup's cpu.max quota/period when it is lower than -// the affinity mask count -- cgroup-aware, like Go's GOMAXPROCS. Compare +// the affinity mask count: cgroup-aware, like Go's GOMAXPROCS. Compare // this program's number against the C++ version's hardware_concurrency() // (which reports the host's cpu count no matter what `--cpus` says). @@ -53,7 +53,7 @@ fn die(err: anyhow::Error) -> ! { } /// Read a whole file, trailing whitespace trimmed. Empty string (not an -/// error) if the file cannot be opened -- cgroup files may be absent on +/// error) if the file cannot be opened: cgroup files may be absent on /// unusual hosts, and callers fall back to "unknown". fn read_trim(path: &str) -> String { let mut buf = String::new(); @@ -134,7 +134,7 @@ fn job_run(seq: &str) -> i32 { } /// Resolve the running binary's own path so serve can re-exec it as -/// "worker"/"job" -- like Go, Rust avoids fork(2)-without-exec (the +/// "worker"/"job": like Go, Rust avoids fork(2)-without-exec (the /// allocator and runtime are not fork-safe once threads exist), so this is /// the idiomatic way to spawn "more of this program" as a real child /// process. @@ -157,7 +157,7 @@ fn serve() -> Result { println!("app: worker started pid={worker_pid}"); // The job spawner: a background thread that periodically re-execs - // "app job " and, in its own thread per child, waits for it -- + // "app job " and, in its own thread per child, waits for it: // that wait() call IS the reap. `stop` lets shutdown cancel the loop // cooperatively instead of detaching a thread that outlives serve(). let stop = Arc::new(AtomicBool::new(false)); @@ -209,7 +209,7 @@ fn serve() -> Result { // THE OTHER HALF OF THE TRAP: naive installs no signal handling whatsoever. // As this container's PID 1, the kernel will not run SIGTERM's default -// action (terminate) for it -- there is no handler, so the signal is simply +// action (terminate) for it: there is no handler, so the signal is simply // dropped. `podman stop` has no way to ask it to leave; only SIGKILL ends it. fn naive_run() -> i32 { print_container_line(); diff --git a/examples/34-our-programs-in-containers/verify.lua b/examples/34-our-programs-in-containers/verify.lua index e8730a2..df718c6 100644 --- a/examples/34-our-programs-in-containers/verify.lua +++ b/examples/34-our-programs-in-containers/verify.lua @@ -5,7 +5,7 @@ -- and asserts observable effects, not merely exit 0: -- -- 1. container-aware resource detection: the CONSTRAINED cpu.max/mem.max --- are reported, not the host's -- and specifically that C++'s +-- are reported, not the host's, and specifically that C++'s -- effective_parallelism is the HOST cpu count (hardware_concurrency's -- trap) while Go's and Rust's match the cgroup quota. -- 2. this process really is PID 1 of its own namespace (pid=1 ppid=0, @@ -15,12 +15,12 @@ -- `podman unshare nsenter` from the host both see the same PID 1. -- 4. pid-1 signal duties: `serve` reacts to SIGTERM almost instantly and -- prints a graceful shutdown line; `naive` never prints that line --- (proving no graceful path ran) and -- for C++/Rust, whose default +-- (proving no graceful path ran) and, for C++/Rust (whose default -- SIGTERM disposition is masked out by the kernel's PID-1 special --- case -- takes the full stop grace period and is SIGKILLed. Go's +-- case), takes the full stop grace period and is SIGKILLed. Go's -- runtime is a real, verified exception: it installs its own handler -- for SIGTERM at startup regardless of signal.Notify, so a "naive" Go --- binary still dies promptly -- just abruptly, with no cleanup. +-- binary still dies promptly, just abruptly, with no cleanup. -- -- Invoked from the example directory with LSP_LANG (cpp|go|rust) and -- EXAMPLE_DIR set; the interpreter may be lua or luajit. Images and diff --git a/examples/35-virtualization-and-kvm/README.md b/examples/35-virtualization-and-kvm/README.md index aab84bb..7d8f525 100644 --- a/examples/35-virtualization-and-kvm/README.md +++ b/examples/35-virtualization-and-kvm/README.md @@ -1,8 +1,8 @@ -# 35-virtualization-and-kvm — overheadbench +# 35-virtualization-and-kvm: overheadbench -Chapter 35's example: `overheadbench`, three tiny host-local microbenchmarks +This chapter's example: `overheadbench`, three tiny host-local microbenchmarks implemented identically in C++23, Go, and Rust. The example itself only -measures **wherever it is run** — it makes no VM/container comparison on its +measures **wherever it is run**: it makes no VM/container comparison on its own. The chapter's point is made by running the *same binary* in three places (bare host, a lab KVM guest, and a rootless podman container) and tabulating the numbers side by side. @@ -21,20 +21,20 @@ bench=mem metric=14.03 unit=GB/s bench=io metric=1480.21 unit=ops/s ``` -- **`syscall`** — a tight `getpid(2)` loop (a raw syscall each iteration, not +- **`syscall`**: a tight `getpid(2)` loop (a raw syscall each iteration, not a libc-cached value). Reports **ns/call**: mean wall time per syscall. This is the number the chapter contrasts against a VM's vmexit-trap cost and a container's near-zero added cost (containers share the host kernel; a syscall inside one *is* a host syscall). -- **`mem`** — allocates a 128 MiB buffer of `u64` words and reads it +- **`mem`**: allocates a 128 MiB buffer of `u64` words and reads it sequentially, summing every word, for a configurable number of passes. Reports **GB/s**: total bytes read across all passes divided by elapsed time. The sum is consumed after the clock stops so the compiler can't prove the scan is dead code and delete it. -- **`io`** — repeats a create → write(4096 bytes) → `fsync(2)` → unlink +- **`io`**: repeats a create → write(4096 bytes) → `fsync(2)` → unlink cycle. Reports **ops/s**: complete cycles per second. The scratch directory is created **relative to the current working directory** - (where `demo.sh` runs the binary from), not the system temp directory — + (where `demo.sh` runs the binary from), not the system temp directory: on many dev hosts `/tmp` is `tmpfs`, where `fsync` is nearly free and the number stops meaning anything as "storage overhead." Comparing this number meaningfully across host/VM/container requires each side to write @@ -50,7 +50,7 @@ milliseconds): `syscall` 200,000 calls, `mem` 16 passes, `io` 200 cycles. Bad usage (an unknown `--bench` value, a flag with a missing value, an unknown flag, or a non-numeric/zero `--iters`) prints `usage: overheadbench [--bench syscall|mem|io|all] [--iters N]` to stderr -and exits 2 — never a silently wrong number. +and exits 2, never a silently wrong number. ## Layout @@ -68,12 +68,12 @@ and exits 2 — never a silently wrong number. Every language directory has a `demo.sh` with the same interface, identical across the whole book: -- `./demo.sh` — build then run (`--bench all`) -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary with the given arguments +- `./demo.sh`: build then run (`--bench all`) +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary with the given arguments - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of - running locally — this is how the chapter gets the VM-side numbers. + running locally. This is how the chapter gets the VM-side numbers. ## Verification @@ -85,18 +85,18 @@ a time. It asserts, per language: in order `syscall`, `mem`, `io`, each parsing as `bench= metric= unit=` with the metric inside a generous but real sanity window (e.g. `syscall` between 1 ns and 5 ms per call, `mem` - between 10 MB/s and 1 TB/s, `io` between 0.1 and 5,000,000 ops/s) — wide + between 10 MB/s and 1 TB/s, `io` between 0.1 and 5,000,000 ops/s), wide enough to tolerate a loaded CI host or unusual hardware, narrow enough to catch a zeroed-out, negative, or nonsensical measurement; - selecting a single bench with `--bench` prints only that bench's row; - `--iters` with a small explicit value (e.g. 10 or 500) still produces a - sane, parseable row — guarding against a divide-by-zero at low iteration + sane, parseable row, guarding against a divide-by-zero at low iteration counts; - bad usage (unknown `--bench` value, missing flag value, unknown flag, non-numeric or zero `--iters`) exits 2 and prints the usage line. Real host numbers from this machine (Fedora 44, kernel 7.1, -`examples/` on an NVMe-backed btrfs filesystem — not `/tmp`): +`examples/` on an NVMe-backed btrfs filesystem, not `/tmp`): | lang | syscall (ns/call) | mem (GB/s) | io (ops/s) | |------|-------------------:|-----------:|-----------:| @@ -104,21 +104,22 @@ Real host numbers from this machine (Fedora 44, kernel 7.1, | go | 90–159 | 12–14 | 1,450–1,670 | | rust | 87–96 | 16–19 | 1,260–1,580 | -The exact numbers vary run to run — that's expected of a microbenchmark; +The exact numbers vary run to run; that's expected of a microbenchmark; `verify.lua` checks shape and sane range, not a fixed value. ## What the chapter does with it (not asserted by `verify.lua`) 1. Run `./demo.sh run` on the bare host (the numbers above). 2. Deploy the same binary to `systems-target` - (`TARGET=systems-target ./demo.sh run`) and record the VM numbers — + (`TARGET=systems-target ./demo.sh run`) and record the VM numbers: expect a real `syscall` gap (vmexit/vmenter round trip on every trapped instruction) and comparable `mem`/`io` numbers once the guest is warmed up, modulo whatever the host's virtio/virtualization layer costs for storage. -3. Build the chapter-34 UBI 10 Containerfile for the same language, run the - binary with `podman run --rm ` on the host, and record the - container numbers — expect `syscall` and `mem` close to bare-host (a +3. Build the containers chapter's UBI 10 Containerfile for the same + language, run the binary with `podman run --rm ` on the host, and + record the container numbers: expect `syscall` and `mem` close to + bare-host (a container's process runs directly on the host kernel and CPU), with `io` depending on the container's storage driver/overlay filesystem. 4. Tabulate host vs VM vs container for the pros/cons discussion: containers diff --git a/examples/35-virtualization-and-kvm/cpp/src/main.cpp b/examples/35-virtualization-and-kvm/cpp/src/main.cpp index ec0b62b..dfb48df 100644 --- a/examples/35-virtualization-and-kvm/cpp/src/main.cpp +++ b/examples/35-virtualization-and-kvm/cpp/src/main.cpp @@ -1,5 +1,5 @@ // overheadbench: three host-local microbenchmarks used to talk about -// virtualization/container overhead in chapter 35 — +// virtualization/container overhead in chapter 35: // (a) syscall latency: a tight getpid(2) loop, ns/call // (b) memory bandwidth: sequential read of a big buffer, GB/s // (c) small-file IO: create/write/fsync/unlink loop, ops/s @@ -7,7 +7,7 @@ // overheadbench [--bench syscall|mem|io|all] [--iters N] // // Prints one line per bench: "bench= metric= unit=". -// This binary always measures wherever it runs (host, VM, or container) — +// This binary always measures wherever it runs (host, VM, or container): // the chapter runs the *same* binary in all three places and tabulates the // numbers; this example itself only asserts the host numbers are sane. @@ -107,7 +107,7 @@ double bench_syscall(std::uint64_t iters) { const auto start = Clock::now(); for (std::uint64_t i = 0; i < iters; ++i) { // A raw syscall (not the libc wrapper) so every iteration really - // crosses into the kernel — this is the syscall-boundary cost the + // crosses into the kernel: this is the syscall-boundary cost the // chapter contrasts against VM (vmexit trap) and container // (near-zero extra) overhead. ::syscall(SYS_getpid); diff --git a/examples/35-virtualization-and-kvm/go/main.go b/examples/35-virtualization-and-kvm/go/main.go index 578b425..51244ef 100644 --- a/examples/35-virtualization-and-kvm/go/main.go +++ b/examples/35-virtualization-and-kvm/go/main.go @@ -1,5 +1,5 @@ // overheadbench: three host-local microbenchmarks used to talk about -// virtualization/container overhead in chapter 35 — +// virtualization/container overhead in chapter 35. // // (a) syscall latency: a tight getpid(2) loop, ns/call // (b) memory bandwidth: sequential read of a big buffer, GB/s @@ -8,7 +8,7 @@ // overheadbench [--bench syscall|mem|io|all] [--iters N] // // Prints one line per bench: "bench= metric= unit=". -// This binary always measures wherever it runs (host, VM, or container) — +// This binary always measures wherever it runs (host, VM, or container); // the chapter runs the *same* binary in all three places and tabulates the // numbers; this example itself only asserts the host numbers are sane. package main @@ -91,7 +91,7 @@ func benchSyscallRun(iters uint64) float64 { start := time.Now() for i := uint64(0); i < iters; i++ { // unix.Getpid issues the raw getpid(2) syscall on every call (the Go - // runtime does not cache it) — every iteration really crosses into + // runtime does not cache it), every iteration really crosses into // the kernel, the syscall-boundary cost this chapter contrasts // against VM (vmexit trap) and container (near-zero extra) overhead. unix.Getpid() diff --git a/examples/35-virtualization-and-kvm/rust/src/main.rs b/examples/35-virtualization-and-kvm/rust/src/main.rs index 59f1cba..d9e8551 100644 --- a/examples/35-virtualization-and-kvm/rust/src/main.rs +++ b/examples/35-virtualization-and-kvm/rust/src/main.rs @@ -1,5 +1,5 @@ // overheadbench: three host-local microbenchmarks used to talk about -// virtualization/container overhead in chapter 35 — +// virtualization/container overhead in chapter 35: // (a) syscall latency: a tight getpid(2) loop, ns/call // (b) memory bandwidth: sequential read of a big buffer, GB/s // (c) small-file IO: create/write/fsync/unlink loop, ops/s @@ -7,7 +7,7 @@ // overheadbench [--bench syscall|mem|io|all] [--iters N] // // Prints one line per bench: "bench= metric= unit=". -// This binary always measures wherever it runs (host, VM, or container) — +// This binary always measures wherever it runs (host, VM, or container): // the chapter runs the *same* binary in all three places and tabulates the // numbers; this example itself only asserts the host numbers are sane. @@ -95,7 +95,7 @@ fn bench_syscall(iters: u64) -> f64 { let start = Instant::now(); for _ in 0..iters { // rustix::process::getpid issues the raw getpid(2) syscall every - // call — every iteration really crosses into the kernel, the + // call: every iteration really crosses into the kernel, the // syscall-boundary cost this chapter contrasts against VM (vmexit // trap) and container (near-zero extra) overhead. black_box(rustix::process::getpid()); diff --git a/examples/35-virtualization-and-kvm/verify.lua b/examples/35-virtualization-and-kvm/verify.lua index f7445c0..37b31e3 100644 --- a/examples/35-virtualization-and-kvm/verify.lua +++ b/examples/35-virtualization-and-kvm/verify.lua @@ -5,7 +5,7 @@ -- * "--bench all" (the default) prints exactly the three rows, in order -- syscall/mem/io, each "bench= metric= unit=" with -- parsing as a finite non-negative number and the expected unit. --- * Each metric lands in a generous but real sanity window — wide enough +-- * Each metric lands in a generous but real sanity window: wide enough -- to tolerate a loaded CI host, narrow enough to catch "printed zero" -- or "printed a bogus huge/negative number" bugs. -- * Selecting a single bench via --bench prints only that bench's row. @@ -13,7 +13,7 @@ -- small explicit value still produces a sane row (guards against a -- divide-by-zero / elapsed==0 edge case at low iteration counts). -- * Bad usage (unknown --bench value, missing flag value, unknown flag, --- non-numeric --iters) exits 2 and prints the usage line — never a +-- non-numeric --iters) exits 2 and prints the usage line, never a -- silent wrong answer. -- -- The chapter separately deploys this same binary to a lab VM @@ -122,7 +122,7 @@ check_row(r.out, "io", lang) -- --------------------------------------------------------------------------- -- 5. Bad usage: unknown --bench value, missing flag value, unknown flag, --- non-numeric --iters all exit 2 with the usage line -- never a silent +-- non-numeric --iters all exit 2 with the usage line, never a silent -- wrong answer. -- --------------------------------------------------------------------------- r = checks.run("./demo.sh " .. lang .. " run --bench bogus") diff --git a/examples/36-proc-sys-and-the-agent/README.md b/examples/36-proc-sys-and-the-agent/README.md index 8e420ab..b96d545 100644 --- a/examples/36-proc-sys-and-the-agent/README.md +++ b/examples/36-proc-sys-and-the-agent/README.md @@ -1,8 +1,8 @@ -# 36-proc-sys-and-the-agent — sysagent v0 +# 36-proc-sys-and-the-agent: sysagent v0 Chapter 36 opens Part 9 (observability). `sysagent` is a small USE-method -metrics collector — the same tool the rest of the Part builds on (chapter 37's -saturation drills, chapter 41's capstone fleet) — reading nothing but +metrics collector, the same tool the rest of the Part builds on (later +saturation drills and the final fleet exercise), reading nothing but `/proc`, `/proc/net`, `/proc/diskstats`, and cgroup PSI. No root, no `/dev/kmsg`, no eBPF: everything here is a plain file a normal user can open. @@ -18,23 +18,23 @@ opts: --interval-ms N (default 200) cpu utilization sampling window - **`sample`** takes exactly one snapshot: it reads `/proc/stat` once, sleeps `--interval-ms`, reads it again, and reports the CPU delta over that - window — alongside `/proc/loadavg`, `/proc/meminfo`, `/proc/diskstats`, + window, alongside `/proc/loadavg`, `/proc/meminfo`, `/proc/diskstats`, `/proc/net/dev`, and cgroup PSI, all gathered **concurrently** with the sleep (C++ `std::jthread`, Go goroutines fanned in over a channel, Rust - scoped threads — three languages, one idea: don't serialize five + scoped threads, three languages, one idea: don't serialize five independent file reads behind a timer). Default text output is `key=value` lines; `--json` emits the single-line JSON object below. - **`serve --port P`** exposes the identical snapshot over a hand-rolled HTTP/1.1 `GET /metrics` endpoint (Go uses `net/http`; C++ and Rust hand-roll the same tiny surface on raw sockets + `poll(2)`, in keeping with the - book's earlier chatterd examples) — one fresh sample per request. Any + book's earlier chatterd examples), serving one fresh sample per request. Any other path 404s. `SIGINT`/`SIGTERM` prints a shutdown line and exits 0. - No/bad arguments print a usage line on stderr and exit 2. ## The deterministic field schema Every field name below is byte-identical across `--json` and `/metrics` in -all three languages — that's the contract this example exists to prove out: +all three languages; that's the contract this example exists to prove out: you can point one Grafana dashboard or one test suite at any of the three. | field | type | source | @@ -62,13 +62,13 @@ readings (a single reading of `/proc/stat` is a cumulative counter since boot, not a percentage). PSI (Pressure Stall Information) is read from **this process's own cgroup** -first — resolved via the single `0::` line in `/proc/self/cgroup` -(cgroup v2 unified hierarchy) joined onto `/sys/fs/cgroup/{cpu,memory,io}.pressure` -— falling back to the system-wide `/proc/pressure/{cpu,memory,io}` files if +first, resolved via the single `0::` line in `/proc/self/cgroup` +(cgroup v2 unified hierarchy) joined onto `/sys/fs/cgroup/{cpu,memory,io}.pressure`, +falling back to the system-wide `/proc/pressure/{cpu,memory,io}` files if the cgroup-scoped ones aren't there. If neither is readable (PSI disabled in the kernel, or a container without a cgroup-v2 delegate), `psi_available` is -`false` and the three `avg10` fields are `0.0` — that's the "if available" -in the chapter spec: a missing PSI source is not an error. +`false` and the three `avg10` fields are `0.0`, matching the "if available" +behavior in the chapter spec: a missing PSI source is not an error. ## Example output @@ -91,25 +91,25 @@ $ kill -INT %1 sysagent: shutting down ``` -(That `sample` line reflects real load on the box this was written on — a +(That `sample` line reflects real load on the box this was written on: a concurrent `cc1plus`-heavy build was running in the background, which is exactly the kind of noise the verification below is built to tolerate.) ## One subject, three concurrency styles -The `/proc` parsing itself is deliberately unglamorous — `open`, read lines, -split fields — so the interesting difference is how each language overlaps +The `/proc` parsing itself is unglamorous (`open`, read lines, +split fields), so the interesting difference is how each language overlaps the five independent reads with the CPU-delta sampling window instead of doing five sequential blocking calls: -- **C++** — `std::jthread` per source (`loadavg`, `meminfo`, `diskstats`, +- **C++**: `std::jthread` per source (`loadavg`, `meminfo`, `diskstats`, `netdev`, `psi`), an `std::atomic` success counter, and a small `ProcFile` RAII wrapper around `std::ifstream` so every `/proc` handle closes itself. Errors flow through `std::expected`. -- **Go** — one goroutine per source, fanned in over a single buffered - `chan namedErr` — the idiomatic Go answer to "wait for N concurrent +- **Go**: one goroutine per source, fanned in over a single buffered + `chan namedErr`, the idiomatic Go answer to "wait for N concurrent results and keep the first error." Errors are wrapped with `%w` throughout. -- **Rust** — `std::thread::scope` with one scoped closure per source, so the +- **Rust**: `std::thread::scope` with one scoped closure per source, so the borrows into the final `Snapshot` don't need `Arc`/`Mutex`; each closure returns a `Result` collected via `JoinHandle::join()`. Errors flow through `Result`/`?`. @@ -117,7 +117,7 @@ doing five sequential blocking calls: The HTTP layer follows the same three-languages-one-idea pattern as the book's chatterd examples: Go leans on its stdlib `net/http`; C++ and Rust hand-roll the request line + `poll(2)` + `signalfd`/`SignalFd` shutdown -doorbell that chapter 24 already established, because there's no reason to +doorbell established earlier in the book's chatterd chapters, because there's no reason to pull in a framework for one endpoint. ## Layout @@ -136,7 +136,7 @@ pull in a framework for one endpoint. Each language directory has a `demo.sh` with the identical interface used book-wide: `./demo.sh build`, `./demo.sh run [args]`, and a bare `./demo.sh` that builds then runs. With `TARGET` set, `run` deploys to that lab VM -instead of running locally — unused here since this chapter is local-only +instead of running locally; unused here since this chapter is local-only (`mode: local` in the manifest); it reads the current host's `/proc`, not a VM's. `run` execs straight into the built binary so signals (`SIGINT` on `serve`) reach it directly rather than a wrapping shell. @@ -146,8 +146,8 @@ VM's. `run` execs straight into the built binary so signals (`SIGINT` on `verify.lua` (run by the test harness with `LSP_LANG` set to one language at a time) asserts observable behavior, not exit 0: -1. Usage errors — no arguments, an unknown subcommand, `serve` without - `--port` — all exit 2 with a usage line. +1. Usage errors (no arguments, an unknown subcommand, `serve` without + `--port`) all exit 2 with a usage line. 2. `sample --json` carries every field in the schema above with a numeric (or boolean, for `psi_available`) value, `mem_total_kb` is positive, `cpu_util_pct` is in `[0, 100]`, the `disks` array has a named entry, and @@ -157,7 +157,7 @@ a time) asserts observable behavior, not exit 0: longer than a `sample --interval-ms 500` window and compares against an immediately-prior idle sample, retried up to three times with a 12 percentage-point margin (or an absolute ≥55% floor, for the case where the - host is already saturated) — robust against running on a real, shared + host is already saturated), robust against running on a real, shared development box rather than an isolated benchmark rig. 4. `serve --port P` answers `GET /metrics` with the same JSON shape, 404s an unknown path, and its `mem_total_kb` agrees **exactly** with a direct diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp b/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp index b265d0b..6b03add 100644 --- a/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp +++ b/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp @@ -1,4 +1,4 @@ -// sysagent: a hand-rolled, single-endpoint HTTP/1.1 server. No framework — +// sysagent: a hand-rolled, single-endpoint HTTP/1.1 server. No framework: // same philosophy as the raw-socket chatterd examples earlier in the book. #pragma once diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp b/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp index beb11b7..b97278a 100644 --- a/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp +++ b/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// sysagent v0 (chapter 36: /proc, /sys, and the agent) — a USE-method +// sysagent v0 (chapter 36: /proc, /sys, and the agent): a USE-method // metrics collector reading /proc + /sys/fs/cgroup, no root required. // // sysagent sample [--json] [--interval-ms N] @@ -13,7 +13,7 @@ // HTTP/1.1 /metrics endpoint, one fresh sample per request. // // The field names in both the --json and /metrics output are the -// deterministic, cross-language schema documented in README.md — Go and +// deterministic, cross-language schema documented in README.md: Go and // Rust sysagent emit byte-for-byte the same keys. #include #include @@ -32,7 +32,7 @@ void usage() { "serve --port P [--interval-ms N]"); } -// Minimal hand-rolled flag scan — no getopt_long, to keep the three +// Minimal hand-rolled flag scan, no getopt_long, to keep the three // languages' argument handling directly comparable. struct Flags { bool json = false; diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp b/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp index 8f16dd8..41c5e74 100644 --- a/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp +++ b/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp @@ -17,7 +17,7 @@ namespace { // RAII wrapper around an ifstream: names the path (for error messages) and // guarantees the fd is closed when the wrapper goes out of scope, exactly -// like every other UniqueFd-style RAII type in this book — just for the +// like every other UniqueFd-style RAII type in this book: just for the // stdio-buffered stream form /proc parsing normally uses. class ProcFile { public: @@ -153,7 +153,7 @@ std::expected, std::string> read_diskstats() { iss >> major >> minor >> name >> reads_completed >> reads_merged >> sectors_read >> ms_reading >> writes_completed >> writes_merged >> sectors_written; if (name.empty()) continue; - // Skip loop/ram devices and partitions with no traffic at all — keeps + // Skip loop/ram devices and partitions with no traffic at all: keeps // the array focused on the disks that actually matter for USE-method // reads, without hardcoding any device-naming scheme. if (name.starts_with("loop") || name.starts_with("ram")) continue; diff --git a/examples/36-proc-sys-and-the-agent/go/httpd.go b/examples/36-proc-sys-and-the-agent/go/httpd.go index 2b2b6d0..311fb50 100644 --- a/examples/36-proc-sys-and-the-agent/go/httpd.go +++ b/examples/36-proc-sys-and-the-agent/go/httpd.go @@ -1,4 +1,4 @@ -// sysagent: the /metrics HTTP endpoint, using net/http (Go's stdlib server — +// sysagent: the /metrics HTTP endpoint, using net/http (Go's stdlib server; // the C++ and Rust versions hand-roll the same tiny surface on raw sockets). package main @@ -45,15 +45,15 @@ func Serve(port, intervalMs int) int { fmt.Printf("sysagent: listening on 0.0.0.0:%d\n", port) - sigCh := make(chan os.Signal, 1) - signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() select { - case <-sigCh: + case <-ctx.Done(): fmt.Println("sysagent: shutting down") - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() - _ = srv.Shutdown(ctx) + _ = srv.Shutdown(shutdownCtx) return 0 case err := <-serveErr: if err != nil && !errors.Is(err, http.ErrServerClosed) { diff --git a/examples/36-proc-sys-and-the-agent/go/main.go b/examples/36-proc-sys-and-the-agent/go/main.go index 37eef7e..b4648cd 100644 --- a/examples/36-proc-sys-and-the-agent/go/main.go +++ b/examples/36-proc-sys-and-the-agent/go/main.go @@ -1,4 +1,4 @@ -// sysagent v0 (chapter 36: /proc, /sys, and the agent) — a USE-method +// sysagent v0 (chapter 36: /proc, /sys, and the agent): a USE-method // metrics collector reading /proc + /sys/fs/cgroup, no root required. // // sysagent sample [--json] [--interval-ms N] @@ -13,7 +13,7 @@ // endpoint, one fresh sample per request. // // The field names in both the --json and /metrics output are the -// deterministic, cross-language schema documented in README.md — C++ and +// deterministic, cross-language schema documented in README.md; C++ and // Rust sysagent emit byte-for-byte the same keys. package main diff --git a/examples/36-proc-sys-and-the-agent/go/procfs.go b/examples/36-proc-sys-and-the-agent/go/procfs.go index b736ef5..c80f78c 100644 --- a/examples/36-proc-sys-and-the-agent/go/procfs.go +++ b/examples/36-proc-sys-and-the-agent/go/procfs.go @@ -28,7 +28,7 @@ type NetStat struct { // Snapshot is one fully-formed sample. Field names here are the canonical, // deterministic names shared with the C++ and Rust implementations (see -// README.md) — only the encoding (Go struct vs C++ struct vs Rust struct) +// README.md); only the encoding (Go struct vs C++ struct vs Rust struct) // differs. type Snapshot struct { CPUUtilPct float64 @@ -299,7 +299,7 @@ type namedErr struct { // TakeSnapshot takes two /proc/stat readings intervalMs apart and fills in // every other field concurrently in between, via goroutines fanned in over a -// channel — the Go analogue of the C++ jthread pool and the Rust scoped +// channel, the Go analogue of the C++ jthread pool and the Rust scoped // threads. func TakeSnapshot(intervalMs int) (Snapshot, error) { t0, err := readCPUTicks() diff --git a/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs b/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs index 9aff97b..fab4710 100644 --- a/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs +++ b/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs @@ -1,4 +1,4 @@ -//! sysagent: a hand-rolled, single-endpoint HTTP/1.1 server. No framework — +//! sysagent: a hand-rolled, single-endpoint HTTP/1.1 server. No framework: //! same philosophy as the raw-socket chatterd examples earlier in the book, //! and the same poll(2) + signalfd(2) shutdown doorbell as their Rust code. diff --git a/examples/36-proc-sys-and-the-agent/rust/src/main.rs b/examples/36-proc-sys-and-the-agent/rust/src/main.rs index 6e75397..5b379d8 100644 --- a/examples/36-proc-sys-and-the-agent/rust/src/main.rs +++ b/examples/36-proc-sys-and-the-agent/rust/src/main.rs @@ -1,4 +1,4 @@ -// sysagent v0 (chapter 36: /proc, /sys, and the agent) — a USE-method +// sysagent v0 (chapter 36: /proc, /sys, and the agent): a USE-method // metrics collector reading /proc + /sys/fs/cgroup, no root required. // // sysagent sample [--json] [--interval-ms N] @@ -13,7 +13,7 @@ // HTTP/1.1 /metrics endpoint, one fresh sample per request. // // The field names in both the --json and /metrics output are the -// deterministic, cross-language schema documented in README.md — C++ and +// deterministic, cross-language schema documented in README.md: C++ and // Go sysagent emit byte-for-byte the same keys. mod httpd; mod procfs; diff --git a/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs b/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs index bc605af..37a1020 100644 --- a/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs +++ b/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs @@ -21,7 +21,7 @@ pub struct NetStat { } /// One fully-formed sample. Field names here are the canonical, deterministic -/// names shared with the C++ and Go implementations (see README.md) — only +/// names shared with the C++ and Go implementations (see README.md): only /// the encoding (Rust struct vs C++ struct vs Go struct) differs. #[derive(Default)] pub struct Snapshot { @@ -272,7 +272,7 @@ fn read_psi() -> Option { } /// Takes two /proc/stat readings `interval_ms` apart and fills in every other -/// field concurrently in between, via scoped threads — the Rust analogue of +/// field concurrently in between, via scoped threads: the Rust analogue of /// the C++ jthread pool and the Go goroutines fanned in over a channel. pub fn take_snapshot(interval_ms: u64) -> Result { let t0 = read_cpu_ticks()?; diff --git a/examples/36-proc-sys-and-the-agent/verify.lua b/examples/36-proc-sys-and-the-agent/verify.lua index c1dcf0e..6a14dc0 100644 --- a/examples/36-proc-sys-and-the-agent/verify.lua +++ b/examples/36-proc-sys-and-the-agent/verify.lua @@ -5,7 +5,7 @@ -- exit 2 with a usage line on stderr. -- 2. `sample --json` emits every expected key with a numeric value, in the -- deterministic cross-language schema (README.md), with sane ranges. --- 3. A cpu-busy child measurably raises cpu_util_pct vs an idle baseline — +-- 3. A cpu-busy child measurably raises cpu_util_pct vs an idle baseline: -- compared with a robust margin and a couple of retries, since this -- runs on a real, possibly-noisy host rather than an isolated lab VM. -- 4. `serve --port P` answers GET /metrics with the same JSON shape, 404s @@ -121,7 +121,7 @@ checks.expect_match(margin_ok and "yes" or "no", "yes", lang .. ": a cpu-busy child raises cpu_util_pct vs idle (margin>=12pp or busy>=55%)") -- --------------------------------------------------------------------------- --- 4. `serve` — /metrics answers, 404s elsewhere, agrees with a direct +-- 4. `serve`: /metrics answers, 404s elsewhere, agrees with a direct -- /proc read, and shuts down cleanly on SIGINT. -- --------------------------------------------------------------------------- diff --git a/examples/37-use-method-60s-analysis/README.md b/examples/37-use-method-60s-analysis/README.md index 39b523c..6c41638 100644 --- a/examples/37-use-method-60s-analysis/README.md +++ b/examples/37-use-method-60s-analysis/README.md @@ -1,7 +1,7 @@ -# _template — hello-syscall +# _template: hello-syscall The seed for every runnable example in the book. It implements the same tiny -program three times — C++23, Go, and Rust — each printing exactly one line: +program three times, in C++23, Go, and Rust, each printing exactly one line: ``` pid on () @@ -26,9 +26,9 @@ _template/ Every language directory has a `demo.sh` with the same interface, identical across the whole book: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. diff --git a/examples/37-use-method-60s-analysis/cpp/src/main.cpp b/examples/37-use-method-60s-analysis/cpp/src/main.cpp index be5abef..eac6d45 100644 --- a/examples/37-use-method-60s-analysis/cpp/src/main.cpp +++ b/examples/37-use-method-60s-analysis/cpp/src/main.cpp @@ -9,8 +9,8 @@ // mpstat, pidstat, iostat, free, sar, ss, top) exactly as a human running the // checklist would, parses their plain-text tables generically (by column // *name*, read from each tool's own header row, never a hardcoded position), -// and for each of the four resources reports which USE signals — Utilization -// and Saturation — fired against a fixed threshold. Errors is reported too, +// and for each of the four resources reports which USE signals (Utilization +// and Saturation) fired against a fixed threshold. Errors is reported too, // fixed false: this chapter induces load, not kernel-logged faults, so an // real checklist run finds none. // @@ -79,7 +79,7 @@ std::vector split_lines(const std::string& text) { // A parsed table row: column names (from a tool's own header line) zipped // with this row's values, so every parser below reads fields *by name* -// rather than by a hardcoded position — robust to sysstat/procps column +// rather than by a hardcoded position: robust to sysstat/procps column // reordering across versions. struct Row { std::vector names; @@ -113,7 +113,7 @@ Row make_row(const std::vector& header_tokens, size_t header_skip, // that returns std::expected, per book convention. std::expected run_capture(const std::string& cmd) { // Grouped in a subshell so a single trailing "2>&1" captures every stage - // of a pipeline (e.g. "dmesg | tail -n 5") — without the parens, 2>&1 + // of a pipeline (e.g. "dmesg | tail -n 5"): without the parens, 2>&1 // binds only to the pipeline's last stage and an earlier stage's stderr // (e.g. dmesg's own permission-denied message) leaks past the capture. std::string full = "( env LC_ALL=C LANG=C " + cmd + " ) 2>&1"; @@ -129,7 +129,7 @@ std::expected run_capture(const std::string& cmd) // Start `env LC_ALL=C LANG=C ` in the background, stdout+stderr // redirected to outfile; returns the child pid (or -1 on fork failure). Used -// for the five interval samplers so they run *concurrently* — a 12s analyze +// for the five interval samplers so they run *concurrently*: a 12s analyze // window costs ~12s wall-clock, not 5x that from running them one at a time. pid_t spawn_to_file(const std::vector& argv, const std::string& outfile) { pid_t pid = ::fork(); @@ -167,7 +167,7 @@ std::string read_all(const std::string& path) { // ------------------------------------------------------- checklist parsers -- // Each parser reads a header line's column names from the tool's own output -// and zips subsequent data lines against that name list — see Row::get above. +// and zips subsequent data lines against that name list: see Row::get above. struct VmstatResult { double run_queue_max = 0.0; @@ -175,7 +175,7 @@ struct VmstatResult { }; // vmstat has no aggregate "Average:" line (unlike mpstat/pidstat), and its -// *first* sample is a since-boot average, not a live interval — both quirks +// *first* sample is a since-boot average, not a live interval: both quirks // are handled here: we scan every data row after the header, skip row 0, and // take the max across the rest (the steady-state saturation signal, not // diluted by an average across the whole window). @@ -208,7 +208,7 @@ struct MpstatResult { }; // mpstat -P ALL's own "Average:" row over the whole window (excludes the -// since-boot first report automatically — sysstat's job, not ours). +// since-boot first report automatically: sysstat's job, not ours). MpstatResult parse_mpstat(const std::string& text) { MpstatResult res; std::vector header, data; @@ -249,7 +249,7 @@ struct IostatResult { }; // iostat -xz has no Average line either, AND -z hides idle devices entirely -// (they just don't appear in a quiet interval) — so we scan every per-second +// (they just don't appear in a quiet interval), so we scan every per-second // block, skip block 0 (since-boot), skip loopback/removable/compressed // pseudo-devices, and take the max over the guest's real block device. IostatResult parse_iostat(const std::string& text) { @@ -563,7 +563,7 @@ int cmd_analyze(int seconds) { std::println("analyze: metric resource=net name=pkts value={:.2f} unit=per_s", net_pkts); // signal thresholds: Utilization/Saturation pair per resource. cpu's - // saturation threshold scales with this guest's own core count — a run + // saturation threshold scales with this guest's own core count: a run // queue longer than (cpus + headroom) means threads are waiting for a // CPU, the textbook USE definition of saturation. struct Candidate { diff --git a/examples/37-use-method-60s-analysis/go/main.go b/examples/37-use-method-60s-analysis/go/main.go index f7f8c33..3d041a5 100644 --- a/examples/37-use-method-60s-analysis/go/main.go +++ b/examples/37-use-method-60s-analysis/go/main.go @@ -9,8 +9,8 @@ // mpstat, pidstat, iostat, free, sar, ss, top) exactly as a human running the // checklist would, parses their plain-text tables generically (by column // *name*, read from each tool's own header row, never a hardcoded position), -// and for each of the four resources reports which USE signals — Utilization -// and Saturation — fired against a fixed threshold. Errors is reported too, +// and for each of the four resources reports which USE signals (Utilization +// and Saturation) fired against a fixed threshold. Errors is reported too, // fixed false: this chapter induces load, not kernel-logged faults, so a // real checklist run finds none. // @@ -81,7 +81,7 @@ func splitLines(text string) []string { // row is a parsed table row: column names (from a tool's own header line) // zipped with this row's values, so every parser below reads fields *by -// name* rather than by a hardcoded position — robust to sysstat/procps +// name* rather than by a hardcoded position, robust to sysstat/procps // column reordering across versions. type row struct { names []string @@ -119,7 +119,7 @@ func makeRow(headerTokens []string, headerSkip int, dataTokens []string, dataSki // captureResult is the Go analogue of std::expected: err // is set only when the shell itself could not be started (the "truly // fallible operation" per the book's convention), never when the captured -// command exits non-zero — its output is still valid text either way. +// command exits non-zero; its output is still valid text either way. type captureResult struct { out string err error @@ -129,7 +129,7 @@ type captureResult struct { // stdout+stderr. Forced to the C locale so decimal points/dates are // unambiguous regardless of the guest's configured locale. Grouped in a // subshell so a single trailing "2>&1" captures every stage of a pipeline -// (e.g. "dmesg | tail -n 5") — without the parens, 2>&1 binds only to the +// (e.g. "dmesg | tail -n 5"); without the parens, 2>&1 binds only to the // pipeline's last stage and an earlier stage's stderr (e.g. dmesg's own // permission-denied message) leaks past the capture. func runCapture(cmd string) captureResult { @@ -137,7 +137,7 @@ func runCapture(cmd string) captureResult { out, err := exec.Command("sh", "-c", full).Output() if err != nil { if _, ok := err.(*exec.ExitError); ok { - // the command ran and produced output but exited non-zero -- + // the command ran and produced output but exited non-zero; // that's not a capture failure, just like popen()/pclose() // never surfacing the child's exit status to the caller here. return captureResult{out: string(out)} @@ -149,7 +149,7 @@ func runCapture(cmd string) captureResult { // runToFile runs `env LC_ALL=C LANG=C ` to completion, redirecting // stdout+stderr to outfile. Used for the five interval samplers, each on its -// own goroutine, so they run *concurrently* — a 12s analyze window costs +// own goroutine, so they run *concurrently*; a 12s analyze window costs // ~12s wall-clock, not 5x that from running them one at a time. func runToFile(argv []string, outfile string) { full := append([]string{"env", "LC_ALL=C", "LANG=C"}, argv...) @@ -174,7 +174,7 @@ func readAll(path string) string { // ------------------------------------------------------- checklist parsers -- // Each parser reads a header line's column names from the tool's own output -// and zips subsequent data lines against that name list — see row.getOr +// and zips subsequent data lines against that name list; see row.getOr // above. type vmstatResult struct { @@ -183,7 +183,7 @@ type vmstatResult struct { } // vmstat has no aggregate "Average:" line (unlike mpstat/pidstat), and its -// *first* sample is a since-boot average, not a live interval — both quirks +// *first* sample is a since-boot average, not a live interval; both quirks // are handled here: we scan every data row after the header, skip row 0, and // take the max across the rest (the steady-state saturation signal, not // diluted by an average across the whole window). @@ -228,7 +228,7 @@ type mpstatResult struct { } // mpstat -P ALL's own "Average:" row over the whole window (excludes the -// since-boot first report automatically — sysstat's job, not ours). +// since-boot first report automatically (sysstat's job, not ours). func parseMpstat(text string) mpstatResult { res := mpstatResult{} var header, data []string @@ -291,7 +291,7 @@ type iostatResult struct { } // iostat -xz has no Average line either, AND -z hides idle devices entirely -// (they just don't appear in a quiet interval) — so we scan every per-second +// (they just don't appear in a quiet interval), so we scan every per-second // block, skip block 0 (since-boot), skip loopback/removable/compressed // pseudo-devices, and take the max over the guest's real block device. func parseIostat(text string) iostatResult { @@ -680,7 +680,7 @@ func cmdAnalyze(seconds int) int { fmt.Printf("analyze: metric resource=net name=pkts value=%.2f unit=per_s\n", netPkts) // signal thresholds: Utilization/Saturation pair per resource. cpu's - // saturation threshold scales with this guest's own core count — a run + // saturation threshold scales with this guest's own core count; a run // queue longer than (cpus + headroom) means threads are waiting for a // CPU, the textbook USE definition of saturation. type candidate struct { diff --git a/examples/37-use-method-60s-analysis/rust/src/main.rs b/examples/37-use-method-60s-analysis/rust/src/main.rs index e7fffcc..9d8a989 100644 --- a/examples/37-use-method-60s-analysis/rust/src/main.rs +++ b/examples/37-use-method-60s-analysis/rust/src/main.rs @@ -9,14 +9,14 @@ // mpstat, pidstat, iostat, free, sar, ss, top) exactly as a human running the // checklist would, parses their plain-text tables generically (by column // *name*, read from each tool's own header row, never a hardcoded position), -// and for each of the four resources reports which USE signals -- Utilization -// and Saturation -- fired against a fixed threshold. Errors is reported too, +// and for each of the four resources reports which USE signals (Utilization +// and Saturation) fired against a fixed threshold. Errors is reported too, // fixed false: this chapter induces load, not kernel-logged faults, so a // real checklist run finds none. // // Rust idioms used throughout, in place of the C++ reference's jthread pools, // std::atomic counters, and std::expected: std::thread::scope (the jthread -// analogue -- worker threads are RAII-joined when the scope block ends), +// analogue: worker threads are RAII-joined when the scope block ends), // AtomicU64 counters shared across them by reference, std::process::Command // for every subprocess capture and spawn, and OpenOptionsExt::custom_flags // plus FileExt::write_at for the O_DSYNC pwrite path. Every observable line @@ -66,7 +66,7 @@ fn split_lines(text: &str) -> Vec { // A parsed table row: column names (from a tool's own header line) zipped // with this row's values, so every parser below reads fields *by name* -// rather than by a hardcoded position -- robust to sysstat/procps column +// rather than by a hardcoded position: robust to sysstat/procps column // reordering across versions. struct Row { names: Vec, @@ -103,7 +103,7 @@ fn make_row( // std::expected here). fn run_capture(cmd: &str) -> io::Result { // Grouped in a subshell so a single trailing "2>&1" captures every stage - // of a pipeline (e.g. "dmesg | tail -n 5") -- without the parens, 2>&1 + // of a pipeline (e.g. "dmesg | tail -n 5"): without the parens, 2>&1 // binds only to the pipeline's last stage and an earlier stage's stderr // (e.g. dmesg's own permission-denied message) leaks past the capture. let full = format!("( env LC_ALL=C LANG=C {cmd} ) 2>&1"); @@ -113,7 +113,7 @@ fn run_capture(cmd: &str) -> io::Result { // Start `env LC_ALL=C LANG=C ` in the background, stdout+stderr // redirected to outfile; returns the child (or None on spawn failure). Used -// for the five interval samplers so they run *concurrently* -- a 12s analyze +// for the five interval samplers so they run *concurrently*: a 12s analyze // window costs ~12s wall-clock, not 5x that from running them one at a time. fn spawn_to_file(argv: &[&str], outfile: &Path) -> Option { let out = File::create(outfile).ok()?; @@ -140,7 +140,7 @@ fn read_all(path: &str) -> String { // ------------------------------------------------------- checklist parsers -- // Each parser reads a header line's column names from the tool's own output -// and zips subsequent data lines against that name list -- see Row::get +// and zips subsequent data lines against that name list: see Row::get // above. #[derive(Default)] @@ -150,7 +150,7 @@ struct VmstatResult { } // vmstat has no aggregate "Average:" line (unlike mpstat/pidstat), and its -// *first* sample is a since-boot average, not a live interval -- both quirks +// *first* sample is a since-boot average, not a live interval; both quirks // are handled here: we scan every data row after the header, skip row 0, and // take the max across the rest (the steady-state saturation signal, not // diluted by an average across the whole window). @@ -191,7 +191,7 @@ struct MpstatResult { } // mpstat -P ALL's own "Average:" row over the whole window (excludes the -// since-boot first report automatically -- sysstat's job, not ours). +// since-boot first report automatically: sysstat's job, not ours). fn parse_mpstat(text: &str) -> MpstatResult { let mut res = MpstatResult::default(); let mut header: Vec = Vec::new(); @@ -250,7 +250,7 @@ struct IostatResult { } // iostat -xz has no Average line either, AND -z hides idle devices entirely -// (they just don't appear in a quiet interval) -- so we scan every per-second +// (they just don't appear in a quiet interval), so we scan every per-second // block, skip block 0 (since-boot), skip loopback/removable/compressed // pseudo-devices, and take the max over the guest's real block device. fn parse_iostat(text: &str) -> IostatResult { @@ -331,7 +331,7 @@ fn sat_cpu(seconds: u64, workers: usize) -> (u64, u64) { checksum.fetch_add(x, Ordering::Relaxed); // keeps x provably live (no DCE) }); } - }); // worker threads join here -- thread::scope is the jthread analogue + }); // worker threads join here: thread::scope is the jthread analogue ( total_iters.load(Ordering::Relaxed), checksum.load(Ordering::Relaxed), @@ -649,7 +649,7 @@ fn cmd_analyze(seconds: i32) -> i32 { println!("analyze: metric resource=net name=pkts value={net_pkts:.2} unit=per_s"); // signal thresholds: Utilization/Saturation pair per resource. cpu's - // saturation threshold scales with this guest's own core count -- a run + // saturation threshold scales with this guest's own core count: a run // queue longer than (cpus + headroom) means threads are waiting for a // CPU, the textbook USE definition of saturation. struct Candidate { diff --git a/examples/37-use-method-60s-analysis/verify.lua b/examples/37-use-method-60s-analysis/verify.lua index d6600ed..8bbeffd 100644 --- a/examples/37-use-method-60s-analysis/verify.lua +++ b/examples/37-use-method-60s-analysis/verify.lua @@ -1,45 +1,45 @@ -- Verify loadmix (37-use-method-60s-analysis) for LSP_LANG. A vm-mode -- example: driven directly against systems-target via plain scp/ssh to --- /tmp/lsp37-/app, never `demo.sh run` under TARGET -- +-- /tmp/lsp37-/app, never `demo.sh run` under TARGET: -- deploy-to-vm.sh's `ssh -t` injects CRLF that breaks Lua patterns, and its -- fixed /home/fedora/app path can be poisoned by a stale binary left over -- from an unrelated example (this happened once during golden capture). --- Every remote invocation below uses the full absolute remote_bin path -- +-- Every remote invocation below uses the full absolute remote_bin path: -- never a leading `cd` that a later `;`-separated command could silently -- fail to inherit. -- -- Asserts observable behavior, identical in shape across cpp/go/rust: --- 1. build, then stage to /tmp/lsp37-/app -- both must exit 0. +-- 1. build, then stage to /tmp/lsp37-/app, both must exit 0. -- 2. error shapes: no args / bad --resource / --seconds 0 / bad analyze -- flag all exit 2 with the two-line usage on stderr. -- 3. saturate cpu/io/net (and mem) at a short --seconds: exit 0, the -- start/done line shapes, the documented worker-count formula against --- the guest's real nproc, ops>0 (mem excepted -- the golden capture +-- the guest's real nproc, ops>0 (mem excepted, the golden capture -- found ops legitimately 0 at a short duration, since allocation alone -- can consume the whole budget), and the elapsed wall time actually --- tracks --seconds for cpu/io/net -- this is exactly what would have +-- tracks --seconds for cpu/io/net: this is exactly what would have -- caught the pre-fix saturator bug where sat_* exited in milliseconds -- regardless of the requested duration. mem gets a generous (x5) -- ceiling instead of a tight window, since it can legitimately -- *overrun* --seconds by several seconds under real swap pressure -- (the inner touch loop only checks the stop flag between passes, per -- the golden-capture calibration notes). --- 4. idle `analyze --seconds N`: the exact skeleton -- 1 start line, 5 +-- 4. idle `analyze --seconds N`: the exact skeleton, 1 start line, 5 -- tool lines, 8 metric lines (each %d+%.%d%d), 12 signal lines in -- cpu/mem/io/net x Utilization/Saturation/Errors order, 1 verdict -- line, and "analyze: done" as the last line of the app's own output. -- 5. load attribution: a background `--resource cpu` saturator makes a -- concurrent `analyze` see busy_pct and run_queue rise, Utilization --- fire, and the verdict land on cpu -- retried up to 3 attempts +-- fire, and the verdict land on cpu, retried up to 3 attempts -- against a generous margin (calibrated from 3 real captures showing -- busy_pct 1.34->100.00 and run_queue 3.00->7.00-8.00, both clearing -- the gate by a large margin). -- 6. wall-clock sanity: `analyze --seconds N` finishes in close to N --- seconds, not 5N -- proving the five interval samplers +-- seconds, not 5N, proving the five interval samplers -- (vmstat/mpstat/iostat/sar/pidstat) run concurrently, not serially. -- -- Never compares formatted decimals across languages (C++, Go, and Rust --- disagree on halfway rounding) -- only shapes (%d+%.%d%d) and numeric +-- disagree on halfway rounding), only shapes (%d+%.%d%d) and numeric -- ranges/ratios are asserted. -- -- Invoked from the example directory with LSP_LANG, TARGET, and REPO_ROOT @@ -89,7 +89,7 @@ end -- Boolean assertion via the checks.lua pattern-match API: match "ok" against -- "ok" when true, or force a mismatch (with the real detail in the label) --- when false -- same helper as example 39's verify.lua. +-- when false, same helper as example 39's verify.lua. local function expect_true(cond, label, detail) if cond then checks.expect_match("ok", "ok", label) @@ -104,7 +104,7 @@ local remote_dir = "/tmp/lsp37-" .. lang local remote_bin = remote_dir .. "/app" -- --------------------------------------------------------------------------- --- 1. build, then stage to /tmp/lsp37-/app -- named "app" so /proc comm +-- 1. build, then stage to /tmp/lsp37-/app, named "app" so /proc comm -- is "app", the book-wide rule. -- --------------------------------------------------------------------------- @@ -126,7 +126,7 @@ local nproc_r = ssh_run("nproc") local nproc = tonumber(nproc_r.out:match("%d+")) or 1 -- --------------------------------------------------------------------------- --- 2. error shapes -- exit 2, usage on stderr. +-- 2. error shapes: exit 2, usage on stderr. -- --------------------------------------------------------------------------- local function expect_usage(args, label) @@ -141,7 +141,7 @@ expect_usage("--resource cpu --seconds 0", "--resource cpu --seconds 0") expect_usage("analyze --bogus", "analyze --bogus") -- --------------------------------------------------------------------------- --- 3. saturate mode -- cpu/io/net (tight elapsed window) plus mem (generous +-- 3. saturate mode: cpu/io/net (tight elapsed window) plus mem (generous -- ceiling, no ops>0 requirement), per the golden-capture calibration. -- --------------------------------------------------------------------------- @@ -234,7 +234,7 @@ do local elapsed_ms = tonumber(r.out:match("SAT_ELAPSED_MS=(%d+)")) -- mem can OVERRUN --seconds by several seconds under real swap pressure - -- (a 12.6s actual for an 8s request was observed) -- a generous x5 + -- (a 12.6s actual for an 8s request was observed), a generous x5 -- ceiling, not the tight cpu/io/net window. expect_true(elapsed_ms ~= nil and elapsed_ms <= mem_secs * 5000, lang .. ": mem finishes within a generous ceiling (secondsx5), tolerating the known swap overrun", @@ -284,7 +284,7 @@ for _, m in ipairs({ end -- All 12 signal lines, in cpu/mem/io/net x Utilization/Saturation/Errors --- order -- one pattern enforces both presence and order via lazy ".-" gaps. +-- order, one pattern enforces both presence and order via lazy ".-" gaps. local signal_order = {} for _, res in ipairs({ "cpu", "mem", "io", "net" }) do table.insert(signal_order, @@ -313,7 +313,7 @@ expect_true(an_elapsed_ms ~= nil and an_elapsed_ms < analyze_secs * 5000, -- --------------------------------------------------------------------------- -- 5. load attribution: a background --resource cpu saturator makes a -- concurrent analyze attribute the load to cpu. Retried up to 3 attempts --- (per the plan's own hedge, mirroring example 36's cpu-busy check) -- +-- (per the plan's own hedge, mirroring example 36's cpu-busy check): -- the golden capture cleared this gate by a large margin every time -- (busy_pct 100.00 vs. 60 threshold; run_queue 7-8 vs. nproc+2=4), so a -- noisy guest should still pass, not produce a false red. diff --git a/examples/38-opentelemetry-to-lgtm/README.md b/examples/38-opentelemetry-to-lgtm/README.md index 39b523c..6c41638 100644 --- a/examples/38-opentelemetry-to-lgtm/README.md +++ b/examples/38-opentelemetry-to-lgtm/README.md @@ -1,7 +1,7 @@ -# _template — hello-syscall +# _template: hello-syscall The seed for every runnable example in the book. It implements the same tiny -program three times — C++23, Go, and Rust — each printing exactly one line: +program three times, in C++23, Go, and Rust, each printing exactly one line: ``` pid on () @@ -26,9 +26,9 @@ _template/ Every language directory has a `demo.sh` with the same interface, identical across the whole book: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. diff --git a/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp b/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp index 49005fa..53be3a0 100644 --- a/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp +++ b/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// lsp-otel-cpp — a traced request/response loop (chapter 38: OpenTelemetry +// lsp-otel-cpp: a traced request/response loop (chapter 38: OpenTelemetry // to LGTM). // // app serve --port PORT [--otel-endpoint URL] @@ -9,14 +9,14 @@ // ("work", "respond"), a monotonic counter (requests_total) is incremented, // and a histogram (request_duration, milliseconds) records the parent // span's wall time. drive is an untraced TCP client that sends N requests -// down one connection and waits for N replies -- the load generator for +// down one connection and waits for N replies: the load generator for // serve's telemetry, not a telemetry source itself. // // Every signal appears exactly once in the OTLP stream: serve's own // SIGINT/SIGTERM handling flushes and shuts down the SDK before exit, the // C++ way of RAII (an explicit flush_and_shutdown() call from the signal // path, since a detached accept thread and a process-ending exit give no -// destructor a guaranteed chance to run -- the same reasoning the Rust port +// destructor a guaranteed chance to run: the same reasoning the Rust port // documents for its own explicit shutdown() call). #include "telemetry.hpp" @@ -73,7 +73,7 @@ std::string_view trim(std::string_view s) { } // write_all loops over write(2) so short writes (always possible on a -// socket) never truncate a reply -- the syscall-level equivalent of Go's +// socket) never truncate a reply: the syscall-level equivalent of Go's // bufio.Writer.Flush() / Rust's Write::write_all(). bool write_all(int fd, std::string_view data) { std::size_t off = 0; @@ -211,8 +211,8 @@ void serve_conn(const telemetry::Handle& t, int fd) { ::close(fd); } -// Event unifies the two sources serve waits on -- an incoming connection -// and a shutdown signal -- into the single queue a Go select over +// Event unifies the two sources serve waits on (an incoming connection +// and a shutdown signal) into the single queue a Go select over // sigCh/connCh, or the Rust port's mpsc::channel over its Event enum, would // use. struct Event { @@ -280,12 +280,12 @@ int run_serve(const std::vector& args) { // A shell running "app serve ... &" as a background job (the exact // pattern demo.sh/verify.lua use) sets SIGINT (and SIGQUIT) to SIG_IGN - // in the child before exec -- standard POSIX shell behavior, inherited + // in the child before exec: standard POSIX shell behavior, inherited // across exec. Blocking + sigwait() alone does not override an // inherited "ignore" disposition; installing a real handler does, which // is exactly what Go's signal.Notify and Rust's signal-hook do under // the hood. The handler itself never runs (the signal stays blocked - // below, so sigwait() -- not the handler -- is what actually consumes + // below, so sigwait() (not the handler) is what actually consumes // it); this call exists purely to force the disposition away from // SIG_IGN. struct sigaction sa {}; @@ -295,9 +295,9 @@ int run_serve(const std::vector& args) { ::sigaction(SIGTERM, &sa, nullptr); // Block SIGINT/SIGTERM in this thread BEFORE spawning (or triggering - // the SDK to spawn) any others: every thread created from here on -- - // including the BatchSpanProcessor/PeriodicExportingMetricReader - // background threads telemetry::init() starts -- inherits the blocked + // the SDK to spawn) any others: every thread created from here on + // (including the BatchSpanProcessor/PeriodicExportingMetricReader + // background threads telemetry::init() starts) inherits the blocked // mask, so only the dedicated sigwait() thread below ever consumes // them. Doing this BEFORE telemetry::init() matters: those SDK threads // would otherwise start with the signal unblocked and the kernel could @@ -334,7 +334,7 @@ int run_serve(const std::vector& args) { // Signal watcher: sigwait() blocks (in this thread only) until one of // the blocked signals is pending, then forwards it onto the shared - // event queue -- the same role Go's signal.Notify channel and the + // event queue: the same role Go's signal.Notify channel and the // Rust port's signal-hook thread play. std::thread signal_thread([&channel, mask] { int sig = 0; @@ -347,7 +347,7 @@ int run_serve(const std::vector& args) { // Accept loop: forwards each accepted connection onto the same event // queue. The process exits by returning all the way out of main, not - // by joining this thread -- same lifecycle as Go's accept goroutine + // by joining this thread: same lifecycle as Go's accept goroutine // (cut short by os.Exit) and the Rust port's accept thread (never // joined). std::thread accept_thread([&channel, listen_fd] { @@ -382,7 +382,7 @@ int run_serve(const std::vector& args) { // ------------------------------------------------------------------- drive // run_drive is a plain, untraced TCP client: it generates the load that -// makes serve's telemetry interesting, but emits none of its own -- the +// makes serve's telemetry interesting, but emits none of its own: the // service under observation is serve, not drive. int run_drive(const std::vector& args) { std::int64_t n = 0; diff --git a/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp index 9aeada8..5a1841e 100644 --- a/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp +++ b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp @@ -53,7 +53,7 @@ bool Handle::flush_and_shutdown(std::chrono::milliseconds timeout) { Handle init(const std::string& endpoint) { // The SDK's default log handler writes Warning/Error to stderr and - // Info/Debug to stdout -- the latter would corrupt serve's stdout + // Info/Debug to stdout: the latter would corrupt serve's stdout // contract (exactly two lines, ever). A no-op handler keeps this build // as quiet as the Go and Rust ports, which have no internal SDK logging // to begin with. diff --git a/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp index ba088fe..190dd00 100644 --- a/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp +++ b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp @@ -1,9 +1,9 @@ -// telemetry.hpp — OpenTelemetry wiring for lsp-otel-cpp (chapter 38: +// telemetry.hpp: OpenTelemetry wiring for lsp-otel-cpp (chapter 38: // OpenTelemetry to LGTM). Mirrors the Go reference's initTelemetry: a // TracerProvider batching spans over OTLP/HTTP, a MeterProvider exporting a // counter and a histogram on the same short interval, and one Handle that -// owns both so serve's signal path can flush-then-shutdown them explicitly -// -- the C++ RAII equivalent of the Go program's deferred cleanup func, +// owns both so serve's signal path can flush-then-shutdown them explicitly: +// the C++ RAII equivalent of the Go program's deferred cleanup func, // called out by hand rather than left to a destructor because "flush before // exit" has a deadline the process must observe (see flush_and_shutdown). #pragma once @@ -37,7 +37,7 @@ inline constexpr std::chrono::milliseconds export_interval{2000}; // Handle owns the trace/metric providers plus the two instruments serve // records against; it is move-only (Counter/Histogram are non-copyable -// nostd::unique_ptr) and deliberately has no destructor-driven shutdown -- +// nostd::unique_ptr) and deliberately has no destructor-driven shutdown: // flush_and_shutdown() must be called explicitly from the signal path so the // caller controls the timeout instead of an unobservable process-exit race. struct Handle { @@ -57,11 +57,11 @@ struct Handle { // init sets up the resource (service.name/version/deployment.environment), // the OTLP/HTTP trace and metric exporters against `endpoint` (a full URL, -// e.g. "http://localhost:4318" -- the /v1/traces and /v1/metrics paths are +// e.g. "http://localhost:4318": the /v1/traces and /v1/metrics paths are // appended here), and the requests_total counter + request_duration // histogram instruments. Aborts the process via die() semantics is the // caller's job; init itself only throws opentelemetry-cpp's own exceptions -// (none, in practice -- the factories are noexcept-safe by construction). +// (none, in practice: the factories are noexcept-safe by construction). Handle init(const std::string& endpoint); } // namespace telemetry diff --git a/examples/38-opentelemetry-to-lgtm/go/main.go b/examples/38-opentelemetry-to-lgtm/go/main.go index 54096ca..78e8f24 100644 --- a/examples/38-opentelemetry-to-lgtm/go/main.go +++ b/examples/38-opentelemetry-to-lgtm/go/main.go @@ -1,4 +1,4 @@ -// lsp-otel — a traced request/response loop (chapter 38: OpenTelemetry to LGTM). +// lsp-otel: a traced request/response loop (chapter 38: OpenTelemetry to LGTM). // // app serve --port PORT [--otel-endpoint URL] // app drive --n N --addr HOST:PORT @@ -8,7 +8,7 @@ // ("work", "respond"), a monotonic counter (requests_total) is incremented, // and a histogram (request_duration, milliseconds) records the parent // span's wall time. drive is an untraced TCP client that sends N requests -// down one connection and waits for N replies — the load generator for +// down one connection and waits for N replies; the load generator for // serve's telemetry, not a telemetry source itself. // // Every signal appears exactly once in the OTLP stream: serve's own @@ -291,7 +291,7 @@ func runServe(args []string) int { // ------------------------------------------------------------------- drive // runDrive is a plain, untraced TCP client: it generates the load that -// makes serve's telemetry interesting, but emits none of its own — the +// makes serve's telemetry interesting, but emits none of its own; the // service under observation is serve, not drive. func runDrive(args []string) int { n := 0 diff --git a/examples/38-opentelemetry-to-lgtm/rust/src/main.rs b/examples/38-opentelemetry-to-lgtm/rust/src/main.rs index 66e064f..7111fcf 100644 --- a/examples/38-opentelemetry-to-lgtm/rust/src/main.rs +++ b/examples/38-opentelemetry-to-lgtm/rust/src/main.rs @@ -1,4 +1,4 @@ -// lsp-otel-rust — a traced request/response loop (chapter 38: OpenTelemetry +// lsp-otel-rust: a traced request/response loop (chapter 38: OpenTelemetry // to LGTM). // // app serve --port PORT [--otel-endpoint URL] @@ -9,7 +9,7 @@ // ("work", "respond"), a monotonic counter (requests_total) is incremented, // and a histogram (request_duration, milliseconds) records the parent // span's wall time. drive is an untraced TCP client that sends N requests -// down one connection and waits for N replies — the load generator for +// down one connection and waits for N replies: the load generator for // serve's telemetry, not a telemetry source itself. // // Every signal appears exactly once in the OTLP stream: serve's own @@ -65,7 +65,7 @@ struct Telemetry { impl Telemetry { // shutdown flushes and closes both providers. Errors are reported but - // never fatal — the process is exiting regardless, this is best effort + // never fatal: the process is exiting regardless, this is best effort // draining of whatever the batch/periodic exporters still hold. fn shutdown(&self) { if let Err(e) = self.tracer_provider.shutdown() { @@ -78,11 +78,11 @@ impl Telemetry { } // init_telemetry wires up the OTLP HTTP/protobuf exporters (reqwest blocking -// client — no tokio runtime needed), a short-interval batch span processor +// client, no tokio runtime needed), a short-interval batch span processor // and periodic metric reader, and the two instruments serve records against. -// The per-signal OTLP HTTP exporters POST to exactly the URL given -- unlike +// The per-signal OTLP HTTP exporters POST to exactly the URL given: unlike // the base-endpoint env-var path, an explicit with_endpoint() does NOT append -// the signal path in opentelemetry-otlp 0.32 -- so the caller's base +// the signal path in opentelemetry-otlp 0.32, so the caller's base // (http://localhost:4318) is joined to /v1/traces and /v1/metrics here. // Posting to the bare base instead 404s at the collector and the export // silently fails (surfacing only as a shutdown error). @@ -238,8 +238,8 @@ fn serve_conn(t: &Telemetry, mut stream: TcpStream) { } } -// Event unifies the two sources serve waits on — an incoming connection and -// a shutdown signal — into the single channel a Go select over sigCh/connCh +// Event unifies the two sources serve waits on (an incoming connection and +// a shutdown signal) into the single channel a Go select over sigCh/connCh // would use. enum Event { Signal(&'static str), @@ -310,7 +310,7 @@ fn run_serve(args: &[String]) -> i32 { // Accept loop: a dedicated thread forwards each accepted connection onto // the same event channel. The process exits (via std::process::exit) // from the shutdown path below rather than closing the listener, so this - // thread never needs to be joined — same lifecycle as Go's accept + // thread never needs to be joined: same lifecycle as Go's accept // goroutine, which os.Exit simply cuts short. { let tx = tx.clone(); @@ -348,7 +348,7 @@ fn run_serve(args: &[String]) -> i32 { // ------------------------------------------------------------------- drive // run_drive is a plain, untraced TCP client: it generates the load that -// makes serve's telemetry interesting, but emits none of its own — the +// makes serve's telemetry interesting, but emits none of its own: the // service under observation is serve, not drive. fn run_drive(args: &[String]) -> i32 { let mut n: i64 = 0; diff --git a/examples/38-opentelemetry-to-lgtm/verify.lua b/examples/38-opentelemetry-to-lgtm/verify.lua index 1b805ba..0d11482 100644 --- a/examples/38-opentelemetry-to-lgtm/verify.lua +++ b/examples/38-opentelemetry-to-lgtm/verify.lua @@ -4,14 +4,14 @@ -- host: it builds the binary, runs `serve` in the background, drives N traced -- requests through it, sends SIGINT so the SDK flushes and shuts down, then -- proves the telemetry actually LANDED IN THE STACK by querying Prometheus --- (metrics) and Tempo (traces) through Grafana's datasource proxy -- not by +-- (metrics) and Tempo (traces) through Grafana's datasource proxy, not by -- trusting the program's own "I exported" claim. -- -- The query surface: only Grafana (:3000, admin/admin) is published off the -- otel-lgtm container; it proxies to the internal Prometheus (uid=prometheus, -- metrics) and Tempo (uid=tempo, traces). Each language tags its telemetry --- with a distinct service.name -- lsp-otel-cpp / lsp-otel-go / lsp-otel-rust --- -- so the three languages' signals never collide in the shared stack and +-- with a distinct service.name (lsp-otel-cpp / lsp-otel-go / lsp-otel-rust), +-- so the three languages' signals never collide in the shared stack and -- each run is queried by its own service_name. -- -- Asserts observable behavior, identical in shape across cpp/go/rust: diff --git a/examples/39-benchmarking-without-lies/README.md b/examples/39-benchmarking-without-lies/README.md index 1df6226..bc78c2b 100644 --- a/examples/39-benchmarking-without-lies/README.md +++ b/examples/39-benchmarking-without-lies/README.md @@ -1,8 +1,9 @@ # 39-benchmarking-without-lies Chapter 39's example: `benchlab`, a microbenchmark harness for the chatterd -frame codec (`encode_frame`/`decode_frame`, introduced ch21) — implemented -three times, C++23, Go, and Rust, with one shared observable contract. The +frame codec (`encode_frame`/`decode_frame`, introduced with the TCP chatterd +server), implemented three times, C++23, Go, and Rust, with one shared +observable contract. The point of this chapter is not the codec; it's the two different ways to measure it, side by side in the same binary. @@ -13,19 +14,19 @@ benchlab --op encode|decode|roundtrip --iters N [--warmup W] # the real thing benchlab --lie [--op encode|decode|roundtrip] # the anti-pattern ``` -`--op` selects what's timed against a fixed workload — one chatterd `DELIVER` +`--op` selects what's timed against a fixed workload: one chatterd `DELIVER` frame (`alice\0the quick brown fox jumps over the lazy dog, three times, for benchlab`): -- `encode` — build the frame from the payload bytes. -- `decode` — parse a pre-built frame back into `(type, payload)`. -- `roundtrip` — both, per iteration. +- `encode`: build the frame from the payload bytes. +- `decode`: parse a pre-built frame back into `(type, payload)`. +- `roundtrip`: both, per iteration. ### The real thing -`--iters N --warmup W` (default `W=1000`) runs `W` untimed warmup iterations -— letting the allocator, branch predictor, and (on Go) the runtime reach -something like steady state — then times each of the next `N` iterations +`--iters N --warmup W` (default `W=1000`) runs `W` untimed warmup iterations, +letting the allocator, branch predictor, and (on Go) the runtime reach +something like steady state, then times each of the next `N` iterations individually with a monotonic clock, and reports: ``` @@ -36,26 +37,26 @@ benchlab: co_p99_ns=914420 expected_interval_ns=87 co_n=113003 benchlab: checksum=000000000079c3e0 ``` -- `min_ns`/`median_ns`/`p99_ns`/`max_ns` — the raw latency distribution in +- `min_ns`/`median_ns`/`p99_ns`/`max_ns`: the raw latency distribution in nanoseconds, nearest-rank percentiles over the sorted sample. -- `co_p99_ns` — the **coordinated-omission-corrected** p99 (see below); +- `co_p99_ns`: the **coordinated-omission-corrected** p99 (see below); always `>= p99_ns`. -- `expected_interval_ns` — the interval the correction assumes, derived from +- `expected_interval_ns`: the interval the correction assumes, derived from the run's own raw median (documented, not a hidden knob). -- `co_n` — how many samples the corrected distribution actually holds +- `co_n`: how many samples the corrected distribution holds (`>= n`, since correction only ever adds synthetic samples). -- `checksum` — a running sum of a value derived from every call's result +- `checksum`: a running sum of a value derived from every call's result (the encoded frame's last byte, or the decoded payload's length). Two purposes: it stops the compiler from proving the loop body's result is - unused and deleting it, and — because the workload is fixed — it is + unused and deleting it, and (because the workload is fixed) it is **bit-for-bit identical every run and across all three languages** (`encode` above: `00000000000fb388` in C++, Go, *and* Rust, for the exact same input). If the codec is behaving identically everywhere, the checksums must match; a diverging checksum is a bug, not noise. -The `min`/`median`/`p99`/`max` numbers above are one machine's numbers — like +The `min`/`median`/`p99`/`max` numbers above are one machine's numbers: like every other latency figure in this book, their magnitude is run- and -hardware-dependent. What's invariant, and what `verify.lua` actually checks, +hardware-dependent. What's invariant, and what `verify.lua` checks, is the *shape*: `min <= median <= p99 <= max`, `co_p99 >= p99`, `co_n >= n`, and a stable checksum across repeated runs of the same workload. @@ -63,8 +64,8 @@ and a stable checksum across repeated runs of the same workload. This harness is a **closed loop**: iteration *N+1* never starts until iteration *N* returns. That's also the classic setup for **coordinated -omission** (Gil Tene's term): if iteration *N* stalls — a page fault, a -context switch, a GC-adjacent pause, a neighbor process getting the core — +omission** (Gil Tene's term): if iteration *N* stalls (a page fault, a +context switch, a GC-adjacent pause, a neighbor process getting the core), the *one* sample recorded for it is that stall's full latency, but a fixed-rate caller (a real client sending one request every `E` nanoseconds) would have had several requests queue up behind that stall, each observing a @@ -76,9 +77,9 @@ The correction (HdrHistogram's `recordValueWithExpectedInterval`): for every raw sample `v` above the expected interval `E`, backfill synthetic samples `v - E, v - 2E, ...` down to (but not below) `E`, representing the virtual requests that would have been queued behind the stall. `E` here is set to the -run's own raw median — i.e., "assume this loop could sustain its typical -per-op rate" — so no extra flag is needed and the two numbers come from one -run. A safety cap (`co_n <= 5,000,000`) stops a pathological single giant +run's own raw median (in other words, "assume this loop could sustain its +typical per-op rate"), so no extra flag is needed and the two numbers come +from one run. A safety cap (`co_n <= 5,000,000`) stops a pathological single giant stall from producing an unbounded backfill. `co_p99_ns` in the example above (`914420`) vs. the raw `p99_ns` (`124`) is @@ -87,7 +88,7 @@ to the raw p99 (only 1000 of 100000 samples count toward it) but dominates the corrected one, because that single stall represented what would have been thousands of queued requests in a real fixed-rate system. -### `--lie` — the benchmark this chapter argues against +### `--lie`: the benchmark this chapter argues against ``` $ ./demo.sh cpp run --lie --op roundtrip @@ -95,15 +96,15 @@ benchlab: lie op=roundtrip (no warmup, single wall-clock sample, ignores varianc benchlab: lie elapsed_ns=3069 sink=76 ``` -No warmup, one call, one wall-clock delta, no percentiles, no variance — and -therefore no way to tell a JIT/allocator-cold first call, a scheduler -preemption, or a page fault apart from steady-state performance. This is not -a strawman: it is the shape of `time single_call()` benchmarks that get -pasted into commit messages. `--lie`'s output never contains `p99_ns=` or -`median_ns=` — there is nothing here to compute a percentile *from* — and +No warmup, one call, one wall-clock delta, no percentiles, no variance: +there's no way to tell a JIT/allocator-cold first call, a scheduler +preemption, or a page fault apart from steady-state performance. +`time single_call()` benchmarks with this shape get pasted into commit +messages routinely. `--lie`'s output never contains `p99_ns=` or +`median_ns=`: there is nothing here to compute a percentile *from*, and `verify.lua` asserts that absence directly. -## Wire format (the codec under test, unchanged since ch21) +## Wire format (the codec under test, unchanged since the TCP chatterd server introduced it) ``` byte: 0 1 2 3 4 5 6 ............. 6+LEN-1 @@ -114,11 +115,12 @@ pasted into commit messages. `--lie`'s output never contains `p99_ns=` or ``` `benchlab` re-implements only `encode_frame`/`decode_frame` against this -format (a `DELIVER`, type `0x03`) — not the daemon; see ch21/22/27 for the -networked chatterd that actually speaks this frame over a socket. `decode` +format (a `DELIVER`, type `0x03`), not the daemon. See the chatterd servers +elsewhere in the book for the networked chatterd that speaks this frame over +a socket. `decode` and `roundtrip` use `std::expected` (C++), `Result<(u8, Vec), String>` with `?` (Rust), and a wrapped `error` via -`%w` (Go) to report a malformed frame — on this fixed, always-valid workload +`%w` (Go) to report a malformed frame. On this fixed, always-valid workload that path is never taken, but it exists because a hand-rolled codec harness should look exactly like calling code would, not a stripped-down happy path. @@ -136,22 +138,22 @@ should look exactly like calling code would, not a stripped-down happy path. Every language uses a small hand-rolled harness rather than a benchmarking framework (Google Benchmark / `go test -bench` / criterion): `benchlab` is a CLI that prints one shared, cross-language table shape, which a framework's -own report format would not give us for free, and there is no third-party -dependency to pin or fetch for any of the three languages — `rust/Cargo.lock` +own report format would not produce, and there is no third-party +dependency to pin or fetch for any of the three languages: `rust/Cargo.lock` carries no external crates. ## The demo contract Every language directory has a `demo.sh` with the book-wide interface: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary (e.g. `./demo.sh run --op decode --iters 100000 --warmup 5000`) - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh` instead of running locally. -This is a host-only example — pure in-process computation, no sockets, no VM. +This is a host-only example: pure in-process computation, no sockets, no VM. Exit codes: `0` a run completed and printed its table (or its lie), `1` the codec broke on its own fixed workload (a bug, never expected here), `2` a usage error (missing/unknown flags, or `--warmup >= --iters`). @@ -168,7 +170,7 @@ time and asserts, per language: (the run's own raw median, floored at 1); and a 64-bit hex checksum is printed; 2. running the identical fixed workload twice yields the identical checksum - both times — the codec's output is deterministic even though the ns + both times: the codec's output is deterministic even though the ns timings are not; 3. `--lie` exits 0, identifies itself as the lie, reports exactly one `elapsed_ns` sample, and contains **no** `p99_ns=`/`median_ns=` anywhere diff --git a/examples/39-benchmarking-without-lies/cpp/src/main.cpp b/examples/39-benchmarking-without-lies/cpp/src/main.cpp index c85d3f4..c71b426 100644 --- a/examples/39-benchmarking-without-lies/cpp/src/main.cpp +++ b/examples/39-benchmarking-without-lies/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// benchlab (C++23) — proper benchmarking of the chatterd frame codec (ch21). +// benchlab (C++23): proper benchmarking of the chatterd frame codec (ch21). // // Usage: // app --op encode|decode|roundtrip --iters N [--warmup W] # the real thing @@ -7,13 +7,13 @@ // The "real thing" times each iteration individually with a monotonic clock, // discards a warmup phase, and reports min/median/p99/max plus a // coordinated-omission-corrected p99 (see co_correct below). `--lie` times a -// single unwarmed call with wall-clock and reports one number — exactly the +// single unwarmed call with wall-clock and reports one number: exactly the // benchmark the rest of this example exists to discredit. // // Wire format (canonical chatterd chat frame, introduced ch21): // [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] -// This file re-implements only encode_frame/decode_frame — the codec under -// test — not the daemon; see ch21/22/27 for the networked version. +// This file re-implements only encode_frame/decode_frame (the codec under +// test), not the daemon; see ch21/22/27 for the networked version. #include #include @@ -29,7 +29,7 @@ namespace { // --------------------------------------------------------------------------- -// Frame codec under test — canonical chatterd chat frame (see file header). +// Frame codec under test: canonical chatterd chat frame (see file header). // --------------------------------------------------------------------------- constexpr std::size_t kHeaderLen = 6; constexpr std::uint8_t kMagic0 = 0x43; @@ -56,7 +56,7 @@ std::string encode_frame(std::uint8_t type, std::string_view body) { return out; } -// Decodes exactly one frame that fills `buf` completely — no partial-read +// Decodes exactly one frame that fills `buf` completely, no partial-read // reassembly here, that lives in the networked chatterd (ch21/22/27); this // harness only measures the codec's per-call cost. [[nodiscard]] std::expected decode_frame(std::string_view buf) { @@ -114,7 +114,7 @@ std::string_view op_name(Op op) { // One iteration of the requested op against the fixed workload. Returns a // small scalar derived from the result so the caller can fold it into a -// checksum — this is what keeps the optimizer from proving the call's result +// checksum: this is what keeps the optimizer from proving the call's result // is unused and deleting it. std::uint64_t run_once(Op op, const std::string& body, const std::string& prebuilt_frame) { switch (op) { @@ -156,7 +156,7 @@ std::int64_t percentile(const std::vector& sorted, double p) { } // Coordinated-omission correction (HdrHistogram's recordValueWithExpectedInterval): -// this harness is a closed loop — iteration N+1 never starts until N returns — +// this harness is a closed loop (iteration N+1 never starts until N returns), // so a stall (a page fault, a context switch, a scheduling hiccup) is // recorded as a single slow sample. A fixed-rate caller would instead have // had many requests queue up behind that stall, each experiencing a similar diff --git a/examples/39-benchmarking-without-lies/go/main.go b/examples/39-benchmarking-without-lies/go/main.go index 7ba2436..ce7f881 100644 --- a/examples/39-benchmarking-without-lies/go/main.go +++ b/examples/39-benchmarking-without-lies/go/main.go @@ -1,4 +1,4 @@ -// benchlab (Go) — proper benchmarking of the chatterd frame codec (ch21). +// benchlab (Go): proper benchmarking of the chatterd frame codec (ch21). // // Usage: // @@ -9,15 +9,15 @@ // (time.Now's monotonic reading), discards a warmup phase, and reports // min/median/p99/max plus a coordinated-omission-corrected p99 (see // coCorrect below). `--lie` times a single unwarmed call with wall-clock and -// reports one number — exactly the benchmark this example exists to +// reports one number, exactly the benchmark this example exists to // discredit. // // Wire format (canonical chatterd chat frame, introduced ch21): // // [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] // -// This file re-implements only encodeFrame/decodeFrame — the codec under -// test — not the daemon; see ch21/22/27 for the networked version. +// This file re-implements only encodeFrame/decodeFrame (the codec under +// test), not the daemon; see ch21/22/27 for the networked version. package main import ( @@ -31,7 +31,7 @@ import ( ) // --------------------------------------------------------------------------- -// Frame codec under test — canonical chatterd chat frame (see file header). +// Frame codec under test: canonical chatterd chat frame (see file header). // --------------------------------------------------------------------------- const ( headerLen = 6 @@ -53,7 +53,7 @@ func encodeFrame(typ byte, body []byte) []byte { return out } -// decodeFrame decodes exactly one frame that fills buf completely — no +// decodeFrame decodes exactly one frame that fills buf completely; no // partial-read reassembly here, that lives in the networked chatterd // (ch21/22/27); this harness only measures the codec's per-call cost. func decodeFrame(buf []byte) (byte, []byte, error) { @@ -134,7 +134,7 @@ func (o op) String() string { // runOnce performs one iteration of the requested op against the fixed // workload, returning a small scalar derived from the result so the caller -// can fold it into a checksum — this is what keeps the compiler (and a human +// can fold it into a checksum; this is what keeps the compiler (and a human // skimming the loop) from mistaking the call for dead code. A codec bug // (never expected on this fixed workload) is wrapped and propagated. func runOnce(o op, body, prebuiltFrame []byte) (uint64, error) { @@ -172,8 +172,8 @@ func percentile(sorted []int64, p float64) int64 { } // coCorrect applies the coordinated-omission correction (HdrHistogram's -// recordValueWithExpectedInterval): this harness is a closed loop — -// iteration N+1 never starts until N returns — so a stall (a GC pause, a +// recordValueWithExpectedInterval): this harness is a closed loop; +// iteration N+1 never starts until N returns, so a stall (a GC pause, a // page fault, a scheduling hiccup) is recorded as a single slow sample. A // fixed-rate caller would instead have had many requests queue up behind // that stall, each experiencing a similar multiple-of-the-target-interval diff --git a/examples/39-benchmarking-without-lies/rust/src/main.rs b/examples/39-benchmarking-without-lies/rust/src/main.rs index a8cafb0..488d46e 100644 --- a/examples/39-benchmarking-without-lies/rust/src/main.rs +++ b/examples/39-benchmarking-without-lies/rust/src/main.rs @@ -1,4 +1,4 @@ -// benchlab (Rust) — proper benchmarking of the chatterd frame codec (ch21). +// benchlab (Rust): proper benchmarking of the chatterd frame codec (ch21). // // Usage: // app --op encode|decode|roundtrip --iters N [--warmup W] # the real thing @@ -7,19 +7,19 @@ // The "real thing" times each iteration individually with a monotonic clock, // discards a warmup phase, and reports min/median/p99/max plus a // coordinated-omission-corrected p99 (see co_correct below). `--lie` times a -// single unwarmed call with wall-clock and reports one number — exactly the +// single unwarmed call with wall-clock and reports one number: exactly the // benchmark the rest of this example exists to discredit. // // Wire format (canonical chatterd chat frame, introduced ch21): // [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] -// This file re-implements only encode_frame/decode_frame — the codec under -// test — not the daemon; see ch21/22/27 for the networked version. +// This file re-implements only encode_frame/decode_frame (the codec under +// test), not the daemon; see ch21/22/27 for the networked version. use std::process::ExitCode; use std::time::Instant; // --------------------------------------------------------------------------- -// Frame codec under test — canonical chatterd chat frame (see file header). +// Frame codec under test: canonical chatterd chat frame (see file header). // --------------------------------------------------------------------------- const HEADER_LEN: usize = 6; const MAGIC0: u8 = 0x43; @@ -39,7 +39,7 @@ fn encode_frame(typ: u8, body: &[u8]) -> Vec { out } -// Decodes exactly one frame that fills `buf` completely — no partial-read +// Decodes exactly one frame that fills `buf` completely: no partial-read // reassembly here, that lives in the networked chatterd (ch21/22/27); this // harness only measures the codec's per-call cost. fn decode_frame(buf: &[u8]) -> Result<(u8, Vec), String> { @@ -103,7 +103,7 @@ impl Op { // One iteration of the requested op against the fixed workload. Returns a // small scalar derived from the result so the caller can fold it into a -// checksum — this is what keeps the optimizer from proving the call's result +// checksum: this is what keeps the optimizer from proving the call's result // is unused and deleting it. Propagates a codec bug (should never trigger on // this fixed workload) via `?` up to main. fn run_once(op: Op, body: &[u8], prebuilt_frame: &[u8]) -> Result { @@ -133,7 +133,7 @@ fn percentile(sorted: &[i64], p: f64) -> i64 { } // Coordinated-omission correction (HdrHistogram's recordValueWithExpectedInterval): -// this harness is a closed loop — iteration N+1 never starts until N returns — +// this harness is a closed loop (iteration N+1 never starts until N returns), // so a stall (a page fault, a context switch, a scheduling hiccup) is // recorded as a single slow sample. A fixed-rate caller would instead have // had many requests queue up behind that stall, each experiencing a similar @@ -280,7 +280,7 @@ fn main() -> ExitCode { let min_ns = sorted[0]; let median_ns = percentile(&sorted, 0.5); let p99_ns = percentile(&sorted, 0.99); - let max_ns = *sorted.last().unwrap(); + let max_ns = *sorted.last().expect("iters >= 2 guarantees at least one sample"); let expected_interval_ns = if median_ns > 0 { median_ns } else { 1 }; const CO_CAP: usize = 5_000_000; diff --git a/examples/39-benchmarking-without-lies/verify.lua b/examples/39-benchmarking-without-lies/verify.lua index f0076b8..764720b 100644 --- a/examples/39-benchmarking-without-lies/verify.lua +++ b/examples/39-benchmarking-without-lies/verify.lua @@ -8,9 +8,9 @@ -- checksum proving the codec actually ran (not skipped/optimized away) -- 2. the same fixed workload run twice produces the same checksum both -- times (the codec's output is deterministic even though the ns --- timings are not) — the "repeatable-order" contract +-- timings are not): the "repeatable-order" contract -- 3. --lie runs, exits 0, and its output contains no percentile table at --- all — that absence is the whole point of the contrast +-- all: that absence is the whole point of the contrast -- 4. error shapes: no flags, an unknown --op, and --warmup >= --iters all -- exit 2 with a usage line -- @@ -112,7 +112,7 @@ for _, op in ipairs({ "encode", "decode", "roundtrip" }) do end -- --------------------------------------------------------------------------- --- 3. --lie: runs, exits 0, and — the entire point — never emits a percentile +-- 3. --lie: runs, exits 0, and (the entire point) never emits a percentile -- table, only a single untrusted wall-clock sample. -- --------------------------------------------------------------------------- diff --git a/examples/40-low-latency-fastpath/README.md b/examples/40-low-latency-fastpath/README.md index 39b523c..6c41638 100644 --- a/examples/40-low-latency-fastpath/README.md +++ b/examples/40-low-latency-fastpath/README.md @@ -1,7 +1,7 @@ -# _template — hello-syscall +# _template: hello-syscall The seed for every runnable example in the book. It implements the same tiny -program three times — C++23, Go, and Rust — each printing exactly one line: +program three times, in C++23, Go, and Rust, each printing exactly one line: ``` pid on () @@ -26,9 +26,9 @@ _template/ Every language directory has a `demo.sh` with the same interface, identical across the whole book: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. diff --git a/examples/40-low-latency-fastpath/cpp/src/main.cpp b/examples/40-low-latency-fastpath/cpp/src/main.cpp index 7e76796..1f9c365 100644 --- a/examples/40-low-latency-fastpath/cpp/src/main.cpp +++ b/examples/40-low-latency-fastpath/cpp/src/main.cpp @@ -1,4 +1,4 @@ -// chatterd-fastpath — a stripped, pinned, allocation-free derivative of the +// chatterd-fastpath: a stripped, pinned, allocation-free derivative of the // chatterd echo/pub-sub hot path (chapter 21) built purely to answer one // question: what does a low-latency network hot path actually cost, and what // does removing that cost look like at the syscall level? @@ -20,14 +20,14 @@ // // app measure --target HOST:PORT --n N [--warmup W] // Connect, then run N synchronous ping-pong round trips (send frame, -// block for the echo, repeat — never more than one frame in flight) and +// block for the echo, repeat: never more than one frame in flight) and // print a p50/p90/p99/p99.9 latency table in nanoseconds. W warmup // round trips are discarded before recording starts (they pay for TCP // slow-start / first-touch page faults / connection setup, none of // which are what this is measuring). // // THE FASTPATH FRAME (fixed 64 bytes; this example's own wire format, a -// deliberately smaller cousin of the chatterd CHAT FRAME from ch21 — no +// deliberately smaller cousin of the chatterd CHAT FRAME from ch21, no // variable-length payload, so nothing here ever needs to size a buffer at // runtime): // @@ -36,7 +36,7 @@ // | 2B | 1B | 1B | 8B BE | 8B BE | 44B | // +-------+---------+------+-----------+-----------------+---------+ // -// magic is "CF" (chatterd-fastpath); the server never inspects seq/send_ns — +// magic is "CF" (chatterd-fastpath); the server never inspects seq/send_ns: // it reads 64 bytes and writes the identical 64 bytes back. All timing is // done by the client against its own clock; seq/send_ns are echoed back only // so the client can confirm it got its own frame back rather than another @@ -45,7 +45,7 @@ // C++23: RAII Socket owns every fd; std::expected carries fallible setup // steps; std::atomic + a non-restarting SIGINT/SIGTERM handler unblock // the accept wait and the read in progress, mirroring chatterd's -// stop_callback-via-shutdown(2) pattern without needing a second thread — the +// stop_callback-via-shutdown(2) pattern without needing a second thread: the // whole point of this example is that the hot loop runs on exactly one // thread, so a hand-off to a jthread is deliberately not in the loop. @@ -206,7 +206,7 @@ Result connect_tcp(const std::string& host, std::uint16_t port) { } // Blocking read of exactly n bytes; false on EOF, hard error, or a signal -// landing (EINTR) — the caller decides whether that means "stop". +// landing (EINTR): the caller decides whether that means "stop". bool read_full_blocking(int fd, void* buf, std::size_t n) { auto* p = static_cast(buf); std::size_t off = 0; @@ -224,8 +224,8 @@ bool read_full_blocking(int fd, void* buf, std::size_t n) { } // Non-blocking spin read: the socket must already be O_NONBLOCK. Spins on -// EAGAIN with no syscall other than recv(2) itself — no sched_yield, no -// nanosleep — which is the entire point of "busy-poll". +// EAGAIN with no syscall other than recv(2) itself, no sched_yield, no +// nanosleep, which is the entire point of "busy-poll". std::atomic* g_stop_for_busy_read = nullptr; bool read_full_busy(int fd, void* buf, std::size_t n) { diff --git a/examples/40-low-latency-fastpath/go/main.go b/examples/40-low-latency-fastpath/go/main.go index 3440284..6e73e21 100644 --- a/examples/40-low-latency-fastpath/go/main.go +++ b/examples/40-low-latency-fastpath/go/main.go @@ -1,4 +1,4 @@ -// chatterd-fastpath — a stripped, pinned, allocation-free derivative of the +// chatterd-fastpath: a stripped, pinned, allocation-free derivative of the // chatterd echo/pub-sub hot path (chapter 21) built purely to answer one // question: what does a low-latency network hot path actually cost, and what // does removing that cost look like at the syscall level? @@ -20,8 +20,8 @@ // // WHERE THE GO PORT DIVERGES FROM THE C++ ONE, AND WHY: // -// The C++ version installs a SIGINT handler with sa_flags = 0 — deliberately -// NOT SA_RESTART — so a blocking read(2) or poll(2) already in progress fails +// The C++ version installs a SIGINT handler with sa_flags = 0, deliberately +// NOT SA_RESTART, so a blocking read(2) or poll(2) already in progress fails // with EINTR and the loop notices the stop flag. That technique is not // available here. The Go runtime installs its own signal handlers with // SA_RESTART, and a "blocking" read on a net.Conn does not sit in a syscall @@ -30,7 +30,7 @@ // // So Go reaches the same observable behavior by a different mechanism: a // signal.Notify goroutine sets the stop flag and then CLOSES the listener and -// the live connection. Closing is what unblocks a parked netpoller read — +// the live connection. Closing is what unblocks a parked netpoller read; // the same shutdown(2)-driven wakeup that chapter 21's chatterd uses via // stop_callback, and the Go-idiomatic answer to "interrupt a blocked I/O // wait". The accept loop additionally carries a 200ms deadline, mirroring the @@ -38,7 +38,7 @@ // accept already in flight. // // The second divergence is pinning. sched_setaffinity(2) acts on a THREAD, -// and a goroutine is not a thread — it migrates between Ms at the scheduler's +// and a goroutine is not a thread; it migrates between Ms at the scheduler's // discretion. Pinning here therefore needs runtime.LockOSThread to nail the // hot loop to one M, plus GOMAXPROCS(1) so the runtime does not keep other // worker threads on other CPUs. See pinToCPU for why the affinity call is @@ -179,7 +179,7 @@ func installSignalHandlers(s *stopper) { // port's sysconf(_SC_NPROCESSORS_ONLN). runtime.NumCPU() is deliberately NOT // used: it returns the size of this process's CPU affinity MASK, so under the // bench driver's `taskset -c 1` it would report 1 and reject a pin to CPU 1 as -// "out of range" — the classic case of a count that lies about its container, +// "out of range", the classic case of a count that lies about its container, // hit here via taskset rather than a cgroup. Reading the kernel's own online // list sidesteps the mask. Falls back to runtime.NumCPU() only if the sysfs // node is unreadable. @@ -213,14 +213,14 @@ func onlineCPUs() int { return count } -// pinToCPU restricts this process to a single CPU, and — critically — makes +// pinToCPU restricts this process to a single CPU, and, critically, makes // that restriction visible in /proc//status, which is where verify.lua // reads Cpus_allowed_list to prove the pinning happened at the kernel level // rather than trusting the line we print. // // The affinity call is made twice on purpose. sched_setaffinity(0, ...) acts // on the CALLING thread, which after LockOSThread is the one running the hot -// loop — that call is what makes the pinning real. But /proc//status +// loop; that call is what makes the pinning real. But /proc//status // reports the mask of the THREAD GROUP LEADER (the thread whose tid == pid), // and the main goroutine is not guaranteed to be running on it. Passing // os.Getpid() explicitly targets the leader, so the observable proof matches @@ -308,7 +308,7 @@ func setNoDelay(c net.Conn) { } // readFullBusy spins a non-blocking recv(2) with no other syscall in the loop -// — no sched_yield, no nanosleep — which is the entire point of "busy-poll". +// with no sched_yield and no nanosleep, which is the entire point of "busy-poll". func readFullBusy(fd int, buf []byte, s *stopper) bool { off := 0 for off < len(buf) { diff --git a/examples/40-low-latency-fastpath/rust/src/main.rs b/examples/40-low-latency-fastpath/rust/src/main.rs index 865971f..4283f81 100644 --- a/examples/40-low-latency-fastpath/rust/src/main.rs +++ b/examples/40-low-latency-fastpath/rust/src/main.rs @@ -1,4 +1,4 @@ -// chatterd-fastpath — a stripped, pinned, allocation-free derivative of the +// chatterd-fastpath: a stripped, pinned, allocation-free derivative of the // chatterd echo/pub-sub hot path (chapter 21) built purely to answer one // question: what does a low-latency network hot path actually cost, and what // does removing that cost look like at the syscall level? @@ -22,9 +22,9 @@ // // This is the port that follows the C++ original most literally, and it can: // Rust's std makes real blocking syscalls (there is no M:N runtime parking -// tasks off the syscall the way Go's netpoller does), so the C++ technique — a +// tasks off the syscall the way Go's netpoller does), so the C++ technique (a // signal handler installed WITHOUT SA_RESTART, so a blocking read(2)/poll(2) -// already in progress returns EINTR and the loop notices the stop flag — works +// already in progress returns EINTR and the loop notices the stop flag) works // verbatim here. The Go port had to substitute a close-the-fd wakeup for // exactly this reason; Rust does not. // @@ -32,12 +32,12 @@ // retries it internally (cvt_r), and Read::read_exact treats // ErrorKind::Interrupted as "try again". Both would swallow the very signal // this example is about. So the servers are built on raw fds via nix, where -// every EINTR is handled explicitly — which also makes OwnedFd the natural +// every EINTR is handled explicitly, which also makes OwnedFd the natural // equivalent of the C++ RAII Socket: it closes its descriptor on drop, is // move-only, and needs no hand-written destructor. // // Pinning is simpler than in Go. sched_setaffinity(2) acts on a thread, and -// this program is single-threaded — the main thread's tid equals the pid — so +// this program is single-threaded (the main thread's tid equals the pid), so // one sched_setaffinity(0, ...) call both pins the hot loop and shows up in // /proc//status, where verify.lua reads Cpus_allowed_list to prove the // pinning happened at the kernel level. No LockOSThread / GOMAXPROCS dance is @@ -68,7 +68,7 @@ const TYPE_ECHO: u8 = 0x01; type Frame = [u8; FRAME_SIZE]; -// A fallible setup step carrying an already-formatted, user-facing message — +// A fallible setup step carrying an already-formatted, user-facing message: // the Rust stand-in for the C++ port's std::expected. type StepResult = Result; @@ -93,13 +93,13 @@ fn frame_header_ok(f: &Frame) -> bool { } fn seq_of(f: &Frame) -> u64 { - u64::from_be_bytes(f[4..12].try_into().unwrap()) + u64::from_be_bytes(f[4..12].try_into().expect("slice is exactly 8 bytes")) } // --------------------------------------------------------------------------- // Shutdown plumbing: a plain (non-SA_RESTART) signal handler so a blocking // read(2)/accept-wait poll(2) in progress is interrupted with EINTR rather -// than transparently resumed — exactly the C++ port's technique. +// than transparently resumed: exactly the C++ port's technique. // --------------------------------------------------------------------------- static STOP: AtomicBool = AtomicBool::new(false); @@ -132,7 +132,7 @@ fn stop_requested() -> bool { fn pin_to_cpu(cpu: i32) -> StepResult<()> { // sysconf(_SC_NPROCESSORS_ONLN) reports CPUs online SYSTEM-WIDE and ignores - // this process's affinity mask — the same call the C++ port uses. That + // this process's affinity mask: the same call the C++ port uses. That // matters under the bench driver's `taskset -c 1`: a mask-aware count // (like Go's runtime.NumCPU) would report 1 and wrongly reject a pin to // CPU 1. sysconf sidesteps that. @@ -145,14 +145,14 @@ fn pin_to_cpu(cpu: i32) -> StepResult<()> { set.set(cpu as usize) .map_err(|e| format!("sched_setaffinity({cpu}): {e}"))?; // Pid 0 == the calling thread, which is the main (and only) thread, whose - // tid == pid — so this is also what /proc//status reports. + // tid == pid, so this is also what /proc//status reports. sched_setaffinity(Pid::from_raw(0), &set) .map_err(|e| format!("sched_setaffinity({cpu}): {e}"))?; Ok(()) } // --------------------------------------------------------------------------- -// Sockets — OwnedFd is the RAII Socket: closes on drop, move-only. +// Sockets: OwnedFd is the RAII Socket (closes on drop, move-only). // --------------------------------------------------------------------------- fn listen_tcp(port: u16) -> StepResult { @@ -167,7 +167,8 @@ fn listen_tcp(port: u16) -> StepResult { let addr = SockaddrIn::new(0, 0, 0, 0, port); // 0.0.0.0:port bind(sock.as_raw_fd(), &addr).map_err(|e| format!("bind 0.0.0.0:{port}: {e}"))?; - listen(&sock, Backlog::new(16).unwrap()).map_err(|e| format!("listen: {e}"))?; + listen(&sock, Backlog::new(16).expect("16 is a valid backlog size")) + .map_err(|e| format!("listen: {e}"))?; Ok(sock) } @@ -190,7 +191,7 @@ fn connect_tcp(host: &str, port: u16) -> StepResult { } // Blocking read of exactly n bytes; false on EOF, hard error, or a signal -// landing (EINTR) — the caller decides whether that means "stop". +// landing (EINTR): the caller decides whether that means "stop". fn read_full_blocking(fd: RawFd, buf: &mut [u8]) -> bool { let bfd = unsafe { BorrowedFd::borrow_raw(fd) }; let mut off = 0; @@ -205,8 +206,8 @@ fn read_full_blocking(fd: RawFd, buf: &mut [u8]) -> bool { } // Non-blocking spin read: the socket must already be O_NONBLOCK. Spins on -// EAGAIN with no syscall other than recv(2) itself — no sched_yield, no -// nanosleep — which is the entire point of "busy-poll". +// EAGAIN with no syscall other than recv(2) itself: no sched_yield, no +// nanosleep, which is the entire point of "busy-poll". fn read_full_busy(fd: RawFd, buf: &mut [u8]) -> bool { let mut off = 0; while off < buf.len() { @@ -415,8 +416,8 @@ fn cmd_measure(host: &str, port: u16, n: u64, warmup: u64, tag: &str) -> i32 { let p90 = percentile_ns(&samples, 90.0); let p99 = percentile_ns(&samples, 99.0); let p999 = percentile_ns(&samples, 99.9); - let mn = *samples.first().unwrap(); - let mx = *samples.last().unwrap(); + let mn = *samples.first().expect("n > 0 guarantees at least one sample"); + let mx = *samples.last().expect("n > 0 guarantees at least one sample"); let sum: f64 = samples.iter().map(|&v| v as f64).sum(); let mean = sum / samples.len() as f64; diff --git a/examples/40-low-latency-fastpath/verify.lua b/examples/40-low-latency-fastpath/verify.lua index 4fabc1f..7abd3f5 100644 --- a/examples/40-low-latency-fastpath/verify.lua +++ b/examples/40-low-latency-fastpath/verify.lua @@ -1,20 +1,20 @@ -- Verify chatterd-fastpath (40-low-latency-fastpath) for LSP_LANG. A vm-mode -- example: driven directly against systems-target via plain scp/ssh to --- /tmp/lsp40-/, never `demo.sh run` under TARGET -- +-- /tmp/lsp40-/, never `demo.sh run` under TARGET: -- deploy-to-vm.sh's `ssh -t` injects CRLF that breaks Lua patterns, and its -- fixed /home/fedora/app path can be poisoned by a stale binary left over -- from an unrelated example (this happened once during golden capture). -- Every remote invocation below uses the full absolute remote_bin / --- remote_driver path -- never a leading `cd` relied on to apply across a +-- remote_driver path: never a leading `cd` relied on to apply across a -- later `;`-separated command in the same string. Every multi-statement --- remote script here is joined with "\n", never "; " -- a line ending in +-- remote script here is joined with "\n", never "; ": a line ending in -- `&` (backgrounding) followed by `; ` produces `cmd & ; next`, a bash -- syntax error that silently makes the whole remote script a no-op parse -- (a real bug hit while writing example 37's verify.lua). -- -- Asserts observable behavior, identical in shape across cpp/go/rust: -- 1. build, then stage app + run-latency-bench.sh to /tmp/lsp40-/, --- chmod 755 both -- both must exit 0. +-- chmod 755 both, both must exit 0. -- 2. error shapes: no args / frobnicate / fastpath without --pin / naive -- without --port / measure without --n all exit 2 with the 3-line -- "usage: app fastpath ..." usage text. @@ -26,20 +26,20 @@ -- 5. the real run-latency-bench.sh (N=20000, WARMUP=500): exit 0, and its -- output proves --pin actually restricts scheduling AT THE KERNEL -- LEVEL (read from /proc//status by the driver script, not the --- program's own claim) -- fastpath-cpus-allowed is exactly the single +-- program's own claim): fastpath-cpus-allowed is exactly the single -- pinned CPU while naive-cpus-allowed is the full online set. -- 6. all six percentiles_ns lines (naive-1..3, fastpath-1..3) parse, each -- with n=20000, a monotonic min<=p50<=p90<=p99<=p99.9<=max, a mean that -- parses as a float, and every value > 0. -- 7. median-of-three per variant: median(fastpath p50) < median(naive p50) --- * 0.90 -- a conservative 10% requirement against an observed ~30-33% +-- * 0.90, a conservative 10% requirement against an observed ~30-33% -- margin across three independent golden-capture bench runs (1.425x - --- 1.488x, never marginal or inverted) -- both medians are always +-- 1.488x, never marginal or inverted); both medians are always -- printed so a marginal run is diagnosable from the log even when this -- assertion passes. -- 8. SIGINT shutdown, tested separately per server with a bounded -- (10s) wait via `timeout 10 tail --pid=$PID -f /dev/null` so a hung --- server surfaces as a clear FAIL rather than a runner-level timeout -- +-- server surfaces as a clear FAIL rather than a runner-level timeout: -- this is the assertion Go is most likely to fail, since Go's runtime -- installs SA_RESTART handlers and a blocking read will not see EINTR -- on its own. naive --port P: "app: naive shutting down", exit 0. @@ -48,7 +48,7 @@ -- "app: fastpath shutting down", exit 0. -- -- Never compares formatted decimals across languages (C++, Go, and Rust --- disagree on halfway rounding) -- only shapes and numeric ranges/ratios are +-- disagree on halfway rounding), only shapes and numeric ranges/ratios are -- asserted; percentile values (p50 etc.) are plain integers so they ARE -- compared numerically across the naive/fastpath medians, but never against -- a fixed expected number. @@ -101,7 +101,7 @@ end -- Boolean assertion via the checks.lua pattern-match API: match "ok" against -- "ok" when true, or force a mismatch (with the real detail in the label) --- when false -- same helper as examples 37/39's verify.lua. +-- when false, same helper as examples 37/39's verify.lua. local function expect_true(cond, label, detail) if cond then checks.expect_match("ok", "ok", label) @@ -141,7 +141,7 @@ local nproc_r = ssh_run("nproc") local nproc = tonumber(nproc_r.out:match("%d+")) or 1 -- --------------------------------------------------------------------------- --- 2. error shapes -- exit 2, "usage: app fastpath ..." on stderr. +-- 2. error shapes: exit 2, "usage: app fastpath ..." on stderr. -- --------------------------------------------------------------------------- local function expect_usage(args, label) @@ -157,7 +157,7 @@ expect_usage("naive", "naive (no port)") expect_usage("measure --target 127.0.0.1:9999", "measure --target ... (no --n)") -- --------------------------------------------------------------------------- --- 3. fastpath --pin out of range -- exit 1, cpu N out of range. +-- 3. fastpath --pin out of range: exit 1, cpu N out of range. -- --------------------------------------------------------------------------- local bad_cpu = nproc + 50 @@ -167,7 +167,7 @@ checks.expect_match(pin_r.out, "app: error: cpu %d+ out of range", lang .. ": out-of-range --pin prints the cpu-out-of-range error") -- --------------------------------------------------------------------------- --- 4. measure error shapes -- connect refused, non-IPv4 host. +-- 4. measure error shapes: connect refused, non-IPv4 host. -- --------------------------------------------------------------------------- local connect_r = ssh_run(remote_bin .. " measure --target 127.0.0.1:1 --n 10") @@ -200,7 +200,7 @@ expect_true(bench_exit == 0, tostring(bench_exit)) -- 5. Kernel-level pinning proof (phase A of the driver script), read from --- /proc//status by the driver -- not the program's own printed claim. +-- /proc//status by the driver, not the program's own printed claim. local naive_allowed = br.out:match("app: naive%-cpus%-allowed (%S+)") local fast_allowed = br.out:match("app: fastpath%-cpus%-allowed (%S+)") @@ -245,8 +245,8 @@ expect_true(matched == 6, -- 7. Median-of-three per variant, conservative threshold, both medians -- always printed (per the golden-capture calibration: observed ratio --- 1.425x-1.488x across 3 independent runs, so a 0.90 factor -- a 10% --- requirement -- has comfortable headroom against a noisier CI run without +-- 1.425x-1.488x across 3 independent runs, so a 0.90 factor (a 10% +-- requirement) has comfortable headroom against a noisier CI run without -- being a tautology). local function median3(t) if #t ~= 3 then return nil end diff --git a/examples/41-capstone-fleet/README.md b/examples/41-capstone-fleet/README.md index 39b523c..6c41638 100644 --- a/examples/41-capstone-fleet/README.md +++ b/examples/41-capstone-fleet/README.md @@ -1,7 +1,7 @@ -# _template — hello-syscall +# _template: hello-syscall The seed for every runnable example in the book. It implements the same tiny -program three times — C++23, Go, and Rust — each printing exactly one line: +program three times, in C++23, Go, and Rust, each printing exactly one line: ``` pid on () @@ -26,9 +26,9 @@ _template/ Every language directory has a `demo.sh` with the same interface, identical across the whole book: -- `./demo.sh` — build then run -- `./demo.sh build` — build only -- `./demo.sh run [args]` — run the built binary +- `./demo.sh`: build then run +- `./demo.sh build`: build only +- `./demo.sh run [args]`: run the built binary - With env `TARGET` set, `run` deploys the binary to that lab VM via `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of running locally. diff --git a/examples/41-capstone-fleet/cpp/src/caps.cpp b/examples/41-capstone-fleet/cpp/src/caps.cpp index b2f11d9..d4952bc 100644 --- a/examples/41-capstone-fleet/cpp/src/caps.cpp +++ b/examples/41-capstone-fleet/cpp/src/caps.cpp @@ -1,4 +1,4 @@ -// caps.cpp — PR_CAPBSET_DROP for every known capability, then +// caps.cpp: PR_CAPBSET_DROP for every known capability, then // PR_SET_NO_NEW_PRIVS. See the Go caps.go for the full rationale: this // shrinks the bounding set inherited by every fork/exec descendant, whether // or not the calling process currently holds any capability at all. diff --git a/examples/41-capstone-fleet/cpp/src/caps.hpp b/examples/41-capstone-fleet/cpp/src/caps.hpp index d9a91a4..7e007e0 100644 --- a/examples/41-capstone-fleet/cpp/src/caps.hpp +++ b/examples/41-capstone-fleet/cpp/src/caps.hpp @@ -1,4 +1,4 @@ -// caps.hpp — capability-bounding-set drop (see caps.cpp for rationale). +// caps.hpp: capability-bounding-set drop (see caps.cpp for rationale). #pragma once namespace caps { diff --git a/examples/41-capstone-fleet/cpp/src/chatterd.cpp b/examples/41-capstone-fleet/cpp/src/chatterd.cpp index f657585..7859515 100644 --- a/examples/41-capstone-fleet/cpp/src/chatterd.cpp +++ b/examples/41-capstone-fleet/cpp/src/chatterd.cpp @@ -1,4 +1,4 @@ -// chatterd.cpp — the book's recurring peer-to-peer chat daemon (ch21-27), +// chatterd.cpp: the book's recurring peer-to-peer chat daemon (ch21-27), // reduced to what the capstone needs: serve local clients, and (new in this // chapter) bridge two chatterd instances across hosts so a message sent on // one node is delivered to clients on the other. The wire frame (proto.hpp) @@ -14,7 +14,7 @@ // // A local MSG is broadcast as DELIVER to every registered connection // (including a bridge connection dialed in from the other side) exactly like -// any other client — no protocol change. When a bridge *worker* itself +// any other client, no protocol change. When a bridge *worker* itself // receives a DELIVER over the connection it dialed, it re-broadcasts locally // with include_bridges=false, so the message never goes back out over any // bridge: no ping-pong between two nodes. @@ -61,7 +61,7 @@ namespace trace_api = opentelemetry::trace; [[nodiscard]] std::string errno_msg() { return std::string(std::strerror(errno)); } // --------------------------------------------------------------------------- -// RAII fd — same shape as every other example's. +// RAII fd: same shape as every other example's. // --------------------------------------------------------------------------- class Fd { @@ -158,7 +158,7 @@ class Fd { } // --------------------------------------------------------------------------- -// Hub — the registry of every connection (local client or bridge worker) +// Hub: the registry of every connection (local client or bridge worker) // currently attached to this server. // --------------------------------------------------------------------------- @@ -347,7 +347,7 @@ int serve(const std::string& host, int port, const std::string& node, const std: } // --------------------------------------------------------------------------- -// send / listen — minimal test clients used by verify to prove cross-host +// send / listen: minimal test clients used by verify to prove cross-host // delivery deterministically (this capstone trims ch27's fuller "loadtest"). // --------------------------------------------------------------------------- diff --git a/examples/41-capstone-fleet/cpp/src/chatterd.hpp b/examples/41-capstone-fleet/cpp/src/chatterd.hpp index 4a6a1e6..a0c0357 100644 --- a/examples/41-capstone-fleet/cpp/src/chatterd.hpp +++ b/examples/41-capstone-fleet/cpp/src/chatterd.hpp @@ -1,4 +1,4 @@ -// chatterd.hpp — the book's recurring peer-to-peer chat daemon (ch21-27), +// chatterd.hpp: the book's recurring peer-to-peer chat daemon (ch21-27), // reduced to what the capstone needs: serve local clients, and bridge two // chatterd instances across hosts so a message sent on one node is delivered // to clients on the other. See chatterd.cpp for the bridging design (mirrors diff --git a/examples/41-capstone-fleet/cpp/src/fwatch.cpp b/examples/41-capstone-fleet/cpp/src/fwatch.cpp index 7a120b8..f9d5711 100644 --- a/examples/41-capstone-fleet/cpp/src/fwatch.cpp +++ b/examples/41-capstone-fleet/cpp/src/fwatch.cpp @@ -1,13 +1,13 @@ -// fwatch.cpp — the book's recurring file watcher (ch07 polling -> ch09 +// fwatch.cpp: the book's recurring file watcher (ch07 polling -> ch09 // inotify/epoll -> ch33 Landlock+seccomp), reduced here to what the fleet // needs: watch a directory, print one line per raw create/write/delete -// inotify event (no debounce/merge — go/fwatch.go prints every event as it +// inotify event (no debounce/merge: go/fwatch.go prints every event as it // arrives, and this port matches that exactly), and optionally run under a // Landlock ruleset restricting reads to that directory. The raw // landlock_create_ruleset/add_rule/restrict_self syscalls and the RAII Fd // pattern are carried over verbatim from ch33's main.cpp ( // has no libc wrapper); full seccomp allow-listing was proven once there and -// is not re-derived here — capability-bounding-set drop (caps.cpp) plus +// is not re-derived here: capability-bounding-set drop (caps.cpp) plus // Landlock are this capstone's sandboxing layers, applied by pmon before it // execs fwatch. #include "fwatch.hpp" @@ -41,7 +41,7 @@ namespace fs = std::filesystem; [[nodiscard]] std::string errno_msg() { return std::string(std::strerror(errno)); } -// RAII file descriptor — same shape as every other example's Fd/Socket type. +// RAII file descriptor: same shape as every other example's Fd/Socket type. class Fd { public: Fd() = default; @@ -72,7 +72,7 @@ class Fd { }; // --------------------------------------------------------------------------- -// Landlock — raw syscalls, no libc wrapper exists (ch33's technique). +// Landlock: raw syscalls, no libc wrapper exists (ch33's technique). // --------------------------------------------------------------------------- [[nodiscard]] long sys_landlock_create_ruleset(const landlock_ruleset_attr* attr, std::size_t size, @@ -98,7 +98,7 @@ class Fd { // Creates a ruleset handling READ_FILE + READ_DIR, adds one path-beneath rule // granting exactly those rights under `dir`, then restrict_self. On success -// this process — and everything it execs from here on — can only read +// this process, and everything it execs from here on, can only read // inside `dir`. [[nodiscard]] bool apply_landlock(const std::string& dir, std::string& err) { constexpr std::uint64_t access = LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR; @@ -137,7 +137,7 @@ class Fd { } // classify mirrors go/fwatch.go's eventName: create, then delete, then -// modify (CLOSE_WRITE or MODIFY), else "other" — checked in that priority +// modify (CLOSE_WRITE or MODIFY), else "other": checked in that priority // order because a single inotify event's mask can carry more than one bit. [[nodiscard]] std::string_view classify(std::uint32_t mask) { if (mask & IN_CREATE) return "create"; diff --git a/examples/41-capstone-fleet/cpp/src/fwatch.hpp b/examples/41-capstone-fleet/cpp/src/fwatch.hpp index e34a051..86ea257 100644 --- a/examples/41-capstone-fleet/cpp/src/fwatch.hpp +++ b/examples/41-capstone-fleet/cpp/src/fwatch.hpp @@ -1,4 +1,4 @@ -// fwatch.hpp — the book's recurring file watcher, reduced to what the fleet +// fwatch.hpp: the book's recurring file watcher, reduced to what the fleet // needs (see the .cpp for the Landlock rationale carried over from ch33). #pragma once diff --git a/examples/41-capstone-fleet/cpp/src/main.cpp b/examples/41-capstone-fleet/cpp/src/main.cpp index 3cc63b5..3652874 100644 --- a/examples/41-capstone-fleet/cpp/src/main.cpp +++ b/examples/41-capstone-fleet/cpp/src/main.cpp @@ -1,11 +1,11 @@ -// 41-capstone-fleet — pmon supervises chatterd + sysagent + fwatch across two +// 41-capstone-fleet: pmon supervises chatterd + sysagent + fwatch across two // lab hosts, capability-dropped and Landlock-sandboxed, all telemetry // exported to the host LGTM stack. One binary, four subcommands (the same // self-re-exec shape ch34's container entrypoint uses): pmon is the fleet's // init; chatterd/sysagent/fwatch are the services it supervises. // // This file is a direct, line-for-line port of go/main.go's flag parsing -// (argFlag/hasFlag/firstPositional/atoiOr) and dispatch table — the usage +// (argFlag/hasFlag/firstPositional/atoiOr) and dispatch table: the usage // banner text below is byte-identical to the Go reference's, since // verify.lua and every language's demo.sh depend on it matching exactly. #include @@ -40,7 +40,7 @@ void usage() { // argFlag scans args for "--name value" and returns (value, true), or // (def, true) if absent, or (_, false) if the flag is present but has no -// following value — mirrors go/main.go's argFlag exactly. +// following value: mirrors go/main.go's argFlag exactly. std::pair arg_flag(const std::vector& args, const std::string& name, const std::string& def) { for (std::size_t i = 0; i < args.size(); ++i) { @@ -65,7 +65,7 @@ bool has_flag(const std::vector& args, const std::string& name) { // firstPositional: walks args, treating every "--xxx" token as consuming // exactly one following token (its value), and returns the first token that -// survives that skipping — mirrors go/main.go's firstPositional exactly, +// survives that skipping: mirrors go/main.go's firstPositional exactly, // including its "skip the very next token no matter what it is" quirk. std::string first_positional(const std::vector& args) { bool skip = false; diff --git a/examples/41-capstone-fleet/cpp/src/pmon.cpp b/examples/41-capstone-fleet/cpp/src/pmon.cpp index a792b40..e376a08 100644 --- a/examples/41-capstone-fleet/cpp/src/pmon.cpp +++ b/examples/41-capstone-fleet/cpp/src/pmon.cpp @@ -1,4 +1,4 @@ -// pmon.cpp — the book's recurring process supervisor (ch11-14, 18-19, 32, +// pmon.cpp: the book's recurring process supervisor (ch11-14, 18-19, 32, // 34), grown here into the capstone's fleet init: it drops the capability // bounding set once (caps.cpp), then fork/execs (self-re-exec via // /proc/self/exe, the same technique ch34's container entrypoint uses) @@ -76,7 +76,7 @@ struct ChildState { }; // run_child_supervisor: fork+exec `self_exe args...` in a loop, restarting on -// unexpected exit — the per-service goroutine from go/pmon.go, one std thread +// unexpected exit: the per-service goroutine from go/pmon.go, one std thread // per service here too. void run_child_supervisor(const std::string& self_exe, const ChildSpec& spec, ChildState& cs, std::atomic& shutting_down) { diff --git a/examples/41-capstone-fleet/cpp/src/pmon.hpp b/examples/41-capstone-fleet/cpp/src/pmon.hpp index 279dd7a..54d033f 100644 --- a/examples/41-capstone-fleet/cpp/src/pmon.hpp +++ b/examples/41-capstone-fleet/cpp/src/pmon.hpp @@ -1,4 +1,4 @@ -// pmon.hpp — the fleet's init: drops the capability bounding set, then +// pmon.hpp: the fleet's init: drops the capability bounding set, then // self-re-execs chatterd/sysagent/fwatch as supervised children. See // pmon.cpp for the fork/exec + sigwait shutdown design. #pragma once diff --git a/examples/41-capstone-fleet/cpp/src/proto.hpp b/examples/41-capstone-fleet/cpp/src/proto.hpp index d3070ee..3746007 100644 --- a/examples/41-capstone-fleet/cpp/src/proto.hpp +++ b/examples/41-capstone-fleet/cpp/src/proto.hpp @@ -1,4 +1,4 @@ -// proto.hpp — the canonical chatterd chat frame (ch21, unchanged since): +// proto.hpp: the canonical chatterd chat frame (ch21, unchanged since): // magic "CH", version 1, one byte type, big-endian u16 length, then payload. #pragma once diff --git a/examples/41-capstone-fleet/cpp/src/sysagent.cpp b/examples/41-capstone-fleet/cpp/src/sysagent.cpp index 1b169fb..249bf0f 100644 --- a/examples/41-capstone-fleet/cpp/src/sysagent.cpp +++ b/examples/41-capstone-fleet/cpp/src/sysagent.cpp @@ -1,4 +1,4 @@ -// sysagent.cpp — /proc/stat, /proc/meminfo, /proc/loadavg readers and the +// sysagent.cpp: /proc/stat, /proc/meminfo, /proc/loadavg readers and the // sample loop, ported field-for-field from go/sysagent.go so the printed // cpu_pct/mem_pct/load1 numbers are computed by the identical formula (not // just "close enough"): idle is specifically the 4th space-separated value @@ -138,7 +138,7 @@ std::expected take_sample(CpuTimes& prev) { } // --------------------------------------------------------------------------- -// OTel observable gauges — one shared, atomically-updated snapshot the +// OTel observable gauges: one shared, atomically-updated snapshot the // meter's periodic collection callbacks read from, the same shape as the Go // reference's mutex-guarded `latest sample` plus RegisterCallback closure. // --------------------------------------------------------------------------- @@ -248,7 +248,7 @@ int run(const std::string& node, int interval_ms, bool once, telemetry::Handle& return rc; } -// saturate — chaos helper: busy-spin threads (cpu) or hold allocated, touched +// saturate: chaos helper: busy-spin threads (cpu) or hold allocated, touched // memory (mem) for --seconds, so a concurrently-running sysagent's // cpu_pct/mem_pct visibly rises (this fleet's USE-method callback). int saturate(const std::string& resource, int seconds, int workers, int mb) { diff --git a/examples/41-capstone-fleet/cpp/src/sysagent.hpp b/examples/41-capstone-fleet/cpp/src/sysagent.hpp index 4dac669..13bc5f2 100644 --- a/examples/41-capstone-fleet/cpp/src/sysagent.hpp +++ b/examples/41-capstone-fleet/cpp/src/sysagent.hpp @@ -1,4 +1,4 @@ -// sysagent.hpp — the book's /proc-reading metrics agent (ch36), reduced to +// sysagent.hpp: the book's /proc-reading metrics agent (ch36), reduced to // three signals (cpu%, mem%, load1) exported both as a stdout line and, when // OTEL_EXPORTER_OTLP_ENDPOINT is set, as OTel gauges over OTLP. See // sysagent.cpp for the /proc/stat delta math, ported field-for-field from the @@ -15,7 +15,7 @@ namespace sysagent { int run(const std::string& node, int interval_ms, bool once, telemetry::Handle& tel); -// saturate — chaos helper: busy-spin threads (cpu) or hold touched memory +// saturate: chaos helper: busy-spin threads (cpu) or hold touched memory // (mem) for `seconds`, so a concurrently-running sysagent's cpu_pct/mem_pct // visibly rises. int saturate(const std::string& resource, int seconds, int workers, int mb); diff --git a/examples/41-capstone-fleet/cpp/src/telemetry.cpp b/examples/41-capstone-fleet/cpp/src/telemetry.cpp index 9ef07ee..ed6f45d 100644 --- a/examples/41-capstone-fleet/cpp/src/telemetry.cpp +++ b/examples/41-capstone-fleet/cpp/src/telemetry.cpp @@ -39,9 +39,9 @@ namespace internal_log = opentelemetry::sdk::common::internal_log; // Counts every Error-level internal SDK log line and prints it, the closest // opentelemetry-cpp equivalent of Go's otel.SetErrorHandler callback (which // both increments exportErrors and Fprintf's to stderr on every export -// failure). LogLevel is pinned to Error below, so Info/Debug/Warning lines — -// the ones that would otherwise corrupt the stdout contract, ch38's hard-won -// lesson — never reach this handler at all. +// failure). LogLevel is pinned to Error below, so Info/Debug/Warning lines +// (the ones that would otherwise corrupt the stdout contract, ch38's +// hard-won lesson) never reach this handler at all. class CountingLogHandler : public internal_log::LogHandler { public: explicit CountingLogHandler(std::string service) : service_(std::move(service)) {} diff --git a/examples/41-capstone-fleet/cpp/src/telemetry.hpp b/examples/41-capstone-fleet/cpp/src/telemetry.hpp index 941a0d0..7ba37e1 100644 --- a/examples/41-capstone-fleet/cpp/src/telemetry.hpp +++ b/examples/41-capstone-fleet/cpp/src/telemetry.hpp @@ -1,8 +1,8 @@ -// telemetry.hpp — OTel wiring shared by chatterd and sysagent, following the +// telemetry.hpp: OTel wiring shared by chatterd and sysagent, following the // book's house C++ pattern established in ch38 (opentelemetry-cpp via Conan, // OTLP/HTTP, a NoopLogHandler-derived quiet SDK). Endpoint comes from the // standard OTEL_EXPORTER_OTLP_ENDPOINT env var, the same variable -// scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT — the +// scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT: the // exporters are pointed at "/v1/traces" and "/v1/metrics" // explicitly here (opentelemetry-cpp's own OTEL_EXPORTER_OTLP_ENDPOINT // auto-read does exactly this same per-signal suffixing per the OTLP env-var @@ -23,7 +23,7 @@ namespace telemetry { // Count of OTel export failures observed via the SDK's internal Error-level -// log line — opentelemetry-cpp has no direct equivalent of Go's +// log line: opentelemetry-cpp has no direct equivalent of Go's // otel.SetErrorHandler, so this is the deterministic stand-in the internal // log handler feeds: printed as ": otel export error: ..." exactly // once per Error-level line, mirroring what Go's handler does on every @@ -51,7 +51,7 @@ struct Handle { // Flushes and shuts down both providers, bounded by a short timeout. A // no-op when telemetry is disabled. The Go reference never gets a chance // to run its own deferred shutdown (every subcommand's dispatch in - // main.go wraps its run in os.Exit(), which skips deferred calls) — this + // main.go wraps its run in os.Exit(), which skips deferred calls): this // is deliberately stricter than that, not a divergence in anything // observable: it prints nothing either way, and only improves the odds // of the last batch actually reaching the collector. @@ -59,7 +59,7 @@ struct Handle { }; // init sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT is set; -// otherwise returns a disabled, no-op handle (the service still runs — OTel +// otherwise returns a disabled, no-op handle (the service still runs: OTel // is additive, never a hard dependency of the fleet coming up). Handle init(const std::string& service, const std::string& node); diff --git a/examples/41-capstone-fleet/cpp/src/util.hpp b/examples/41-capstone-fleet/cpp/src/util.hpp index a77f995..6e61aaa 100644 --- a/examples/41-capstone-fleet/cpp/src/util.hpp +++ b/examples/41-capstone-fleet/cpp/src/util.hpp @@ -1,8 +1,8 @@ -// util.hpp — small helpers shared across the fleet's subcommands, the C++ +// util.hpp: small helpers shared across the fleet's subcommands, the C++ // counterpart of util.go's installSignalFlag: chatterd/sysagent/fwatch each // run their own poll loop already (accept-with-timeout, interval sleep, // poll(2) with a timeout), so a level-triggered flag is all any of them -// needs from a signal — no EINTR-sensitive syscall unwinding required, unlike +// needs from a signal, no EINTR-sensitive syscall unwinding required, unlike // ex40's fastpath server. pmon (a supervisor, not a poller) does its own // dedicated sigwait() handling in pmon.cpp instead of using this. #pragma once diff --git a/examples/41-capstone-fleet/go/caps.go b/examples/41-capstone-fleet/go/caps.go index 1023936..3402891 100644 --- a/examples/41-capstone-fleet/go/caps.go +++ b/examples/41-capstone-fleet/go/caps.go @@ -1,4 +1,4 @@ -// caps.go — capability-bounding-set drop, the cheap, always-available half of +// caps.go: capability-bounding-set drop, the cheap, always-available half of // this chapter's sandboxing story (the other half, Landlock, lives in // fwatch.go and was fully proven with seccomp alongside it in ch33; this // capstone does not re-derive the seccomp BPF program, only reuses the @@ -8,7 +8,7 @@ // execve of a setuid-root binary can never regain it, for this process and // every descendant (the bounding set is inherited across fork/exec and can // only shrink). It is meaningful regardless of whether the calling process -// currently holds any capabilities at all — that is exactly pmon's case when +// currently holds any capabilities at all; that is exactly pmon's case when // deployed unprivileged on a lab guest. package main diff --git a/examples/41-capstone-fleet/go/chatterd.go b/examples/41-capstone-fleet/go/chatterd.go index 311b4ca..d9f4f06 100644 --- a/examples/41-capstone-fleet/go/chatterd.go +++ b/examples/41-capstone-fleet/go/chatterd.go @@ -1,4 +1,4 @@ -// chatterd.go — the book's recurring peer-to-peer chat daemon (ch21-27), +// chatterd.go: the book's recurring peer-to-peer chat daemon (ch21-27), // reduced to what the capstone needs: serve local clients, and (new in this // chapter) bridge two chatterd instances across hosts so a message sent on // one node is delivered to clients on the other. The wire frame (proto.go) @@ -14,7 +14,7 @@ // // A local MSG is broadcast as DELIVER to every registered connection // (including a bridge connection dialed in from the other side) exactly like -// any other client — no protocol change. When a bridge *worker* itself +// any other client, no protocol change. When a bridge *worker* itself // receives a DELIVER over the connection it dialed, it re-broadcasts locally // with includeBridges=false, so the message never goes back out over any // bridge: no ping-pong between two nodes. @@ -207,7 +207,7 @@ func bridgeWorker(peerAddr, localNode, peerNode string, h *hub, sig interface{ L } // --------------------------------------------------------------------------- -// send / listen — minimal test clients used by verify to prove cross-host +// send / listen: minimal test clients used by verify to prove cross-host // delivery deterministically (this capstone trims ch27's fuller "loadtest"). // --------------------------------------------------------------------------- diff --git a/examples/41-capstone-fleet/go/fwatch.go b/examples/41-capstone-fleet/go/fwatch.go index b6cea42..cb89046 100644 --- a/examples/41-capstone-fleet/go/fwatch.go +++ b/examples/41-capstone-fleet/go/fwatch.go @@ -1,4 +1,4 @@ -// fwatch.go — the book's recurring file watcher (ch07 polling -> ch09 +// fwatch.go: the book's recurring file watcher (ch07 polling -> ch09 // inotify/epoll -> ch33 Landlock+seccomp), reduced here to what the fleet // needs: watch a directory, print one line per create/write/delete event, // and optionally run under a Landlock ruleset that restricts reads to that @@ -18,7 +18,7 @@ import ( ) // --------------------------------------------------------------------------- -// Landlock — raw syscalls; x/sys/unix has the SYS_LANDLOCK_* numbers but no +// Landlock: raw syscalls; x/sys/unix has the SYS_LANDLOCK_* numbers but no // wrapper. Read-only ruleset scoped to one directory (ch33's technique). // --------------------------------------------------------------------------- @@ -94,7 +94,7 @@ func applyLandlock(dir string) error { } // --------------------------------------------------------------------------- -// watch loop — inotify on the directory (non-recursive, matches ch09). +// watch loop: inotify on the directory (non-recursive, matches ch09). // --------------------------------------------------------------------------- func fwatchWatch(dir string, sandbox bool, timeoutMs int) int { diff --git a/examples/41-capstone-fleet/go/main.go b/examples/41-capstone-fleet/go/main.go index d79f55a..bc985c2 100644 --- a/examples/41-capstone-fleet/go/main.go +++ b/examples/41-capstone-fleet/go/main.go @@ -1,4 +1,4 @@ -// 41-capstone-fleet — pmon supervises chatterd + sysagent + fwatch across +// 41-capstone-fleet: pmon supervises chatterd + sysagent + fwatch across // two lab hosts, capability-dropped and Landlock-sandboxed, all telemetry // exported to the host LGTM stack. One binary, four subcommands (the same // self-re-exec shape ch34's container entrypoint uses): pmon is the fleet's @@ -110,8 +110,11 @@ func main() { peer, _ := argFlag(args, "--peer", "") peerNode, _ := argFlag(args, "--peer-node", "") tel := initTelemetry(context.Background(), "chatterd", node) - defer tel.shutdown(context.Background()) - os.Exit(chatterdServe(host, atoiOr(portS, 47100), node, peer, peerNode, tel)) + code := chatterdServe(host, atoiOr(portS, 47100), node, peer, peerNode, tel) + // os.Exit below would otherwise skip a deferred shutdown entirely, + // dropping any telemetry batched since the last periodic export. + _ = tel.shutdown(context.Background()) + os.Exit(code) case "send": args := rest[1:] host, _ := argFlag(args, "--host", "127.0.0.1") @@ -153,8 +156,11 @@ func main() { intervalS, _ := argFlag(rest, "--interval-ms", "2000") once := hasFlag(rest, "--once") tel := initTelemetry(context.Background(), "sysagent", node) - defer tel.shutdown(context.Background()) - os.Exit(sysagentRun(node, atoiOr(intervalS, 2000), once, tel)) + code := sysagentRun(node, atoiOr(intervalS, 2000), once, tel) + // os.Exit below would otherwise skip a deferred shutdown entirely, + // dropping any telemetry batched since the last periodic export. + _ = tel.shutdown(context.Background()) + os.Exit(code) case "fwatch": if len(rest) == 0 { diff --git a/examples/41-capstone-fleet/go/pmon.go b/examples/41-capstone-fleet/go/pmon.go index 9f4e320..e6258a5 100644 --- a/examples/41-capstone-fleet/go/pmon.go +++ b/examples/41-capstone-fleet/go/pmon.go @@ -1,4 +1,4 @@ -// pmon.go — the book's recurring process supervisor (ch11-14, 18-19, 32, +// pmon.go: the book's recurring process supervisor (ch11-14, 18-19, 32, // 34), grown here into the capstone's fleet init: it drops the capability // bounding set once, then fork/execs (self-re-exec, same technique ch34's // container entrypoint uses) chatterd, sysagent, and fwatch as children, @@ -70,7 +70,6 @@ func pmonRun(node, sandboxDir, peer, peerNode string, chatterdPort int, healthIn var wg sync.WaitGroup for _, spec := range specs { - spec := spec cs := states[spec.name] wg.Add(1) go func() { diff --git a/examples/41-capstone-fleet/go/proto.go b/examples/41-capstone-fleet/go/proto.go index 074b5d7..f081ac3 100644 --- a/examples/41-capstone-fleet/go/proto.go +++ b/examples/41-capstone-fleet/go/proto.go @@ -1,4 +1,4 @@ -// proto.go — the canonical chatterd chat frame (introduced ch21, unchanged +// proto.go: the canonical chatterd chat frame (introduced ch21, unchanged // since): magic "CH", version 1, one byte type, big-endian u16 length, then // payload. This capstone speaks only JOIN/MSG/DELIVER, the same three types // every prior chatterd chapter uses. diff --git a/examples/41-capstone-fleet/go/sysagent.go b/examples/41-capstone-fleet/go/sysagent.go index 37ab72d..62ebbfb 100644 --- a/examples/41-capstone-fleet/go/sysagent.go +++ b/examples/41-capstone-fleet/go/sysagent.go @@ -1,4 +1,4 @@ -// sysagent.go — the book's /proc-reading metrics agent (ch36), reduced to +// sysagent.go: the book's /proc-reading metrics agent (ch36), reduced to // three signals (cpu%, mem%, load1) exported both as a stdout line (cheap, // always-on local observability) and as OTel gauges over OTLP when // OTEL_EXPORTER_OTLP_ENDPOINT is set. Also carries the chaos driver's @@ -187,7 +187,7 @@ func sysagentRun(node string, intervalMs int, once bool, tel *telemetry) int { return 0 } -// saturate — chaos helper: busy-spin goroutines (cpu) or hold allocated, +// saturate: chaos helper; busy-spin goroutines (cpu) or hold allocated, // touched memory (mem) for --seconds, so a concurrently-running sysagent's // cpu_pct/mem_pct visibly rises (this is the fleet's USE-method callback). func saturate(resource string, seconds, workers, mb int) int { diff --git a/examples/41-capstone-fleet/go/telemetry.go b/examples/41-capstone-fleet/go/telemetry.go index 66b74c0..455f348 100644 --- a/examples/41-capstone-fleet/go/telemetry.go +++ b/examples/41-capstone-fleet/go/telemetry.go @@ -1,8 +1,8 @@ -// telemetry.go — OTel wiring shared by chatterd and sysagent. Both signals +// telemetry.go: OTel wiring shared by chatterd and sysagent. Both signals // (traces for chatterd deliveries, metrics for sysagent's /proc samples) are // exported over OTLP/HTTP. Endpoint comes from the standard // OTEL_EXPORTER_OTLP_ENDPOINT env var (the exporters read it themselves, per -// the OTel env-var spec) — this is exactly the variable +// the OTel env-var spec); this is exactly the variable // scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT, so a // service started under pmon on a lab guest picks it up with no extra flag. package main @@ -29,7 +29,7 @@ import ( // exportErrors counts OTel export failures observed via the global error // handler. verify polls this (via the "otel: export_errors=N" line each // service prints at exit / on request) as the deterministic stand-in for -// "the exporter got a 200" — the batch/periodic exporters only ever call the +// "the exporter got a 200"; the batch/periodic exporters only ever call the // error handler on a non-2xx response or a transport failure. var exportErrors atomic.Int64 @@ -51,7 +51,7 @@ func noopTelemetry(service string) *telemetry { // initTelemetry sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT // is set; otherwise it returns a disabled, no-op telemetry (the service still -// runs — OTel is additive, never a hard dependency of the fleet coming up). +// runs; OTel is additive, never a hard dependency of the fleet coming up). func initTelemetry(ctx context.Context, service, node string) *telemetry { endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") if endpoint == "" { diff --git a/examples/41-capstone-fleet/go/util.go b/examples/41-capstone-fleet/go/util.go index 39240cc..1eb7d0f 100644 --- a/examples/41-capstone-fleet/go/util.go +++ b/examples/41-capstone-fleet/go/util.go @@ -1,4 +1,4 @@ -// util.go — small helpers shared across the fleet's subcommands. +// util.go: small helpers shared across the fleet's subcommands. package main import ( diff --git a/examples/41-capstone-fleet/rust/src/caps.rs b/examples/41-capstone-fleet/rust/src/caps.rs index 448981f..4ae09c5 100644 --- a/examples/41-capstone-fleet/rust/src/caps.rs +++ b/examples/41-capstone-fleet/rust/src/caps.rs @@ -1,4 +1,4 @@ -// caps.rs — capability-bounding-set drop, the cheap, always-available half of +// caps.rs: capability-bounding-set drop, the cheap, always-available half of // this chapter's sandboxing story (the other half, Landlock, lives in // fwatch.rs). See go/caps.go for the full rationale. // @@ -6,7 +6,7 @@ // execve of a setuid-root binary can never regain it, for this process and // every descendant (the bounding set is inherited across fork/exec and can // only shrink). It is meaningful regardless of whether the calling process -// currently holds any capabilities at all — that is exactly pmon's case when +// currently holds any capabilities at all: that is exactly pmon's case when // deployed unprivileged on a lab guest. // lastKnownCap is CAP_CHECKPOINT_RESTORE (40) on a 6.x+ kernel; probing past diff --git a/examples/41-capstone-fleet/rust/src/chatterd.rs b/examples/41-capstone-fleet/rust/src/chatterd.rs index d667c7c..d57b0f6 100644 --- a/examples/41-capstone-fleet/rust/src/chatterd.rs +++ b/examples/41-capstone-fleet/rust/src/chatterd.rs @@ -1,4 +1,4 @@ -// chatterd.rs — the book's recurring peer-to-peer chat daemon (ch21-27), +// chatterd.rs: the book's recurring peer-to-peer chat daemon (ch21-27), // reduced to what the capstone needs: serve local clients, and (new in this // chapter) bridge two chatterd instances across hosts so a message sent on // one node is delivered to clients on the other. The wire frame (proto.rs) @@ -14,7 +14,7 @@ // // A local MSG is broadcast as DELIVER to every registered connection // (including a bridge connection dialed in from the other side) exactly like -// any other client — no protocol change. When a bridge *worker* itself +// any other client: no protocol change. When a bridge *worker* itself // receives a DELIVER over the connection it dialed, it re-broadcasts locally // with include_bridges=false, so the message never goes back out over any // bridge: no ping-pong between two nodes. @@ -23,7 +23,7 @@ // chatterd.cpp's own RAII-Fd approach: a poll(2)-with-timeout accept loop // (so the process can check the shutdown flag between accepts) instead of // closing a blocked accept() from another thread, which std::net cannot do -// safely. OwnedFd is the RAII socket — closes on drop, move-only, no +// safely. OwnedFd is the RAII socket: closes on drop, move-only, no // hand-written destructor needed. use std::net::{Ipv4Addr, SocketAddrV4}; use std::os::fd::{AsFd, AsRawFd, FromRawFd, OwnedFd, RawFd}; @@ -46,7 +46,7 @@ use crate::telemetry::{self, Telemetry}; use crate::util; // --------------------------------------------------------------------------- -// TCP helpers — IPv4-literal only, matching chatterd.cpp's inet_pton-based +// TCP helpers: IPv4-literal only, matching chatterd.cpp's inet_pton-based // listen_tcp/connect_tcp (every caller in this example, verify.lua included, // only ever passes literal addresses, never a hostname to resolve). // --------------------------------------------------------------------------- @@ -122,7 +122,7 @@ fn connect_tcp(host: &str, port: u16, timeout_ms: i64) -> Result ch09 +// fwatch.rs: the book's recurring file watcher (ch07 polling -> ch09 // inotify/epoll -> ch33 Landlock+seccomp), reduced here to what the fleet // needs: watch a directory, print one line per raw create/write/delete -// inotify event (no debounce/merge — go/fwatch.go prints every event as it +// inotify event (no debounce/merge: go/fwatch.go prints every event as it // arrives, and this port matches that exactly, unlike ex09/ex33's fuller // debounced watcher), and optionally run under a Landlock ruleset that // restricts reads to that directory. Landlock is wired exactly as ex33's @@ -9,7 +9,7 @@ // probe since landlock_create_ruleset has no libc/crate wrapper for the // version-query form); the inotify fd comes from ex09's `nix::sys::inotify` // usage. Full seccomp allow-listing was proven once in ex33 and is not -// re-derived here — capability-bounding-set drop (caps.rs) plus Landlock are +// re-derived here: capability-bounding-set drop (caps.rs) plus Landlock are // this capstone's sandboxing layers, applied by pmon before it execs fwatch. use std::fs; use std::os::fd::AsFd; @@ -27,7 +27,7 @@ use nix::sys::inotify::{AddWatchFlags, InitFlags, Inotify}; use crate::util; // --------------------------------------------------------------------------- -// Landlock — raw syscall for the ABI probe (no libc/crate wrapper exists for +// Landlock: raw syscall for the ABI probe (no libc/crate wrapper exists for // the version-query form of landlock_create_ruleset); the ruleset itself // goes through the `landlock` crate, exactly ex33's technique. // --------------------------------------------------------------------------- @@ -79,7 +79,7 @@ fn apply_landlock(dir: &str) -> Result<(), String> { } // classify mirrors go/fwatch.go's eventName: create, then delete, then -// modify (CLOSE_WRITE or MODIFY), else "other" — checked in that priority +// modify (CLOSE_WRITE or MODIFY), else "other": checked in that priority // order because a single inotify event's mask can carry more than one bit. fn classify(mask: AddWatchFlags) -> &'static str { if mask.contains(AddWatchFlags::IN_CREATE) { diff --git a/examples/41-capstone-fleet/rust/src/main.rs b/examples/41-capstone-fleet/rust/src/main.rs index 79d5f4d..03c5292 100644 --- a/examples/41-capstone-fleet/rust/src/main.rs +++ b/examples/41-capstone-fleet/rust/src/main.rs @@ -1,11 +1,11 @@ -// main.rs — 41-capstone-fleet — pmon supervises chatterd + sysagent + fwatch +// main.rs (41-capstone-fleet): pmon supervises chatterd + sysagent + fwatch // across two lab hosts, capability-dropped and Landlock-sandboxed, all // telemetry exported to the host LGTM stack. One binary, four subcommands // (the same self-re-exec shape ch34's container entrypoint uses): pmon is // the fleet's init; chatterd/sysagent/fwatch are the services it supervises. // // This file is a direct port of go/main.go's flag parsing -// (argFlag/hasFlag/firstPositional/atoiOr) and dispatch table — the usage +// (argFlag/hasFlag/firstPositional/atoiOr) and dispatch table: the usage // banner text below is byte-identical to the Go reference's (cross-checked // against cpp/src/main.cpp's std::print of the same banner), since // verify.lua and every language's demo.sh depend on it matching exactly. @@ -38,7 +38,7 @@ fn usage() { /// arg_flag scans args for "--name value" and returns (value, true), or /// (def, true) if absent, or (_, false) if the flag is present but has no -/// following value — mirrors go/main.go's argFlag exactly. +/// following value; mirrors go/main.go's argFlag exactly. fn arg_flag(args: &[String], name: &str, def: &str) -> (String, bool) { for (i, a) in args.iter().enumerate() { if a == name { @@ -57,7 +57,7 @@ fn has_flag(args: &[String], name: &str) -> bool { /// first_positional: walks args, treating every "--xxx" token as consuming /// exactly one following token (its value), and returns the first token that -/// survives that skipping — mirrors go/main.go's firstPositional exactly, +/// survives that skipping; mirrors go/main.go's firstPositional exactly, /// including its "skip the very next token no matter what it is" quirk. fn first_positional(args: &[String]) -> String { let mut skip = false; diff --git a/examples/41-capstone-fleet/rust/src/pmon.rs b/examples/41-capstone-fleet/rust/src/pmon.rs index f94e422..793ee20 100644 --- a/examples/41-capstone-fleet/rust/src/pmon.rs +++ b/examples/41-capstone-fleet/rust/src/pmon.rs @@ -1,4 +1,4 @@ -// pmon.rs — the book's recurring process supervisor (ch11-14, 18-19, 32, +// pmon.rs: the book's recurring process supervisor (ch11-14, 18-19, 32, // 34), grown here into the capstone's fleet init: it drops the capability // bounding set once (caps.rs), then fork/execs (self-re-exec via // /proc/self/exe, the same technique ch34's container entrypoint uses) @@ -87,7 +87,7 @@ fn describe_wait_status(status: nix::Result) -> String { } // run_child_supervisor: fork+exec `self_exe args...` in a loop, restarting on -// unexpected exit — the per-service goroutine from go/pmon.go, one OS thread +// unexpected exit: the per-service goroutine from go/pmon.go, one OS thread // per service here too. fn run_child_supervisor( self_exe: &CString, @@ -148,25 +148,15 @@ fn run_child_supervisor( } fn print_health(order: &[&str], states: &HashMap<&str, Arc>) { - let mut line = String::from("pmon: health "); - for (i, name) in order.iter().enumerate() { - if i > 0 { - line.push(' '); - } - line.push_str(name); - line.push('='); - line.push_str(states[name].get_state().name()); - } - line.push_str(" restarts="); - for (i, name) in order.iter().enumerate() { - if i > 0 { - line.push(','); - } - line.push_str(name); - line.push(':'); - line.push_str(&states[name].restarts.load(Ordering::Relaxed).to_string()); - } - println!("{line}"); + let health: Vec = order + .iter() + .map(|name| format!("{name}={}", states[name].get_state().name())) + .collect(); + let restarts: Vec = order + .iter() + .map(|name| format!("{name}:{}", states[name].restarts.load(Ordering::Relaxed))) + .collect(); + println!("pmon: health {} restarts={}", health.join(" "), restarts.join(",")); } pub fn run( diff --git a/examples/41-capstone-fleet/rust/src/proto.rs b/examples/41-capstone-fleet/rust/src/proto.rs index 96d6311..c0654b3 100644 --- a/examples/41-capstone-fleet/rust/src/proto.rs +++ b/examples/41-capstone-fleet/rust/src/proto.rs @@ -1,4 +1,4 @@ -// proto.rs — the canonical chatterd chat frame (introduced ch21, unchanged +// proto.rs: the canonical chatterd chat frame (introduced ch21, unchanged // since): magic "CH", version 1, one byte type, big-endian u16 length, then // payload. This capstone speaks only JOIN/MSG/DELIVER, the same three types // every prior chatterd chapter uses. diff --git a/examples/41-capstone-fleet/rust/src/sysagent.rs b/examples/41-capstone-fleet/rust/src/sysagent.rs index 7ca4e2c..9b99b06 100644 --- a/examples/41-capstone-fleet/rust/src/sysagent.rs +++ b/examples/41-capstone-fleet/rust/src/sysagent.rs @@ -1,4 +1,4 @@ -// sysagent.rs — /proc/stat, /proc/meminfo, /proc/loadavg readers and the +// sysagent.rs: /proc/stat, /proc/meminfo, /proc/loadavg readers and the // sample loop, ported field-for-field from go/sysagent.go so the printed // cpu_pct/mem_pct/load1 numbers are computed by the identical formula (not // just "close enough"): idle is specifically the 4th space-separated value @@ -151,7 +151,7 @@ pub fn run(node: &str, interval_ms: u64, once: bool, tel: &Telemetry) -> i32 { 0 } -// saturate — chaos helper: busy-spin threads (cpu) or hold allocated, +// saturate: chaos helper, busy-spin threads (cpu) or hold allocated, // touched memory (mem) for --seconds, so a concurrently-running sysagent's // cpu_pct/mem_pct visibly rises (this fleet's USE-method callback). pub fn saturate(resource: &str, seconds: i64, workers: i64, mb: i64) -> i32 { diff --git a/examples/41-capstone-fleet/rust/src/telemetry.rs b/examples/41-capstone-fleet/rust/src/telemetry.rs index fa0c713..0a03bb0 100644 --- a/examples/41-capstone-fleet/rust/src/telemetry.rs +++ b/examples/41-capstone-fleet/rust/src/telemetry.rs @@ -1,7 +1,7 @@ -// telemetry.rs — OTel wiring shared by chatterd and sysagent. Both signals +// telemetry.rs: OTel wiring shared by chatterd and sysagent. Both signals // (traces for chatterd deliveries, metrics for sysagent's /proc samples) are // exported over OTLP/HTTP. Endpoint comes from the standard -// OTEL_EXPORTER_OTLP_ENDPOINT env var — this init deliberately never calls +// OTEL_EXPORTER_OTLP_ENDPOINT env var: this init deliberately never calls // `.with_endpoint()`: opentelemetry-otlp 0.32's exporters only append the // per-signal "/v1/traces" / "/v1/metrics" suffix when they resolve the // endpoint themselves from OTEL_EXPORTER_OTLP_ENDPOINT (an explicit @@ -14,7 +14,7 @@ // export failures through its internal `otel_error!` tracing events, which // require a `tracing` subscriber to go anywhere. ex38 already established // that installing one (even a filtered one) risks corrupting the stdout/ -// stderr contract this capstone's verify.lua depends on line-for-line — so +// stderr contract this capstone's verify.lua depends on line-for-line, so // none is installed here, and (unlike go/telemetry.go's exportErrors counter, // fed by otel.SetErrorHandler) this port has no live export-failure signal to // report; the "otel export_errors=" line below is therefore always 0, a @@ -92,10 +92,10 @@ impl Telemetry { } /// Sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT is set; -/// otherwise returns a disabled, no-op handle (the service still runs — OTel +/// otherwise returns a disabled, no-op handle (the service still runs, OTel /// is additive, never a hard dependency of the fleet coming up). `service` is /// always one of this binary's own subcommand names ("chatterd"/"sysagent"), -/// passed as a `&'static str` literal by every caller — the Meter/Tracer +/// passed as a `&'static str` literal by every caller: the Meter/Tracer /// provider APIs require that lifetime. pub fn init(service: &'static str, node: &str) -> Telemetry { let endpoint = std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").unwrap_or_default(); @@ -124,7 +124,7 @@ fn init_enabled(service: &'static str, node: &str, _endpoint: &str) -> Result SysagentGauges { diff --git a/examples/41-capstone-fleet/rust/src/util.rs b/examples/41-capstone-fleet/rust/src/util.rs index 039090a..0df8766 100644 --- a/examples/41-capstone-fleet/rust/src/util.rs +++ b/examples/41-capstone-fleet/rust/src/util.rs @@ -1,8 +1,8 @@ -// util.rs — small helpers shared across the fleet's subcommands, the Rust +// util.rs: small helpers shared across the fleet's subcommands, the Rust // counterpart of util.go's installSignalFlag: chatterd/sysagent/fwatch each // run their own poll loop already (accept-with-timeout, interval sleep, // poll(2) with a timeout), so a level-triggered flag is all any of them -// needs from a signal — no EINTR-sensitive syscall unwinding required, unlike +// needs from a signal: no EINTR-sensitive syscall unwinding required, unlike // ex40's fastpath server. pmon (a supervisor, not a poller) does its own // dedicated child-reaping loop in pmon.rs instead of using this. use std::sync::OnceLock; diff --git a/examples/41-capstone-fleet/verify.lua b/examples/41-capstone-fleet/verify.lua index 873d3f0..e1070dd 100644 --- a/examples/41-capstone-fleet/verify.lua +++ b/examples/41-capstone-fleet/verify.lua @@ -1,5 +1,5 @@ -- Verify the capstone fleet (41-capstone-fleet) for LSP_LANG. This lane is --- mode:vm-peer, but -- following the convention example 23 established -- the +-- mode:vm-peer, but (following the convention example 23 established) the -- verify itself is a LOCAL assertion run entirely on loopback: the whole fleet -- (pmon supervising chatterd + sysagent + fwatch), the cross-node chatterd -- bridge (two nodes on 127.0.0.1 with distinct ports), and the OTLP export to @@ -11,13 +11,13 @@ -- Asserts observable behavior, identical in shape across cpp/go/rust: -- 1. usage/error shapes: no args, `chatterd` / `fwatch` with no subcommand, -- `chatterd send` missing --nick/--text, `sysagent saturate` missing --- --resource -- all exit 2 with the usage banner. +-- --resource, all exit 2 with the usage banner. -- 2. sysagent --once: exit 0, the `sysagent: node=.. cpu_pct=.. mem_pct=.. -- load1=.. ts=..` line, and the `otel disabled` line when no endpoint set. -- 3. fwatch snapshot DIR lists the directory; fwatch watch DIR reports -- `event: create|modify|delete PATH` lines then `(timeout)`; fwatch watch -- --sandbox installs a Landlock ruleset (`fwatch: landlock ABI=N enforced --- dir=..`) -- the kernel-level sandbox proof, not the program's own claim. +-- dir=..`), the kernel-level sandbox proof, not the program's own claim. -- 4. chatterd: a message sent to one node is `received from=NICK text=..` by -- a local listener; and across the BRIDGE (node A --peer-> node B), a -- message sent on A is delivered to a listener on B as @@ -88,7 +88,7 @@ local build = checks.run("./demo.sh " .. lang .. " build") checks.expect_exit(build, 0, lang .. ": demo.sh build exits 0") -- --------------------------------------------------------------------------- --- 1. usage / error shapes -- exit 2, usage banner. +-- 1. usage / error shapes: exit 2, usage banner. -- --------------------------------------------------------------------------- local function expect_usage(args, label) local r = sh("@BIN@ " .. args) diff --git a/examples/42-embedded-lua-scripting/README.md b/examples/42-embedded-lua-scripting/README.md index 64caa9e..458c3b6 100644 --- a/examples/42-embedded-lua-scripting/README.md +++ b/examples/42-embedded-lua-scripting/README.md @@ -1,7 +1,7 @@ -# 42 — embedded-lua-scripting +# 42: embedded-lua-scripting `pmon` lifts its restart/backoff decision out of the host binary and into a -shared **`policy.lua`**. Each host — C++ (sol2), Go (gopher-lua), Rust (mlua) — +shared **`policy.lua`**. Each host, C++ (sol2), Go (gopher-lua), Rust (mlua), embeds a Lua interpreter, sandboxes it, and calls `on_exit(info)` after each supervised child dies. Lua returns a `{ action, delay_ms, reason }` decision the host obeys. Edit `policy.lua`, send **SIGHUP**, and the running supervisor @@ -24,16 +24,16 @@ policy --host.now_ms()--> host -- monotonic-ish clock, for time-aware policies The sandbox opens only `base`, `string`, `table`, `math`, and removes `load`/`loadfile`/`dofile`/`require`; `os`, `io`, `package` and `debug` are never loaded. A policy therefore cannot touch the filesystem, spawn processes, -or pull in new code — `policy-evil.lua` proves it. +or pull in new code; `policy-evil.lua` proves it. ## Policy files | file | role | |---|---| -| `policy.lua` | v1 — base backoff 500ms, give up after 8 restarts | -| `policy-v2.lua` | v2 — base backoff 100ms, give up after 2 restarts (the SIGHUP edit) | -| `policy-evil.lua` | calls `os.execute` — rejected by the sandbox at runtime | -| `policy-broken.lua` | a syntax error — rejected at load | +| `policy.lua` | v1: base backoff 500ms, give up after 8 restarts | +| `policy-v2.lua` | v2: base backoff 100ms, give up after 2 restarts (the SIGHUP edit) | +| `policy-evil.lua` | calls `os.execute`: rejected by the sandbox at runtime | +| `policy-broken.lua` | a syntax error: rejected at load | `policy.lua` uses only the intersection of Lua 5.1 (gopher-lua) and Lua 5.4 (sol2/mlua): no `//`, no bitwise operators, no ``/``, no diff --git a/examples/42-embedded-lua-scripting/cpp/src/main.cpp b/examples/42-embedded-lua-scripting/cpp/src/main.cpp index 7ae188b..94aa510 100644 --- a/examples/42-embedded-lua-scripting/cpp/src/main.cpp +++ b/examples/42-embedded-lua-scripting/cpp/src/main.cpp @@ -1,14 +1,14 @@ -// pmon — chapter 42: an embedded-Lua restart policy via sol2. +// pmon: chapter 42: an embedded-Lua restart policy via sol2. // // The restart/backoff decision lives entirely in policy.lua, not in this // binary: the host loads it into a *sandboxed* Lua 5.4 state (base/string/ -// table/math only — no os, io, package, debug; load/loadfile/dofile/require/ +// table/math only, no os, io, package, debug; load/loadfile/dofile/require/ // collectgarbage/loadstring removed even though their libraries were never // opened, since a global with that name could otherwise be reintroduced by a // careless policy), calls `on_exit(info)` for every simulated child exit, and -// prints exactly what the policy hands back. A real SIGHUP — blocked with +// prints exactly what the policy hands back. A real SIGHUP: blocked with // sigprocmask and consumed as data through a signalfd + poll, the same -// pattern ch12/ch13 use for SIGCHLD — reloads the policy file with no +// pattern ch12/ch13 use for SIGCHLD: reloads the policy file with no // restart of pmon itself: the running supervisor keeps its child-tracking // state, only the decision logic changes underneath it. @@ -41,7 +41,7 @@ using std::chrono::steady_clock; // Every stdout write is a bare fflush away from a race with the parent shell // that greps this process's redirected output file for a sentinel line (see -// demo.sh's `run()`) — std::println is fully buffered once stdout is not a +// demo.sh's `run()`): std::println is fully buffered once stdout is not a // tty, so every print here is followed by an explicit flush. void say(std::string_view line) { std::println("{}", line); @@ -159,7 +159,7 @@ void print_decision(std::string_view name, const Decision& dec) { // --------------------------------------------------------------------------- // SIGHUP, the systems-programming content: block it process-wide up front so // there is never a window where it could interrupt a syscall or fire a -// default handler, then consume it as ordinary readability on a signalfd — +// default handler, then consume it as ordinary readability on a signalfd: // same pattern ch12/ch13 use for SIGCHLD, just a different signal. // --------------------------------------------------------------------------- @@ -196,7 +196,7 @@ void wait_for_sighup(int sigfd) { } // Fresh sandbox + host table + protected load + on_load(), all rebuilt from -// scratch — the simplest correct way to guarantee reload leaves no state +// scratch: the simplest correct way to guarantee reload leaves no state // from the previous policy behind. [[nodiscard]] std::expected reload(const std::string& policy_path) { sol::state lua = make_sandbox(); @@ -250,7 +250,7 @@ int cmd_run(const std::string& policy_path) { say("pmon: reload requested"); ::close(*sigfd); - // Re-read the same --policy path — its content is now policy-v2.lua. + // Re-read the same --policy path: its content is now policy-v2.lua. auto phase2 = reload(policy_path); if (!phase2) { std::println(stderr, "pmon: policy error: {}", phase2.error()); diff --git a/examples/42-embedded-lua-scripting/go/main.go b/examples/42-embedded-lua-scripting/go/main.go index 8445286..2d16c26 100644 --- a/examples/42-embedded-lua-scripting/go/main.go +++ b/examples/42-embedded-lua-scripting/go/main.go @@ -1,5 +1,5 @@ // pmon: a tiny process supervisor whose restart/backoff policy is not -// compiled in -- it is a Lua script (policy.lua), loaded into a sandboxed +// compiled in; it is a Lua script (policy.lua), loaded into a sandboxed // gopher-lua state and re-loaded on demand with a real SIGHUP. See the // chapter for why this beats a JSON config: the policy can compute (Lua // closures, exponential backoff, crash-loop detection) instead of merely @@ -38,7 +38,7 @@ type event struct { } // decision is what the policy's on_exit returned: what to do about the -// child, when, and why. reason is Lua's string, verbatim -- the host never +// child, when, and why. reason is Lua's string, verbatim; the host never // constructs decision text itself. type decision struct { action string @@ -85,7 +85,7 @@ func parsePolicyFlag(args []string) (string, error) { // die is the terminal-error path for `run`: print the fully wrapped error // chain and exit 1. Every error reaching here was built with fmt.Errorf's // %w, so the chain (CLI step -> Lua-load step -> Lua's own message) survives -// intact -- this is "errors, three ways" step one: wrapped errors. +// intact; this is "errors, three ways" step one: wrapped errors. func die(err error) { fmt.Fprintln(os.Stderr, "pmon:", err) os.Exit(1) @@ -104,7 +104,7 @@ func luaErr(err error) error { } // newSandbox builds a fresh Lua state with only base/string/table/math -// opened -- no os, io, package, debug, coroutine -- and the loader globals +// opened (no os, io, package, debug, coroutine), and the loader globals // that could pull in more code or touch the filesystem removed. Every // (re)load, at startup and again on each SIGHUP, gets its own state built // from scratch: a misbehaving policy can never leave residue for the next @@ -132,7 +132,7 @@ func newSandbox() *lua.LState { // The unopened libraries (os, io, package, debug, coroutine) are left as // plain nils by SkipOpenLibs, which gopher-lua's VM reports generically - // ("attempt to index a non-table object(nil) with key 'execute'") -- it + // ("attempt to index a non-table object(nil) with key 'execute'"); it // never names the global the way PUC-Rio Lua does. Trade a nil for a // guard table whose __index raises a message that names the blocked // global explicitly: same sandboxing (the real library is still never @@ -167,7 +167,7 @@ func newSandbox() *lua.LState { } // loadPolicy compiles and executes the policy chunk under the sandbox (a -// protected load -- a syntax error becomes a returned error, never a panic), +// protected load; a syntax error becomes a returned error, never a panic), // then calls on_load() if the policy defines one, also protected. func loadPolicy(L *lua.LState, path string) error { if err := L.DoFile(path); err != nil { @@ -260,7 +260,7 @@ func runPhase(L *lua.LState, events []event) error { // systems-programming content of this chapter (same pattern as ch12): pmon // is a long-running process that picks up a new policy without restarting. // signal.Notify + a blocking receive on a buffered channel is Go's idiomatic -// async-signal-safe equivalent of signalfd+poll -- the actual OS-level +// async-signal-safe equivalent of signalfd+poll; the actual OS-level // signal handling happens in the runtime's signal thread, never in // user Go code, so there is nothing unsafe to do inside a handler here. func cmdRun(policyPath string) error { @@ -298,8 +298,8 @@ func cmdRun(policyPath string) error { } // cmdCheck loads one policy under the sandbox and probes on_exit once. Any -// failure -- a syntax error at load, an on_load error, or a sandbox -// violation reached from on_exit -- is reported the same way: the operator +// failure (a syntax error at load, an on_load error, or a sandbox +// violation reached from on_exit) is reported the same way: the operator // deciding whether to trust this file doesn't need to know which step // failed, only that it did. func cmdCheck(policyPath string) int { diff --git a/examples/42-embedded-lua-scripting/policy-broken.lua b/examples/42-embedded-lua-scripting/policy-broken.lua index 6324097..ed3b985 100644 --- a/examples/42-embedded-lua-scripting/policy-broken.lua +++ b/examples/42-embedded-lua-scripting/policy-broken.lua @@ -1,10 +1,10 @@ --- policy-broken.lua -- a policy with a syntax error. [validate: expect-syntax-error] +-- policy-broken.lua: a policy with a syntax error. [validate: expect-syntax-error] -- -- The host loads policies with a protected load, so a syntax error surfaces as -- data (a returned error), not a crash. `pmon check --policy policy-broken.lua` -- reports the parse failure and exits non-zero; the running supervisor would -- keep its previous good policy. This is the load-time half of "Errors, three --- ways" -- the runtime half is policy-evil.lua. +-- ways"; the runtime half is policy-evil.lua. POLICY_VERSION = "broken" @@ -12,6 +12,6 @@ function on_exit(info) -- Deliberate syntax error: `end` is missing to close the `if`. if info.exit_code == 0 then return { action = "stop", delay_ms = 0, reason = "clean" } - -- (no `end` here) -- parser reports "'end' expected" + -- (no `end` here): parser reports "'end' expected" return { action = "restart", delay_ms = 0, reason = "crash" } end diff --git a/examples/42-embedded-lua-scripting/policy-evil.lua b/examples/42-embedded-lua-scripting/policy-evil.lua index 28312a2..40a7d31 100644 --- a/examples/42-embedded-lua-scripting/policy-evil.lua +++ b/examples/42-embedded-lua-scripting/policy-evil.lua @@ -1,11 +1,11 @@ --- policy-evil.lua -- a policy that tries to break out of the sandbox. +-- policy-evil.lua: a policy that tries to break out of the sandbox. -- -- The host loads every policy inside a sandbox: os, io, package and debug are -- never opened, so `os` is not available to the script. This policy tries to -- shell out anyway. Reaching for `os.execute` raises a Lua error (the exact --- wording differs per host -- PUC-Lua names the nil global, gopher-lua's guard +-- wording differs per host: PUC-Lua names the nil global, gopher-lua's guard -- table names the blocked field), which the host catches through its protected --- call and reports as a rejected policy -- the supervisor keeps running, the +-- call and reports as a rejected policy: the supervisor keeps running, the -- malicious policy never takes effect. `pmon check --policy policy-evil.lua` -- shows the rejection. diff --git a/examples/42-embedded-lua-scripting/policy-v2.lua b/examples/42-embedded-lua-scripting/policy-v2.lua index ea34bea..733db76 100644 --- a/examples/42-embedded-lua-scripting/policy-v2.lua +++ b/examples/42-embedded-lua-scripting/policy-v2.lua @@ -1,4 +1,4 @@ --- policy-v2.lua -- the "edited" policy the demo drops in before sending SIGHUP. +-- policy-v2.lua: the "edited" policy the demo drops in before sending SIGHUP. -- -- Identical in shape to policy.lua; only CONFIG and POLICY_VERSION change. The -- host re-reads its --policy path on SIGHUP, so swapping this file in and diff --git a/examples/42-embedded-lua-scripting/policy.lua b/examples/42-embedded-lua-scripting/policy.lua index a5e0bcf..279c1ac 100644 --- a/examples/42-embedded-lua-scripting/policy.lua +++ b/examples/42-embedded-lua-scripting/policy.lua @@ -1,4 +1,4 @@ --- policy.lua -- pmon restart/backoff policy. +-- policy.lua: pmon restart/backoff policy. -- -- Loaded once at startup and again on every SIGHUP. The host (C++/sol2 and -- Rust/mlua on Lua 5.4, Go/gopher-lua on Lua 5.1) calls on_exit() after each @@ -8,14 +8,14 @@ -- the exact same file drives all three hosts. -- -- Contract: --- on_exit(info) -> decision -- required --- on_load() -- optional, called once right after (re)load --- host.log(msg) -- host callback, available inside this file --- host.now_ms() -- host callback, monotonic-ish wall clock ms +-- on_exit(info) -> decision : required +-- on_load() : optional, called once right after (re)load +-- host.log(msg) : host callback, available inside this file +-- host.now_ms() : host callback, monotonic-ish wall clock ms -- -- The host sandboxes this script: os, io, package and debug are NOT loaded, -- and load/loadfile/dofile/require are removed, so a policy cannot touch the --- filesystem, spawn processes, or pull in new code -- see the chapter. +-- filesystem, spawn processes, or pull in new code; see the chapter. -- Tunables. Nothing outside this table needs to change to retune the policy, -- which is the whole point of shipping it as data-shaped Lua instead of a @@ -31,7 +31,7 @@ local CONFIG = { POLICY_VERSION = "1.0.0" -- Compute the next backoff delay from the delay the host actually used last --- time (info.last_backoff_ms), not from a restart counter -- the host doesn't +-- time (info.last_backoff_ms), not from a restart counter: the host doesn't -- need to know the policy's math, and the policy doesn't need the host's -- counters to stay in lockstep with its own. It just doubles whatever it is -- handed back, with a floor and a cap. @@ -92,7 +92,7 @@ function on_exit(info) end -- Crash-looping: several failures in a row, each dying almost immediately. - -- This is the case a flat restart counter misses -- 8 restarts spread over + -- This is the case a flat restart counter misses: 8 restarts spread over -- an hour is normal churn, 3 restarts in 3 seconds is a broken binary. local fast = uptime_ms >= 0 and uptime_ms < CONFIG.fast_crash_ms if fast and consecutive_failures >= CONFIG.fast_crash_streak then diff --git a/examples/42-embedded-lua-scripting/rust/src/main.rs b/examples/42-embedded-lua-scripting/rust/src/main.rs index e38c488..3b19d36 100644 --- a/examples/42-embedded-lua-scripting/rust/src/main.rs +++ b/examples/42-embedded-lua-scripting/rust/src/main.rs @@ -1,11 +1,11 @@ -// pmon -- a restart/backoff supervisor whose policy lives in Lua, not in this +// pmon: a restart/backoff supervisor whose policy lives in Lua, not in this // binary. `policy.lua` (and its SIGHUP-swapped sibling `policy-v2.lua`) is // loaded into a *sandboxed* mlua state: only base/string/table/math are // opened, and the loader globals that could read files or eval new code from // inside the sandbox (`load`, `loadfile`, `dofile`, `require`, // `collectgarbage`, `loadstring`) are removed after the fact. `os`, `io`, // `package` and `debug` are simply never opened, so a policy that reaches for -// `os.execute` hits a nil global -- see `run_check` and `policy-evil.lua`. +// `os.execute` hits a nil global: see `run_check` and `policy-evil.lua`. // // Errors, three ways: CLI parse failures `bail!` out of `parse_args`; // fallible host-side operations return `anyhow::Result` composed with @@ -26,7 +26,7 @@ use anyhow::{Context, Result, bail}; use mlua::{Function, Lua, LuaOptions, StdLib, Table, Value}; /// Wall-clock milliseconds since the UNIX epoch, exposed to policies as -/// `host.now_ms()`. "Monotonic-ish" is all the contract asks for -- policies +/// `host.now_ms()`. "Monotonic-ish" is all the contract asks for: policies /// only use it for human-readable timestamps, never for scheduling. fn now_ms() -> i64 { SystemTime::now() @@ -88,7 +88,7 @@ fn load_policy(lua: &Lua, path: &str) -> mlua::Result<()> { /// One supervised-child exit event, exactly as the shared scenario spec /// describes it. `signal` is never set in this chapter's scenario (every /// event here is an exit-code death), so the `info` table simply never gets -/// a `signal` field -- matching "a nil field is absent from the table". +/// a `signal` field, matching "a nil field is absent from the table". struct ExitEvent { name: &'static str, exit_code: i64, @@ -111,7 +111,7 @@ fn info_table(lua: &Lua, ev: &ExitEvent) -> mlua::Result { /// Call `on_exit(info)` under a protected call and print the resulting /// decision. The `reason` text is entirely Lua's (built with -/// `string.format` in policy.lua) -- this only formats the wrapping +/// `string.format` in policy.lua); this only formats the wrapping /// `pmon: decision ...` line around whatever the policy returned. fn decide(lua: &Lua, ev: &ExitEvent) -> mlua::Result<()> { let info = info_table(lua, ev)?; @@ -199,7 +199,7 @@ fn phase2(lua: &Lua) -> mlua::Result<()> { /// installs an async-signal-safe handler that only flips an `AtomicBool` (no /// allocation, no Lua call, no I/O inside the signal handler itself), and /// this thread polls that flag instead of doing any real work inside signal -/// context -- the same discipline ch12 uses for signalfd/self-pipe. The flag +/// context: the same discipline ch12 uses for signalfd/self-pipe. The flag /// is registered *before* we announce readiness, so a SIGHUP that lands the /// instant after the "awaiting" line prints is never missed. fn cmd_run(policy_path: &str) -> Result<()> { @@ -247,7 +247,7 @@ fn cmd_run(policy_path: &str) -> Result<()> { /// Load `policy_path` under the sandbox and run one probe `on_exit` call. /// Returns the policy's declared version on success; the `mlua::Error` on /// failure already carries the Lua-side message (a syntax error's location, -/// or -- for `policy-evil.lua` -- "attempt to index a nil value (global +/// or (for `policy-evil.lua`) "attempt to index a nil value (global /// 'os')", which is how the sandbox rejection gets reported without the host /// knowing anything about what the policy tried to do). fn run_check(policy_path: &str) -> mlua::Result { diff --git a/examples/42-embedded-lua-scripting/verify.lua b/examples/42-embedded-lua-scripting/verify.lua index accf063..55248a6 100644 --- a/examples/42-embedded-lua-scripting/verify.lua +++ b/examples/42-embedded-lua-scripting/verify.lua @@ -1,7 +1,7 @@ -- Verify the embedded-Lua policy engine for LSP_LANG (cpp|go|rust). -- -- The C++/Go/Rust hosts all embed the SAME policy.lua and must produce the --- SAME observable decisions -- that identity is what this script asserts, not +-- SAME observable decisions: that identity is what this script asserts, not -- merely a zero exit. Every assertion is a checks.expect_match against the -- host's fixed decision-line format, so it is deterministic and language- -- neutral (no wall-clock timing, no process introspection). diff --git a/examples/43-rust-macros-for-systems/README.md b/examples/43-rust-macros-for-systems/README.md index 95816ad..cf48fc1 100644 --- a/examples/43-rust-macros-for-systems/README.md +++ b/examples/43-rust-macros-for-systems/README.md @@ -1,17 +1,17 @@ -# 43 — rust-macros-for-systems +# 43: rust-macros-for-systems The book's first **single-language** example (Rust only): the deep dives don't mirror across three languages. It shows three metaprogramming techniques a -systems programmer actually reaches for, as a small cargo **workspace** — a +systems programmer reaches for, as a small cargo **workspace**: a proc-macro crate plus the binary that uses it: -- **`#[derive(SysError)]`** — a derive macro that generates `Display` + `std::error::Error` +- **`#[derive(SysError)]`**: a derive macro that generates `Display` + `std::error::Error` from `#[error("...")]` attributes (a miniature `thiserror`), for errno-style error enums. -- **`#[instrument]`** — an attribute macro that wraps a function to log its - entry (name + args), exit (return value), and wall-clock duration — tracing +- **`#[instrument]`**: an attribute macro that wraps a function to log its + entry (name + args), exit (return value), and wall-clock duration, tracing without a runtime. -- **A sound `unsafe` unaligned read**, validated with **miri** — plus a +- **A sound `unsafe` unaligned read**, validated with **miri**, plus a defect-named `--oob-read` path that miri flags as undefined behaviour. ## Layout @@ -38,7 +38,7 @@ proc-macro crate plus the binary that uses it: ## Verify `verify.lua` (Rust only; skips other langs) asserts the generated `Display` -strings, the `#[instrument]` entry/exit/timing lines, and the parsed record — +strings, the `#[instrument]` entry/exit/timing lines, and the parsed record: behaviour, not exit codes. The gate runs on the **pinned stable** toolchain; miri (nightly) is exercised by the chapter, not the gate. diff --git a/examples/43-rust-macros-for-systems/rust/src/main.rs b/examples/43-rust-macros-for-systems/rust/src/main.rs index 67800a1..d8c9c62 100644 --- a/examples/43-rust-macros-for-systems/rust/src/main.rs +++ b/examples/43-rust-macros-for-systems/rust/src/main.rs @@ -3,9 +3,9 @@ //! //! - `#[derive(SysError)]` (from the `sysmacros` proc-macro crate) generates //! `Display` + `std::error::Error` for an errno-style error enum from -//! `#[error("...")]` attributes — a miniature `thiserror`. +//! `#[error("...")]` attributes: a miniature `thiserror`. //! - `#[instrument]` wraps `hash_chunk` to log its entry, exit, and -//! wall-clock duration — tracing without a runtime. +//! wall-clock duration: tracing without a runtime. //! - `read_record` is a sound `unsafe` unaligned read of a `repr(C)` struct //! out of a byte buffer, validated with miri; `--oob-read` deliberately //! reads 4 bytes past the buffer so miri reports the undefined behaviour. @@ -24,7 +24,7 @@ enum PmonError { #[instrument] fn hash_chunk(len: usize) -> u64 { - // Deterministic FNV-1a over 0..len — the exact value doesn't matter, only + // Deterministic FNV-1a over 0..len: the exact value doesn't matter, only // that it's reproducible across runs. let mut h = 0xcbf29ce484222325u64; for i in 0..len { diff --git a/examples/43-rust-macros-for-systems/rust/sysmacros/src/lib.rs b/examples/43-rust-macros-for-systems/rust/sysmacros/src/lib.rs index 027ca89..1fe8b98 100644 --- a/examples/43-rust-macros-for-systems/rust/sysmacros/src/lib.rs +++ b/examples/43-rust-macros-for-systems/rust/sysmacros/src/lib.rs @@ -1,4 +1,4 @@ -//! `sysmacros` — the proc-macro crate backing chapter 43's two techniques: +//! `sysmacros`, the proc-macro crate backing chapter 43's two techniques: //! //! - `#[derive(SysError)]`: a miniature `thiserror`. Each enum variant carries //! `#[error("...")]`; this derive generates `Display` (interpolating the @@ -11,7 +11,7 @@ //! //! Malformed input (a `#[derive(SysError)]` on a non-enum, or a variant //! missing `#[error(...)]`) is reported as a real `compile_error!` via -//! `syn::Error::to_compile_error()` — never a panic. +//! `syn::Error::to_compile_error()`, never a panic. use proc_macro::TokenStream; use proc_macro2::TokenStream as TokenStream2; diff --git a/examples/44-go-runtime-for-systems/README.md b/examples/44-go-runtime-for-systems/README.md index 163468d..40b0f09 100644 --- a/examples/44-go-runtime-for-systems/README.md +++ b/examples/44-go-runtime-for-systems/README.md @@ -1,23 +1,23 @@ -# 44 — go-runtime-for-systems +# 44: go-runtime-for-systems A **single-language** example (Go only): a program that observes its own Go runtime across four themes, printing self-computed verdict lines so the behaviour is checkable, not just describable. -- **`sched`** — the GMP scheduler: a rendezvous barrier proves N goroutines are +- **`sched`**: the GMP scheduler: a rendezvous barrier proves N goroutines are concurrently alive, a worker pool proves work distribution, and a busy-loop under `GOMAXPROCS=1` proves signal-based **async preemption** (a starved goroutine still makes progress). -- **`gc`** — GC pacing: with auto-GC off, one `runtime.GC()` advances `NumGC` by +- **`gc`**: GC pacing: with auto-GC off, one `runtime.GC()` advances `NumGC` by exactly 1; then the same churn under `GOGC=100` vs `800`, and under a tight `GOMEMLIMIT`, shows the pacer's CPU/memory trade-off in collection counts. -- **`netpoll`** — 2000 goroutines block on pipe reads while the OS **thread** +- **`netpoll`**: 2000 goroutines block on pipe reads while the OS **thread** count stays bounded (`<= 4*GOMAXPROCS+16`): goroutines are not threads. -- **`knobs`** — echoes the effective `GOMAXPROCS`/`NumCPU`/`GOGC`/`GOMEMLIMIT`/ +- **`knobs`**: echoes the effective `GOMAXPROCS`/`NumCPU`/`GOGC`/`GOMEMLIMIT`/ `GODEBUG`. On **Go 1.26** this runs with the Green Tea GC on by default and container-aware -`GOMAXPROCS` (both since 1.25/1.26) — see the chapter. +`GOMAXPROCS` (both since 1.25/1.26); see the chapter. ## Run it @@ -37,7 +37,7 @@ GODEBUG=asyncpreemptoff=1 ./go/bin/app sched # the preempt check flips to FAIL ## Verify `verify.lua` (Go only; skips other langs) matches each theme's fixed verdict -label ending in the literal `ok`, plus the `ALL: PASS` roll-up — behaviour the +label ending in the literal `ok`, plus the `ALL: PASS` roll-up: behaviour the program asserts about itself, run on the pinned Go 1.26.5 toolchain. ```bash @@ -45,4 +45,4 @@ LSP_LANG=go lua verify.lua ``` Mode: `local`. Stdlib only (`runtime`, `runtime/debug`, `os`, `sync`, -`sync/atomic`, `bufio`) — no external modules. +`sync/atomic`, `bufio`); no external modules. diff --git a/examples/44-go-runtime-for-systems/go/main.go b/examples/44-go-runtime-for-systems/go/main.go index bba7c1f..e5fd486 100644 --- a/examples/44-go-runtime-for-systems/go/main.go +++ b/examples/44-go-runtime-for-systems/go/main.go @@ -1,6 +1,6 @@ // go-runtime-for-systems: a program that observes its own Go runtime across -// four themes -- scheduler (GMP), GC pacing, netpoller/syscalls, and effective -// knobs -- printing self-computed verdict lines so the behavior is checkable, +// four themes: scheduler (GMP), GC pacing, netpoller/syscalls, and effective +// knobs; printing self-computed verdict lines so the behavior is checkable, // not just describable. Every "