From 91a33ea0d923f2138fa62ab057b5c51062b613fc Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Fri, 23 Oct 2026 18:06:29 -0400 Subject: [PATCH] docs: add Learning objectives + Why this matters spine to body chapters Add the chapter-spine to the 39 substantive body chapters (03-24 and 28-44, excluding Setting-Up 00-02, the redirect stubs 25-27, and the Appendices/ Compendium 45-56): a '## Learning objectives' section ('By the end of this chapter you can:' + action-verb capability bullets) followed by '## Why this matters', as the first two H2s after the front matter and before the existing hook. Objectives authored per chapter from its real content (syscalls/APIs/ tools and the tri-language artifacts it builds); Why-this-matters authored fresh (the existing hook stays as the hook). Matches the spine now documented in CLAUDE.md. Additive (+1068 lines); existing content untouched; Jekyll build clean; residual ban-tier unchanged (justified deliberately<=1/ch + delve toolname). --- _docs/03-observability-stack.md | 40 +++++++++++++++++++++++ _docs/04-syscalls-and-the-abi.md | 41 +++++++++++++++++++++++ _docs/05-errors-three-ways.md | 41 +++++++++++++++++++++++ _docs/06-concurrency-three-ways.md | 45 ++++++++++++++++++++++++++ _docs/07-fds-raii-and-the-vfs.md | 39 ++++++++++++++++++++++ _docs/08-page-cache-and-durability.md | 42 ++++++++++++++++++++++++ _docs/09-event-loops-epoll-inotify.md | 17 ++++++++++ _docs/10-io-uring.md | 16 +++++++++ _docs/11-process-lifecycle.md | 17 ++++++++++ _docs/12-signals-and-safe-handling.md | 17 ++++++++++ _docs/13-pidfd-modern-process-mgmt.md | 17 ++++++++++ _docs/14-identity-and-privilege.md | 16 +++++++++ _docs/15-virtual-memory.md | 35 ++++++++++++++++++++ _docs/16-mmap-and-shared-mappings.md | 33 +++++++++++++++++++ _docs/17-allocators-and-gc-runtimes.md | 35 ++++++++++++++++++++ _docs/18-pipes-fifos-splice.md | 33 +++++++++++++++++++ _docs/19-unix-sockets-fd-passing.md | 33 +++++++++++++++++++ _docs/20-shared-memory-coordination.md | 36 +++++++++++++++++++++ _docs/21-tcp-sockets.md | 36 +++++++++++++++++++++ _docs/22-scaling-the-server.md | 37 +++++++++++++++++++++ _docs/23-udp-and-peer-discovery.md | 36 +++++++++++++++++++++ _docs/24-time-timers-deadlines.md | 36 +++++++++++++++++++++ _docs/28-gdb-and-remote-debugging.md | 35 ++++++++++++++++++++ _docs/29-valgrind-sanitizers-miri.md | 42 ++++++++++++++++++++++++ _docs/30-ebpf-observation-toolkit.md | 41 +++++++++++++++++++++++ _docs/31-per-language-toolbelts.md | 18 +++++++++++ _docs/32-namespaces-and-cgroups.md | 18 +++++++++++ _docs/33-seccomp-and-landlock.md | 18 +++++++++++ _docs/34-our-programs-in-containers.md | 18 +++++++++++ _docs/35-virtualization-and-kvm.md | 18 +++++++++++ _docs/36-proc-sys-and-the-agent.md | 18 +++++++++++ _docs/37-use-method-60s-analysis.md | 18 +++++++++++ _docs/38-opentelemetry-to-lgtm.md | 18 +++++++++++ _docs/39-benchmarking-without-lies.md | 18 +++++++++++ _docs/40-low-latency-fastpath.md | 18 +++++++++++ _docs/41-capstone-fleet.md | 18 +++++++++++ _docs/42-embedded-lua-scripting.md | 18 +++++++++++ _docs/43-rust-macros-for-systems.md | 18 +++++++++++ _docs/44-go-runtime-for-systems.md | 18 +++++++++++ 39 files changed, 1068 insertions(+) diff --git a/_docs/03-observability-stack.md b/_docs/03-observability-stack.md index f3ea642..ce5d091 100644 --- a/_docs/03-observability-stack.md +++ b/_docs/03-observability-stack.md @@ -6,6 +6,46 @@ description: "Stand up the Grafana LGTM all-in-one container with podman compose duration: "30 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Stand up the Grafana `grafana/otel-lgtm` all-in-one container with `podman + compose`, pinned to a fixed image tag for reproducible output +- Enable the `podman.socket` systemd user unit so `podman compose` has a + Docker API endpoint to delegate to +- Explain OTLP's two transports (gRPC on `:4317`, HTTP on `:4318`) and why + this book standardizes on the curl-able HTTP path +- Post a minimal OTLP/JSON trace to `:4318/v1/traces` with `curl` and query + it back through Grafana's Tempo datasource proxy +- Declare an example's telemetry dependency with `requires: [lgtm]` and + explain why the harness marks a missing backend SKIP, never FAIL +- Reach the host's LGTM stack from inside a lab VM through the libvirt + gateway address and forward `OTEL_ENDPOINT` past `sudo`'s environment strip +- Cross-check `podman compose`'s claims independently with `podman ps` and + Grafana's own `/api/health` endpoint + +## Why this matters + +Every language runtime and every syscall wrapper in this book eventually +needs somewhere to send what it observes about itself: a span, a counter, a +log line. Without a backend that telemetry is just `printf` with extra +ceremony, read once and discarded. Standing up the pipeline here, before any +instrumented example exists, means every later chapter can treat +"telemetry works" as a given rather than a per-chapter chore, and it means +the choice of transport (OTLP over HTTP) and the shape of the check +(`requires: [lgtm]`, skip-not-fail) get decided once instead of twenty-six +times. + +It also previews a discipline that recurs constantly in systems programming: +building the smallest thing that gives you the real interface. A production +observability stack is four independently scaled services; a workbench needs +the interface those services expose, not their scale. Knowing how to collapse +infrastructure to its essential shape, and verifying the collapse with the +same tools (`curl`, `podman ps`) you would use against the real thing, is a +skill this book leans on again when the lab VMs and later the page cache +force the same kind of judgment call. + Chapter 1 gave you three toolchains; Chapter 2 gave you two disposable VMs. This chapter adds the last piece of the workbench: somewhere for telemetry to *go*. Many chapters in this book emit metrics, traces, or logs (a syscall diff --git a/_docs/04-syscalls-and-the-abi.md b/_docs/04-syscalls-and-the-abi.md index 850177e..abc6b2e 100644 --- a/_docs/04-syscalls-and-the-abi.md +++ b/_docs/04-syscalls-and-the-abi.md @@ -6,6 +6,47 @@ description: "What a system call really is: the syscall instruction, the registe duration: 40 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- State the x86-64 syscall register ABI: the syscall number in `rax`, + arguments in `rdi rsi rdx r10 r8 r9`, and the result or `-errno` back in + `rax`, including why `r10` stands in for the C ABI's `rcx` +- Trace a syscall's round trip from the `syscall` trap instruction through + `entry_SYSCALL_64` and `sys_call_table` dispatch back to ring 3 +- Identify vDSO-served calls (`clock_gettime`, `vgetrandom`) that never trap, + using `/proc/self/maps` and `strace -c` counts +- Compare how glibc wrappers, the Go runtime's own stubs, and rustix's + `linux_raw` backend each reach the kernel, using `objdump`, `ldd`, and + `strace` +- Build and run `sysprobe`, a four-step syscall demonstrator (open, write, + sleep, random) implemented identically in C++23, Go, and Rust +- Read `strace` output closely enough to spot wrapper-layer substitutions + (`nanosleep` traced as `clock_nanosleep`) and partial vDSO interception +- Attribute a binary's total syscall count, via `strace -c`, to runtime + startup overhead versus the four steps of actual application work + +## Why this matters + +Every abstraction in this book, RAII wrappers, error types, concurrency +primitives, sits on top of the same fact: a user-space program can only ask +the kernel for anything by trapping into it, and that trap is a register +convention, not a function call. Until that boundary is precise in your head, +every later claim about performance, safety, or portability is resting on a +guess. This chapter replaces the guess with instruments: a register-level +description of the trap, and three ways of watching it happen (`strace`, +`objdump`, `/proc/self/maps`) that stay in your toolkit for the rest of the +book. + +It also sets up the book's central comparative method. C++, Go, and Rust are +not just three syntaxes for the same program; they are three different +relationships with the kernel ABI, from zero inlined `syscall` instructions +to 511 to 32. Seeing that difference now, with a trivial four-step program, +means later chapters on concurrency, I/O multiplexing, and signals can point +at a runtime behavior and explain it in terms you already have, instead of +treating each language's quirks as unrelated trivia. + Chapter 1 ended on a number it did not explain: the C++ hello made 64 syscalls, the Go hello made 217, and we waved at "the runtime" and moved on. This chapter is where that debt gets paid. With the lab and observability stack built, Part diff --git a/_docs/05-errors-three-ways.md b/_docs/05-errors-three-ways.md index ed6eb5a..d0bf399 100644 --- a/_docs/05-errors-three-ways.md +++ b/_docs/05-errors-three-ways.md @@ -6,6 +6,47 @@ description: "One error's journey from a failing write(2) into std::expected, a duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- State the kernel's error contract (a failing syscall returns `-errno` in + `rax`) and explain how glibc's thread-local `errno`, Go's `unix.Errno` + value, and rustix's `Errno` wrapper each deliver that same integer +- Implement `copyx`'s error taxonomy, distinct exit codes for source-side + versus destination-side failures, using `std::expected`, a wrapped Go + `error`, and a `thiserror` enum +- Apply the book's EINTR-retry and short-write-resume policies inside + `read`/`write` loops in all three languages +- Draw the values-versus-exceptions line per language: `std::expected` + versus exceptions at subsystem boundaries in C++, `error` versus `panic` + in Go, `Result` versus an abort-only `panic!` in Rust +- Reproduce `ENOSPC` deterministically with `/dev/full` and confirm + byte-identical error messages and exit codes across the three binaries +- Use `strace` to catch the exact failing `write(2)` syscall, before any + runtime has dressed the failure up as a language-level error + +## Why this matters + +A syscall failure is not an exceptional event the kernel escalates to you; +it is an ordinary return value that your language's runtime chooses how to +represent, and that choice has consequences for every program you write +afterward. `copyx` turns that choice into something you can hold in your +hands: one file copier, one kernel fact (`ENOSPC` is 28, no more and no +less), three +type systems arguing about what to do with it, and one shared contract +(exit codes, message format, retry policy) that makes the disagreement +irrelevant from the outside. + +That contract is not local to this chapter. Every program in the rest of the +book, the concurrent hasher in Chapter 6, the file watcher built across Part +2, the benchmark in Chapter 8, inherits the EINTR-retry and short-write-resume +rules established here, and every one of them reaches for the same +values-versus-exceptions line when a new failure mode shows up. Getting the +taxonomy right once, with a program small enough to read start to finish, +is what lets later chapters spend their words on new material instead of +re-litigating what a correct error policy looks like. + Chapter 4 walked a syscall down to the boundary and back, and ended on the detail this chapter is built from: when a syscall fails, the kernel does not throw, log, or explain: it puts a small negative integer in a register and diff --git a/_docs/06-concurrency-three-ways.md b/_docs/06-concurrency-three-ways.md index 123bfa8..e279355 100644 --- a/_docs/06-concurrency-three-ways.md +++ b/_docs/06-concurrency-three-ways.md @@ -6,6 +6,51 @@ description: "The second thread that runs the whole book: how each language shar duration: 50 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Compare C++23's `jthread`/`stop_token`/`condition_variable_any`, Go's + goroutines with channels and `context.Context`, and Rust's scoped threads + with `Send`/`Sync` as three vocabularies over the same `clone(2)`/`futex(2)` + kernel primitives +- Build `parhash`, a four-worker parallel file hasher that produces + byte-identical stdout across all three languages despite each one walking + the tree in a different order +- Implement safe cross-thread SIGINT cancellation per language: blocking the + signal and consuming it with `sigtimedwait` in C++, `signal.NotifyContext` + in Go, and a `signal_hook`-driven `AtomicBool` in Rust +- Explain why `mpsc::Receiver` is not `Sync`, why that forces + `Arc>`, and how `thread::scope` lets workers borrow stack + data without reference counting +- Verify a cancellation-drain contract live, stop accepting new work, finish + what is in flight, report what completed, exit 130, by sending a real + SIGINT mid-run +- Inspect a running process's threads with `/proc//task` and each + thread's `wchan` to confirm all three language models reduce to the same + kernel threads + +## Why this matters + +Every language in this book claims to make concurrency easier, and each one +is telling the truth about a different part of the problem. C++ gives you +the raw parts and trusts you to assemble a correct queue and signal +discipline; Go gives you a runtime that hides the threads entirely behind +channels and a scheduler; Rust gives you a compiler that refuses to let a +shared-mutation bug type-check. `parhash` makes that difference concrete +instead of rhetorical: the same four-worker pool, three times, with +`/proc//task` proving that underneath the vocabulary it is always +`clone(2)` threads parked on `futex(2)`. + +This chapter is also where the book's two permanent threads meet: Chapter 5's +error policy has to survive concurrent execution, and the cancellation drain +built here, stop accepting, finish what you hold, report what completed, is +the pattern every later concurrent example reuses, from the signal handling +in Part 7 to the event loops in Chapter 27. Understanding *why* each language +reaches for its own idiom here, rather than just copying the syntax, is what +makes those later chapters readable at a glance instead of a fresh puzzle +each time. + Chapter 5 pinned down the first of this book's two permanent threads: how each language turns an `errno` into something a caller can act on. This chapter pins down the second, how each language *shares work*, and it does it the diff --git a/_docs/07-fds-raii-and-the-vfs.md b/_docs/07-fds-raii-and-the-vfs.md index b6ce37e..adbb2d4 100644 --- a/_docs/07-fds-raii-and-the-vfs.md +++ b/_docs/07-fds-raii-and-the-vfs.md @@ -6,6 +6,45 @@ description: "The three-layer model behind every file descriptor (fd table, open duration: 45 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Explain the three-layer fd model: the per-process fd table (home of the + close-on-exec bit), the system-wide open file description (flags and + offset), and the VFS inode (type, size, mtime) +- Justify `O_CLOEXEC` at open time as the only race-free way to stop an fd + leaking into a `fork`/`exec`'d child, versus a later `fcntl` call +- Walk a directory tree safely with `openat`/`fstatat` anchored on a + directory fd, avoiding the re-resolution races a path-based walk invites +- Build and compare owning fd wrappers across languages: a move-only C++ + `Fd` class with an explicit `release()`, Go's `os.File` with a deferred + `Close`, and Rust's `OwnedFd`/`BorrowedFd` +- Implement `fwatch` v0, a deterministic snapshot/diff tool, identically in + C++23, Go, and Rust, including its `+`/`-`/`~` diff format +- Verify fd-table, open-file-description, and inode claims independently + using `/proc//fd`, `/proc//fdinfo`, `strace`, `ls -li`, and `lsof` + +## Why this matters + +Most bugs involving file descriptors come from treating "the fd" as a single +thing when it is really three: a slot in your process, a shared kernel +object, and the file itself. Conflating them is how a `dup`'d descriptor +surprises you by sharing an offset, how a privileged fd leaks across `exec`, +or how a deleted file keeps its storage alive because something still holds +it open. This chapter gives you the vocabulary to say precisely which layer +a bug lives on, and the tools (`/proc//fd`, `fdinfo`, `strace`) to prove +it rather than guess. + +That precision also opens the book's second long-running artifact. `fwatch` +starts here as a plain snapshot/diff tool with no events at all, by design: +before it can react to filesystem changes in later chapters (inotify, epoll, +io_uring, and eventually a Landlock sandbox), it needs an owning fd +abstraction and a race-free directory walk that will not need to be rebuilt +once real concurrency and real kernel notifications enter the picture. The +RAII-style wrapper and the anchored `openat`/`fstatat` walk built here are +required by every version of `fwatch` that follows. + This chapter starts the second of the book's long-running artifacts: `fwatch`, the file watcher that will grow inotify, epoll, io_uring, and finally a Landlock sandbox over the next several chapters. Version 0 has no events at diff --git a/_docs/08-page-cache-and-durability.md b/_docs/08-page-cache-and-durability.md index 67e9918..6f9ecde 100644 --- a/_docs/08-page-cache-and-durability.md +++ b/_docs/08-page-cache-and-durability.md @@ -6,6 +6,48 @@ description: "What write(2) actually promises, and what durability costs: dirty duration: 45 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Explain what a successful buffered `write(2)` actually guarantees: a + dirty page in the page cache, visible to readers, nothing about the device +- Distinguish `fsync`, `fdatasync`, and `sync_file_range` by exactly what + each flushes, and explain why `sync_file_range` is not a durability + primitive despite touching the device +- Use `O_DIRECT` correctly: respect its block-alignment requirements and + treat an `EINVAL` at `open(2)` as a documented, filesystem-dependent + outcome rather than a generic error +- Implement the tmpfile → `fsync` → `rename` → `fsync(dir)` pattern for + crash-safe file replacement +- Build and run `iobench`, a tri-language write benchmark with buffered, + fsync-every, and direct modes, and read its `MiB/s` and `fsync_ms` output + as two different claims +- Measure writeback and durability cost independently of the benchmark's own + numbers, using `/proc/meminfo`'s `Dirty:` counter, `/proc//io`, and + `strace -c` + +## Why this matters + +"The write succeeded" is one of the most quietly overloaded sentences in +systems programming. A successful `write(2)` return code says only that your +bytes reached kernel memory; whether they survive a power loss a second later +depends on a sync call you have to ask for explicitly, and getting that +choice wrong is how real databases lose real data after passing every test +that never pulled the power cord. This chapter turns that gap from folklore +into a measured quantity: 21 ms to "write" 64 MiB, 138 ms to write it +durably, watched directly in `/proc/meminfo` and `/proc//io` rather than +taken on faith. + +It also closes the loop Chapter 7 opened. That chapter gave you a precise +model of what a file descriptor is; this one gives you a precise model of +what happens to the bytes you push through it, which is the other half of +"I wrote a file" that most introductions skip. The durability tools +introduced here, `fsync`/`fdatasync` semantics, the atomic-rename pattern, +`O_DIRECT`'s tradeoffs, are exactly the primitives Part 2's remaining +chapters (event loops, and eventually `fwatch`'s own durability story) build +on next. + Chapter 7 ended with a file descriptor safely wrapped in RAII and a byte successfully written through the VFS. This chapter is about the uncomfortable question that follows: written *where*? A successful `write(2)` puts your diff --git a/_docs/09-event-loops-epoll-inotify.md b/_docs/09-event-loops-epoll-inotify.md index dd8ed8d..6716bf1 100644 --- a/_docs/09-event-loops-epoll-inotify.md +++ b/_docs/09-event-loops-epoll-inotify.md @@ -6,6 +6,23 @@ description: "The readiness model: nonblocking fds multiplexed by one epoll loop duration: 50 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Register nonblocking file descriptors with `epoll_create1`/`epoll_ctl` and explain why a kernel-side interest list lets `epoll_wait` scale where `poll`/`select` cannot. +- Distinguish level-triggered from edge-triggered (`EPOLLET`) readiness, state the drain-to-`EAGAIN` discipline each requires, and identify when `EPOLLONESHOT` applies. +- Build an inotify watch with `inotify_init1`/`inotify_add_watch`, parse packed `inotify_event` records, and recognize `IN_Q_OVERFLOW` as the signal to fall back to a snapshot/diff rescan. +- Fold `timerfd` and `signalfd` into one epoll loop as ordinary readable fds, including the one-shot re-arm pattern and the block-then-read signalfd prerequisite. +- Compare the explicit C++/Rust single-threaded epoll loop against Go's goroutine-plus-channel `select` design built on the runtime's netpoller. +- Inspect a running watcher's interest list and fd table through `/proc//fdinfo` and read `strace` output for `epoll_wait`, inotify, timerfd, and signalfd calls. + +## Why this matters + +Every scalable network service, file watcher, and terminal multiplexer rests on the same decision: decouple waiting from the work the wait is for. The readiness model is how a single thread tracks thousands of file descriptors without a thread per source, and it is the direct ancestor of the event loops behind editors, sync clients, and the servers Part 6 builds. + +This chapter also establishes the comparison that drives the rest of this part: an explicit, visible-in-source epoll loop versus a runtime that hides the same machinery behind channels and goroutines. Chapter 10 builds on this foundation by introducing the completion model, io_uring, as the second half of the readiness-versus-completion pair. + Chapter 8 closed on a question durability cannot answer: "which of ten thousand fds is ready?" This chapter answers it with the **readiness model**: the kernel will tell you *when an fd has something for you*, and a program diff --git a/_docs/10-io-uring.md b/_docs/10-io-uring.md index 75d34f3..8ef757d 100644 --- a/_docs/10-io-uring.md +++ b/_docs/10-io-uring.md @@ -6,6 +6,22 @@ description: "fwatch grows a batched sync engine and meets the second of the two duration: 60 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Explain the difference between the readiness model (epoll asks "may I?") and the completion model (io_uring reports "it is done"). +- Construct an io_uring instance with `io_uring_setup`, map the SQ ring, CQ ring, and SQE array via the canonical `mmap`s, and submit batched operations with `io_uring_enter`. +- Write and link submission queue entries (`IORING_OP_READ`/`IORING_OP_WRITE`) with `IOSQE_IO_LINK` to express intra-pair ordering without a userspace round trip. +- Measure when completion-based I/O helps and when it costs more than a synchronous read/write loop, using `strace -c` syscall counts and wall-clock timings. +- Account for io_uring's security exposure, including why container seccomp profiles deny it by default, and explain Go's deliberate refusal to expose a uring engine. + +## Why this matters + +Readiness and completion are the two mental models underlying every I/O-heavy system on Linux, and knowing which one a workload needs is a design decision with measurable consequences: this chapter's own numbers show a completion-based engine running roughly ten times slower than a synchronous loop on warm, small files, while the same engine would win decisively against deep queues on real storage or busy sockets. Fewer syscalls is not automatically faster, and this chapter proves it rather than asserting it. + +The chapter also surfaces a security tradeoff worth carrying forward: io_uring's opcode-dispatched interface can bypass syscall filters written against `read`/`write`/`connect`, which is why container runtimes deny it by default. That tension between a powerful kernel interface and a shrinking attack surface returns when Part 7 moves these programs into containers. + Chapter 9 ended with `fwatch` blocked in an event loop, and with a promise: epoll would not be the last word on Linux I/O. Epoll answers one question: *which of these fds may I touch without blocking?* Then it leaves every diff --git a/_docs/11-process-lifecycle.md b/_docs/11-process-lifecycle.md index 2bc6b7e..6beae7a 100644 --- a/_docs/11-process-lifecycle.md +++ b/_docs/11-process-lifecycle.md @@ -6,6 +6,23 @@ description: "fork, exec, and wait walked end to end as pmon v0 spawns and reaps duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Trace a process from `fork(2)` through `execve(2)` to `wait4(2)`/`waitid(2)`, and explain what copy-on-write actually copies. +- Build a `O_CLOEXEC` self-pipe that lets a parent distinguish "exec failed" from "the program ran and exited." +- Observe a zombie process live in `ps` and `/proc//stat`, and explain why a parent must reap it before its pid can be reused. +- Distinguish exit codes from termination signals (`WIFEXITED`/`WEXITSTATUS` versus `WIFSIGNALED`/`WTERMSIG`) and apply the shell's `128+n` convention. +- Compare the `clone`/`clone3` flavors that glibc's `fork`, Go's `os/exec`, and Rust's `Command` each issue, verified with `strace -f`. +- Explain orphan reparenting to a subreaper (`PR_SET_CHILD_SUBREAPER`) and why an orphaned child is never left permanently abandoned. + +## Why this matters + +Fork, exec, and wait are the three syscalls underneath every process a Linux system runs, from a shell pipeline to a container runtime launching an entry point. Getting the gap between fork and exec right, and reaping promptly, is the difference between a supervisor that works and one that silently leaks zombies or orphans until `pid_max` runs out. + +This chapter opens `pmon`, the process monitor that carries through the rest of this part: signal handling in Chapter 12, pidfd-based supervision in Chapter 13, and privilege dropping in Chapter 14 all build directly on the spawn-and-reap discipline established here. + This chapter opens the book's third part with a new long-running artifact: `pmon`, a process monitor that will grow signal forwarding, pidfd-based supervision, and privilege dropping over the next three chapters. Version 0 diff --git a/_docs/12-signals-and-safe-handling.md b/_docs/12-signals-and-safe-handling.md index 4890013..412ec34 100644 --- a/_docs/12-signals-and-safe-handling.md +++ b/_docs/12-signals-and-safe-handling.md @@ -6,6 +6,23 @@ description: "Dispositions, per-thread masks, and the pending set, then three ra duration: 50 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Describe the three kernel-level signal states: per-process disposition, per-thread mask, and the coalescing pending set. +- Explain async-signal-safety and why most library and runtime code (`malloc`, `printf`, locks) must never run inside a handler. +- Implement the self-pipe trick and `signalfd(2)`, and compare how pmon's C++, Go, and Rust builds each turn a signal into ordinary data. +- Build a restarting supervisor (pmon v1) around a single blocking wait whose timeout doubles as debounce and backoff. +- Reset a child's signal mask between fork and exec so an inherited blocked SIGTERM cannot make the child unstoppable. +- Read `/proc//status` `Sig*` bitmasks and interpret `strace` evidence of `rt_sigaction` and `tgkill` to compare the three languages' signal-handling internals. + +## Why this matters + +A signal handler runs at an arbitrary instant inside arbitrary code, which makes it one of the easiest places in a Linux program to introduce a deadlock or a race that only shows up under load. The discipline this chapter teaches, converting signals into data and handling them in a plain loop, is what separates a supervisor that restarts cleanly from one that occasionally hangs on SIGTERM. + +This chapter turns pmon from a one-shot runner (Chapter 11) into an actual supervisor: restart on crash, clean shutdown, and configuration reload all ride the same signal-to-data conversion. Chapter 13 closes the one race this design still carries, a recycled pid receiving a forwarded signal meant for a dead child. + Chapter 11 taught `pmon` to run one child and report how it died. Version 1 makes it a real supervisor: restart a crashing child with a doubling backoff, shut down cleanly on SIGTERM or SIGINT, and replace the child on SIGHUP. That diff --git a/_docs/13-pidfd-modern-process-mgmt.md b/_docs/13-pidfd-modern-process-mgmt.md index 480731b..8b9c68e 100644 --- a/_docs/13-pidfd-modern-process-mgmt.md +++ b/_docs/13-pidfd-modern-process-mgmt.md @@ -6,6 +6,23 @@ description: "The pid-reuse race constructed step by step, then closed for good: duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Construct the five-step pid-reuse race and explain why an integer pid carries no identity guarantee once it is reaped. +- Open a pidfd with `pidfd_open(2)`, or obtain one atomically at spawn time with `clone3`'s `CLONE_PIDFD` flag. +- Poll a pidfd for exit readiness, reap it with `waitid(P_PIDFD, ...)`, and signal it with `pidfd_send_signal(2)`. +- Explain why `pidfd_send_signal` returns `ESRCH` instead of silently signaling a recycled pid. +- Compare how Go's runtime (which adopts `CLONE_PIDFD` for every child) and Rust's stable standard library (which keeps pidfd support behind the unstable `linux_pidfd` feature) handle process identity today, verified with `strace`. +- Inspect a pidfd's target process through the `Pid:` line of `/proc//fdinfo/`. + +## Why this matters + +A supervisor that names its children by pid carries a race it usually never notices: any reap it does not control, a runtime, a library, a signal handler, can free that pid for reuse before the supervisor acts on it again. On a busy build server or a container with a small `pid_max`, that window is not theoretical. + +Pidfds close the race by making a process a referenceable object instead of a reused number, and they compose directly with the event-loop model Chapter 9 built: a process exit becomes ordinary fd readiness. This chapter makes pidfds pmon's default engine, completing the supervision story Chapters 11 and 12 started before Chapter 14 turns to privilege. + `pmon` v1 ended the last chapter supervising a crashing child correctly: no async handlers, SIGCHLD arriving as data through a signalfd, a `WNOHANG` reap classifying every death. But it still carries a lie it inherited from 1970s diff --git a/_docs/14-identity-and-privilege.md b/_docs/14-identity-and-privilege.md index 9acc627..496de85 100644 --- a/_docs/14-identity-and-privilege.md +++ b/_docs/14-identity-and-privilege.md @@ -6,6 +6,22 @@ description: "Real, effective, and saved uids as one state machine that setresui duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Explain the three-uid state machine (real, effective, saved) and why `setresuid(2)` replaces the error-prone `setuid`/`seteuid`/`setreuid` sequence. +- Apply the umask to explain how file-creation modes are filtered, and why a staged binary needs a world-readable path once a process drops privilege. +- Distinguish the five process capability sets (permitted, effective, inheritable, bounding, ambient) from the three file capability sets, and state why the ambient set exists. +- Execute the ordering-sensitive privilege-drop sequence (`PR_SET_KEEPCAPS`, `setgroups`/`setresgid`/`setresuid`, `capset`, `PR_CAP_AMBIENT_RAISE`, `execve`) and explain why reordering any step breaks it. +- Verify a privilege drop with a positive/negative control pair, using `getpcaps`, the `Cap*` lines of `/proc//status`, `capsh --decode`, and a `bpftrace` probe on `cap_capable`. + +## Why this matters + +Most daemons start as root because they need a privileged resource (a port below 1024, a protected path) and must stop being root the moment they no longer do. Getting that transition wrong in either direction, leaving root behind by accident or stripping the one capability the service needed, is a recurring real-world vulnerability class, not a theoretical one. + +This chapter gives pmon the final piece of its supervision story: dropping privilege correctly while carrying exactly one capability across an identity change and an `execve`, proven with a paired positive and negative run rather than a single claim. It closes out Part 3's process-and-identity arc before Part 4 turns to virtual memory. + `pmon` can spawn a child, shepherd it through signals, and supervise it race-free through a pidfd. Version 3 faces the question every real supervisor meets next: it starts as root (it has to, to become another user) and diff --git a/_docs/15-virtual-memory.md b/_docs/15-virtual-memory.md index 1bf6482..029b4ca 100644 --- a/_docs/15-virtual-memory.md +++ b/_docs/15-virtual-memory.md @@ -6,6 +6,41 @@ description: "The address space read line by line from /proc/self/maps, brk vers duration: 45 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Read a process's address space line by line from `/proc/self/maps` and + identify VMAs, the `[heap]`, the mmap region, `[stack]`, and the + `[vvar]`/`[vdso]` pages +- Trace an allocation with `strace -e trace=mmap,brk,madvise` to tell + whether `malloc`, Rust's default allocator, or Go's runtime services it + with `brk` or `mmap` +- Measure minor and major page faults with `getrusage(2)` deltas and explain + fault-around batching for cached file reads +- Distinguish VSZ, RSS, and PSS, and read `/proc//smaps_rollup` to find + the number that sums correctly across a fleet +- Read overcommit policy (`vm.overcommit_memory`, `CommitLimit`, + `Committed_AS`) and the OOM killer's `oom_score_adj` without writing to + `/proc/sys/vm` +- Apply `madvise(2)` hints (`MADV_WILLNEED`, `MADV_DONTNEED`, `MADV_FREE`, + `MADV_HUGEPAGE`) to a mapped range +- Build and run `memmap` in C++, Go, and Rust to inspect a process's own + address space under five allocation modes + +## Why this matters + +A segfault, a stalled service, or a container killed without warning +usually traces back to a wrong assumption about memory: that an allocation +reserves RAM, that `free` returns pages to the kernel, or that RSS measures +what a process costs a host. None of those assumptions hold, and the gap +between what a program asked for and what the kernel delivered is exactly +where capacity planning goes wrong under load. + +This chapter builds the vocabulary, VMAs, faults, VSZ/RSS/PSS, overcommit, +that every later memory and IPC chapter assumes, including `shmkv`, the +key-value store the rest of this part builds directly on mapped pages. + Part 3 closed with `pmon` dropping privileges in chapter 14; this part turns inward, to the memory every one of those processes lives in. The one new idea is that **an address is a promise, not a page**: when `malloc`, `make`, or diff --git a/_docs/16-mmap-and-shared-mappings.md b/_docs/16-mmap-and-shared-mappings.md index 22745e9..3bd3096 100644 --- a/_docs/16-mmap-and-shared-mappings.md +++ b/_docs/16-mmap-and-shared-mappings.md @@ -6,6 +6,39 @@ description: "File-backed vs anonymous and MAP_SHARED vs MAP_PRIVATE copy-on-wri duration: 50 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Distinguish file-backed from anonymous mappings and `MAP_SHARED` from + `MAP_PRIVATE`, and demonstrate copy-on-write with a live mmap scribble +- Use `msync(MS_SYNC)` to force dirty page-cache pages to disk, and explain + why visibility between `MAP_SHARED` mappers needs no syscall at all +- Perform the `ftruncate`-then-`mmap` lifecycle correctly and explain why + touching a mapped page past EOF raises `SIGBUS` +- Create and seal an anonymous `memfd_create` file with `F_SEAL_SHRINK`, + `F_SEAL_GROW`, and `F_SEAL_FUTURE_WRITE` +- Define a byte-exact binary format (a `u32` little-endian field, NUL-padded + slots) that C++, Go, and Rust read and write interchangeably, verified + with `cmp` +- Build `shmkv` v0 and read `strace`, `/proc//maps` (the `rw-s` + permission bits), and `cmp` output to prove one physical page is shared + across processes + +## Why this matters + +Treating a mapped file like ordinary memory is a reasonable mental model +right up until two processes write to it at once, or a peer truncates the +file out from under your mapping. `MAP_SHARED` collapses memory, the page +cache, and the file on disk into one set of physical pages, which is fast +and risky in exactly the ways a systems programmer needs to see directly +rather than take on faith. + +This chapter gives you the mapping quadrant (file-backed versus anonymous, +shared versus private) and the discipline, `msync` for durability, +`ftruncate` before `mmap`, sealed memfds for untrusted peers, that `shmkv` +v0 needs to be a safe, cross-language, byte-exact store. + The previous chapter took the address space apart (VMAs, page tables, faults) without ever asking where a mapping's bytes come from. This chapter answers that with the book's next artifact: `shmkv`, a key/value store whose entire diff --git a/_docs/17-allocators-and-gc-runtimes.md b/_docs/17-allocators-and-gc-runtimes.md index 110b89c..ffc2850 100644 --- a/_docs/17-allocators-and-gc-runtimes.md +++ b/_docs/17-allocators-and-gc-runtimes.md @@ -6,6 +6,41 @@ description: "What actually happens between your allocation and the kernel's pag duration: 50 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Explain why `free(3)` returns chunks to glibc's arena and tcache rather + than pages to the kernel, and read `mallinfo2` output to see fragmentation + directly +- Prove with `nm -D` and `ltrace` that Rust's default global allocator is + glibc malloc, the same allocator C++ uses +- Build arena allocators with `std::pmr::monotonic_buffer_resource` (C++), + bumpalo's `Bump` (Rust), and `sync.Pool` (Go), and state what each one + changes, and does not change, about peak RSS +- Tune Go's GC pacer live with `GOGC` and `GOMEMLIMIT` and read `gc_cycles` + against `peak_rss` to show that memory use is a policy outcome +- Measure peak RSS and wall time with `getrusage(RUSAGE_SELF)` and + cross-check the result against `valgrind --tool=massif`, + `/usr/bin/time -v`, and `go tool pprof` +- Run `allocbench` across C++, Go, and Rust to compare per-object free, + arena reclamation, and traced garbage collection on one workload + +## Why this matters + +Every allocation-heavy service eventually asks the same question: where did +the memory go? Answering it requires knowing what your language's allocator +actually does with a `free`, a dropped `Box`, or a Go object the collector +has not traced yet, because a dashboard only reports the outcome, not the +mechanism, and "my program freed everything but RSS didn't drop" is a +question that costs an on-call engineer real time if the mechanism is not +already familiar. + +This chapter opens the allocator and the GC pacer directly: glibc's arena +and tcache, what `pmr`/`bumpalo` arenas really buy you, and the live +`GOGC`/`GOMEMLIMIT` knobs that turn Go's memory footprint into a negotiated +policy rather than a mystery. + The last two chapters dealt in wholesale memory: pages from the kernel, mapped and shared with `mmap`. This chapter is retail. Between your `std::string`, your `strconv.Itoa`, your `format!` and those pages sits an allocator: glibc diff --git a/_docs/18-pipes-fifos-splice.md b/_docs/18-pipes-fifos-splice.md index 08ed21d..e0455b7 100644 --- a/_docs/18-pipes-fifos-splice.md +++ b/_docs/18-pipes-fifos-splice.md @@ -6,6 +6,39 @@ description: "pmon v4 turns the supervisor into a log pipeline: the pipe as a ri duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Read a pipe's capacity live with `fcntl(F_GETPIPE_SZ/F_SETPIPE_SZ)` and + explain why a full or empty ring applies back-pressure instead of losing + data +- State what `PIPE_BUF` atomicity guarantees, and does not guarantee, about + message framing, large writes, and reads +- Create a FIFO with `mkfifo` and reason through the `open(2)` blocking + table, including the `O_WRONLY|O_NONBLOCK`/`ENXIO` retry pattern +- Handle `SIGPIPE`/`EPIPE` as the writer's contract for a long-lived relay + that must survive a detached reader +- Move bytes between a file and a pipe with `splice(2)` without a userspace + copy, and explain why the file offset advances only for delivered bytes +- Extend `pmon` (v4) to capture a child's stdout and stderr into a framed + log and republish it through a FIFO, verified with `strace`, `/proc`, and + a `dd` throughput run + +## Why this matters + +Every shell pipeline and every process supervisor depends on pipes behaving +correctly under load: back-pressure instead of data loss, a predictable end +when a reader vanishes, and a parent that does not accidentally keep its +own pipe open forever. Getting any of those wrong produces a supervisor +that hangs, a log that silently stops mid-stream, or a pipeline a shell +cannot terminate cleanly. + +This chapter treats the pipe as what it is, a bounded ring of kernel pages, +and builds `pmon` v4's log pipeline on that model: framed capture that +never relies on `PIPE_BUF` for message boundaries, and a `splice`-based +FIFO relay that moves bytes without a userspace copy. + This part of the book is about processes talking to each other, and it opens with the primitive every shell has used since 1973. `pmon` ended the process part supervising a crashing child with two byte-identical engines, pidfd and diff --git a/_docs/19-unix-sockets-fd-passing.md b/_docs/19-unix-sockets-fd-passing.md index 9388583..ba37037 100644 --- a/_docs/19-unix-sockets-fd-passing.md +++ b/_docs/19-unix-sockets-fd-passing.md @@ -6,6 +6,39 @@ description: "pmon grows a control plane on a UNIX stream socket: filesystem vs duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Choose among `SOCK_STREAM`, `SOCK_DGRAM`, and `SOCK_SEQPACKET` for a UNIX + domain socket based on message shape and authentication needs +- Bind a filesystem-path `AF_UNIX` socket versus an abstract (NUL-prefixed) + one, and read both back with `ss -x` +- Authenticate a peer with `SO_PEERCRED` instead of trusting a self-reported + claim +- Send and receive an open file descriptor across a socket with + `sendmsg`/`recvmsg` and `SCM_RIGHTS`, using the `CMSG_SPACE`/`CMSG_LEN`/ + `CMSG_DATA` macros correctly +- Explain why `SCM_RIGHTS` transfers a reference to an open file + description, with its shared offset and flags, rather than a number, and + confirm it with `/proc` fdinfo on both processes at once +- Extend `pmon` (v5) with a control-plane socket (`status`/`stop`/`logfd`) + that a stranger process can find and use + +## Why this matters + +A process that wants to be found and trusted by a stranger needs more than +a shared pipe: it needs an address, a way to check who is really calling, +and sometimes a way to hand over an already-open resource instead of +re-deriving it. UNIX domain sockets are the one primitive on the system +built for exactly that, and `SCM_RIGHTS` gets used constantly without +anyone being clear on what actually crosses the wire. + +This chapter builds `pmon` v5's control plane on `SO_PEERCRED` for +kernel-vouched identity and `SCM_RIGHTS` for descriptor passing, then +proves with `/proc` on both processes at once that an open file +description, not a number, is what moved. + Chapter 18 gave `pmon` a log pipeline built from pipes and FIFOs (fast, kernel-buffered, and strictly byte-shaped): a pipe moves data, and only between processes that inherited its ends. Version 5 needs something pipes diff --git a/_docs/20-shared-memory-coordination.md b/_docs/20-shared-memory-coordination.md index 922ae44..2330f69 100644 --- a/_docs/20-shared-memory-coordination.md +++ b/_docs/20-shared-memory-coordination.md @@ -6,6 +6,42 @@ description: "The coordination ladder from a 1 ms poll loop through POSIX messag duration: 60 minutes --- +## Learning objectives + +By the end of this chapter you can: + +- Explain why `futex(2)` is the kernel's one real coordination primitive and + how `FUTEX_WAIT`/`FUTEX_WAKE` close the check-then-sleep race that plain + polling cannot +- Distinguish a private futex (keyed by virtual address) from a shared + futex (keyed by inode and offset) and choose correctly for unrelated + processes +- Implement a seqlock for read-mostly shared data, including why the writer + bumps the sequence word twice per update +- Bind process-shared atomics onto mapped memory with `std::atomic_ref`, + `AtomicU32::from_ptr`, and Go's `sync/atomic`, and state each language's + safety contract +- Measure and rank a coordination ladder (poll loop, POSIX message queues, + futex, pipe, eventfd, UNIX socket) by p50/p99 latency +- Extend `shmkv` (v1) with a watcher that sleeps on a futex instead of + polling, verified with `strace` and `/proc`'s `voluntary_ctxt_switches` + +## Why this matters + +Polling a shared value in a loop is the first instinct for interprocess +coordination, and it is also the most common unnecessary cost in a +latency-sensitive system: measured on this host, a 1 ms poll loop runs +three orders of magnitude slower than the alternative. Every mutex, +condition variable, and semaphore in a C++, Go, or Rust runtime is +userspace protocol built on one kernel call, and understanding that call +directly is what lets you choose correctly between polling, message +queues, and shared-memory wakeups. + +This chapter builds `shmkv` v1 on `futex(2)` and a seqlock: writers that +never block, readers that retry on a torn copy instead of taking a lock, +and a measured latency ladder that ranks every coordination channel this +book has built so far. + Chapter 16 (mmap and shared mappings) left `shmkv` in an awkward state: two processes share a page of key-value slots, but a reader who wants to *notice* a change has no option except reading the page again, and again, and again. diff --git a/_docs/21-tcp-sockets.md b/_docs/21-tcp-sockets.md index 18064e3..3a9ad32 100644 --- a/_docs/21-tcp-sockets.md +++ b/_docs/21-tcp-sockets.md @@ -6,6 +6,42 @@ description: "chatterd v0 opens Part 6 with a thread-per-connection TCP chat roo duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Build a listening TCP socket with the four ordered calls + `socket`/`bind`/`listen`/`accept4`, filling in `sockaddr_in` by hand with + `htons` and `inet_pton` +- Explain why `SO_REUSEADDR` is required to rebind a port still held in + `TIME_WAIT` +- Design a length-prefixed wire frame (magic, version, type, big-endian + `u16` length) because TCP delivers bytes with no message boundaries +- Implement a read-exactly-n loop (`read_full`/`io.ReadFull`/`read_exact`) + that never mistakes a short read for a complete message +- Build `chatterd` v0, a thread-per-connection TCP chat room, in C++, Go, + and Rust with an RAII/move-only socket that owns every fd +- Identify the two failure modes of thread-per-connection service + (per-connection memory cost, lock-held-across-I/O blocking) that chapter + 22 addresses +- Verify a live server with `ss -tlnp`/`-tnp` and read the wire bytes at the + `read`/`write` syscall boundary with `strace` + +## Why this matters + +A TCP server looks like a pipe with an address until the first production +incident proves otherwise: a restart that cannot rebind its own port, a +message that arrives split across two `read` calls, or a slow client that +freezes every other connection behind a shared lock. Each of those is a +direct consequence of what TCP actually promises, ordered bytes with no +message boundaries, not what a quick read-write test suggested. + +This chapter builds `chatterd` v0 from the four raw syscalls that make a +listening socket, establishes the length-prefixed frame every later +networking chapter in this book reuses, and names the two failure modes, +per-connection cost and a lock held across I/O, that make +thread-per-connection the wrong model once scale shows up. + Part 5 spent nine chapters moving bytes between processes that already knew each other: pipes an ancestor set up, UNIX sockets a stranger found by a path on the local machine. `chatterd` opens Part 6 by dropping the last diff --git a/_docs/22-scaling-the-server.md b/_docs/22-scaling-the-server.md index 51665c5..c686e20 100644 --- a/_docs/22-scaling-the-server.md +++ b/_docs/22-scaling-the-server.md @@ -6,6 +6,43 @@ description: "chatterd v1 holds many connections from one thread: nonblocking ac duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Build a per-connection state machine (`inbuf`/`outbuf`) that survives across + `epoll_wait` wakeups, treating `EAGAIN` as a signal to wait rather than an + error. +- Implement backpressure with an outbound queue and `EPOLLOUT`, arming and + clearing write-interest via `epoll_ctl` as the queue fills and drains. +- Drain `accept4` to `EAGAIN` on a nonblocking listener so one `epoll_wait` + wakeup can accept a whole burst of pending connections. +- Compare Go's goroutine-per-connection model to a hand-rolled epoll loop, and + locate the runtime's hidden `epoll_pwait` calls with `strace`. +- Guard a send to a dead peer with `MSG_NOSIGNAL`, and defer reaping dead + connections to the end of an event batch instead of mid-iteration. +- Read `/proc//status` thread counts and `ss -tp` to confirm how many OS + threads actually back a given connection-handling design. +- Measure the C10K cost directly with `strace -c`, comparing `clone3` counts + between the epoll and threaded engines under load. + +## Why this matters + +A server that spends one OS thread per connection runs out of room long +before it runs out of clients: ten thousand idle sockets mean ten thousand +stacks and a scheduler doing more context-switching than work. The readiness +model this chapter builds, nonblocking sockets plus one `epoll_wait`, is how +a single thread holds that many connections without any of them blocking +another, and the backpressure discipline (outbound queue, `EPOLLOUT`) is what +keeps a slow reader from stalling the rest. + +It also sets up the comparison every polyglot team eventually has to make: +Go's goroutine-per-connection code looks nothing like this epoll loop, yet +underneath it the runtime is running the identical mechanism. Knowing that +lets you read a Go service's concurrency as threads parked on an epoll +instance, not as magic, and lets you choose the right language for a given +connection-handling budget instead of guessing. + Chapter 21 gave `chatterd` a length-prefixed frame protocol and the obvious server: one OS thread per connection, each blocked in `recv`. That design is correct and it is easy to reason about, right up to the point a decade of diff --git a/_docs/23-udp-and-peer-discovery.md b/_docs/23-udp-and-peer-discovery.md index 470e1f2..84bf8f4 100644 --- a/_docs/23-udp-and-peer-discovery.md +++ b/_docs/23-udp-and-peer-discovery.md @@ -6,6 +6,42 @@ description: "chatterd learns to find peers it was never told about: UDP multica duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Open and configure a UDP multicast socket with `IP_ADD_MEMBERSHIP`, + `IP_MULTICAST_IF`, `IP_MULTICAST_TTL`, and `IP_MULTICAST_LOOP` to join and + send to an administratively-scoped group. +- Explain why connectionless, message-oriented, best-effort UDP suits + discovery while TCP suits the conversation that follows, and why a beacon + must stay inside one unfragmented MTU. +- Dedupe repeated beacons by peer name and bridge a first-seen discovery into + a TCP dial carrying the existing canonical chat frame. +- Coordinate shutdown of a blocking UDP receive loop with a socket read + timeout plus a stop signal (`stop_token`, cancelled `context`, or + `AtomicBool`, depending on language). +- Verify multicast group membership and socket state with `ip maddr` and + `ss -uap`, and capture beacons on the wire with `tcpdump -X`. +- Run a two-host discovery exchange across a libvirt bridge and explain why a + TTL-1 multicast beacon stays link-local without any NAT workaround. + +## Why this matters + +A peer that has to be told every address in advance does not scale past a +demo: real services find each other. UDP multicast gives one `sendto` a way +to reach every listener in a group without the sender knowing who or how many +they are, and the dedup-then-dial pattern this chapter builds, a +connectionless announcement bridged into a connection-oriented conversation, +is the shape behind most service-discovery protocols you will meet later, +whatever their wire format. + +The chapter also draws a hard line worth keeping: discovery tolerates loss +and reorder, so it belongs on UDP, while the conversation that follows needs +ordering and reliability, so it belongs on TCP. Picking the wrong protocol for +either half either drops messages you needed or pays connection overhead you +did not. + Chapters 21 and 22 built `chatterd` on TCP: the canonical chatterd chat frame (magic `"CH"`, version, type, a big-endian length, and a payload) carried between two endpoints you already knew, since you typed the peer's diff --git a/_docs/24-time-timers-deadlines.md b/_docs/24-time-timers-deadlines.md index 2d3ad29..d849600 100644 --- a/_docs/24-time-timers-deadlines.md +++ b/_docs/24-time-timers-deadlines.md @@ -6,6 +6,42 @@ description: "chatterd v3 learns to survive a peer that vanishes: the POSIX cloc duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Distinguish `CLOCK_REALTIME`, `CLOCK_MONOTONIC`, `CLOCK_BOOTTIME`, and + `CLOCK_PROCESS_CPUTIME_ID`, and explain why any duration or deadline must be + measured on a monotonic clock. +- Explain why `clock_gettime` on the common clocks is served from the vDSO + with no syscall, and confirm it with `strace` showing zero `clock_gettime` + lines. +- Create and arm a `timerfd(CLOCK_MONOTONIC)` as a periodic heartbeat that + drops into the same `poll`/`epoll` loop as a socket and a `signalfd`. +- Implement a liveness deadline that resets on every received frame and + declares a peer dead only when the monotonic deadline elapses, not on a raw + socket error. +- Implement jittered exponential backoff for reconnection and explain why the + jitter term prevents a synchronized thundering herd on recovery. +- Trace a live heartbeat/timeout/backoff session with `strace` to confirm the + sequence of `timerfd_settime`, `sendto`, and the shrinking `poll` timeout. + +## Why this matters + +A peer that vanishes without closing its socket is the common case in +production, not the exception: a pulled cable or a killed process leaves the +other end blocked in `read` forever unless something measures silence and +acts on it. Getting that measurement right depends entirely on which clock +you read, and the wrong choice (a wall-clock deadline that NTP can step +backward) turns a heartbeat into a liability instead of a safety net. + +The timerfd pattern here also pays off beyond chat: any program that needs a +periodic tick inside an existing `poll`/`epoll` loop, without a separate +thread or a signal handler, uses this exact mechanism. Pairing it with +jittered exponential backoff is the standard discipline for reconnecting to a +recovering service without a fleet of clients hammering it back down the +moment it comes up. + Chapters 21–23 built `chatterd` into a peer-to-peer chat daemon that links two hosts over TCP and passes framed messages. It has one fatal naïveté: it assumes the peer stays. Pull the cable, kill the process, or let a switch drop the diff --git a/_docs/28-gdb-and-remote-debugging.md b/_docs/28-gdb-and-remote-debugging.md index 2a01f76..717ef24 100644 --- a/_docs/28-gdb-and-remote-debugging.md +++ b/_docs/28-gdb-and-remote-debugging.md @@ -6,6 +6,41 @@ description: "app crashes by design, three ways: a real SIGSEGV and a systemd-co duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Extract a `systemd-coredump` core with `coredumpctl` and open it in `gdb` + for a full DWARF-driven backtrace, instead of trusting a lightweight + summary trace that can silently omit frames. +- Read pretty-printed values in `gdb` (`std::string_view`, struct fields) and + layout-aware Rust values in `rust-gdb` (`Option`/enum variants by name) + rather than raw memory offsets. +- Explain how C++, Go, and Rust each handle the same fault differently: a + hardware `SIGSEGV` and a core file, a runtime-converted panic, and an + unwind-only `Option::unwrap()` panic with no signal at all. +- Recognize `gdb`'s blind spot on Go (it models OS threads, not goroutines) + and identify when a goroutine-aware tool like `delve` is needed instead. +- Set up remote debugging with `gdbserver` and `gdb`'s `target remote`, + keeping every symbol table and pretty-printer on the host side of the link. +- Cross-check a `gdb` backtrace's addresses against the DWARF `.debug_line` + table independently, using `readelf` and `objdump`. + +## Why this matters + +A `println` tells you what the code did up to the point you placed it, and +only if you remembered to put one there before the bug happened. A debugger +reads a crash, or a running process, after the fact, without changing a line +of source, which is the only approach that works once a bug only reproduces +occasionally or only in an environment you cannot freely recompile in. + +The three languages also make the same bug look completely different at the +tooling layer: a core file is the right artifact for C++, useless for Rust's +panic-and-unwind, and only accidentally adequate for Go once goroutines enter +the picture. Knowing which tool actually sees your language's failure mode, +and which one just looks like it does, is what keeps a debugging session from +turning into a search for frames that were never going to be there. + Every debugger so far in this book has been a `println` you had to recompile to move. This chapter replaces that habit with the real tool: a crash that already happened, frozen in a core file or a live process, that you can diff --git a/_docs/29-valgrind-sanitizers-miri.md b/_docs/29-valgrind-sanitizers-miri.md index 089643e..8670ff1 100644 --- a/_docs/29-valgrind-sanitizers-miri.md +++ b/_docs/29-valgrind-sanitizers-miri.md @@ -6,6 +6,48 @@ description: "bugfarm seeds five defects: leak, use-after-free, an uninitialised duration: "60 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Distinguish runtime binary-translation tools (valgrind's `memcheck`, + `helgrind`, `massif`) from compile-time source instrumentation + (`ASan`/`UBSan`/`TSan`/`LSan`), and state the rebuild/overhead trade-off + each implies. +- Run valgrind `memcheck` against an unmodified binary to catch a leak, an + uninitialized read, a use-after-free, and a heap overflow, and read its + addressability/definedness shadow-bit model. +- Build and run `ASan`/`TSan`-instrumented binaries and read a shadow-memory + report (redzone, freed-block, `heap-use-after-free`) down to the exact + tripped byte. +- Explain why valgrind misreports or misses defects in a Go binary whose + runtime manages memory outside libc, and why `go build -race` (which links + ThreadSanitizer directly) does not. +- Use `cargo miri` to catch undefined behavior reachable only through + `unsafe` Rust, naming the exact allocation and deallocation sites of a + use-after-free. +- Compare three independent data-race detectors (ThreadSanitizer, helgrind, + `go -race`) converging on the same colliding source line by three different + mechanisms. + +## Why this matters + +Memory and concurrency bugs rarely announce themselves where they happen: a +use-after-free can run clean for years before the freed block gets reused, +and a data race can print the right answer nine times out of ten. The tools +in this chapter catch five such defects at the moment they occur rather than +however many steps later they happen to corrupt something visibly, and each +tool does it with a different cost: no rebuild but an order-of-magnitude +slowdown for valgrind, a required rebuild but near-native speed for a +sanitizer, symbolic interpretation with no compiled binary at all for miri. + +Which tool fits depends on the language and the situation: a borrow checker +or a garbage collector can rule a whole bug class out before any tool runs, +and knowing which three of the five defects C++, Go, and Rust each still +need to worry about (directly from this chapter's own catch/miss matrix) is +what turns "run every tool on everything" into picking the right one for the +binary in front of you. + Chapter 28 read a crash after it happened. This chapter catches five bugs *before* they get the chance to crash anything; or, in one case, watches a tool fail to catch one at all. `bugfarm` is a single CLI, diff --git a/_docs/30-ebpf-observation-toolkit.md b/_docs/30-ebpf-observation-toolkit.md index e57a38c..f5b2333 100644 --- a/_docs/30-ebpf-observation-toolkit.md +++ b/_docs/30-ebpf-observation-toolkit.md @@ -6,6 +6,47 @@ description: "app work --seconds N runs an ordinary, unmodified workload on syst duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Point bcc-tools (`execsnoop`, `opensnoop`, `funccount`, `offcputime`, + `biolatency`) at an unmodified running binary to observe file opens, exec + calls, function hit counts, off-CPU time, and block I/O latency with no + rebuild and no restart. +- Write `bpftrace` one-liners and scripts that attach to a tracepoint, kprobe, + or uprobe, filter on built-ins like `pid`/`comm`/`argN`, and aggregate + results with `hist()`/`count()`. +- Attach a `uprobe`/`uretprobe` pair to time a userspace function's + entry-to-return latency and read the resulting histogram. +- Use `bpftool prog show`/`map show` to independently confirm what the kernel + actually has loaded, cross-checking what `bpftrace` itself reports. +- Demangle Itanium (C++) and Rust v0 mangled symbol names with `c++filt` to + find the exact string a uprobe must attach to. +- Explain why Go's register-based ABIInternal breaks the System-V-assuming + `arg0`/`argN` shorthand, and verify the mismatch by reading `RDI` against + `RAX` on a live uprobe. +- Compare this eBPF-based observation path against SystemTap's kernel-module + and dyninst backends for the same kind of probe. + +## Why this matters + +Every technique earlier in this part needed something extra attached to the +target: debug info for `gdb`, a sanitizer rebuild for ASan/TSan. The eBPF +observation stack this chapter uses watches a plain, unmodified binary from +outside, live, while it runs: the kernel loads a small verified program onto +a probe point and runs it on every hit, with no recompilation and no +downtime. That is the technique this chapter teaches: pointing existing, +purpose-built tools (bcc-tools, `bpftrace`, `bpftool`) at your own userspace +programs, never writing or loading kernel code of your own. + +It also surfaces a real trap worth knowing before you hit it live: tools that +read function arguments by position assume the System V calling convention, +and Go's compiler does not follow it. A uprobe can report a plausible-looking +wrong number instead of an error, and the only defense is knowing which +register a given language and function actually use, confirmed here by +reading `RDI` next to `RAX` on the same call. + Chapter 28 read a crash gdb already knew about; Chapter 29 caught bugs a compile-time instrumentation pass or a binary-translating emulator watched for. Both chapters needed something extra glued onto the target: debug diff --git a/_docs/31-per-language-toolbelts.md b/_docs/31-per-language-toolbelts.md index fd3a17d..7846ada 100644 --- a/_docs/31-per-language-toolbelts.md +++ b/_docs/31-per-language-toolbelts.md @@ -6,6 +6,24 @@ description: "app hot and app alloc are the same trial-division prime count and duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Read a flamegraph in one scan by tracing the widest column from the root frame to the leaf, regardless of which language produced it +- Capture a CPU profile with `perf record --call-graph fp` and summarize it with `perf report --stdio` against a native ELF binary +- Drive Go's in-process profilers (`runtime/pprof`'s `--cpuprofile`/`--memprofile`, `runtime/trace`'s `--trace`) and read their output with `go tool pprof --top` and `go tool trace` +- Render a `cargo flamegraph` SVG and explain how `inferno` collapses and draws the same samples `perf report` tabulates as text +- Script a `dlv`/`gdb`-style breakpoint session to confirm a profiler's named hot frame independently +- Run static analyzers (`clang-tidy`, GCC `-fanalyzer`, `cargo clippy`) and distinguish a "this is slow" finding from a "this pattern is suspicious" finding +- Cross-check three unrelated profiling tools against one shared `app hot`/`app alloc` target to confirm they agree on the hot path + +## Why this matters + +A profiler only answers "where does the time go" if you can read what it hands back, and every language hands it back differently: `perf` attaches to any ELF binary from the outside, Go's profilers live inside the binary as libraries the program calls into, and Rust sits in between, native enough for `perf` but shipping its own `cargo flamegraph` wrapper on top. A systems programmer who only knows one of these toolbelts is stuck guessing on every other stack. + +The payoff is the cross-check this chapter runs at the end: three profiling tools, built by three unrelated projects, reading three different kinds of instrumentation, all converge on the same function name at the same overwhelming share of samples. That agreement, not any single tool's output, is what makes a profiling result trustworthy enough to act on. + Chapter 28 opened this part by reading a crash gdb already knew about; Chapter 29 broke code by design and watched a sanitizer catch it. Both chapters answer "what is wrong". This one answers a different question that diff --git a/_docs/32-namespaces-and-cgroups.md b/_docs/32-namespaces-and-cgroups.md index 268ee5b..ffa28fd 100644 --- a/_docs/32-namespaces-and-cgroups.md +++ b/_docs/32-namespaces-and-cgroups.md @@ -6,6 +6,24 @@ description: "The eight Linux namespaces and cgroup v2's unified hierarchy as th duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Name the job of each of the eight Linux namespace types and identify which four `pmon containerize` requests +- Choose between `unshare(2)`, `setns(2)`, and `clone3(2)` based on whether you're moving the caller, joining an existing namespace, or creating a child and its namespaces atomically +- Explain the pid-1 reparenting and reaping obligations a process takes on inside a fresh PID namespace, and why `pmon containerize` sidesteps both by `exec`ing CMD immediately +- Build a cgroup v2 leaf by writing `cgroup.subtree_control`, `memory.max`, and `cpu.max`, and read `memory.pressure` (PSI) as a per-cgroup stall signal distinct from utilization +- Verify a cgroup memory ceiling with a negative control, confirming the OOM killer fires and reading its `dmesg` evidence (`oom_memcg=`) +- Diagnose the `ENOMEM` a threaded Go runtime hits calling `unshare(CLONE_NEWPID)` directly, and fix it with `exec.Cmd.SysProcAttr.Cloneflags` instead of `runtime.LockOSThread()` +- Inspect namespace identity directly with `readlink /proc//ns/*` and demonstrate a missing `/proc` remount with `nsenter` + +## Why this matters + +"Container" names no single kernel object: it is namespaces and cgroups, two independent primitives any userspace program can assemble by hand. Seeing them built from raw syscalls, rather than through `podman run`, is what makes the pid-1 reparenting rules, the per-thread nature of `unshare`, and a cgroup's enforcement all concrete instead of folklore. + +The Go `ENOMEM` failure in this chapter is a real, reproduced defect, not a hypothetical: `unshare(CLONE_NEWPID)` caps the caller to exactly one child in the new namespace, and `os/exec`'s internal probe fork silently spends it. Knowing that restriction, and the `SysProcAttr.Cloneflags` fix that avoids it entirely, is the difference between a container tool that works on the host you tested and one that fails unpredictably on every other kernel. + Every `pmon` version so far has supervised, signaled, and identified a child, but the child has always lived in the same world as its parent: same process-id space, same hostname, same view of every device and byte of diff --git a/_docs/33-seccomp-and-landlock.md b/_docs/33-seccomp-and-landlock.md index c7ac168..8b340a3 100644 --- a/_docs/33-seccomp-and-landlock.md +++ b/_docs/33-seccomp-and-landlock.md @@ -6,6 +6,24 @@ description: "fwatch sandboxes itself with two independent, unprivileged kernel duration: "55 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Build a seccomp-bpf syscall allowlist from a real `strace -f -c` profile rather than guessing which syscalls a program needs +- Install a filter with `seccomp_init`/`seccomp_rule_add`/`seccomp_load` (or Go's hand-emitted cBPF program) and choose a default action with `prctl(PR_SET_NO_NEW_PRIVS)` set first +- Contrast `SCMP_ACT_ERRNO` (the caller keeps running and can clean up) against `SCMP_ACT_KILL_PROCESS` (an uncatchable `SIGSYS`, no return) and pick the right one for a given sandbox posture +- Construct a Landlock ruleset with `landlock_create_ruleset`/`landlock_add_rule`/`landlock_restrict_self` that restricts filesystem reads to one directory +- Probe the Landlock ABI version at runtime instead of assuming it, and gate a feature on the version number it actually needs +- Prove a sandbox's effect with paired positive and negative controls, reading the resulting `EPERM`/`EACCES` and the kernel's own `/proc//status` seccomp fields +- Apply both mechanisms process-wide in a threaded runtime using `TSYNC` plus `runtime.LockOSThread()`, instead of restricting only the calling thread + +## Why this matters + +seccomp-bpf and Landlock let an unprivileged process restrict itself, no root, no namespace, no VM required. That makes them two of the cheapest isolation techniques available to any program this book has built, and they compose directly with the namespaces and cgroups from Chapter 32 as part of what a real container runtime assembles. + +Both are also easy to get wrong in ways that look fine until tested: a filter built from a guessed syscall list breaks the program it was meant to protect, an ABI assumed rather than probed misbehaves on a different kernel, and a restriction applied to only one thread in a multithreaded runtime leaves every other thread unsandboxed. The design discipline this chapter teaches, profile first, probe the ABI, verify with a negative control, is what turns seccomp and Landlock from documentation into a sandbox you can trust. + Chapter 32 gave `fwatch` its own view of the process tree and its own resource limits through namespaces and cgroup v2 -- isolation the kernel grants to a group of processes. This chapter hands the same idea to a diff --git a/_docs/34-our-programs-in-containers.md b/_docs/34-our-programs-in-containers.md index de71563..7dbcb86 100644 --- a/_docs/34-our-programs-in-containers.md +++ b/_docs/34-our-programs-in-containers.md @@ -6,6 +6,24 @@ description: "The same three daemons this book has built all along, now shipped duration: "60 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Write a multi-stage Containerfile that builds in a full UBI toolchain image and ships only a static-linked binary copied into a minimal `ubi10-micro` runtime stage +- Explain how rootless podman's user-namespace uid mapping (`/etc/subuid`, `podman unshare cat /proc/self/uid_map`) lets container root remain an unprivileged host user +- Implement a container entrypoint that handles PID 1's signal and reaping duties explicitly, instead of relying on a default action that the kernel never supplies +- Read a container's own cgroup-imposed CPU and memory ceiling from `/proc/self/cgroup` and `/sys/fs/cgroup`, and contrast `hardware_concurrency()` against cgroup-aware `GOMAXPROCS`/`available_parallelism()` +- Reset an inherited signal mask after `fork(2)` so a forwarded `SIGTERM` is not left permanently pending in a child +- Reach a container's PID 1 from the bare host with `podman exec`, `podman unshare nsenter`, and a direct `gdb -p` attach, and explain why plain `nsenter` fails +- Probe podman's default seccomp profile to show it silently denies `io_uring` syscalls that an unmodified program might depend on + +## Why this matters + +Shipping a daemon as a container changes facts the program itself can observe and must handle correctly: what counts as CPU parallelism, what happens to an unhandled signal on PID 1, and which syscalls are even reachable. Getting any of these wrong doesn't fail loudly, a C++ build that trusts `hardware_concurrency()` inside a `--cpus=2` container just oversubscribes silently, and a naive entrypoint just sits through a full `podman stop` grace period before being killed. + +This chapter also closes the loop on two mechanisms the book built from raw syscalls in Chapters 32 and 33: a container runtime assembles namespaces, cgroups, and a seccomp profile for you, but the pid-1 obligations and the per-thread signal-mask hazards underneath them are the same ones you'd hit building a container tool by hand. + Chapter 32 gave one of our programs its own namespaces and cgroup directly, by calling `unshare`/`clone3` itself. Chapter 33 had `fwatch` sandbox itself with seccomp and Landlock. This chapter hands both jobs to something else: diff --git a/_docs/35-virtualization-and-kvm.md b/_docs/35-virtualization-and-kvm.md index 7269c56..85e7d1d 100644 --- a/_docs/35-virtualization-and-kvm.md +++ b/_docs/35-virtualization-and-kvm.md @@ -6,6 +6,24 @@ description: "How a VM actually runs: VMX root/non-root modes, EPT, and virtio d duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Explain the VMX root/non-root mode split, what causes a VM exit, and why a guest syscall like `getpid(2)` never triggers one while `fsync(2)` against `virtio-blk` does +- Describe the virtio-blk round trip (guest descriptor, kick, VM exit, QEMU backend `fsync`, VM entry) and how Extended Page Tables amortize ordinary memory access to near-native cost +- Place the four points on the isolation spectrum, bare process, container, microVM, full KVM/QEMU VM, against escape surface and boot/overhead cost +- Identify the rust-vmm crates (`kvm-ioctls`, `vm-memory`, `virtio-queue`) underlying both Firecracker and Cloud Hypervisor +- Write a raw-syscall microbenchmark (`bench_syscall`, `bench_mem`, `bench_io`) with anti-dead-code guards appropriate to each language and a monotonic clock +- Run the same binary unmodified across a bare host, a KVM guest, and a rootless Podman container, and interpret which metrics track the host and which diverge +- Cross-check a benchmark's claimed syscall count with `strace -c`, and explain why `ptrace` overhead disqualifies it as a timing source + +## Why this matters + +Choosing between a process, a container, a microVM, and a full VM is a cost decision, not an abstract security ranking, and the cost only becomes real once you measure it in the same units this book has used throughout: nanoseconds per syscall, gigabytes per second, operations per second. `overheadbench` turns "which deployment option" into a question with a number attached, run on the exact hardware and kernel you'll actually ship on. + +The VMX mental model matters beyond benchmarking too: knowing that a VM exit is triggered by touching state the guest doesn't own, not merely by "leaving the guest", explains why a program's syscall-heavy and I/O-heavy paths can have wildly different overhead profiles inside the same guest, a distinction that shapes where to spend optimization effort. + Every daemon in this book has, until now, run either directly on the host or inside a Podman container sharing the host's kernel. This chapter puts a third option on the table: a full KVM guest with its own kernel, and asks the diff --git a/_docs/36-proc-sys-and-the-agent.md b/_docs/36-proc-sys-and-the-agent.md index 2b7811f..fcb88a4 100644 --- a/_docs/36-proc-sys-and-the-agent.md +++ b/_docs/36-proc-sys-and-the-agent.md @@ -6,6 +6,24 @@ description: "sysagent v0 is a USE-method metrics collector that reads /proc and duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Compute CPU utilization as a delta between two `/proc/stat` reads `--interval-ms` apart, rather than treating a single cumulative-tick read as a percentage +- Gather independent `/proc` sources (`loadavg`, `meminfo`, `diskstats`, `net/dev`, cgroup PSI) concurrently with the sampling sleep so a snapshot costs `max(interval_ms, gather_time)`, not the sum +- Resolve a process's own cgroup v2 PSI files via the `0::` line in `/proc/self/cgroup`, with a system-wide `/proc/pressure` fallback +- Design a schema field (`psi_available`) that lets a missing signal degrade a result instead of failing the whole collection +- Hand-write a JSON serializer with a fixed field order and consistent float formatting so three languages' output is byte-identical, instead of relying on each language's default library +- Expose the same snapshot through both a one-shot `sample --json` CLI and a `serve --port P` HTTP `/metrics` endpoint +- Choose a concurrency shape per language, shared-struct writes to disjoint fields in C++/Go versus `thread::scope` with owned return values in Rust, driven by each language's memory model + +## Why this matters + +A metrics agent that's going to run continuously in production can't afford to fork a subprocess or shell out to a tool for every sample; reading `/proc` and `/sys/fs/cgroup` directly is the only approach cheap enough to run every second for months between deploys. Getting the `/proc/stat` delta wrong, or treating a cumulative counter as an instantaneous reading, is one of the most common correctness bugs in hand-rolled monitoring code. + +The cross-language schema discipline here also matters beyond this one example: a dashboard, a `jq` filter, or a test suite that has to understand three different field-naming and serialization conventions across a C++, Go, and Rust build of the same tool is not a tool anyone will actually standardize on. Byte-identical output is what makes the three builds interchangeable in practice, not just in principle. + Chapter 35's `overheadbench` measured syscall, memory, and I/O overhead across a bare process, a container, and a VM, but every one of those numbers came from watching `strace -c` and hand-timed loops wrapped around each run. That's diff --git a/_docs/37-use-method-60s-analysis.md b/_docs/37-use-method-60s-analysis.md index 0163faf..95b0305 100644 --- a/_docs/37-use-method-60s-analysis.md +++ b/_docs/37-use-method-60s-analysis.md @@ -6,6 +6,24 @@ description: "loadmix is Brendan Gregg's Utilization/Saturation/Errors checklist duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Apply Brendan Gregg's USE method (Utilization, Saturation, Errors) as a fixed threshold table against a resource, rather than an informal checklist +- Parse `vmstat`, `mpstat -P ALL`, `iostat -xz`, `sar -n DEV`, and `pidstat` output by column name from each tool's own header row, so a parser survives a `sysstat`/`procps-ng` version bump that reorders columns +- Handle each sampled tool's specific quirk (the since-boot first row in `vmstat`/`iostat`, `mpstat`'s `Average:` aggregate rows, `dmesg`'s permission-denied path) without failing the whole run +- Spawn five interval samplers concurrently and join them once so an N-second checklist costs N seconds, not five times N +- Scale a saturation threshold to the machine it runs on (`run_queue >= cpus + 2`) where that is the accurate signal, instead of a flat constant +- Rank resources by Saturation-value-to-threshold ratio rather than raw Saturation value, so a high-scale metric doesn't always win +- Diagnose a `std::jthread` RAII ordering bug (`request_stop()` firing before a worker's own deadline) and fix it by holding the parent scope open with `sleep_until` + +## Why this matters + +Most engineers triaging a live system reach for `uptime`, `vmstat`, `mpstat`, `iostat`, and `pidstat`, not a custom collector, because those tools are already trusted and already parse a running kernel correctly. `loadmix analyze` encodes that exact workflow as a program, so understanding how it parses tool output and scales its thresholds is understanding how an on-call engineer's checklist actually works under the hood. + +The column-name parsing discipline and the concurrent-sampler design generalize past this one tool: any program that wraps an external CLI's plain-text output needs the same defense against a version bump reordering fields, and any program running several blocking interval samples needs the same concurrency shape to avoid turning a one-minute checklist into a five-minute one. + Chapter 36's `sysagent` built a USE-method collector from first principles: reading `/proc/stat`, `/proc/net`, `/proc/diskstats`, and cgroup PSI directly, no external process, no tool you'd have to trust. That is the right call for a diff --git a/_docs/38-opentelemetry-to-lgtm.md b/_docs/38-opentelemetry-to-lgtm.md index 23869d2..f93fc6c 100644 --- a/_docs/38-opentelemetry-to-lgtm.md +++ b/_docs/38-opentelemetry-to-lgtm.md @@ -6,6 +6,24 @@ description: "lsp-otel is a traced request/response server that exports OpenTele duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Emit a parent/child trace tree per request with `StartSpan`/`tracer.Start`/`span_builder` across C++, Go, and Rust +- Record a monotonic counter (`requests_total`) and a histogram (`request_duration`) alongside that trace +- Carry parent/child span context three ways: Go's explicit `context.Context`, Rust's `Context::current_with_span`, and C++'s `Scope` guard +- Export spans and metrics over OTLP/HTTP to a collector, batching on a short interval and flushing on shutdown +- Verify the export landed by querying Tempo and Prometheus through Grafana's datasource proxy instead of trusting the program's own output +- Tag each language's exporter with a distinct `service.name` so three services share one LGTM stack without colliding +- Reconcile each language's shutdown signal handling (a signal-driven flush, `select`, or an `mpsc` channel) against the same OTLP contract + +## Why this matters + +A program that claims to export telemetry and a program that actually exports telemetry look identical from the terminal: both print a clean exit and move on. The only way to tell them apart is to go read the backend the export was supposed to reach, which is exactly what this chapter's verification step does instead of trusting `serve`'s own stdout. + +That distinction matters because the usual failure mode here is silent: a wrong OTLP endpoint path, a default SDK log flooding the output contract, or a batch that never flushes before the process exits all produce a program that looks healthy and ships nothing. Querying Tempo and Prometheus back is what turns "I instrumented the service" into a claim you can actually stand behind. + Chapter 37's `loadmix analyze` printed a USE-method verdict to a terminal, and Chapter 36's `sysagent` served one JSON snapshot per HTTP request. Both make you *present* to read the number: someone has to be running the command, or scraping diff --git a/_docs/39-benchmarking-without-lies.md b/_docs/39-benchmarking-without-lies.md index f6b5cc9..8f8131a 100644 --- a/_docs/39-benchmarking-without-lies.md +++ b/_docs/39-benchmarking-without-lies.md @@ -6,6 +6,24 @@ description: "benchlab measures the ch21 chatterd frame codec two ways in the sa duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Discard a warmup period before timing, separating cold-start cost from steady-state latency +- Collect per-iteration samples and report percentiles (min/median/p99/max) instead of a single average +- Correct a closed-loop benchmark for coordinated omission by backfilling synthetic samples down to an expected interval +- Read the CPU frequency governor (`scaling_governor`, `cpupower frequency-info`) and treat it as a confound in tail latency +- Prevent dead-code elimination by folding every result into an observable checksum, and confirm the difference by reading `objdump` output +- Cross-check wall-clock percentiles against `perf stat` cycles and instructions counts +- Implement the identical warmup/percentile/coordinated-omission algorithm in C++, Go, and Rust and verify matching behavior across all three + +## Why this matters + +A microbenchmark you wrote yourself is the easiest number in the whole book to get quietly wrong, because nothing stops it from compiling cleanly and printing a confident, reproducible figure for code that never ran, or for a stall a closed loop recorded once when a real caller would have felt it hundreds of times over. `benchlab` is built to show both traps happening in the same binary, so the gap between a misleading number and a real one is something you can watch rather than take on faith. + +The payoff carries past this chapter: Chapter 40 spends the discipline earned here, pinning a core and reusing a buffer to shrink exactly the tail this chapter learned to measure accurately. + Chapter 37 gave you the USE method's 60-second checklist for a system already in trouble, and Chapter 38 wired that telemetry into the LGTM stack you can query back. Both assume the numbers arriving are real. This chapter asks the diff --git a/_docs/40-low-latency-fastpath.md b/_docs/40-low-latency-fastpath.md index e3a4156..16b87f9 100644 --- a/_docs/40-low-latency-fastpath.md +++ b/_docs/40-low-latency-fastpath.md @@ -6,6 +6,24 @@ description: "chatterd-fastpath strips the ch21 echo hot path down to a pinned c duration: "50 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Pin a process to one CPU with `sched_setaffinity(2)` and confirm placement through `/proc//status`'s `Cpus_allowed_list` +- Eliminate per-message heap allocation by reusing a single stack buffer across the life of a connection +- Replace a blocking `read(2)` with a non-blocking `recv` busy-poll loop that spins on `EAGAIN` instead of sleeping +- Count voluntary context switches through `/proc//status` to prove a server never blocks, rather than trusting what it prints +- Resolve a CPU-count discrepancy between `sysconf(_SC_NPROCESSORS_ONLN)` and Go's `runtime.NumCPU()` under a `taskset` affinity mask +- Shut down cleanly across three mechanisms: a non-`SA_RESTART` handler consuming `EINTR`, and Go's poller-driven fd close +- Benchmark a naive server against a fastpath server under identical `taskset` placement and report the percentile gap + +## Why this matters + +Chapter 39 earned the right to trust a `p99`. This chapter spends that trust on the next question: once you believe the number, how do you make it small? Pinning, zero allocation, and busy-poll each remove one specific, nameable source of tail latency, and each carries a real cost, most of all busy-poll's whole dedicated core, so the chapter's real lesson is picking the right trade for a latency-critical path rather than applying every trick everywhere. + +The proof discipline matters as much as the techniques. `Cpus_allowed_list` and `voluntary_ctxt_switches` come from the kernel's own accounting in `/proc//status`, not from anything the server prints about itself, which is what turns "this server is pinned and never sleeps" from a claim into something you can check. + Chapter 39 taught you to trust a latency number: warm the code, time each iteration, correct for coordinated omission, and report the tail instead of the mean. This chapter spends that discipline. If you now believe a `p99` you diff --git a/_docs/41-capstone-fleet.md b/_docs/41-capstone-fleet.md index 9528175..5a5a55e 100644 --- a/_docs/41-capstone-fleet.md +++ b/_docs/41-capstone-fleet.md @@ -6,6 +6,24 @@ description: "41-capstone-fleet is one binary with four subcommands that assembl duration: "60 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Drop the capability bounding set and set `no_new_privs` with `prctl(2)` before spawning any child process +- Self-re-exec one binary as three supervised services (`chatterd`, `sysagent`, `fwatch`) through fork/exec and `waitpid` +- Restart a crashed child with a backoff while forwarding `SIGTERM`/`SIGINT` to every supervised child +- Bridge `chatterd` across two hosts by joining a peer connection under a `bridge@` nick +- Export `sysagent`'s cpu/mem/load1 gauges over OTLP when `OTEL_EXPORTER_OTLP_ENDPOINT` is set +- Sandbox `fwatch` under a Landlock ruleset scoped to one directory and probe the kernel's enforced ABI +- Verify a two-host fleet end to end: health lines from each supervisor, a cross-host message delivery, and a gauge reaching Prometheus through the libvirt gateway + +## Why this matters + +This is the final project: four of the book's six recurring programs, assembled into one supervised, sandboxed, observed system running on two real hosts instead of four separate demos on one. `pmon` plays the fleet's own init, which means every sandboxing and signal-handling discipline from earlier chapters (capability drop, Landlock, self-re-exec, three different ways to consume a shutdown signal) has to cooperate inside a single process tree instead of standing alone. + +The chapter also closes the loop this book keeps returning to: prove the claim, don't just print it. A healthy-looking `pmon: health` line is not proof the fleet works; a message that crosses two real hosts and a gauge that lands in the host's own Prometheus through the libvirt gateway are. + Chapter 0's course map promised six programs that would grow chapter by chapter into real systems software. By now four of them have grown enough to stand on their own: `pmon` learned to fork, exec, wait, drop privilege, and diff --git a/_docs/42-embedded-lua-scripting.md b/_docs/42-embedded-lua-scripting.md index 17f58f4..bddade0 100644 --- a/_docs/42-embedded-lua-scripting.md +++ b/_docs/42-embedded-lua-scripting.md @@ -6,6 +6,24 @@ description: "lua-policy moves pmon's restart/backoff decision out of the compil duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Move a restart/backoff decision out of compiled code into a sandboxed `policy.lua` behind one `on_exit(info) -> decision` contract +- Embed the same Lua file in three structurally different runtimes: sol2 (C++/PUC-Rio 5.4), gopher-lua (Go/Lua 5.1), and mlua (Rust/PUC-Rio 5.4) +- Build a sandbox that never opens `os`, `io`, `package`, or `debug`, and nils `load`/`loadfile`/`dofile`/`require`/`collectgarbage`/`loadstring` +- Call into a policy under a protected call (`sol::protected_function`, `CallByParam{Protect:true}`, `mlua::Result`) so a script error returns as data, not a crash +- Reload a policy live on a real `SIGHUP` by rebuilding the Lua state from scratch, with no process restart +- Write a shared-subset policy file that runs identically on Lua 5.1 and Lua 5.4 +- Compare three languages' sandbox-rejection diagnostics for the identical malicious policy + +## Why this matters + +A restart policy baked into compiled C++, Go, or Rust means a tuning change costs a rebuild and a redeploy. Moving that decision into one sandboxed `policy.lua` instead means `pmon` can change its crash-looping threshold or its backoff curve on a live `SIGHUP`, with the same process and the same PID throughout, which is the capability an operator actually wants during an incident. + +The harder question this chapter answers is whether "the same policy" really means the same behavior once it is embedded three structurally different ways. Proving sol2, gopher-lua, and mlua reach byte-identical restart decisions from one unmodified file, across a Lua 5.1/5.4 version gap, is what makes "one policy.lua, three host runtimes" more than a slogan. + Chapter 12 taught `pmon` to treat a signal as data instead of an interruption: block `SIGCHLD` process-wide, then read it off a `signalfd` on the next turn of an ordinary `poll` loop. Chapter 41's final-project fleet leaned diff --git a/_docs/43-rust-macros-for-systems.md b/_docs/43-rust-macros-for-systems.md index 7219073..eb8d580 100644 --- a/_docs/43-rust-macros-for-systems.md +++ b/_docs/43-rust-macros-for-systems.md @@ -6,6 +6,24 @@ description: "sysmacros is chapter 43's proc-macro crate, powering two Rust-only duration: "35 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Build a `proc-macro = true` crate that parses tokens with `syn 2.0` and emits code with `quote!` +- Derive a `Display`/`Error` implementation from one `#[error("...")]` attribute per enum variant, across `Fields::Named`, `Unnamed`, and `Unit` shapes +- Write an `#[instrument]` attribute macro that wraps a function body to log its entry, exit, and `Instant`-measured duration +- Perform a sound `unsafe` unaligned read with `core::ptr::read_unaligned`, guarded by an explicit bounds `assert!` +- Validate the read's memory safety with `cargo +nightly miri run` and read a located Undefined Behavior diagnostic +- Convert a malformed macro input into a `compile_error!` via `syn::Error::to_compile_error()` +- Distinguish miri's correctness-checking role from profiling by comparing a native run's duration against an interpreted one + +## Why this matters + +A hand-written error type and a hand-written timing wrapper are both boilerplate you have written before, which makes them a reasonable place to ask how much a proc-macro can write for you instead: `#[derive(SysError)]` and `#[instrument]` both turn an annotation into real code, generated once at compile time, with zero runtime cost once `app` is built. + +The second half of the chapter asks a sharper question: how do you prove an `unsafe` block is still doing exactly what its comment claims? `assert!` enforces the bound a sound read depends on, and `miri` is what catches the one line that skips it, turning a silent out-of-bounds read that a normal `cargo run` sails straight through into a located, reproducible diagnostic. + Every chapter so far has mirrored the same behavior across C++23, Go, and Rust, because the point of those chapters was a Linux mechanism (a syscall, a signal, a namespace) that all three languages sit on top of diff --git a/_docs/44-go-runtime-for-systems.md b/_docs/44-go-runtime-for-systems.md index fb10ed8..2f349f0 100644 --- a/_docs/44-go-runtime-for-systems.md +++ b/_docs/44-go-runtime-for-systems.md @@ -6,6 +6,24 @@ description: "go-runtime-for-systems is a single Go binary that observes its own duration: "45 minutes" --- +## Learning objectives + +By the end of this chapter you can: + +- Explain the GMP scheduler model (goroutines, machines, processors) and how work-stealing moves half a queue between P's +- Force signal-based async preemption of a non-yielding goroutine under `GOMAXPROCS=1` and name `SIGURG`'s role in it +- Tune GC pacing with `GOGC` and `GOMEMLIMIT` (`debug.SetGCPercent`, `debug.SetMemoryLimit`) and measure the resulting cycle-count effect +- Distinguish a blocking read from a netpoller-parked read, and explain why 2,000 blocked goroutines need only a handful of OS threads +- Read `/proc/self/status`'s `Threads:` field to cross-check the runtime's own goroutine and thread bookkeeping +- Interpret `GODEBUG=gctrace=1` and `GODEBUG=schedtrace=1000,scheddetail=1` output +- Disable async preemption with `GODEBUG=asyncpreemptoff=1` and observe the scheduler's own verdict flip to FAIL + +## Why this matters + +Every earlier Go chapter in this book leaned on the runtime without looking at it: goroutines that seemed to run in parallel, garbage that seemed to collect itself, sockets that seemed to block without blocking a thread. This chapter makes the runtime answer for itself, asking the scheduler, the collector, and the netpoller direct questions and printing the answers as computed verdicts rather than taking any of it on faith. + +The netpoller theme is the clearest payoff: 2,000 goroutines blocked on a read and a thread count that stays in the double digits is the concrete, measured reason Go programs can hold far more concurrent work in flight than their OS thread count would suggest, and the chapter proves it against the kernel's own `/proc/self/status` accounting, not the runtime's self-report. + Chapter 43 spent its whole budget on Rust's compiler doing extra work at build time -- a proc-macro crate that runs once, produces tokens, and gets out of the way. This chapter is the mirror image: a runtime that never gets