Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ example doubles as a standalone demo.
`systems-peer`) for anything privileged, kernel-observing, or two-host;
Podman LGTM stack (Grafana/Loki/Tempo/Mimir) + Performance Co-Pilot for
observability chapters.
- **eBPF scope**: tooling only (bcc-tools, bpftrace, bpftool) — we observe our
userspace programs; we do not write kernel eBPF.
- **eBPF scope**: tooling only (bcc-tools, bpftrace, bpftool); we observe our
userspace programs and do not write kernel eBPF.

## Local development

Expand All @@ -24,7 +24,7 @@ bundle exec jekyll serve --baseurl "" # http://127.0.0.1:4000/
## Verification model

CI validates site integrity and that examples *build*. It cannot run VM, root,
or observability demos — the authoritative gate is a host run of
or observability demos: the authoritative gate is a host run of
`scripts/test-all-examples.py --mode all` with the lab up. A demo is marked
**verified** only when it produced its claimed observable effect, never merely
because it compiled.
Expand All @@ -33,12 +33,12 @@ because it compiled.

This project is dual-licensed to separate the code from the writing:

- **Code** — everything under `examples/`, plus build and helper scripts and
inline code snippets — is licensed under the **Apache License 2.0**
- **Code**: everything under `examples/`, plus build and helper scripts and
inline code snippets, is licensed under the **Apache License 2.0**
([`LICENSE`](LICENSE)).
- **Content** — the tutorial and book prose (chapters under `_docs/`),
- **Content**: the tutorial and book prose (chapters under `_docs/`),
diagrams and images under `assets/`, and the slide decks under
`presentation/` — is licensed under **Creative Commons Attribution 4.0
`presentation/`, is licensed under **Creative Commons Attribution 4.0
International (CC BY 4.0)** ([`LICENSE-docs.md`](LICENSE-docs.md)).

Copyright 2026 PatternCatalyst (Robert Sedor).
6 changes: 3 additions & 3 deletions _docs/00-outline.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ switchable code tabs, and every example doubles as a standalone demo.

{% include excalidraw.html
file="00-course-map"
alt="Course map: four journey stages - Foundations, Core systems, Operating context, and Mastery - shown as bands containing the fourteen parts with their chapter ranges, and the six artifacts fwatch, shmkv, chatterd, pmon, bugfarm, and sysagent along the bottom, each with an arrow to the part where it grows."
alt="Course map: four stages, Foundations, Core systems, Operating context, and Mastery, shown as bands containing the fourteen parts with their chapter ranges, and the six artifacts fwatch, shmkv, chatterd, pmon, bugfarm, and sysagent along the bottom, each with an arrow to the part where it grows."
caption="Figure 0.1: the course at a glance." %}

Two threads run through the whole book and never stop:
Expand Down Expand Up @@ -63,8 +63,8 @@ artifacts that grow chapter by chapter:

Examples run **locally** on your Fedora 44 machine unless they need root, kernel
observation, or a second host; those run in the disposable KVM lab
(`systems-target`, and `systems-peer` for two-host networking) that Chapter 2
provisions. Observability chapters use a local Podman LGTM stack (Grafana,
(`systems-target`, and `systems-peer` for two-host networking) that the lab
setup provisions. Observability chapters use a local Podman LGTM stack (Grafana,
Loki, Tempo, Mimir) plus Performance Co-Pilot from its container image. Nothing
experimental ever runs un-snapshotted on your workstation.

Expand Down
10 changes: 5 additions & 5 deletions _docs/01-prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ book was written against, which is why this chapter is less "install these
packages" and more "here is *why* each pin exists, and here is one script that
tells you whether your host matches." By the end you will have run the book's
first program, `examples/01-hello-syscalls`, in all three languages, and
cross-checked with `strace` that each of them really made the syscalls the
source code claims.
cross-checked with `strace` that each of them made the syscalls the source
code claims.

The code is in `examples/01-hello-syscalls/`. `./demo.sh` there builds and
runs all three implementations; its `README.md` covers the run-script contract
Expand Down Expand Up @@ -81,7 +81,7 @@ per run; this one accumulates every result into a table and exits non-zero
only at the end, so one pass gives you the full shopping list. Second, it
distinguishes **hard requirements** (`fail`, which gates the exit code) from
**soft** ones (`warn`, informational). That split encodes what the book
actually needs versus what is merely convenient.
needs versus what is merely convenient.

The hard requirements, and the reason each is hard:

Expand Down Expand Up @@ -299,7 +299,7 @@ are `CStr`-like views, and `to_string_lossy()` handles the (theoretical)
non-UTF-8 case instead of panicking. Three languages, three defaults: C++
makes you construct the error discipline yourself, Go hands you an error value
and conventions, and Rust moves part of the manpage into the type system.
Chapter 5 turns this comparison into policy.
This comparison is the seed of the book's errors-three-ways policy.

## Build, run, observe

Expand Down Expand Up @@ -383,7 +383,7 @@ step ever requires the IDE.
- Every example follows one `demo.sh` contract: `build`, `run`, and
`TARGET=<vm>` for deploy-to-guest. It is *verified* only by its observable
output, which `verify.lua` pattern-matches.
- `strace` confirms what a binary actually asks the kernel for, and already
- `strace` confirms what a binary asks the kernel for, and already
reveals runtime differences: 64 syscalls for the C++ hello, 217 for Go's.

Next, the **KVM lab**: provisioning the `systems-target` and `systems-peer`
Expand Down
17 changes: 9 additions & 8 deletions _docs/02-lab-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ description: "Provision two disposable Fedora KVM guests with cloud-init: an obs
duration: 45 minutes
---

Chapter 1 proved that three toolchains build and run the same program on your
workstation. This chapter builds the other half of the book's infrastructure:
The hello-syscalls example already proved that three toolchains build and run
the same program on your workstation. This chapter builds the other half of
the book's infrastructure:
two disposable Fedora VMs (`systems-target` and `systems-peer`) that absorb
everything you should never do to the machine you work on. A systems-programming
book keeps asking you to run code as root, watch the kernel from the inside,
Expand Down Expand Up @@ -36,15 +37,15 @@ a path on your workstation is a bad afternoon. **Kernel observation** (bpftrace
one-liners, perf profiles, the BCC tools) attaches instrumentation to a live
kernel, and you want that kernel to be one you can reboot without losing your
editor. **Destructive demos** (OOM kills, disk-full behavior, broken network
configs) only teach if you actually break something, which
configs) only teach if you break something, which
means you need something safe to break.

A VM answers all three at once, and adds the property no container gives you:
**its own kernel**. Namespaces share the host kernel, so a container can't show
you kernel-version drift, can't be probed as an isolated kernel, and can't be
snapshot-reverted with its kernel state. The lab guests run their own
`6.19.10-300.fc44` kernel while the host runs `7.1`, a visible
difference this chapter turns into its cross-check.
difference used below as a cross-check.

That said, the VM is not the default. The book's decision rule, applied at the
top of every example:
Expand Down Expand Up @@ -148,8 +149,8 @@ base keeps provisioning fast and the snapshot meaningful.
*and* disk, because the guest is running, so a revert restores it
*running*, tools warm, in seconds. The discipline has two halves:

- **Snapshot immediately after provisioning** (and after any deliberate
addition you want to keep). This is your known-good tooled baseline.
- **Snapshot immediately after provisioning** (and after any addition you
want to keep). This is your known-good tooled baseline.
- **Revert after every destructive demo**: `scripts/lab/revert-vm.sh
systems-target` puts the guest back to `lab-ready`, no forensics, no
"did that experiment leave state behind?" doubt.
Expand Down Expand Up @@ -291,8 +292,8 @@ lab: built once, reverted forever.

## Cross-check: prove the code runs *there*

Chapter 1's example prints the kernel it runs on, which makes it a perfect
independent check that `deploy-to-vm.sh` really executes on the guest rather
The hello-syscalls example prints the kernel it runs on, which makes it a
perfect independent check that `deploy-to-vm.sh` executes on the guest rather
than, say, quietly running the local binary. Run it both ways:

```bash
Expand Down
18 changes: 9 additions & 9 deletions _docs/03-observability-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ same tools (`curl`, `podman ps`) you would use against the real thing, is a
skill this book leans on again when the lab VMs and later the page cache
force the same kind of judgment call.

Chapter 1 gave you three toolchains; Chapter 2 gave you two disposable VMs.
This chapter adds the last piece of the workbench: somewhere for telemetry to
The prerequisites gave you three toolchains; lab setup gave you two disposable
VMs. This chapter adds the last piece of the workbench: somewhere for telemetry to
*go*. Many chapters in this book emit metrics, traces, or logs (a syscall
counter here, a request span there), and printing them to a terminal only gets
you so far. What you want is a real backend you can query, graph, and
Expand All @@ -57,8 +57,8 @@ image, run rootless under Podman, listening on exactly two ports you will
memorize: **4318** for sending and **3000** for looking.

The infrastructure is in `examples/_infra/`: one compose file and a README.
There is no per-chapter example here; every telemetry-emitting example from
Chapter 38 onward depends on what you stand up now.
There is no per-chapter example here; every telemetry-emitting example, once
OpenTelemetry instrumentation begins, depends on what you stand up now.

{% include excalidraw.html
file="03-telemetry-pipeline"
Expand Down Expand Up @@ -146,7 +146,7 @@ polls Grafana's `/api/health` from inside the container with a generous
`start_period`, because first boot unpacks and starts all five services and
takes ~30 seconds; the health state is what scripts wait on instead of
sleeping. Finally the network gets the fixed name `lsp-observability` so any
future container (the PCP suite in Chapter 45, for instance) can join it by
future container (the PCP suite, for instance) can join it by
name rather than by guessing a generated one.

## The prerequisite: the Podman user socket
Expand Down Expand Up @@ -333,12 +333,12 @@ endpoint, two network paths, both answering.
`.122.1`), and `deploy-to-vm.sh` forwards `OTEL_ENDPOINT` through `sudo
env` because sudo strips the environment.

The stack sits idle until Chapter 38 instruments real programs in all three
languages against it, and Chapter 45 adds the PCP, Cockpit, and SystemTap
suites alongside. With the toolchains, the lab, and the telemetry pipeline in
The stack sits idle until OpenTelemetry instrumentation puts real programs in
all three languages against it, and the Linux analysis suites add PCP,
Cockpit, and SystemTap alongside. With the toolchains, the lab, and the telemetry pipeline in
place, the workbench is complete; next, Foundations begins where everything
in this book bottoms out: the syscall boundary and the ABI beneath it.

---

<p><span class="status status--verified">verified</span>: on the Fedora 44 host: <code>podman.socket</code> enabled, <code>podman compose -f examples/_infra/compose-lgtm.yaml up -d</code> brought <code>lsp-lgtm</code> (otel-lgtm:0.8.1) to healthy, <code>/api/health</code> returned ok, the OTLP HTTP POST to <code>:4318/v1/traces</code> returned 200, and the Tempo query by trace ID through Grafana's datasource proxy returned the span <code>r02-gate-smoke</code> (service <code>lsp-smoke</code>). Not yet exercised: exporting telemetry from inside a lab VM via the <code>192.168.124.1</code> gateway; the endpoint-forwarding path is verified only as far as <code>deploy-to-vm.sh</code> running host-built binaries on the guest; a VM-side OTLP export lands with Chapter 38.</p>
<p><span class="status status--verified">verified</span>: on the Fedora 44 host: <code>podman.socket</code> enabled, <code>podman compose -f examples/_infra/compose-lgtm.yaml up -d</code> brought <code>lsp-lgtm</code> (otel-lgtm:0.8.1) to healthy, <code>/api/health</code> returned ok, the OTLP HTTP POST to <code>:4318/v1/traces</code> returned 200, and the Tempo query by trace ID through Grafana's datasource proxy returned the span <code>r02-gate-smoke</code> (service <code>lsp-smoke</code>). Not yet exercised: exporting telemetry from inside a lab VM via the <code>192.168.124.1</code> gateway; the endpoint-forwarding path is verified only as far as <code>deploy-to-vm.sh</code> running host-built binaries on the guest; a VM-side OTLP export lands with OpenTelemetry instrumentation.</p>
22 changes: 11 additions & 11 deletions _docs/04-syscalls-and-the-abi.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ means later chapters on concurrency, I/O multiplexing, and signals can point
at a runtime behavior and explain it in terms you already have, instead of
treating each language's quirks as unrelated trivia.

Chapter 1 ended on a number it did not explain: the C++ hello made 64 syscalls,
the Go hello made 217, and we waved at "the runtime" and moved on. This chapter
The prerequisites left a number unexplained: the C++ hello made 64 syscalls,
the Go hello made 217, and "the runtime" took the blame without proof. This chapter
is where that debt gets paid. With the lab and observability stack built, Part
1 starts at the only interface that actually matters in this book, the
1 starts at the only interface that matters in this book, the
user/kernel boundary, and answers three questions precisely: what *is* a
system call at the instruction level, why do some libc calls never reach the
kernel at all (the vDSO), and what path does each of our three languages take
Expand Down Expand Up @@ -314,8 +314,8 @@ unix.EOPNOTSUPP)` still matches through the added context. rustix never
materializes a global at all: the negative `rax` *is* the `Err(Errno)`, and
the fallback triage becomes a match arm:
`Err(Errno::OPNOTSUPP | Errno::ISDIR | Errno::INVAL)`. Same kernel fact,
three idioms; Chapter 5 turns this three-way comparison into the book's
error-handling policy.
three idioms; this three-way comparison becomes the book's error-handling
policy.

### Concurrency lens

Expand All @@ -330,7 +330,7 @@ C++ and Rust stay a single thread from `execve` to exit, which is exactly
why their traces need no `-f`. The portable lesson is the `EINTR` loops:
once any process has signal handlers (as Go's runtime just demonstrated),
*every* blocking syscall can return early, so retry-on-`EINTR` is not
defensive decoration but the baseline discipline; Chapter 12 (signals)
defensive decoration but the baseline discipline; safe signal handling
makes it precise.

## Build, run, observe
Expand Down Expand Up @@ -382,7 +382,7 @@ distinct boundary signatures: none of them wrong, all of them visible.

## Cross-check: counting every trap

Chapter 1 counted 64 syscalls for the C++ hello and 217 for Go's. `strace
The prerequisites counted 64 syscalls for the C++ hello and 217 for Go's. `strace
-c` on the three `sysprobe` binaries (`-f` for Go) deepens that finding with
a full accounting; trimmed to the key rows from this session's
runs:
Expand All @@ -395,8 +395,8 @@ rust : 73 total — 12 mmap, 5 openat, 6 write, 2 getrandom, 1 nanosleep
```

The program's own work is four steps in every case; the rest is runtime
overhead, and the ratio matches Chapter 1's almost exactly (roughly 4× for
Go, near-parity for C++ and Rust). C++'s 21 `mmap`s are the dynamic loader
overhead, and the ratio almost exactly matches what the prerequisites
counted (roughly 4× for Go, near-parity for C++ and Rust). C++'s 21 `mmap`s are the dynamic loader
mapping four libraries; Rust's 73 is C++-shaped because its *startup* still
goes through glibc even though its syscalls do not. Go's 283 is dominated by
runtime self-management: signals, threads, and its sleeping monitor. As an
Expand All @@ -405,7 +405,7 @@ instructions) confirms *where* each binary keeps its boundary crossings:
nowhere (glibc's), everywhere (a static runtime's), and inline at 32 rustix
call sites. Two tools, one story.

<p><span class="status status--unverified">unverified</span>: <strong>On the lab VM:</strong> the kernel-side view of this chapter (watching <code>sys_enter_*</code> tracepoints fire with <code>bpftrace</code> as sysprobe runs, counting traps from inside the kernel instead of via ptrace) needs root and a disposable kernel, so it is not run here; chapter 30 (Debugging part) exercises exactly this on <code>systems-target</code>.</p>
<p><span class="status status--unverified">unverified</span>: <strong>On the lab VM:</strong> the kernel-side view of this chapter (watching <code>sys_enter_*</code> tracepoints fire with <code>bpftrace</code> as sysprobe runs, counting traps from inside the kernel instead of via ptrace) needs root and a disposable kernel, so it is not run here; the eBPF observation toolkit exercises exactly this on <code>systems-target</code>.</p>

## What you learned

Expand All @@ -421,7 +421,7 @@ call sites. Two tools, one story.
leaves a distinct fingerprint in `strace`.
- `strace -c` puts numbers on runtime overhead: 70 (C++) vs 283 (Go, all
threads) vs 73 (Rust) traps for four steps of real work, the same shape
as Chapter 1's 64-vs-217, now fully attributed.
as the prerequisites' 64-vs-217, now fully attributed.

Next, **errors, three ways**: the `-errno` fact each wrapper just handed us
becomes a policy (`std::expected`, wrapped Go errors, and `thiserror`-style
Expand Down
Loading
Loading