diff --git a/_includes/header.html b/_includes/header.html index 6148fdf..5271ebe 100644 --- a/_includes/header.html +++ b/_includes/header.html @@ -5,6 +5,8 @@ {{ site.title | default: "Tutorial" }} + Home Chapters on () + +`dmesg status=denied` is the expected shape for an unprivileged account: +`analyze` treats a permission-denied tool as a reportable condition, not a +crash. + +## Two modes, one binary + +``` +app --resource cpu|mem|io|net --seconds N saturate that resource +app analyze [--seconds N] run the checklist and name + the saturated resource +``` + +- `--resource cpu` spins a worker pool doing xorshift arithmetic only, no + syscalls, no allocation in the hot loop. +- `--resource mem` touches 1.35x the guest's `MemTotal` across a worker pool, + overshooting available RAM so pages land in swap, without OOM-killing the + guest. +- `--resource io` opens one file per worker with `O_DSYNC` and `pwrite`s a + 16 KiB buffer of xorshift bytes to the same offset in a loop, forcing a + synchronous write on every call. +- `--resource net` floods UDP datagrams over loopback from a pool of senders + to one receiver. + +Every saturator prints a `loadmix: start resource=... seconds=... workers=N` +line, runs for approximately the requested duration, and prints a +`loadmix: done resource=... ops=... bytes=...` line. `analyze` induces no +load of its own; it spends its `--seconds` window watching the system +through the nine tools listed above. + +An unrecognized `--resource`, a missing `--seconds`, `--seconds 0` or +negative, or an unrecognized `analyze` flag all print the two-line usage +banner above to stderr and exit `2`. A clean run of either mode exits `0`. +There is no separate runtime-error exit code: `analyze`'s subprocess calls +degrade instead of failing the whole run, a denied `dmesg`, an empty `ss -s`, +or a sampler with no data rows all show up as a `0.00`/`denied`/empty field +rather than a non-zero exit. + +## The USE threshold table + ``` +cpu Utilization: busy_pct >= 60% Saturation: run_queue >= cpus+2 +mem Utilization: used_pct >= 75% Saturation: swap_io >= 300 KB/s +io Utilization: util_pct >= 40% Saturation: iowait_pct >= 8% +net Utilization: pkts/s >= 2000 Saturation: pkts/s >= 8000 +``` + +cpu's Saturation threshold scales with the guest it runs on, `cpus + 2`, +rather than a flat constant: a run queue longer than the core count plus a +little headroom means threads are waiting for a CPU that isn't there. Errors +is printed for every resource but always `fired=false`, since these +saturators induce load, not kernel-logged faults; the line keeps the +Utilization/Saturation/Errors triple complete. The final verdict picks the +resource with the highest Saturation-value-to-threshold *ratio*, not the +highest raw Saturation value, so net's 8000-scale threshold never automatically +loses to cpu's 4-scale one. + +## Parsing by column name, not position + +Every parser reads a tool's header row first and looks fields up by name +against it, never by a hardcoded index. `Row` (C++), `row` (Go), and `Row` +(Rust) all express the same idea: column names from a tool's own header line, +zipped with that row's values, so a `sysstat` or `procps-ng` version bump +that reorders columns doesn't break the parser. + +`parse_mpstat` is the clearest case: `mpstat -P ALL`'s `Average:` rows begin +with a label token (`Average: CPU` / `Average: all`) that has to be skipped +before `%idle` lines up with the real idle percentage, and `busy_pct` is +computed as `100.0 - %idle` rather than summing `%usr + %sys + %nice + ...`, +since idle's complement is "everything else" regardless of what utilization +sub-categories a future sysstat release adds. `vmstat` has no `Average:` row +at all and its first sample is a since-boot average, so its parser discards +row 0 and takes the max run-queue value across the rest. `iostat -xz` has the +same since-boot-first-row quirk, plus `-z` hides idle devices, so its parser +tracks block boundaries by index rather than by content. -via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error -handling (`std::expected` / wrapped errors via `%w` / `rustix`). +## Concurrency, two ways + +`analyze`'s five interval samplers (`vmstat`, `mpstat -P ALL`, `iostat -xz`, +`sar -n DEV`, `pidstat`, each run as ` 1 N`) are real subprocesses +spawned concurrently and joined once, not run one after another; that's the +whole reason a `--seconds 10` checklist finishes in about 10 seconds instead +of the roughly 50 seconds five serial invocations would cost. + +Each saturator's worker pool is a second, unrelated concurrency story: +compute a deadline, spawn workers that loop until stopped or the deadline +passes, then join. C++'s pool is `std::jthread`, whose RAII destructor runs +`request_stop()` before `join()`, so without `std::this_thread::sleep_until(deadline)` +holding the parent scope open, every worker would be told to stop the +instant the spawn loop ends, in single-digit milliseconds, regardless of +`--seconds`. Go and Rust don't need that pattern: each worker checks its own +deadline directly inside its loop condition, so the parent's `wg.Wait()` or +`thread::scope` return is a rendezvous, not part of the termination signal. ## Layout ``` -_template/ +37-use-method-60s-analysis/ ├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] -├── verify.lua # automated check driven by CI -├── cpp/ # CMake preset build, demo.sh -├── go/ # go build, demo.sh -└── rust/ # cargo build, demo.sh +├── verify.lua # automated check, driven against the systems-target VM +├── cpp/ # CMake preset build (src/main.cpp), demo.sh +├── go/ # go build (main.go), demo.sh +└── rust/ # cargo build (src/main.rs), demo.sh ``` ## The demo contract @@ -36,16 +145,47 @@ across the whole book: The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs `cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. +This is a VM example: the saturation signals only mean something measured +against a real, otherwise-quiet kernel. `verify.lua` does not drive it +through `demo.sh run TARGET=...`, since `deploy-to-vm.sh` wraps +commands in `ssh -t`, which injects CRLF that breaks the tool's plain-text +output. Instead it stages the built binary directly to +`/tmp/lsp37-/app` on `systems-target` with plain `scp`/`ssh` and drives +it there. + ## Verification -`verify.lua` is run from this directory with `LSP_LANG` set to one language at -a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and -asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, -77 = skip. +`verify.lua` runs with `LSP_LANG` set to one language at a time and `TARGET` +pointing at the lab VM. It builds, stages the binary, then asserts: + +- all four usage-error shapes (no args, a bad `--resource`, `--seconds 0`, a + bad `analyze` flag) exit `2` with the two-line usage banner on stderr +- `--resource cpu|io|net --seconds 4` each exit `0`, print the documented + worker-count formula for the guest's real `nproc`, report `ops > 0`, and + finish within a tight window around the requested `--seconds` (this is + exactly what would have caught a saturator that exited in milliseconds + regardless of the requested duration) +- `--resource mem` exits `0` without OOM-ing the guest, matches its own + worker-count formula, and stays under a generous ceiling (`ops > 0` is not + required for mem: a short duration can spend the whole budget on the + initial allocation before any touch pass completes) +- an idle `analyze --seconds 8` prints the exact line skeleton, 1 start line, + 5 tool lines, 8 metric lines, 12 signal lines in cpu/mem/io/net x + Utilization/Saturation/Errors order, 1 verdict line, `analyze: done` as the + last line of the app's own output, and finishes well under `5 x --seconds`, + proving the five samplers ran concurrently rather than serially +- under a concurrent background `--resource cpu` saturator, a second + `analyze` run attributes the load: `busy_pct >= 60` or + `run_queue >= nproc + 2`, Utilization `fired=true`, and `verdict=cpu`, + retried up to 3 attempts against a margin calibrated from real captures + +`python3 scripts/test-all-examples.py --only 37-use-method-60s-analysis +--mode vm` builds all three and runs `verify.lua` per language against +`systems-target`. ## Mapping to a chapter -Each chapter's example is a copy of this template (`scripts/new-example.sh -NN-slug`) with the program body replaced by that chapter's code. Keep the -demo contract and `verify.lua` shape intact so CI and the manifest -(`examples/manifest.yaml`) keep working. +This example backs `_docs/37-use-method-60s-analysis.md`, "The USE method in +60 seconds." The chapter walks through the column-name parsing discipline, +the USE threshold table, and the `std::jthread` RAII ordering issue in the +saturator pool in more depth than this README does. diff --git a/examples/38-opentelemetry-to-lgtm/README.md b/examples/38-opentelemetry-to-lgtm/README.md index 6c41638..4e58db5 100644 --- a/examples/38-opentelemetry-to-lgtm/README.md +++ b/examples/38-opentelemetry-to-lgtm/README.md @@ -1,21 +1,91 @@ -# _template: hello-syscall +# 38-opentelemetry-to-lgtm: lsp-otel, a traced server whose export gets checked, not trusted -The seed for every runnable example in the book. It implements the same tiny -program three times, in C++23, Go, and Rust, each printing exactly one line: +`lsp-otel` is a traced request/response server. Every request it handles +produces a parent span wrapping two child spans, a counter increment, and a +histogram observation, all exported over OTLP/HTTP to the local LGTM stack +(Loki, Grafana, Tempo, Mimir plus an OpenTelemetry collector). The point of +the example is the verification, not the server: it queries Tempo and +Prometheus through Grafana's datasource proxy to confirm the export actually +landed, instead of trusting the program's own clean exit. +```console +[host]$ ./go/bin/app serve --port 8080 & +app: serve listening on 127.0.0.1:8080 +[host]$ ./go/bin/app drive --n 10 --addr 127.0.0.1:8080 +app: drive sent 10/10 ok +[host]$ kill -INT %1 +app: shutting down (interrupt) ``` -pid on () -``` -via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error -handling (`std::expected` / wrapped errors via `%w` / `rustix`). +`serve --port PORT [--otel-endpoint URL]` accepts connections and traces each +one; `--otel-endpoint` defaults to `http://localhost:4318`, the collector the +`lsp-lgtm` stack publishes. `drive --n N --addr HOST:PORT` is a plain TCP +client that sends `N` requests and reads `N` replies; it stays untraced by +design, since the service under observation is `serve`, not the thing poking +it. This is a **local-mode example that requires the LGTM stack**: both +subcommands run on the host, and the exporter posts to the collector on +`localhost:4318`. + +## Three signals, one request + +Each request `serve` handles starts a parent span named `request`, carrying a +`request.seq` attribute, with two children nested inside it: + +- **`work`**: a tiny `1 + seq%4` millisecond sleep standing in for actual + computation. +- **`respond`**: writing the `ok ` reply line. + +Alongside that span tree, every request records two metrics: `requests_total`, +a monotonic counter that goes up by one per request, and `request_duration`, +a histogram (in milliseconds) that records one observation per request, the +parent span's wall-clock duration. The counter answers "how many," the +histogram answers "how long," and Tempo later shows the parent/child nesting +itself: whether a request's time went into `work` or `respond`. + +## Carrying the parent span, three ways + +One function, `handleRequest`/`handle_request`, starts the parent span, nests +the two children, writes the reply, and records both metrics. The API shape +is the same across all three languages; how the parent/child relationship is +carried is not: + +- **Go** threads an explicit `context.Context`. `t.tracer.Start(ctx, "request", ...)` + returns a *new* context carrying the parent span, and passing that context + into each child `Start` call is what makes it a child. +- **Rust** makes the same context explicit: `Context::current_with_span(parent_span)` + builds a context holding the parent, and each child is started with + `start_with_context(&t.tracer, &parent_cx)`. +- **C++** uses a `Scope` guard. Constructing `trace_api::Scope parent_scope(parent_span)` + installs the parent as the current span for the lifetime of that RAII + object, so children started while the scope is alive attach to it + implicitly. + +Three mechanisms, the same `request -> {work, respond}` tree. + +## Exporting over OTLP/HTTP + +Each language's telemetry init wires up two exporters (traces and metrics) +against a shared resource, the set of attributes identifying this service in +the stack. The one that matters here is `service.name`, set differently per +language by design: `lsp-otel-cpp`, `lsp-otel-go`, `lsp-otel-rust`. All three +export into the same shared LGTM stack, so the distinct name is what keeps +their signals from colliding and lets verification query each language's +telemetry on its own. + +Both exporters speak OTLP over HTTP/protobuf to the collector and batch on a +short (~2 second) interval, a batch span processor for traces and a periodic +reader for metrics, instead of the SDK defaults (5-second span batches, +60-second metric batches), which are too slow for a demo that starts, drives +ten requests, and stops within a couple of seconds. A flush on shutdown, tied +to the SIGINT handler, is what guarantees the last batch leaves the process +before it exits. ## Layout ``` -_template/ +38-opentelemetry-to-lgtm/ ├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] -├── verify.lua # automated check driven by CI +├── verify.lua # automated check: drives serve+drive, queries Tempo+Prometheus ├── cpp/ # CMake preset build, demo.sh ├── go/ # go build, demo.sh └── rust/ # cargo build, demo.sh @@ -23,29 +93,39 @@ _template/ ## The demo contract -Every language directory has a `demo.sh` with the same interface, identical -across the whole book: - -- `./demo.sh`: build then run -- `./demo.sh build`: build only -- `./demo.sh run [args]`: run the built binary -- With env `TARGET` set, `run` deploys the binary to that lab VM via - `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of - running locally. +Every language directory has a `demo.sh` with the interface used across the +book: `./demo.sh` (build then run), `./demo.sh build` (build only), and +`./demo.sh run [args]` (run the built binary; with `TARGET` set, deploy to a +lab VM instead). The top-level `demo.sh` here is only a dispatcher. -The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs -`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. +`serve`'s exit codes: a usage error (no subcommand, `serve` without `--port`, +`drive` with `--n 0` or missing) prints the two-line usage banner and exits +`2`; a runtime error (`drive` against a dead address) exits `1` with an +`app: error: …` prefix; a clean run exits `0`, `serve` after SIGINT, `drive` +after `N/N` successful replies. ## Verification -`verify.lua` is run from this directory with `LSP_LANG` set to one language at -a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and -asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, -77 = skip. +`verify.lua` runs from this directory with `LSP_LANG` set to one language at +a time. It first checks Grafana's `/api/health` endpoint; if the LGTM stack +isn't reachable, the whole example SKIPs rather than failing. Against a live +stack, it builds, checks the usage/error exit codes and messages, then runs +`serve --port 47380` in the background, drives ten requests through it, +sends SIGINT, and waits for `serve` to exit `0` after its flush. + +With the data presumably exported, it queries the backend instead of trusting +`serve`'s output: `requests_total{service_name="lsp-otel-"}` and +`request_duration_milliseconds_count{service_name="lsp-otel-"}` through +Prometheus, and a trace search with `service.name=lsp-otel-` through +Tempo, both via Grafana's datasource proxy (`/api/datasources/proxy/uid/prometheus/...` +and `.../uid/tempo/...`) on `localhost:3000` with `admin:admin`. Each query is +polled with a bounded retry to absorb the exporter's batch interval and the +backends' own ingest lag. The example passes only once both counters read at +least 10 and at least one matching trace shows up in Tempo. ## Mapping to a chapter -Each chapter's example is a copy of this template (`scripts/new-example.sh -NN-slug`) with the program body replaced by that chapter's code. Keep the -demo contract and `verify.lua` shape intact so CI and the manifest -(`examples/manifest.yaml`) keep working. +This is the example for the OpenTelemetry-to-LGTM chapter in the +Observability part: instrumenting a request/response server with traces and +metrics, exporting them over OTLP/HTTP, and verifying the export by reading +the backend back instead of trusting the program's own word. diff --git a/examples/40-low-latency-fastpath/README.md b/examples/40-low-latency-fastpath/README.md index 6c41638..4c2f74d 100644 --- a/examples/40-low-latency-fastpath/README.md +++ b/examples/40-low-latency-fastpath/README.md @@ -1,51 +1,253 @@ -# _template: hello-syscall +# 40-low-latency-fastpath: chatterd-fastpath, an echo server with nothing left to remove -The seed for every runnable example in the book. It implements the same tiny -program three times, in C++23, Go, and Rust, each printing exactly one line: +`chatterd-fastpath` takes the echo hot path from `chatterd` (the book's chat +server) and strips it to three moves: pin the process to one CPU, reuse a +single stack buffer for the life of a connection, and, optionally, spin on a +non-blocking `recv` instead of sleeping in a blocking `read`. A `naive` server +in the same binary does the opposite of each move, so the two can be raced +under identical placement and the gap between them measured. ``` -pid on () +[host]$ cd examples/40-low-latency-fastpath && ./demo.sh build ``` -via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error -handling (`std::expected` / wrapped errors via `%w` / `rustix`). +The interesting run happens on the lab guest, where `run-latency-bench.sh` +brings up both servers and reads their pinning back from the kernel before +timing them: -## Layout +``` +[vm]$ bash run-latency-bench.sh ./app 20000 500 +app: naive-cpus-allowed 0-1 +app: fastpath-cpus-allowed 1 +app: percentiles_ns tag=naive-1 p50=11957 p90=15325 p99=28509 p99.9=80747 min=10075 max=442827 mean=12975.38 n=20000 +app: percentiles_ns tag=fastpath-1 p50=9282 p90=11225 p99=16626 p99.9=47039 min=6900 max=87199 mean=9607.65 n=20000 +app: percentiles_ns tag=naive-2 p50=11135 ... n=20000 +app: percentiles_ns tag=fastpath-2 p50=7936 ... n=20000 +app: percentiles_ns tag=naive-3 p50=11320 ... n=20000 +app: percentiles_ns tag=fastpath-3 p50=8184 ... n=20000 +``` + +`naive-cpus-allowed` comes back `0-1`, the full online set, because the naive +server never touches its own affinity mask. `fastpath-cpus-allowed` comes back +`1`, a single CPU, because `sched_setaffinity(2)` narrowed it. Both lines are +read from `/proc//status` by the driver script, not printed by the +program itself. + +## The frame and the three subcommands + +One binary, three subcommands, one fixed 64-byte frame: ``` -_template/ -├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] -├── verify.lua # automated check driven by CI -├── cpp/ # CMake preset build, demo.sh -├── go/ # go build, demo.sh -└── rust/ # cargo build, demo.sh +app fastpath --port P --pin CPU [--busy-poll] +app naive --port P +app measure --target HOST:PORT --n N [--warmup W] [--tag TAG] ``` -## The demo contract +`fastpath` pins itself to `CPU` with `sched_setaffinity(2)`, reuses one +stack-allocated buffer for every message, and with `--busy-poll` spins a +non-blocking `recv(2)` instead of blocking on `read(2)`. `naive` is the +baseline chatterd hot loop would run: default scheduling, a fresh heap buffer +per message, a blocking `read(2)`. `measure` connects to either server and +runs `N` synchronous ping-pong round trips, never more than one frame in +flight, timing each against `CLOCK_MONOTONIC` and printing a +coordinated-omission-corrected `p50`/`p90`/`p99`/`p99.9` table in nanoseconds. +`W` warmup round trips (500 by default under the bench driver) are discarded +before recording starts. + +The wire frame is this example's own format, a smaller, fixed-size cousin of +the chatterd chat frame from the TCP chapter: + +``` ++-------+---------+------+-----------+---------------+---------+ +| magic | version | type | seq (u64) | send_ns (u64) | pad | +| 2B | 1B | 1B | 8B BE | 8B BE | 44B | ++-------+---------+------+-----------+---------------+---------+ +``` + +Magic is `CF` (chatterd-fastpath). The server never inspects `seq` or +`send_ns`: it reads 64 bytes and writes the identical 64 bytes back. All +timing happens on the client's own clock; `seq` is only echoed back so +`measure` can confirm it got its own frame back rather than another +connection's or garbage, a correctness check, not a timing mechanism. -Every language directory has a `demo.sh` with the same interface, identical -across the whole book: +## Three moves that shrink the tail -- `./demo.sh`: build then run -- `./demo.sh build`: build only -- `./demo.sh run [args]`: run the built binary -- With env `TARGET` set, `run` deploys the binary to that lab VM via - `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of - running locally. +Each move removes one specific, nameable source of tail latency, and each has +a real cost: -The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs -`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. +- **Pin to one CPU** (`sched_setaffinity(2)`). An unpinned thread can migrate + between messages onto a core that needs to come back from a low C-state and + holds none of the working set in cache, and that migration is itself a + tail-latency event. Pinning trades the freedom to flee a noisy neighbor for + the removal of migration as a source of jitter. +- **Reuse one buffer.** `fastpath` reads into a single 64-byte stack array for + the whole life of a connection; `naive` allocates a fresh heap buffer + (`make_unique` in C++, `make([]byte, …)` in Go, `vec![0u8; …]` in Rust) on + every message and lets it fall out of scope. At 64 bytes the allocator cost + is small but variable: most calls hit a thread-local cache, the occasional + one grabs the arena lock or faults in a page, and that variance lands in the + tail. +- **Busy-poll** (`--busy-poll`). A blocking `read(2)` on an idle socket puts + the thread to sleep; waking it for the next frame costs a scheduler round + trip, a voluntary context switch, on every message. Busy-poll never sleeps: + it spins on a non-blocking `recv` that returns `EAGAIN` until data arrives, + burning the whole pinned core to keep the thread on-CPU and ready. It is the + most expensive move, a dedicated core, and the one with the largest effect + on median latency. + +## Proving pinning and busy-poll from the kernel, not the program + +Both claims are checked against `/proc//status`, never against anything +the server prints about itself. + +`Cpus_allowed_list` proves the pin: if `sched_setaffinity` had silently +failed, `fastpath-cpus-allowed` would read `0-1` like the naive server instead +of a single CPU. + +`voluntary_ctxt_switches` proves busy-poll never sleeps. Running a +20,000-message `measure` against each server, both pinned to CPU 1 with +`taskset`, and reading the counter before and after: + +``` +[vm]$ # naive (blocking read) +before: voluntary_ctxt_switches: 2 +after: voluntary_ctxt_switches: 20500 + +[vm]$ # fastpath (busy-poll) +before: voluntary_ctxt_switches: 2 +after: voluntary_ctxt_switches: 3 +``` + +The blocking server takes one voluntary context switch per message, 20,498 of +them to serve 20,000 requests plus warmup. The busy-poll server sleeps once +for the whole run. On this run the naive `p50` was 11,426 ns and the fastpath +`p50` was 8,148 ns; to a first approximation, the ~3,300 ns gap is the +sleep/wake round trip through the scheduler that busy-poll declined to take +20,000 times. + +## The CPU count that lies about a `taskset` mask + +`pin_to_cpu`'s range check needs a CPU count, and the correct count is the +number of CPUs online system-wide, because the pin argument is an absolute CPU +index. C++ and Rust both call `sysconf(_SC_NPROCESSORS_ONLN)`, which returns +that number (2 on the lab guest) and is blind to the caller's own affinity +mask. Go's obvious equivalent, `runtime.NumCPU()`, is not the same thing: it +returns the size of the process's CPU affinity mask instead. The two numbers +agree on an idle machine and diverge the moment something narrows the mask, +which is exactly what `run-latency-bench.sh` does by launching each server +under `taskset -c 1`. Under that mask, `runtime.NumCPU()` returns `1`, and a +naive port's own range check would reject the `--pin 1` it was told to use: + +``` +app: error: cpu 1 out of range (0..0) +``` + +The fix reads the kernel's own account of the online set instead: +`onlineCPUs()` in the Go port parses `/sys/devices/system/cpu/online` and +falls back to `runtime.NumCPU()` only if that read fails, matching the +`sysconf` semantics the other two languages get automatically. The Go port also +carries two lines neither C++ nor Rust needs: `sched_setaffinity(2)` acts on a +thread, and a goroutine is not a thread, so `runtime.LockOSThread()` pins the +hot loop's goroutine to one OS thread and `GOMAXPROCS(1)` keeps the runtime +from parking other work on other cores before the affinity call means +anything. The affinity call itself runs twice, once against the calling +thread (which restricts the loop) and once against the process (the mask +`/proc//status` reports back). + +## Shutdown: three mechanisms, one observable outcome + +The hot loop is single-threaded by design in all three languages; the +interesting concurrency is in shutdown. C++ and Rust install a `SIGINT` +handler with `sa_flags` explicitly **not** `SA_RESTART`, so a blocking +`read(2)` or `poll(2)` already in flight returns `EINTR` instead of resuming, +the loop sees a stop flag, and it exits. Rust reaches for this directly +through `nix`-owned raw file descriptors, because `std`'s higher-level +wrappers (`TcpListener::accept`, `Read::read_exact`) retry `EINTR` +internally. + +Go cannot use that technique: its runtime installs its own signal handlers +with `SA_RESTART`, and a "blocking" read on a `net.Conn` parks the goroutine +on an `epoll` registration rather than sitting in a syscall a signal could +interrupt. So the Go port reaches the same outcome, a clean +`app: … shutting down` message and exit 0, by a different route: its signal +handler sets a stop flag and closes the listener and the live connection. +Closing an fd is what wakes a poller-parked read. The busy-poll path needs one +extra step, since its socket is a `dup`'d raw fd outside the poller: +`trigger()` also calls `shutdown(SHUT_RDWR)` on it to break the spin. + +## Errors, three ways, same bytes + +A usage error (no subcommand, an unknown one, `fastpath` without `--pin`, +`naive` without `--port`, `measure` without `--n`) prints a three-line banner +to stderr and exits `2`: + +``` +usage: app fastpath --port P --pin CPU [--busy-poll] + app naive --port P + app measure --target HOST:PORT --n N [--warmup W] [--tag TAG] +``` + +A runtime error exits `1` with an `app: error: …` message: `cpu 99 out of +range (0..1)` from the pin range check, `connect 127.0.0.1:1: Connection +refused` when `measure` can't reach a server, `notanip: not an IPv4 address` +when the target host isn't a dotted quad (the servers call `inet_pton`, not a +DNS lookup, so a hostname is a deliberate error), and +`measure: connection closed at iteration N` if a server dies mid-run. C++ +returns `std::expected`, Go returns `error` with the +`strerror` text unwrapped so the message reads like C's, and Rust returns +`Result` pre-formatted at the failure site; the bytes on stderr are +identical across all three, which `verify.lua` asserts directly. + +## Layout + +``` +40-low-latency-fastpath/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── run-latency-bench.sh # staged to the lab guest and run there; the real benchmark +├── verify.lua # behavioral checks, driven against the lab guest +├── cpp/ # CMake preset build, src/main.cpp, demo.sh +├── go/ # go build, main.go, demo.sh +└── rust/ # cargo build, src/main.rs, demo.sh +``` + +## The demo contract + +`./demo.sh [cpp|go|rust] build` builds one language; `./demo.sh build` builds +all three. `./demo.sh [cpp|go|rust] run [args]` runs the built binary +directly; bare `./demo.sh` builds and runs all three with no arguments, which +only exercises the usage banner, since none of the three subcommands has a +sensible zero-argument default. The real measurement lives in +`run-latency-bench.sh`, which this example is a **VM example** for: the +timing numbers only mean something on a quiet, pinned lab guest, so +`verify.lua` builds locally, stages the binary and the driver script to +`systems-target` over `scp`, and runs everything there. ## Verification -`verify.lua` is run from this directory with `LSP_LANG` set to one language at -a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and -asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, -77 = skip. +`verify.lua` builds each language, stages the binary and +`run-latency-bench.sh` to `/tmp/lsp40-/` on the lab guest, and asserts, +per language: the five usage-error shapes exit `2` with the banner; an +out-of-range `--pin` exits `1` with `cpu N out of range`; a refused connect +and a non-IPv4 `measure` target each exit `1` with the matching `app: error:` +text; the real `run-latency-bench.sh` run exits `0`; `Cpus_allowed_list` from +`/proc//status` shows the naive server with the full online CPU set and +the fastpath server with exactly one; all six `percentiles_ns` lines parse +with `n=20000` and monotonic percentiles; the median of three fastpath `p50`s +beats the median of three naive `p50`s by at least 10%; and a bounded +10-second wait after `SIGINT` confirms both servers exit `0` with their +shutdown line, the check most likely to catch a Go port that forgot to close +its listener on signal. + +``` +[host]$ python3 scripts/test-all-examples.py --only 40-low-latency-fastpath --mode vm +verifying... + verify 40-low-latency-fastpath [cpp]: PASS + verify 40-low-latency-fastpath [go]: PASS + verify 40-low-latency-fastpath [rust]: PASS +``` ## Mapping to a chapter -Each chapter's example is a copy of this template (`scripts/new-example.sh -NN-slug`) with the program body replaced by that chapter's code. Keep the -demo contract and `verify.lua` shape intact so CI and the manifest -(`examples/manifest.yaml`) keep working. +This is the example for "The low-latency fast path: pin a core, reuse one +buffer, and stop asking the scheduler for permission" +(`_docs/40-low-latency-fastpath.md`), in the Performance and Low Latency part. diff --git a/examples/41-capstone-fleet/README.md b/examples/41-capstone-fleet/README.md index 6c41638..56e08cc 100644 --- a/examples/41-capstone-fleet/README.md +++ b/examples/41-capstone-fleet/README.md @@ -1,21 +1,162 @@ -# _template: hello-syscall +# 41-capstone-fleet: the final project fleet -The seed for every runnable example in the book. It implements the same tiny -program three times, in C++23, Go, and Rust, each printing exactly one line: +This is the final project: one binary, four subcommands, that assembles four +of the book's six recurring programs (`pmon`, `chatterd`, `sysagent`, +`fwatch`) into a single supervised system and runs it on two real hosts. +`pmon` is the fleet's own init: it drops the capability bounding set and sets +`no_new_privs`, then self-re-execs the same binary three times with three +different `argv[0]` commands, restarting whichever child exits unexpectedly +and forwarding its own `SIGTERM`/`SIGINT` to every child it supervises. ``` -pid on () +app pmon [--node NAME] [--sandbox-dir DIR] [--peer HOST:PORT] [--peer-node NAME] + [--chatterd-port P] [--health-interval-ms N] +app chatterd serve [--host H] [--port P] [--node NAME] [--peer HOST:PORT] [--peer-node NAME] +app chatterd send --host H --port P --nick NICK --text TEXT [--timeout-ms T] +app chatterd listen --host H --port P --nick NICK [--timeout-ms T] +app sysagent [--node NAME] [--interval-ms N] [--once] +app sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M] +app fwatch snapshot DIR +app fwatch watch DIR [--sandbox] [--timeout-ms T] ``` -via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error -handling (`std::expected` / wrapped errors via `%w` / `rustix`). +A loopback run shows the whole fleet in one place: + +```console +$ ./go/bin/app pmon --node p1 --sandbox-dir /tmp/fwatch-sandbox +pmon: capabilities bounding_set_dropped=0 no_new_privs=1 +pmon: started service=chatterd pid= +chatterd: listening on 0.0.0.0:47100 node=p1 +pmon: started service=sysagent pid= +sysagent: node=p1 cpu_pct= mem_pct= load1= ts= +pmon: started service=fwatch pid= +fwatch: landlock ABI=9 enforced dir=/tmp/fwatch-sandbox +pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0 +^C +pmon: shutdown +``` + +`pmon` never execs a *different* binary for the other three services: it +resolves its own running path (`os.Executable()` in Go, +`readlink("/proc/self/exe")` in C++ and Rust) and forks+execs that same path +with three different argument lists, the same self-re-exec technique a +container entrypoint uses elsewhere in this book. + +## pmon: capability drop, self-re-exec, supervise, forward signals + +Before forking anything, `pmon` loops `prctl(PR_CAPBSET_DROP, c, …)` across +every known capability and ignores `EINVAL` once it runs past whatever the +running kernel's last capability is, then calls +`prctl(PR_SET_NO_NEW_PRIVS, 1, …)`. `PR_CAPBSET_DROP` removes a capability +from the *bounding set*, not the current effective set: the guarantee is that +no future `execve` of a setuid-root binary, by this process or any +descendant, can ever regain that capability, because the bounding set only +shrinks across `fork`/`exec`. That guarantee holds even on an +already-unprivileged account, which is why a loopback run prints +`pmon: capabilities bounding_set_dropped=0 no_new_privs=1`: `0` is correct +here, there was nothing left to drop, and the line still proves the `prctl` +calls ran. + +Each of the three services then gets its own supervisor (a goroutine in Go, +a `std::thread` in C++ and Rust) running the same loop: fork+exec, record the +pid, print `pmon: started service=NAME pid=PID`, block on the child's exit, +and if the exit was not requested, count a restart, print +`pmon: restart service=NAME attempt=N reason=…`, back off 300ms, and retry. +A single failed start is logged and retried, never treated as fatal, because +a supervisor's job is to keep trying. `SIGTERM`/`SIGINT` are consumed once +(a dedicated signal-handling goroutine/thread, not a signal handler) and +forwarded to every recorded child pid, and `pmon` prints a +`pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0` +line on every `--health-interval-ms` tick until shutdown, when it prints +`pmon: shutdown` and exits `0`. + +## chatterd, bridged across two hosts + +With `--peer HOST:PORT`/`--peer-node NAME` set on both sides, `chatterd` +bridges across two nodes with no protocol change: a **bridge worker** dials +the other node's server and joins its client registry under the nick +`bridge@`. A local `MSG` is broadcast as `DELIVER` to every +registered connection, including a bridge connection dialed in from the +other side, exactly like any other client; when a bridge worker itself +receives a `DELIVER` over the connection it dialed, it re-broadcasts locally +with `includeBridges=false`, so a message never bounces back out over the +same bridge. The one new piece of logic is tagging: if the delivered nick +has no `@` in it yet, the bridge worker tags it with the *sending* node's +name before re-broadcasting. A listener on the far side sees +`chatterd: received from=dave@target text=…`: the `@target` suffix is the +proof the message crossed the wire, not that it was echoed locally. Either +side can dial first, or restart under `pmon`, because the bridge worker just +retries on a backoff until the connection succeeds. + +## sysagent: cpu/mem/load1 as OTel gauges + +`sysagent` reads the same three signals it always has (`cpu_pct` from +`/proc/stat`, `mem_pct` from `/proc/meminfo`, `load1` from `/proc/loadavg`) +and prints them every `--interval-ms` as +`sysagent: node=… cpu_pct=… mem_pct=… load1=… ts=…`. When +`OTEL_EXPORTER_OTLP_ENDPOINT` is set, those same three numbers are also +registered as OTel observable gauges (`sysagent.cpu.pct`, `sysagent.mem.pct`, +`sysagent.load1`), tagged with a `node` attribute and exported over +OTLP/HTTP on a periodic reader; with no endpoint set, `sysagent` prints +`sysagent: otel disabled (OTEL_EXPORTER_OTLP_ENDPOINT not set)` and runs +otherwise unchanged. `sysagent saturate --resource cpu|mem --seconds N +[--workers K|--mb M]` spins worker threads or holds a resident allocation so +a concurrently-running `sysagent`'s own gauges visibly move. + +## fwatch under an optional Landlock ruleset + +`fwatch watch DIR --sandbox` installs a Landlock ruleset scoped to the +watched directory before the `inotify` loop starts. It probes the running +kernel's enforced ABI with +`landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, builds +the ruleset around that one directory, and calls `landlock_restrict_self` +after setting `no_new_privs`. `fwatch: landlock ABI=N enforced dir=…` prints +only once the kernel has accepted the restriction, and `N` is not a constant +the program carries: it is whatever the running kernel actually supports. +Capability-bounding-set drop (applied once, by `pmon`, before any child +exists) and Landlock (applied per-process, by `fwatch` itself) are this +fleet's two sandbox layers; seccomp is proven in an earlier example and is +not reapplied here. + +## The two-VM lab + +`examples/manifest.yaml` registers this example as `mode: vm-peer`, +`requires: [lgtm]`: the real demo runs `pmon` on both `systems-target` +(192.168.124.7) and `systems-peer` (192.168.124.95), bridged to each other, +each exporting OTLP through the libvirt gateway (192.168.124.1) to the +LGTM stack running on the host. + +```console +[host]$ scp app fedora@192.168.124.7:~/app +[host]$ scp app fedora@192.168.124.95:~/app +[vm]$ OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1:4318 \ + ./app pmon --node target --peer 192.168.124.95:47100 --peer-node peer +[peer]$ OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1:4318 \ + ./app pmon --node peer --peer 192.168.124.7:47100 --peer-node target +``` + +Both guests reach full health (`pmon: health chatterd=up sysagent=up +fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0`), and a message sent on +`target` is delivered to a listener on `peer`: + +```console +[vm]$ ./app chatterd send --host 127.0.0.1 --port 47100 --nick dave --text "hello across hosts" +[peer]$ ./app chatterd listen --host 127.0.0.1 --port 47100 --nick carol +chatterd: received from=dave@target text=hello across hosts +``` + +`fwatch --sandbox` on the lab guest's kernel (6.19.10) enforces a different +Landlock ABI than the host's own loopback run: the identical probe, two +different kernels, by design, not a discrepancy. `sysagent`'s gauges reach +the host's Prometheus labeled by node (`sysagent_cpu_pct{node="peer"}`), +confirming the guest-to-host OTLP path over the libvirt gateway. ## Layout ``` -_template/ +41-capstone-fleet/ ├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] -├── verify.lua # automated check driven by CI +├── verify.lua # automated check driven by CI, run entirely on loopback ├── cpp/ # CMake preset build, demo.sh ├── go/ # go build, demo.sh └── rust/ # cargo build, demo.sh @@ -23,29 +164,52 @@ _template/ ## The demo contract -Every language directory has a `demo.sh` with the same interface, identical -across the whole book: +Each language directory's `demo.sh` has the same interface as every other +example: `build`, `run [args]`, or bare (build then run). With `TARGET` set, +`run` deploys the built binary to that lab VM via +`scripts/lab/deploy-to-vm.sh` instead of running locally, which is how `app` +gets staged onto `systems-target` and `systems-peer` for the two-host run +above. The top-level `demo.sh` here is only a dispatcher. -- `./demo.sh`: build then run -- `./demo.sh build`: build only -- `./demo.sh run [args]`: run the built binary -- With env `TARGET` set, `run` deploys the binary to that lab VM via - `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of - running locally. +## Verification -The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs -`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. +`verify.lua` follows the convention established elsewhere in the book for +`vm-peer` examples: the assertions themselves run entirely on loopback, +including the cross-node bridge (two nodes on distinct `127.0.0.1` ports) +and telemetry to a local LGTM stack, while the real two-host run is the +chapter's demo. Per language, it asserts: -## Verification +1. **Usage and error shapes**: no args, `chatterd`/`fwatch` with no + subcommand, `chatterd send` missing `--nick`/`--text`, and `sysagent + saturate` missing `--resource` all exit 2 with the matching usage banner + (`usage: app …` or `sysagent saturate`'s own, narrower banner). +2. **`sysagent --once`**: exit 0, the `sysagent: node=… cpu_pct=…` line, and + `otel disabled` with no endpoint set. +3. **`fwatch`**: `snapshot DIR` lists directory entries; `watch DIR` reports + `event: create|modify|delete PATH` lines; `watch DIR --sandbox` installs a + Landlock ruleset and prints `fwatch: landlock ABI=N enforced dir=…`. +4. **`chatterd`**: a message sent to one node is received locally as + `received from=NICK text=…`, and across the bridge (node A `--peer->` + node B) a message sent on A is delivered on B as + `received from=dave@A text=…`. +5. **The `pmon` fleet**: `pmon: capabilities bounding_set_dropped=N + no_new_privs=1`, all three services start, health reaches + `chatterd=up sysagent=up fwatch=up`, and `SIGTERM` produces + `pmon: shutdown` and exit 0. +6. **Telemetry**: with `OTEL_EXPORTER_OTLP_ENDPOINT` set, `sysagent` prints + `otel enabled` and its `sysagent_cpu_pct` gauge reaches Prometheus through + Grafana's proxy. Skipped, not failed, if the LGTM stack is unreachable. -`verify.lua` is run from this directory with `LSP_LANG` set to one language at -a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and -asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, -77 = skip. +`python3 scripts/test-all-examples.py --only 41-capstone-fleet --mode vm` +builds and verifies all three languages. -## Mapping to a chapter +## Mapping to the book -Each chapter's example is a copy of this template (`scripts/new-example.sh -NN-slug`) with the program body replaced by that chapter's code. Keep the -demo contract and `verify.lua` shape intact so CI and the manifest -(`examples/manifest.yaml`) keep working. +This example closes the arc that runs through the whole book: `pmon` +(fork/exec/wait, privilege drop, namespaces), `chatterd` (TCP, then epoll, +UDP peer discovery, async, a pinned-core fast path), `sysagent` (`/proc` and +`/sys/fs/cgroup` reading, then exporting), and `fwatch` (polling, then +`inotify`, then a Landlock sandbox around itself) each grew on its own in +earlier examples. This one does not add a new syscall; it assembles those +four into one supervised, sandboxed, observed system running on two real +hosts instead of four separate demos on one.