Build SC Linux as a structured, actionable Jupyter Book

.github/workflows/book.yml

@@ -0,0 +1,21 @@
+name: Jupyter Book
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+permissions:
+  contents: read
+jobs:
+  book:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+      - name: Install pinned Jupyter Book
+        run: |
+          python3 -m venv .venv
+          .venv/bin/python -m pip install --disable-pip-version-check -r requirements-book.txt
+      - name: Build the static book without execution
+        run: .venv/bin/jupyter book build --html --strict

.gitignore

@@ -1,5 +1,6 @@
 /build/
 /dist/
+/_build/
 .venv/
 .ipynb_checkpoints/
 **/.ipynb_checkpoints/

CONTRIBUTING.md

@@ -7,7 +7,8 @@
 | Change | Edit |
 | --- | --- |
 | A build or verification operation | `notebooks/<operation>.ipynb` |
-| Explanation or tutorial | `notebooks/01-*` to `04-*` or `docs/` |
+| Guided OS-building chapters | `book/*.ipynb` and `book/*.md` |
+| Compact operational notebooks | `notebooks/01-*` to `04-*` |
 | Docker or mkosi orchestration | `scripts/*.sh`, `Dockerfile` |
 | OS packages, boot settings or partition layout | `images/*/mkosi.conf` and `mkosi.repart/` |
 | Files installed in the OS | `images/*/overlay/` |
@@ -16,7 +17,9 @@
 
 The notebooks are the **canonical host-side Python sources**, not copies of `scripts/*.py`. The runner in `scripts/notebook.sh` executes them without Jupyter, so the exact notebook cells are used in CI.
 
-Run interactive notebooks with `make lab`. It creates working copies in ignored `build/notebooks/` so execution counters and saved outputs never dirty the committed source. To contribute, copy reviewed edits back into `notebooks/`, clear cell outputs, and commit the canonical version.
+Run canonical operational notebooks with `make lab`; run Jupyter Book chapters with `make book-lab` (ignored working copies). Render the book with `make book-html` without notebook execution.
+
+Run interactive operational notebooks with `make lab`. It creates working copies in ignored `build/notebooks/` so execution counters and saved outputs never dirty the committed source. To contribute, copy reviewed edits back into `notebooks/`, clear cell outputs, and commit the canonical version.
 
 Keep each notebook easy to read from top to bottom: goal, prerequisites, visible arguments, focused cells, and checks. Put import-safe helpers before the final guarded command cell. Do not commit generated outputs, secrets, private keys, kernel traces, or host credentials. Avoid hidden notebook state and unreviewed network access.
 

Makefile

@@ -1,12 +1,15 @@
 .DEFAULT_GOAL := help
-.PHONY: help check builder images smoke repro lab
+.PHONY: help check builder images smoke repro lab book-html book-lab book
 
 help:
 	@printf '%s\n' \
 	  'SC Linux image factory' \
 	  '' \
 	  '  make check    Run unit tests, source checks, and shell syntax checks' \
-	  '  make lab      Open JupyterLab working copies' \
+	  '  make lab      Open operational notebooks in JupyterLab' \
+	  '  make book-lab Open guided book chapters in JupyterLab' \
+	  '  make book-html Render the Jupyter Book site without execution' \
+	  '  make book     Build the static Jupyter Book without executing cells' \
 	  '  make builder  Build the pinned Docker toolchain' \
 	  '  make images   Build the RAW disk and UEFI installer ISO' \
 	  '  make smoke    Boot RAW clones, install the ISO, and boot the result' \
@@ -36,3 +39,12 @@ repro:
 
 lab:
 	bash scripts/lab.sh
+
+book-lab:
+	bash scripts/book-lab.sh
+
+book-html:
+	jupyter book build --html
+
+book:
+	jupyter book build --html --strict

README.md

@@ -19,6 +19,24 @@ make check
 
 The build requires an **amd64 Linux** machine with Docker, Git, Make, OpenSSL and at least **40 GiB free**. It uses privileged containers. Follow [the build guide](docs/building.md) before running it.
 
+## Guided Jupyter Book: build an OS step by step
+
+The [SC Linux build book](book/index.md) separates the workflow into nine self-contained chapters: host prerequisites, image configuration, CA trust, builder, image construction, checksums/footprint, QEMU boot, host seed and reproducibility.
+
+Read the chapters on GitHub or open working copies in JupyterLab with `make book-lab`.
+
+Install the optional documentation tools in a development virtual environment:
+
+```sh
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install -r requirements-book.txt jupyterlab
+make book
+make book-lab
+```
+
+`make book` renders HTML in strict mode without running image-build cells. The book's code cells run local commands and some build privileged Docker images; the public HTML book must **not** execute them. To render the site, install Jupyter Book 2 and use `make book-html` (without `--execute`). The installed OS never includes Jupyter.
+
 ## Learn and operate through notebooks
 
 Open [`notebooks/`](notebooks) in JupyterLab, starting Jupyter from the repository root:

book/01-environment.ipynb

@@ -0,0 +1,72 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "# 1 · Prepare the build host\n\n## Goal\n\nConfirm that the host can build a **Debian 13 amd64** RAW image and boot-test it with QEMU. This chapter changes nothing.\n\n## Prerequisites\n\nUse a Linux x86-64 host with Docker, Git, Make, Python 3.11+, OpenSSL and at least 40 GiB of available space. Docker Desktop is not qualified for this workflow."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "from pathlib import Path\nimport os, subprocess, sys\nSTART = Path.cwd().resolve()\nROOT = next((p for p in (START, *START.parents) if (p/\"myst.yml\").is_file() and (p/\"scripts/build.sh\").is_file()), None)\nassert ROOT is not None, \"Run from the SC Linux checkout (make book-lab)\"\ndef run(*argv):\n    print(\"+\", \" \".join(map(str, argv)), flush=True)\n    return subprocess.run([str(a) for a in argv], cwd=ROOT, check=True)\nprint(\"Source:\", ROOT)"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 1 · Identify the platform"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "import platform, shutil\nprint(\"OS:\", platform.system(), \"Architecture:\", platform.machine())\nassert platform.system()==\"Linux\", \"Use an amd64 Linux host\"\nassert platform.machine() in (\"x86_64\",\"AMD64\"), \"The current image target is amd64\"\nfor binary in (\"docker\",\"git\",\"make\",\"python3\",\"openssl\"):\n    print(binary, shutil.which(binary) or \"MISSING\")\n    assert shutil.which(binary), f\"Install {binary}\""
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 2 · Check storage and Docker"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "disk=shutil.disk_usage(ROOT)\nprint(f\"Free disk: {disk.free/2**30:.1f} GiB\")\nassert disk.free >= 40*2**30, \"Reserve at least 40 GiB\"\nrun(\"docker\",\"info\",\"--format\",\"{{.OSType}} {{.Architecture}}\")"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 3 · Inspect Git state\n\nImage builds require every tracked change to be committed; artifacts belong under ignored `build/`."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "run(\"git\",\"status\",\"--short\")\nrun(\"git\",\"rev-parse\",\"--short\",\"HEAD\")"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Check\n\nThe preceding cells must all complete without errors. Host readiness alone does not imply valid images.\n\n## Next\n\nContinue to [review the build configuration](02-configuration.ipynb)."
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "tags": [
+      "skip-execution"
+    ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}

book/02-configuration.ipynb

@@ -0,0 +1,84 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "# 2 · Understand the image inputs\n\n## Goal\n\nIdentify the pinned software inputs, image layout and packages **before** running a privileged build.\n\n## Prerequisites\n\nComplete the host preflight; all cells below are read-only."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "from pathlib import Path\nimport os, subprocess, sys\nSTART = Path.cwd().resolve()\nROOT = next((p for p in (START, *START.parents) if (p/\"myst.yml\").is_file() and (p/\"scripts/build.sh\").is_file()), None)\nassert ROOT is not None, \"Run from the SC Linux checkout (make book-lab)\"\ndef run(*argv):\n    print(\"+\", \" \".join(map(str, argv)), flush=True)\n    return subprocess.run([str(a) for a in argv], cwd=ROOT, check=True)\nprint(\"Source:\", ROOT)"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 1 · Read pinned versions\n\nThe container uses a Debian snapshot and `mkosi` package defined in `config/build.json`."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "import json\nbuild_config=json.loads((ROOT/\"config/build.json\").read_text())\nfor key, value in sorted(build_config.items()):\n    print(f\"{key:24} {value}\")"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 2 · Review server packages\n\nThese are mkosi package inputs, not a promise of the exact installed footprint."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "import configparser\ndef read_mkosi(name):\n    parser=configparser.ConfigParser(interpolation=None)\n    parser.read(ROOT/\"images\"/name/\"mkosi.conf\")\n    return parser\nserver=read_mkosi(\"server\")\ninstaller=read_mkosi(\"installer\")\nfor role,cfg in ((\"Server\",server),(\"Installer\",installer)):\n    names=cfg.get(\"Content\",\"Packages\").split()\n    print(f\"{role} packages ({len(names)}):\", \", \".join(names))"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 3 · Inspect partitions and boot command\n\nThe server contains an ESP and writable ext4 root. The UKI embeds the kernel and initrd."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "for path in sorted((ROOT/\"images/server/mkosi.repart\").glob(\"*.conf\")):\n    print(\"\\n\", path.name)\n    print(path.read_text())\nprint(\"Kernel command line:\",server.get(\"Content\",\"KernelCommandLine\"))"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 4 · Run static source checks\n\nThis executes the canonical checks without creating image artifacts."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "run(\"make\",\"check\")"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Check\n\nYou should know what is pinned, which packages go into each image, and why the disk has multiple partitions.\n\n## Next\n\n[Validate FreeIPA CA trust](03-trust.ipynb)."
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "tags": [
+      "skip-execution"
+    ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}

book/03-trust.ipynb

@@ -0,0 +1,72 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "# 3 · Verify the FreeIPA trust anchor\n\n## Goal\n\nCheck that the embedded **public CA** is a valid CA certificate and its DER SHA-256 digest matches the local fingerprint file.\n\n## Prerequisites\n\nObtain the expected CA fingerprint independently from your FreeIPA administrator. A matching repository fingerprint alone is **not** independent authentication."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "from pathlib import Path\nimport os, subprocess, sys\nSTART = Path.cwd().resolve()\nROOT = next((p for p in (START, *START.parents) if (p/\"myst.yml\").is_file() and (p/\"scripts/build.sh\").is_file()), None)\nassert ROOT is not None, \"Run from the SC Linux checkout (make book-lab)\"\ndef run(*argv):\n    print(\"+\", \" \".join(map(str, argv)), flush=True)\n    return subprocess.run([str(a) for a in argv], cwd=ROOT, check=True)\nprint(\"Source:\", ROOT)"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 1 · Inspect the checked-in certificate\n\nNo private key or one-time enrollment password belongs in the repository."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "CERT = ROOT/\"trust/ipa-ca.crt\"\nFINGERPRINT = ROOT/\"trust/ipa-ca.sha256\"\nassert CERT.is_file() and FINGERPRINT.is_file(), \"Import a reviewed public CA first\"\nprint(\"Certificate:\", CERT.relative_to(ROOT))\nprint(\"Local recorded DER SHA-256:\", FINGERPRINT.read_text().strip())\nrun(\"openssl\",\"x509\",\"-in\",CERT,\"-noout\",\"-subject\",\"-issuer\",\"-dates\")"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 2 · Check certificate constraints and digest\n\nThe shared runtime library verifies DER digest, validity dates and the CA constraint."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "sys.path.insert(0,str(ROOT/\"lib\"))\nfrom scos.core import verify_ca\nlocal_fingerprint = verify_ca(CERT, FINGERPRINT.read_text())\nprint(\"Local certificate check passed:\", local_fingerprint)"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Action 3 · Compare with independently verified fingerprint\n\nEnter the fingerprint obtained **out-of-band**, not copied from `trust/ipa-ca.sha256`. This cell does not send the value anywhere."
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": "INDEPENDENT_DER_SHA256 = \"\"\nif INDEPENDENT_DER_SHA256:\n    assert local_fingerprint == INDEPENDENT_DER_SHA256.strip().lower().replace(\":\",\"\"), \"Independent CA fingerprint mismatch\"\n    print(\"Independent CA fingerprint matches\")\nelse:\n    print(\"NOT VERIFIED INDEPENDENTLY — obtain the authoritative fingerprint before building\")"
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": "## Check\n\nA local certificate check passes only the structural and digest tests. **Do not build or distribute images until independent CA verification is complete.** If replacing the CA, use `notebooks/trust.ipynb` and commit the reviewed public files.\n\n## Next\n\n[Build the pinned toolchain](04-builder.ipynb)."
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "tags": [
+      "skip-execution"
+    ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}