Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 24 additions & 14 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,35 +6,45 @@

| Change | Edit |
| --- | --- |
| A build or verification operation | `notebooks/<operation>.ipynb` |
| Guided OS-building chapters | `book/*.ipynb` and `book/*.md` |
| Compact operational notebooks | `notebooks/01-*` to `04-*` |
| Host-side validation, staging, packaging or VM tests | Named operation in `notebooks/*.ipynb` |
| Step-by-step teaching chapters | `book/*.ipynb` and `book/*.md` |
| Compact operational walkthrough | `notebooks/01-*` through `04-*` |
| Docker or mkosi orchestration | `scripts/*.sh`, `Dockerfile` |
| OS packages, boot settings or partition layout | `images/*/mkosi.conf` and `mkosi.repart/` |
| OS packages, boot settings or partition layout | `images/*/mkosi.conf` and `images/server/mkosi.repart/` |
| Files installed in the OS | `images/*/overlay/` |
| Runtime checks used by installed commands | `lib/scos/core.py` |
| FreeIPA domain or build baseline | `config/identity.json` or `config/build.json` |
| Shared validation and installed-command helpers | `lib/scos/core.py` |
| FreeIPA identity or build baseline | `config/identity.json` or `config/build.json` |
| Task guides | `docs/` |

The notebooks are the **canonical host-side Python sources**, not copies of `scripts/*.py`. The runner in `scripts/notebook.sh` executes them without Jupyter, so the exact notebook cells are used in CI.
Named operation notebooks are the canonical host-side Python sources. `scripts/notebook.sh` executes their cells directly in Make and CI. Keep build logic there; guided notebooks should call the underlying operations rather than duplicate them. Guest commands and Python regression tests remain ordinary files because they must work without Jupyter.

Run canonical operational notebooks with `make lab`; run Jupyter Book chapters with `make book-lab` (ignored working copies). Render the book with `make book-html` without notebook execution.
## Edit notebooks

Run interactive operational notebooks with `make lab`. It creates working copies in ignored `build/notebooks/` so execution counters and saved outputs never dirty the committed source. To contribute, copy reviewed edits back into `notebooks/`, clear cell outputs, and commit the canonical version.
Follow the [notebook guide](docs/notebooks.md) for setup, working copies and copying reviewed edits back to source. Use `make lab` for operational notebooks or `make book-lab` for teaching chapters.

Keep each notebook easy to read from top to bottom: goal, prerequisites, visible arguments, focused cells, and checks. Put import-safe helpers before the final guarded command cell. Do not commit generated outputs, secrets, private keys, kernel traces, or host credentials. Avoid hidden notebook state and unreviewed network access.
Keep each notebook readable from top to bottom: goal, prerequisites, visible arguments, focused cells and checks. Put import-safe helpers before the final guarded command cell. Avoid hidden notebook state and unreviewed network access.

## Test before opening a PR
Do not commit generated outputs, credentials, private keys, keytabs, kernel traces or seed media. Clear code-cell outputs and execution counts before committing notebooks.

## Verify your changes

```sh
make check
```

This runs the Python unit tests, checks all notebook code-cell syntax, validates configuration and shell scripts, and verifies pinned action references. For image changes, also run `make builder`, `make images` and `make smoke` on supported amd64 Linux. Use `make repro` for deterministic-build work.
This runs unit tests, notebook syntax and structure checks, local documentation-link checks, configuration validation, shell syntax checks and pinned-action checks.

| Change | Additional verification |
| --- | --- |
| Book content or configuration | Install `requirements-book.txt`, then run `make book` |
| Image inputs or build pipeline | On supported amd64 Linux, run `make builder`, `make images` and `make smoke` |
| Deterministic-build behavior | Run `make repro` |
| Provisioning or identity | Test the affected workflow in a real deployment |

**Important:** some Python files remain deliberately: installed guest commands depend on `lib/scos/core.py`, and Python regression tests remain under `tests/`. The deployed image must never require Jupyter.
Image builds require a clean, committed source tree. In your pull request, state what changed and which checks actually ran. Keep source-test results separate from build, boot and live-enrollment evidence.

## Security and license

The image builder uses privileged Docker. Import only independently verified public IPA CA certificates. Protect seed media and signing keys. See [SECURITY.md](SECURITY.md).
The image builder uses privileged Docker. Import only independently verified public IPA CA certificates. Protect seed media and signing keys. See [Security](SECURITY.md).

A source repository is not automatically licensed for redistribution. The project owner must choose a license for original code; upstream dependencies retain their existing licenses.
96 changes: 32 additions & 64 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,89 +1,57 @@
# SC Linux

A small **Debian 13 (Trixie) amd64** server image, built with **mkosi 25.3-7**. One build produces a compressed bootable disk image and an installer ISO. FreeIPA enrollment happens **after** each host is provisioned.
A Debian 13 (`amd64`) server image built with mkosi. One build produces a compressed UEFI disk image and an installer ISO. Hosts are provisioned with cloud-init and enrolled in FreeIPA after installation.

**Status:** development. Source checks, image construction, VM boot and real FreeIPA enrollment are separate acceptance stages. Do not assume a successful source check means an image was tested.
**Development status:** source checks, image builds, VM boot tests and live FreeIPA enrollment are separate verification stages. See [verification](docs/verification.md) for what each stage proves.

## Get started
## Start here

1. Clone the repository and run source checks.
2. Independently verify the fingerprint of the checked-in public FreeIPA CA, or replace it with your own reviewed CA.
3. Commit any configuration or trust changes.
4. Build the images and run the disposable VM tests.
Clone the repository and validate the source. These checks need Python 3.11+, Git, Make, Bash and OpenSSL; they do not need Docker or Jupyter.

```sh
git clone https://github.com/soft-cloud-dev/sc-linux.git
cd sc-linux
make check
```

The build requires an **amd64 Linux** machine with Docker, Git, Make, OpenSSL and at least **40 GiB free**. It uses privileged containers. Follow [the build guide](docs/building.md) before running it.

## Guided Jupyter Book: build an OS step by step

The [SC Linux build book](book/index.md) separates the workflow into nine self-contained chapters: host prerequisites, image configuration, CA trust, builder, image construction, checksums/footprint, QEMU boot, host seed and reproducibility.

Read the chapters on GitHub or open working copies in JupyterLab with `make book-lab`.

Install the optional documentation tools in a development virtual environment:
To build images, use an **amd64 Linux host** with Docker and at least **40 GiB free**. Follow the [build guide](docs/building.md) to review configuration, independently verify the public FreeIPA CA and commit your changes before building. Builds use privileged containers.

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-book.txt jupyterlab
make book
make book-lab
make builder
make images
make smoke
```

`make book` renders HTML in strict mode without running image-build cells. The book's code cells run local commands and some build privileged Docker images; the public HTML book must **not** execute them. To render the site, install Jupyter Book 2 and use `make book-html` (without `--execute`). The installed OS never includes Jupyter.
The main outputs are `build/output/sc-linux.raw.zst` and `build/output/sc-linux.iso`, accompanied by checksums, size reports and VM logs. See the [output reference](docs/building.md#4-inspect-and-boot-test).

## Learn and operate through notebooks
## Choose a workflow

Open [`notebooks/`](notebooks) in JupyterLab, starting Jupyter from the repository root:

| Start here | What you will do |
| Goal | Start here |
| --- | --- |
| [01 · Validate](notebooks/01-validate.ipynb) | Check configuration and run tests |
| [02 · Build](notebooks/02-build.ipynb) | Review inputs and generate RAW + ISO |
| [03 · Inspect](notebooks/03-inspect.ipynb) | Verify hashes, compare sizes |
| [04 · Smoke-test](notebooks/04-smoke.ipynb) | Boot test VMs and review evidence |

The **actual host-side operations** also live in notebooks: [check](notebooks/check.ipynb), [trust](notebooks/trust.ipynb), [prepare](notebooks/prepare.ipynb), [make-iso](notebooks/make-iso.ipynb), [artifacts](notebooks/artifacts.ipynb), [footprint](notebooks/footprint.ipynb), [seed](notebooks/seed.ipynb), and [smoke](notebooks/smoke.ipynb).

You **do not need Jupyter installed to use CI or the Makefile**. Execute any operation with the minimal, Python-standard-library runner:

```sh
bash scripts/notebook.sh notebooks/check.ipynb
make check
```
| Build from the command line | [Build guide](docs/building.md) |
| Learn the build step by step | [Nine-chapter Jupyter Book](book/index.md) |
| Run or edit notebooks locally | [Notebook guide](docs/notebooks.md) |
| Install a machine and configure SSH | [Installation guide](docs/installing.md) |
| Configure CA trust and enroll a host | [FreeIPA identity](docs/identity.md) |
| Understand or change the implementation | [Architecture](docs/architecture.md) · [Contributing](CONTRIBUTING.md) |

The notebook is the executable source. The runner evaluates its Python cells in order; it does not execute Markdown or install dependencies. JupyterLab is an optional interface, not part of the deployed OS.
Jupyter is optional. Make and CI execute the canonical operation notebooks through a Python-standard-library runner. The installed OS does not include Jupyter.

## Open notebooks interactively
## Commands

JupyterLab is needed only when you want to **explore cells interactively**. Install it in a local development environment, not in the built OS:

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install jupyterlab
make lab
```

Run `make lab`, then open `01-validate.ipynb`. This creates **working copies** of the notebooks in ignored `build/notebooks/`, so Jupyter autosave and execution outputs cannot dirty the committed source or block image builds. Repeated launches preserve these copies. To contribute changes, deliberately copy the edited notebook back into `notebooks/`, clear its cell outputs, and commit it. CI and command-line builds continue to use the committed notebooks through `make` and `scripts/notebook.sh`.

## Build and test

After configuring verified CA trust and committing your changes:

```sh
make builder
make images
make smoke
```
| Command | Purpose |
| --- | --- |
| `make` | Show available commands |
| `make check` | Run unit tests and source checks |
| `make builder` | Create the Docker build toolchain |
| `make images` | Build the RAW disk and installer ISO |
| `make smoke` | Boot RAW clones, install the ISO and boot the result |
| `make repro` | Compare two fresh unsigned builds byte for byte |
| `make lab` / `make book-lab` | Open operational notebooks / book chapters |
| `make book` | Render the book in strict mode without executing cells |

Expected outputs in `build/output/`: `sc-linux.raw.zst`, `sc-linux.iso`, `SHA256SUMS`, `artifacts.json`, `footprint.json`, package manifests, and VM logs after smoke testing.
## Scope

For a step-by-step explanation see [Build](docs/building.md), [Install](docs/installing.md), [Identity](docs/identity.md), [Architecture](docs/architecture.md), [Footprint](docs/footprint.md), and [Verification](docs/verification.md).
The image includes systemd, cloud-init NoCloud, SSH and a native FreeIPA client. EFI signing is optional. Authenticated root, remote attestation, automatic updates, Kubernetes orchestration and FreeBSD images are not implemented.

**Scope:** UEFI, systemd, cloud-init NoCloud, SSH and native FreeIPA client. Signing is optional. The repository does **not** implement authenticated root, remote attestation, automatic updates, Kubernetes orchestration, or a FreeBSD image. Read [Security](SECURITY.md) before deploying.
Read [Security](SECURITY.md) before deployment. The [documentation index](docs/README.md) also covers image size, upstream references and historical changes.
30 changes: 30 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Documentation

[Home](../README.md)

## Build and operate

| Guide | Use it to |
| --- | --- |
| [Build](building.md) | Prepare the host, verify trust and produce RAW + ISO images |
| [Notebooks](notebooks.md) | Choose a notebook workflow and set up JupyterLab |
| [Install](installing.md) | Create a host seed, install the image and connect over SSH |
| [Identity](identity.md) | Configure FreeIPA trust and enroll a provisioned host |
| [Verification](verification.md) | Distinguish source, build, VM and live deployment evidence |
| [Footprint](footprint.md) | Inspect and compare disk, ISO and package sizes |

For a guided walkthrough, start with the [Jupyter Book](../book/index.md).

## Understand and contribute

- [Architecture](architecture.md): repository layout, build sequence and runtime boundaries.
- [Contributing](../CONTRIBUTING.md): where to edit and which checks to run.
- [Security](../SECURITY.md): trust boundaries and deployment limitations.
- [Upstream references](upstream.md): documentation for the tools used by the build.

## Historical changes

These records describe earlier revisions, not the current verification status:

- [Repository readability rewrite](changes/2026-10-16-readability.md)
- [Notebook-first source migration](changes/2026-10-16-notebooks.md)
44 changes: 21 additions & 23 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,40 +8,38 @@ SC Linux keeps the *build environment* separate from the *installed operating sy

| Path | Purpose |
| --- | --- |
| `notebooks/01-*` through `04-*` | Guided, interactive workflow |
| `notebooks/*.ipynb` | Executable host-side validation, staging, image packaging and tests |
| `book/` | Nine-chapter teaching sequence and book reference |
| `docs/` | Task guides and historical change records |
| `notebooks/01-*` through `04-*` | Compact, interactive workflow |
| Named `notebooks/*.ipynb` | Canonical host-side validation, staging, image packaging and tests |
| `scripts/notebook.sh` | Execute the same notebook code noninteractively in CI |
| `scripts/*.sh` | Docker, mkosi and reproducibility orchestration |
| `config/` | Version pins and FreeIPA identity |
| `images/server/` | Server mkosi recipe, partition layout and system files |
| `images/installer/` | Installer UKI recipe, units and disk installer |
| `lib/scos/core.py` | Small **runtime** library used by installed commands |
| `lib/scos/core.py` | Validation helpers shared by host operations and installed commands |
| `tests/` | Python unit tests for safety and compatibility |
| `trust/` | Operator-reviewed public IPA CA certificate |
| `build/` | Ignored images, metadata and VM logs |
| `build/` | Ignored images, metadata, VM logs and notebook working copies |

**Why some .py files remain:** installed programs still use Python. A deployed node must not need Jupyter. Python unit tests also remain importable; host utility entrypoints moved to notebooks.
The [notebook guide](notebooks.md) explains the teaching, operational and implementation layers. Installed programs and unit tests use ordinary Python files and run without Jupyter.

## The build sequence

```text
config + verified public CA
|
check.ipynb
|
prepare.ipynb -> server/installer overlay and apt snapshot
|
mkosi server -> sc-linux.raw
| |
zstd payload digests
| |
+-------> mkosi installer -> UKI
|
make-iso.ipynb -> sc-linux.iso
|
artifacts / footprint notebooks
|
smoke.ipynb -> VM evidence
```mermaid
flowchart TD
inputs["Configuration and verified public CA"] --> check["Source checks"]
check --> stage["Stage overlays and apt snapshot"]
stage --> raw["Build server RAW"]
raw --> compress["Compress RAW"]
raw --> digests["Calculate payload digests"]
compress --> digests
digests --> installer["Build installer UKI"]
installer --> iso["Assemble installer ISO"]
compress --> iso
compress --> reports["Artifact hashes and size reports"]
iso --> reports
reports --> smoke["VM smoke tests"]
```

The source tree is mounted read-only into a privileged builder container, then copied into a temporary workspace. Metadata records the committed source revision and timestamp. The installer includes both the compressed RAW and expected payload digests.
Expand Down
6 changes: 3 additions & 3 deletions docs/building.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Build your first image

For an interactive, chapter-by-chapter explanation of the complete image factory, use the [SC Linux Jupyter Book](../book/index.md). The book separates environment checks, version pins, trust, filesystem staging, mkosi image construction, packaging, checksum verification and VM testing. The actual CI image pipeline remains the canonical source of build behavior.

[Home](../README.md) · [Install](installing.md) · [Verification](verification.md)

This guide builds **two files from one server definition**: a compressed RAW disk for a hypervisor, and an ISO that installs the same disk image.

For an interactive walkthrough, use the [SC Linux Jupyter Book](../book/index.md). The [notebook guide](notebooks.md) covers local setup and the shorter operational walkthrough.

## Before you start

Use an **amd64 Linux host**, not Docker Desktop. Install Docker, Git, Make, Python 3.11+ and OpenSSL. Reserve at least **40 GiB** disk space. You need a verified **public** FreeIPA CA. The build uses privileged Docker and requires a clean, committed Git checkout.
Expand Down Expand Up @@ -54,7 +54,7 @@ For a second build, provide a different destination:
bash scripts/build.sh build/run-02
```

For a cell-by-cell build in JupyterLab, activate your JupyterLab virtual environment and run `make lab`. Open `02-build.ipynb` from `build/notebooks/`. Working copies are Git-ignored so saved notebook outputs cannot violate the clean-tree build requirement.
For a cell-by-cell build, follow the [notebook setup guide](notebooks.md#set-up-jupyterlab) and open `02-build.ipynb` in JupyterLab.

## 4. Inspect and boot-test

Expand Down
2 changes: 2 additions & 0 deletions docs/changes/2026-10-16-notebooks.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Notebook-first source migration — 2026-10-16

**Historical record:** this describes the notebook-migration revision. The later nine-chapter book is documented in the [notebook guide](../notebooks.md). Use [verification](../verification.md) for current acceptance stages.

## What changed

Eight host-side Python operations moved from `scripts/*.py` to executable `notebooks/*.ipynb`. The notebook cells are the canonical implementation, not a wrapper that imports deleted scripts. A Python-standard-library runner (`scripts/notebook.sh`) runs those same cells inside Docker and in CI, without requiring an interactive Jupyter server.
Expand Down
2 changes: 2 additions & 0 deletions docs/changes/2026-10-16-readability.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Repository readability rewrite

**Historical record:** this describes the readability-rewrite revision. Its test counts and trust-file state are not the current project status. Use the [documentation index](../README.md) for current guides and [verification](../verification.md) for acceptance stages.

## Goal and scope

Make the supplied image factory easy to navigate, operate, and maintain.
Expand Down
Loading
Loading