diff --git a/.github/workflows/compilation_tests.yml b/.github/workflows/compilation_tests.yml index be36564a..c2988079 100644 --- a/.github/workflows/compilation_tests.yml +++ b/.github/workflows/compilation_tests.yml @@ -85,12 +85,23 @@ jobs: working-directory: impl/rust-cli run: cargo test --verbose - # The ignored stress suite (tests/stress_tests.rs) is not run here. It ran - # under `continue-on-error: true`, which hid its state (Hypatia RE005): - # on 2026-11-16, 5 of its tests fail outright and 2 run for more than two - # minutes each, so this step only ever burned the job's 30-minute timeout. - # Run it by hand with `cargo test --test stress_tests -- --ignored` until - # the suite is repaired and can gate honestly. + # The ignored stress suite (tests/stress_tests.rs) gates the build. It was + # repaired on 2026-11-17 (see docs/audits/2026-11-16-hypatia-remediation.adoc); + # there is deliberately no continue-on-error (Hypatia RE005). + # The shell syncs its journal after every operation, so on a real disk + # this suite mostly measures the disk (82 s on ext4 against 39 s on tmpfs + # locally). TMPDIR points at tmpfs so that the step measures the shell, + # and 5 minutes leaves wide margin. + # Linux only: stress_deep_nesting_1000_levels needs 1000 directory levels + # in one absolute path (~2,000 bytes), which fits Linux's PATH_MAX of 4096 + # but not macOS's 1024. + - name: Run ignored tests (stress) + if: runner.os == 'Linux' + working-directory: impl/rust-cli + timeout-minutes: 5 + env: + TMPDIR: /dev/shm + run: cargo test --test stress_tests -- --ignored --test-threads=1 # ============================================================ # MSRV (Minimum Supported Rust Version) diff --git a/.gitignore b/.gitignore index a3b59c58..a989a5e5 100644 --- a/.gitignore +++ b/.gitignore @@ -122,6 +122,9 @@ htmlcov/ ai-cli-crash-capture/ .cache/ .vsh_state.json +.vsh_state.json.migrated +.vsh_state.journal +.vsh_state.journal.* # ============ # Strategic diff --git a/CLAUDE.md b/CLAUDE.md index b6a2c509..ae4e12d2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -217,7 +217,7 @@ valence-shell/ proofs/ lean4/ # Lean recheck leg + executable model oracle (101 theorems) coq/ # CIC proofs + extraction (131 theorems, 0 admits — re-verified 2026-07-16, Coq 8.18.0) - agda/ # Theory source of truth (owner ruling 2026-11-09); 10 modules, 0 postulates, --safe --without-K --cubical-compatible + agda/ # Theory source of truth (owner ruling 2026-11-09); 14 modules (10 filesystem, 4 journal model), 0 postulates, --safe --without-K --cubical-compatible z3/ # SMT checks (ASSUMED; 2 files definitional, 3 axiomatic) idris2/ # ABI carrier (0 holes, 0 partials, 2 primitive-eq axioms) docs/ # Design docs, roadmaps, PROOF_HOLES_AUDIT.adoc diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index 7f0fc4d8..96624dac 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -116,10 +116,22 @@ reproduce. Absence of a tool is an open obligation, not a passing leg | Same theorems in Agda (the theory source since the 2026-11-09 owner ruling) | proven (2026-11-09) -| `agda --safe --without-K --cubical-compatible All.agda`, Agda 2.7.0.1 + stdlib v2.2: 10/10 modules; CI `.github/workflows/agda.yml` -| `proofs/agda/` (ten modules via `All.agda`) +| `agda --safe --without-K --cubical-compatible All.agda`, Agda 2.7.0.1 + stdlib v2.2: all modules in `All.agda` (ten for these theorems; fourteen with the journal model below); CI `.github/workflows/agda.yml` +| `proofs/agda/` (via `All.agda`) | No postulates, no holes, the estate invocation. Filesystem equality is pointwise (`_≗_`). Grade `PWG-3` is derived from the CI receipt on the merge head, not asserted here. See `docs/audits/2026-11-09-proof-ground-truth.adoc` +| Journal model: after a crash, loading yields the synced events followed by a prefix of the unsynced ones; a cut-short record is rejected, and is classified as torn, never as damaged; rollback leaves no operation of the rolled-back transaction live +| proven (2026-11-17) +| `bash scripts/check-agda.sh`, Agda 2.7.0.1 + stdlib v2.2; ten mutations rejected locally (`proofs/agda/VERIFICATION.adoc`) +| `proofs/agda/JournalModel.agda` `crash-recovery`, `write-atomic`, `crash-tail`; `JournalFraming.agda` `step-torn`, `decode-sound`, `classify-torn`; `JournalReplay.agda` `run-ids`, `rollback-no-live` +| About the model only. The crash model (synced bytes survive; the unsynced part survives as a prefix) is an assumption about storage, and reordered or partly written blocks are outside it. No property of the checksum or of the header check is proved or needed. The converse is not proved and does not hold: damage that leaves a header passing its check and declaring more bytes than the file holds is classified as torn + +| The Rust journal follows the journal model +| tested-only +| `cargo test --test journal_crash_tests`: 3,139 crash images on simulated storage that implements the model's file, and 3,130 crash states on real files, none failing (2026-11-17, local, with the header check); four deliberate bugs were each caught. Unit tests flip every bit of every header and of a whole record and cut a journal at every byte +| `impl/rust-cli/src/journal.rs`, `impl/rust-cli/src/state.rs`, `impl/rust-cli/tests/journal_crash_tests.rs` +| Written to mirror the model, not derived from it or proved against it. One scripted session, not all sessions. The journal file still lives in the session root, so the shell's own commands can delete or alter it; the model does not cover that + | Same theorems restated in Z3 (Isabelle and Mizar archived 2026-11-09) | argued (the ledger word for PROOF-POLICY §5 `ASSUMED`: a direct SMT run with no proof certificate) | `scripts/verify-z3.sh`, exact result sequence per file diff --git a/docs/ARCHITECTURE_DIAGRAMS.adoc b/docs/ARCHITECTURE_DIAGRAMS.adoc index d5e0602c..b5de5565 100644 --- a/docs/ARCHITECTURE_DIAGRAMS.adoc +++ b/docs/ARCHITECTURE_DIAGRAMS.adoc @@ -103,7 +103,7 @@ System Documentation │ │ │ │ │ │ │ │ │ ┌──────▼──────────────────▼──────────────────▼───────┐ │ │ │ │ │ State Persistence (JSON) │ │ │ -│ │ │ ~/.vsh_state.json │ │ │ +│ │ │ /.vsh_state.journal │ │ │ │ │ └────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ @@ -214,7 +214,7 @@ USER INPUT: echo "Hello $USER" > output.txt ┌─────────────────┐ │ 7. UPDATE │ Update shell state │ STATE │ last_exit_code = 0 -│ (state.rs) │ Persist to ~/.vsh_state.json +│ (state.rs) │ Persist to /.vsh_state.journal └────────┬────────┘ │ ▼ @@ -391,7 +391,7 @@ USER: undo ┌────────────┐ │ START │ Initialize shell - │ (clean) │ Load ~/.vsh_state.json (if exists) + │ (clean) │ Load /.vsh_state.journal (if exists) └──────┬─────┘ │ ▼ diff --git a/docs/audits/2026-11-16-hypatia-remediation.adoc b/docs/audits/2026-11-16-hypatia-remediation.adoc index dec3dd02..2c4583a5 100644 --- a/docs/audits/2026-11-16-hypatia-remediation.adoc +++ b/docs/audits/2026-11-16-hypatia-remediation.adoc @@ -56,7 +56,7 @@ They close when the underlying alert closes. | `npx_in_workflow` (2) | fixed | `bunx --no-install --bun rescript build`, which never fetches an unpinned package. | `missing_timeout_minutes`, WH006 | fixed | `labels.yml`, `label-triage.yml` (echo-types versions). | RE001 (3) | fixed | `harden-runner`: `block` to `api.github.com` in `instant-sync.yml`, `block` to the SMTP endpoint in `push-email-notify.yml`, `audit` in `hypatia-scan.yml`. -| RE005 | fixed | The ignored stress suite no longer runs under `continue-on-error` in `compilation_tests.yml`; see <>. +| RE005 | fixed | The ignored stress suite no longer runs under `continue-on-error` in `compilation_tests.yml`; after its repair it gates the Linux test jobs again. See <>. | WH008 (3) | fixed | See the Secret Scanner section. | SD022 (2) | fixed | Both paths named another tree (`echidna/src/rust/`, the Agda tarball's `src/data/lib`); now qualified. | SD009 | left for the owner | `ffi/zig/src/main.zig` has no SPDX header. Licence edits are owner-only (estate doctrine 6). @@ -94,3 +94,134 @@ Run one test at a time on 2026-11-16, 120 s limit each: 5 fail (`stress_many_operations_10000`, `stress_resource_limits_max_history`), 3 pass; `stress_large_file_1gb` was not run. The CI step only ever reached the 30-minute job timeout. Repairing the suite is separate work. + +=== Repair (2026-11-17) + +All 11 tests now pass. The step is back in `compilation_tests.yml` as +`cargo test --test stress_tests -- --ignored --test-threads=1`, Linux only, +with `timeout-minutes: 5` and no `continue-on-error`. No test was deleted or +skipped, and no assertion was loosened. + +Shell bugs fixed (`src/state.rs`, `src/commands.rs`): + +* *O(n²) persistence.* Every recorded or undone operation rewrote the whole + history to `.vsh_state.json` (cloned, then pretty-printed). 1,000 touches + took 5.9 s; 10,000 and 100,000 never finished. The hot path now appends + one line to `.vsh_state.journal`; the snapshot is rewritten only when the + journal reaches half the history length (at least 1,024 entries), which + keeps the cost amortised O(1). Snapshots are written to a temporary file + and renamed, and record the last journal sequence number they contain, so + a crash cannot tear the snapshot or replay an entry twice. A torn journal + line or a sequence gap ends replay and forces the next save to be a full + snapshot, so operations made after a crash are not stranded behind the + bad line (Codex review on #229). After the fix, 1,000 touches take + 0.04 s. Seven new unit tests in `state.rs` cover replay, compaction, trim, + a torn tail, crash-after-rename and transactions; two more cover + operations after a torn tail and `cd -` surviving a restart (`cd` now + journals `previous_dir`). ++ +*Superseded later on 2026-11-17.* Review kept finding crash-consistency +defects in that two-file design (a snapshot beside a journal), and a +crash-point harness confirmed two failing classes. It was replaced by one +file of length-prefixed, checksummed records, each appended and synced, +following an Agda model (`proofs/agda/JournalModel.agda`). The journal is +rewritten as a single snapshot record, all-or-nothing, at the same threshold. +A failed load is now an error instead of an empty history, and an old +`.vsh_state.json` is converted on first start and kept as +`.vsh_state.json.migrated`. The crash harness +(`impl/rust-cli/tests/journal_crash_tests.rs`) finds no failing class on the +new format. The amortised cost is unchanged; each operation now also pays +one sync. +* *O(n²) single-step undo.* Each `undo 1` walked past every already-undone + operation to pick its target, then searched for that target again from the + oldest end of the history to mark it. A checked hint for the block of undone operations removes the rescan: + undoing 10,000 operations one at a time went from 1.8 s to 0.34 s. A unit + test compares it against a full scan through undo, record, trim and an + out-of-order mark. +* *Rollback not persisted.* `rollback` cleared the active transaction + without saving. That had been masked by the next operation's full rewrite, + and the journal would have exposed it. It now goes through + `ShellState::abort_transaction`, which saves. +* *Rollback left reversed operations live* (found in review, also on + `main`): `begin t; touch f; rollback; undo` failed with "No such file or + directory". Rollback now marks each operation undone as it reverses it, + skips operations already undone inside the transaction, and closes the + transaction only when every operation has been reversed. It first checks + each operation's record and refuses the whole rollback, changing nothing, + if one has no inverse or lacks its undo data (owner ruling OD-39). If a + reversal then fails on the disk, the transaction stays open so the + rollback can be run again. + +Test expectations corrected (`tests/stress_tests.rs`): + +* `stress_concurrent_multiple_instances`, `stress_many_operations_10000`: + the listings counted the shell's own `.vsh_state*` files. They now exclude + only those files, and the concurrent test checks the exact set of 1,000 + names. +* `stress_deep_nesting_1000_levels`: `level_NNNN` names give an + 11,000-byte path, past `PATH_MAX`, and the test then ran `mkdir` on paths + that already existed. It now builds the 1,000 levels through the shell + with one-character names (about 2,000 bytes, asserted against `PATH_MAX`) + and undoes them all. +* `stress_deep_nesting_no_stack_overflow`: it only called + `fs::create_dir_all`, which fails with ENAMETOOLONG at 5,000 levels. It + now builds the tree with `mkdirat`/`openat`, checks that shell `mkdir` and + `touch` at that depth fail cleanly without recording anything, and removes + the tree. +* `stress_large_file_undo_streaming`, `stress_large_file_1gb`: both called + `touch` on an existing file, but vsh's `touch` only creates files + (EEXIST). The 100 MB test now runs `rm` (the operation that saves file + content as undo data) plus `undo` and checks the restored bytes. The 1 GB + test times `chmod`, `mv` and their undos. +* `stress_undo_redo_efficiency`: "redo" popped the redo stack without + redoing. It now calls `commands::redo` and checks the files return. +* `stress_resource_limits_max_history`: it now asserts the history is capped + at `max_history_size` and that the persisted state stays under 32 MiB + (measured: 2.3 MB). + +Measured (debug build, 4 cores, each test run alone with the CI flags): +whole suite 29-51 s across three runs. These are the timings of the two-file +design. With the single-file journal and a sync per record, the whole suite +takes 82 s on this machine's ext4 disk and 39 s on tmpfs (one run each). The +CI step therefore puts its temporary directories on tmpfs, so that it +measures the shell's own cost and not the runner's disk. + +[cols="3,1",options="header"] +|=== +| Test | Wall time +| `stress_concurrent_multiple_instances` | 0.6 s +| `stress_concurrent_no_corruption` | 0.2 s +| `stress_deep_nesting_1000_levels` | 0.5 s +| `stress_deep_nesting_no_stack_overflow` | 0.7 s +| `stress_history_memory_efficiency` | 0.5 s +| `stress_large_file_1gb` | 1.8 s +| `stress_large_file_undo_streaming` | 12.6 s +| `stress_many_operations_10000` | 2.3 s +| `stress_many_small_files` | 1.2 s +| `stress_resource_limits_max_history` | 10-32 s +| `stress_undo_redo_efficiency` | 0.2 s +|=== + +Still open: + +* *Undo data is not streamed.* `rm` reads the whole file into + `undo_data`, and the journal stores those bytes as a JSON number array + (about 3 bytes per byte). Removing a 100 MB file takes about 11.5 s in a + debug build and writes about 300 MB of state. The test name promises + streaming; the test comment now says it is not implemented. The fix is + out-of-line undo storage (for example, renaming the file into a store + under the root), which changes the reversibility implementation, so it is + left for the owner. +* *History trimming is O(limit) per operation.* At the cap, each new + operation drains the oldest entry from the front of a `Vec`, moving about + 1.2 MB at the default 10,000 limit. That is about 7 s of the 100,000-op + test. A `VecDeque` would remove it but changes the public + `ShellState::history` type that tests and `CLAUDE.md` document as a `Vec`. +* *macOS.* The suite is not run there: `PATH_MAX` is 1024, so the + 1,000-level test cannot pass with absolute paths, and the rest was not + measured on macOS. +* The journal still lives in the session root, so shell commands can delete + or edit it, and two shells on one root are not kept apart. It is created + with mode 0600, never opened through a symlink, and synced on every + record. Moving it to an owner-only directory under an exclusive lock is + the next step (route plan section 14.3, C3). diff --git a/docs/plans/VSH-PRODUCT-REVIEW-2026-11-17.adoc b/docs/plans/VSH-PRODUCT-REVIEW-2026-11-17.adoc new file mode 100644 index 00000000..526cb96a --- /dev/null +++ b/docs/plans/VSH-PRODUCT-REVIEW-2026-11-17.adoc @@ -0,0 +1,1239 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// Copyright (c) Jonathan D.A. Jewell += valence-shell: product review, comparison and staged model +Jonathan Jewell (hyperpolymath) +:toc: left +:toc-title: Contents +:toclevels: 2 +:doctype: article +:icons: font +:sectanchors: + +*Date:* 2026-11-17 +*Status:* REVIEW. The owner ruled on section 10 on 2026-11-17: route B (engine +first), the truthful baseline next on its own branch, the capture experiment to +decide the first front end, and hole (b) left for the change-set engine. The +rulings are OD-42 to OD-45 in the route plan (`VSH-ROUTE-PLAN-2026-11-04.adoc`). +The text below is as written before those rulings. + +*Asked for:* a harsh review of the whole shell plan: what the product is and +what it is for; how it compares, in function and in modularity, with bash, NGS, +plug-in frameworks such as oh-my-zsh, and other shells; a rating of each aspect, +now and where it is heading; what each direction costs on the way to a usable +and one day production shell; and a development model in which every step +leaves a stable product. + +*Evidence grades:* + +* *Measured*: a command was run or a file read in this session (2026-11-17). +* *Reported*: read on a primary page by a research pass in this session and + linked. Items re-read by hand are marked *checked*. +* *Inference*: reasoned from the above. +* *Unverified*: not checked. + +== 0. Bottom line + +. *As a shell it is not usable today.* On 112 everyday command lines it gives + the same result as bash on 57. `dash`, a minimal POSIX shell, gives the same + result on 91, and brush, an existing shell written in Rust, on 111. The + failures are ordinary: `[ -d / ]` always fails; a quoted `*` or `[` reaches + programs with a backslash added; `cat *.txt` is not expanded; `mkdir -p a/b` + creates a directory named `-p`; `exit 7` exits with 0; a nested `for` loop + gives the wrong answer. Several of these report success. All of them are on + `main` and predate this month's branches (Measured, section 2). +. *The tests did not see it, and neither does the plan.* No test file on `main` + runs the built program; they call its functions. The plan's definition of done + (gates G1 to G11) has no gate for use as a shell, so "v1.0-honest" could ship + with every one of these failures (Measured, sections 2.6 and 8.1). +. *What is distinctive does work.* The history with proof references, the dry + run, checkpoints, transactions, undo and redo all behave as described in an + interactive session, and the two open pull requests add a crash-safe journal + and handle-based containment (Measured, section 2.7). +. *Undo reaches very little of a real session.* It covers the shell's own file + commands, redirections and variables. An edit in an editor, `sed -i`, `git`, + a build: none of it can be undone. This limits usefulness more than any gap + in the language (Measured, section 3). +. *There is a product here, and it is not "another bash".* Nothing in wide use + gives undo and redo, with grouping, for the effects of arbitrary commands. + The nearest tool, `try`, shows the mechanism works, and an experiment in this + session reproduced it at about 4.5 ms per command (sections 4.2 and 7.4). +. *Recommended direction:* treat the engine as the product and the shell + language as one of several front ends, built last. In order: a command-line + tool and library; a plug-in for bash, zsh and fish; capture of any command; + a server for AI agents; and only then an interactive shell of its own, + probably on an existing core. Each step is a product someone can use. The + other routes, their costs and the ratings are in sections 7 and 10. + +''''' + +== 1. What was measured, and how + +[cols="2,5",options="header"] +|=== +|Item |Value + +|Builds |`main` at `63c14a0`; the journal branch at `4f7fa6a` (valence-shell#230); the containment branch at `bb6aa75` (valence-shell#232). Release builds, rustc 1.99.1. +|Machine |Linux 6.18 in a container, as root, 4 cores, 16 GB. Disk is ext4 on a virtual device; `/dev/shm` is tmpfs. +|Reference |GNU bash 5.2.21. Calibration: dash, and brush 0.4.0 built from crates.io. +|Battery |112 command lines in 18 groups: 111 one-liners and one here-document (appendix A). Each runs in a fresh directory under bash and under the shell being tested, through `-c`. A line passes when standard output and exit status both match bash. +|Probes |About 150 further command lines for root causes, the file commands, undo, and what a program receives as arguments. +|Interactive |The interactive shell driven through `tmux` (a real terminal): typing, Tab, arrow keys, Ctrl-C, Ctrl-Z, `less`, `vim`. +|Timing |200 runs for start-up; single runs for the larger workloads. +|=== + +Limits of the method: + +* `rm` lines are not in the automated battery. A safety check in this + environment declines `bash -c` scripts that run `rm`, so the shell's `rm` was + probed directly instead. +* Everything ran as root, so nothing here shows behaviour for an ordinary user. +* Linux only. Nothing was run on macOS or BSD. +* The battery is this review's own. It is small and was written before the + results were known. A public corpus (section 11) should replace it as the + yardstick. + +''''' + +== 2. The shell as a shell + +=== 2.1 The battery + +[cols="3,2,2,2,2,2",options="header"] +|=== +|Shell |Same as bash |POSIX part (91) |bash-only part (21) |Different output |Exit status only + +|dash |91 |91 |0 |21 |0 +|brush 0.4.0 |111 |90 |21 |1 |0 +|*vsh, `main`* |*57* |*53* |*4* |*54* |*1* +|vsh, journal branch |57 |53 |4 |54 |1 +|vsh, containment branch |56 |52 |4 |55 |1 +|=== + +The "POSIX part" is the 91 lines that dash gets right; the other 21 use syntax +only bash has. The one line the containment branch loses is `cd /tmp && pwd`, +which it refuses on purpose (`cd` stays inside the session root). The one line +brush "fails" is an alias used in a non-interactive shell, where bash itself +does not expand it. + +By group, lines that match bash (Measured): + +[cols="2,1,1,1,1",options="header"] +|=== +|Group |Lines |dash |vsh `main` |vsh containment + +|arith |5 |2 |1 |1 +|array |3 |0 |0 |0 +|basic |3 |3 |3 |3 +|builtin |17 |13 |9 |8 +|case |2 |2 |2 |2 +|ext (other programs) |8 |8 |7 |7 +|fs (file commands) |8 |8 |7 |7 +|func |6 |5 |5 |5 +|glob |6 |4 |1 |1 +|if |3 |2 |1 |1 +|logic |7 |6 |3 |3 +|loop |9 |7 |1 |1 +|pipe |5 |5 |4 |4 +|quote |4 |2 |2 |2 +|redir |7 |6 |4 |4 +|sub (subshells, groups) |4 |4 |1 |1 +|subst (command substitution) |4 |4 |0 |0 +|vars |11 |10 |6 |6 +|=== + +=== 2.2 Root causes + +Each was reproduced on the `main` build. "Silent" means the command reports +success and the result is wrong. + +[cols="1,5,4,1",options="header"] +|=== +|# |What happens |Example |Silent + +|R1 |A quoted `*`, `?`, `[`, `{` or `~` is passed on with a backslash in front. The parser marks quoted characters with a backslash in a flat string (`quoted_word_to_string`, `parser.rs:2527`) and nothing takes it off again before the program is started |`grep '[0-9]' f` searches for `\[0-9]`. `awk '{print $1}'` is a syntax error. `pass:[echo "*"]` prints `pass:[\*]` |Yes +|R2 |Unquoted wildcards, braces and `~` are expanded only when the command also has a redirection. The plain path (`execute_external`) never calls the expansion; the redirection path does (`external.rs:41`) |`cat *.txt` fails with "No such file"; `cat *.txt > out` works |No +|R3 |`[ … ]` always fails. The parser removes the closing `]` (`parser.rs:3713`) and the evaluator then requires it (`test_command.rs:248`) |`[ -d / ] && echo yes` prints "test: missing ']'" |No +|R4 |Command substitution works only as the whole value of an assignment, and only for another program |`x=$(date +%Y)` works. `x=$(echo hi)`, `echo $(date)`, `for i in $(seq 3)` and backquotes do not |Partly +|R5 |Arithmetic expansion works only in an assignment |`pass:[i=$((i+1))]` works; `pass:[echo $((1+2))]` prints the text `pass:[$((1+2))]` |Yes +|R6 |`${x#p}`, `${x%p}` and `${x^^}` give an empty string. `$$` is not known |`x=abcdef; echo ${x#abc}` prints an empty line |Yes +|R7 |The file commands take one operand and no options. An option is treated as the operand; further operands are dropped |`mkdir -p a/b` creates a directory named `-p`. `touch a b c` creates `a`. `ls -la` is an error |Yes +|R8 |In `-c` and script mode, a failing built-in command ends the whole script |`ls nosuch; echo after` prints nothing after the error |No +|R9 |`exit N` exits with status 0. `set -e` has no effect. `! cmd`, `break`, `continue` and `wait` do not exist, so a loop that relies on `break` never ends. `&` is accepted only at the end of a line |`exit 7; echo $?` in the caller gives 0 |Partly +|R10 |`export` and `VAR=value cmd` do not reach the program being run |`export FOO=bar; sh -c 'echo $FOO'` prints an empty line |Yes +|R11 |A `for` loop inside a `for` loop runs the inner loop once with the outer variable empty |`for i in a b; do for j in 1 2; do echo $i$j; done; done` prints `1 2` |Yes +|R12 |Not supported: arrays, `pass:[(( ))]`, `let`, the C-style `for`, `[[ -d x ]] && …`, subshells, a brace group or a `while` loop in a pipeline, `read` from a here-string, `type`, `command -v`, `pushd` |`echo a b \| while read x y; do …` gives "Command not found: while" |No +|R13 |A redirection on a built-in command is resolved inside the session root |`ls nosuch 2>/dev/null` fails: "Parent directory does not exist: /dev" |No +|R14 |After starting a program the shell polls for its exit every 50 ms (`external.rs:326`, and two more loops like it) |1,000 runs of `/bin/true` take 50.5 s on the journal build (100 runs take 5.1 s on `main`); bash takes 1.4 s for 1,000 |- +|=== + +R1 and R7 are the dangerous ones. They change what a command does without +saying so. + +R1 to R6 share one cause. The POSIX shell turns a word into arguments in fixed +steps: tilde, parameter, command and arithmetic expansion, then field +splitting, then pathname expansion, then quote removal. Here the steps are +spread over the parser and three execution paths, and "was this character +quoted" is carried as backslashes inside a string. The root `CLAUDE.md` already +lists the consequence as its first priority ("requires quote-context +threading"). This cannot be patched into correctness one case at a time +(Inference); section 9.1 gives the structural answer. + +=== 2.3 The file commands against coreutils + +Twenty-two everyday forms, on the containment build (the parser is the same on +`main`). Starting tree in each case: files `a` and `b`, directories `dir` +(holding `inner`) and `sub`. Measured. + +[cols="3,3,4",options="header"] +|=== +|Typed |coreutils |vsh + +|`cp a c`, `mv a c`, `chmod +x a`, `ln -s a l`, `cd sub` |as expected |Same +|`cd -` with no previous directory |error |Same error +|`cp -r dir dir2`, `cp -p a c`, `mv -f a b`, `chmod -R 700 dir` |as expected |Error: the option is taken for a file name +|`cp a b sub/`, `mv a b sub/` |both go into `sub` |Error: "Destination already exists: b" +|`mv a sub` |`a` moves into `sub` |Error: "Destination already exists: sub" +|`ln -sf a l`, `ls -1` |as expected |Error +|`mkdir -p x/y/z` |creates the chain |*Creates a directory named `-p`, exit 0* +|`mkdir x y`, `touch x y`, `chmod 755 a b` |act on both |*Act on the first only, exit 0* +|`ls dir sub` |lists both |*Lists the first only, exit 0* +|`ln a hard` |hard link |Refused (by design: symbolic links only) +|`cd` |goes to `$HOME` |Goes to the session root (by design on the containment branch) +|=== + +Six forms behave as coreutils do, nine fail with an error, five do something +else and report success, and two differ on purpose. The earlier direct probes +of `rm` gave the same picture: `rm -f x`, `rm -rf x` and `rm -r x` treat the +option as the file to remove. + +The built-in commands also see only the session root, while other programs see +the whole machine: `cp /etc/hostname .` fails, `cat /etc/hostname` works. On +`main` the root itself moves with `cd`, so `cd /etc; touch x` created `/etc/x` +during this review (removed afterwards). That is hole (f) of the route plan and +is closed on valence-shell#232. + +=== 2.4 Interactive use + +Journal build, through `tmux`. Measured. + +[cols="3,5",options="header"] +|=== +|Works |Detail + +|Line editing, history, reverse search |Up arrow and Ctrl-R work (the `reedline` library). History is kept in `~/.vsh_history`. +|Multi-line input |`for … do` shows a continuation prompt and runs on `done`. +|Completion of command names |`ec` then Tab gives `echo`. +|Ctrl-C |At the prompt the shell stays. On a running program it stops the program. +|Full-screen programs |`less` and `vim` run and return. +|Background jobs |`sleep 2 &` then `jobs` lists it. +|The shell's own commands |`history --proofs`, `explain`, `checkpoint`, `diff`, `restore`, `replay`, `graph`, `begin`, `commit`, `rollback`, `undo`, `redo` and the `obliterate` confirmation all respond as documented. +|=== + +[cols="3,5",options="header"] +|=== +|Does not work |Detail + +|`help` |"help: Command not found: help". The banner tells the user to type it. +|`ls -la` |"Error: Not a directory" (R7). `ls -la \| cat` works, because in a pipeline the real `ls` runs. +|Completion of file names |Only when the word starts with `./`. `cat al` then Tab and `cd su` then Tab give "NO RECORDS FOUND". +|Ctrl-Z |Prints `^Z`; the shell goes on waiting and typed text is not read until Ctrl-C. No suspend, so `fg` and `bg` have nothing to act on. +|Prompt |`PS1` is ignored. The prompt is `vsh>` with an operation count. +|Start-up file |None. There is no `~/.vshrc`, no configuration file and no hook. +|Piped input |`printf 'echo hi\n' \| vsh` fails: "Error reading input: No such device or address". Scripts must be passed as a file or with `-c`. +|=== + +=== 2.5 Speed + +Measured; one machine, so treat the ratios as indicative. + +[cols="4,2,2,2,2",options="header"] +|=== +|Workload |bash |vsh `main` |vsh journal |vsh containment + +|Start and run `true` (ms, 200 runs; dash 1.60) |2.25 |2.04 |2.11 |2.33 +|1,000 file creations, built-in `touch`, tmpfs (ms) |1,530 ^1^ |321 |24 |527 +|The same on disk (ms) |1,530 ^1^ |706 |253 |687 +|1,000 runs of `/bin/true` (ms) |1,391 |- |50,514 |- +|Counting loop, 20,000 turns (ms) |67 |- |434 ^2^ |- +|200 MB through `cat \| tr \| wc` (ms) |592 |- |667 |- +|=== + +^1^ bash starts the `touch` program each time. +^2^ Each assignment is recorded as an undoable operation: the loop wrote a +3.1 MB journal. + +Start-up and pipelines are fine. Built-in operations are fast, and the journal +made them about thirteen times faster than `main` on tmpfs, because `main` +rewrites its whole state file on every operation. Two costs stand out: R14 +(50 ms on every program), and the price of recording every variable assignment +durably (on a real disk each record is a flush). + +=== 2.6 Size, tests and releases + +Measured, journal branch unless stated. + +* 25,788 lines of Rust in 34 files. By role (approximate): the interpreter + (parser, expansion, execution of programs, redirection, arithmetic, test, + functions, jobs) about 13,000 lines, of which `parser.rs` is 5,315; the + engine (state, commands, journal, deletion, audit, verification) about 9,400; + the interactive layer about 2,700. +* 140 crates in the dependency tree, 25 of them direct. Release binary 3.7 MB + (brush: 14.4 MB). +* 880 tests pass on the journal branch, 869 on the containment branch. On + `main` and the journal branch *no test file runs the built program*: all 22 + call library functions. The tests named for wildcards call `expand_glob` + directly; the tests for `[` call `parse_test_expr` with the `]` still in + place. That is how R2 and R3 pass their tests and fail in use. The containment + branch adds the first tests that run the program (the fifteen hole + regressions). +* No release tags. No packages. +* Other implementation directories that are not part of the running shell: + `impl/elixir` (6,417 lines; build broken), `impl/zig` and `ffi` (8,877 lines; + builds, not connected), `impl/ocaml` (860 lines; design only), `impl/mcp` + (8 files of AffineScript; its README describes ReScript and Deno files that + are no longer there). The `alkahest-shell-transmuter` directory (49 files) is + a separate alpha project. +* 60 files sit at the top level of the repository, 35 of them documents, + several of which are reports from earlier sessions. + +=== 2.7 What works well + +The parts that carry the project's idea are the best-made parts (Measured): + +* `explain rm f` shows the preconditions, the state change, the inverse and the + theorem, and changes nothing. +* `checkpoint`, `diff` and `restore` name a state, show what separates it from + now, and return to it. +* `begin`, `commit` and `rollback` group operations. On valence-shell#230 + rollback first checks every operation's record and refuses the whole rollback + if one cannot be reversed. A failure that only the file system reveals + part-way through still leaves the transaction open, with the newer operations + reversed (ruling OD-39). +* The journal on valence-shell#230 survives a crash at every point the harness + can reach, and its model is proved in Agda. That holds for one shell. Two + shells on the same root can still lose each other's records until the lock of + step C3. +* On valence-shell#232, thirteen of the fifteen known runtime holes are closed. + +Three things in the product's own output say more than the proof ledger allows, +and should be reworded whatever else is decided: + +* `replay` and `diff` print "all proven reversible ✓" and "All reversals proven". + The theorems are about the model; the link from the model to this code is + `tested-only` in the ledger. One of the theorems named, for variables, is + listed in `history --proofs` as "pending". +* `proofs` prints "✓ Zig FFI layer checks preconditions". That layer is not + connected to the shell. +* `obliterate` describes its three overwrites as "NIST SP 800-88 Purge". In + that standard, overwriting with ordinary write commands is the *Clear* level; + Purge needs device commands or cryptographic erase (Reported: section 5.3). + +''''' + +== 3. What undo covers in a real session + +Undo covers what the shell does itself: `mkdir`, `rmdir`, `touch`, `rm`, `cp`, +`mv`, `ln -s`, `chmod`, `chown`, output redirection and variable assignment. +Measured in an interactive session and with `-c`: + +[cols="4,4",options="header"] +|=== +|Action |`undo` afterwards + +|`touch t`, `rm t`, `mv a b` |Reversed +|Edit and save a file in `vim` |"Nothing to undo" +|`sed -i s/a/X/ f` |Not reversed; `undo` goes to the operation before it +|`git init` |"Nothing to undo" +|`echo hi > f` on `main` and the journal branch |Fails: "Undo touch failed: No such file or directory". Fixed on valence-shell#232 (the redirection records a path relative to the root) +|A file changed by another program since the operation |`undo` overwrites the other program's data (hole b, still open: section 10, decision 4) +|=== + +In ordinary work most changes to files are made by other programs: editors, +version control, build tools, package managers, `sed`, `tar`. So the feature the +project is named for applies to a small share of what a user does, and only if +they type the plain form of a handful of commands (section 2.3). + +Two further limits follow from recording everything durably: + +* The previous value of an overwritten variable is written to the state file + (Measured: after `T=first; T=second` the file holds `first`). A secret held + in a variable therefore reaches the disk. The file is mode 0600 on the + journal branch and 0644 on `main`. +* Contents of removed and overwritten files are kept in the same file until + step C4 of the route plan moves them to a store of their own. + +''''' + +== 4. What this is + +=== 4.1 Three things in one repository + +. *An engine for reversible operations*: operations with inverses, a history, + transactions, checkpoints, a journal, containment, and deliberate + irreversible deletion with a record. About 9,400 lines. Sound in outline, + with known holes being closed. +. *A proof corpus about a model of that engine*: Agda as the source, rechecked + in Lean and Coq. Sound for the model. Not connected to the code by anything + mechanical. +. *A shell interpreter wrapped round the engine*: about half the Rust. It is + the weakest part, the largest part, and the part the roadmap spends most of + its words on ("v0.10.x POSIX coverage"). + +The first two are what make the project different. The third is what a user +meets first. + +=== 4.2 What is new here and what is not + +Reported unless marked; links in section 5.3. + +[cols="2,5,4",options="header"] +|=== +|Feature |Already available elsewhere |Not available elsewhere + +|Undo |Deleted files (`trash-cli`, `gio trash`). Whole volumes back to a snapshot (snapper on btrfs, ZFS, NILFS2). One command inspected before it is applied (`try`). Files an AI agent changed with its own editing tool (agent checkpoints) |Undo and redo of the effects of arbitrary commands, one command at a time, with grouping. No mainstream shell has undo +|Containment |Landlock, bubblewrap, firejail, nsjail, systemd settings: mature, and they wrap any shell |Nothing: this is a property the product must have, not a reason to choose it +|Record of what was done |`auditd`, `sudo` session logs, `tlog`, Teleport; sealing in journald and in transparency logs |A record that ties each command to the changes it made and can show it was not altered +|Deletion with evidence |Device commands (`nvme sanitize`, `blkdiscard --secure`), cryptographic erase |Nothing reliable for a single file on flash or copy-on-write storage, and software cannot change that +|Proofs |Smoosh (a tested semantics of the POSIX shell, not proved); CoLiS (a core calculus proved, the translation trusted) |No machine-checked shell exists. Proofs about the operations, as here, are a smaller target and a reachable one +|=== + +Two signals from that history deserve weight: + +* Microsoft withdrew file-system transactions twice for lack of use: + Transactional NTFS ("extremely limited developer interest") and the + transaction cmdlets in PowerShell ("very limited usage"). Transactions that + developers have to ask for were not wanted (Reported). +* `try` won a best-paper award at OSDI in 2026 (Reported, checked), and every + AI coding agent's documentation says its rewind does not cover what shell + commands did (Reported; checked for one of them). Undo that needs no + cooperation from the command is wanted now. + +=== 4.3 The jobs it could do + +[cols="1,4,4,3",options="header"] +|=== +|# |Job |What people use today |What it would take + +|J1 |"I am about to do something risky to this directory tree and want to be able to take it back exactly" |Copy the directory first; git, if it is a repository; a snapshot, if the file system has them |Undo for any command, in the shell they already use, at no noticeable cost +|J2 |"An AI agent is about to run commands in my project. I want it confined, I want to see what it changed, and I want one step back" |Containers and sandboxes (isolate, do not undo); git; agent checkpoints (their own edits only) |The same engine behind a tool interface for agents +|J3 |"I must be able to show what was done, and what was deleted" |`sudo` logs, `auditd`, ticket systems, device erase certificates |A sealed record, and deletion methods that match the storage +|J4 |"I study or teach reversible computing and verified systems" |Papers |The model, the proofs and a small demonstrator +|=== + +None of the four needs a new shell language. J1 and J2 need coverage of +arbitrary commands and low friction. J3 needs integrity of the record. J4 needs +the proofs. A new interpreter is a way of delivering them, and it is the most +expensive way. + +=== 4.4 A statement of identity to rule on + +Proposed for the project's deed once the owner has ruled. Until then it is a +proposal and no descriptive file is changed. + +*IS:* an environment for accountable operations on a directory tree. Every +change it makes is either reversible, with a machine-checked account of why the +reversal is right, or deliberately irreversible, with a durable record that it +happened. Shells, scripts and agents are ways of reaching it. + +*IS NOT:* a replacement for bash; a POSIX shell; a sandbox for hostile code; a +backup system; a compliance product; a proof that a storage device forgot +anything. + +*The name still fits:* a valence shell is the outer layer through which an atom +bonds. Here it is the layer through which a tree is changed. + +''''' + +== 5. Comparison + +=== 5.1 Shells as products + +The research pass for this table stopped part-way on a usage limit. The rows +marked *checked* were read by hand on the project's own page on 2026-11-17, and +the battery column is Measured. The other rows are from general knowledge and +are *Unverified* here; they are kept short for that reason. + +[cols="2,5,2,4,2",options="header"] +|=== +|Shell |What it is |Same as bash on the 112 lines |How it is extended |Grade + +|bash |The default shell on most Linux systems; POSIX with many extensions |112 (the reference) |Sourced files, functions, a prompt variable, a hook before each prompt, programmable completion. Collections: bash-it, oh-my-bash |Unverified +|dash |A small, strict POSIX shell used for system scripts |91 |Hardly at all; that is its point |Measured +|zsh with oh-my-zsh |zsh is a POSIX-like shell with extensions. Oh My Zsh is "an open source, community-driven framework for managing your zsh configuration": 300+ optional plugins, 140+ themes, 2,500+ contributors, about 191,000 stars, MIT |Not run |A plugin is a file that zsh reads at start-up. It is switched on by naming it in `plugins=(...)` in `~/.zshrc`; a user's own go in `custom/plugins/` |Oh My Zsh *checked*; zsh Unverified +|fish |Not POSIX. Known for what it does unconfigured: suggestions, highlighting, completion |Not run |Its own configuration language; functions and completions as files; plugin managers such as fisher |Unverified +|nushell |Not POSIX. Pipelines carry tables and records, not text |Not run |Its own language with modules; plugins are separate programs that speak a serialised protocol |Unverified +|NGS |"Next Generation Shell is a modern programming language for DevOps." It holds that bash and Python are both "inadequate for Ops tasks". Multiple dispatch, user-defined types, exceptions for non-zero exit codes. Written in C, GPL-3.0, about 1,500 stars. Its README says "The work on the shell has just started" and that NGS "does not have a REPL yet" |Not run |As a programming language: methods and types |*checked* +|Oils |Two languages in one project: OSH, which aims to run existing bash scripts, and YSH, a new language. Tested with a large corpus of cases run against several shells |Not run |- |Unverified +|brush 0.4.0 |"bash/POSIX-compatible shell implemented in Rust". "Bash-compatible. Embeddable. Extensible." More than 2,500 test cases "run the same script under brush and Bash and compare stdout, stderr, exit status, and filesystem side effects". Rated by its authors as "daily-driver quality" on Linux and on macOS (Apple silicon). Still missing: `select`, `wait -n`, `disown`, some traps. MIT, about 2,300 stars |111 |"brush-core is a tested implementation of Bash semantics that your own Rust software can embed and extend" |*checked*; battery Measured +|PowerShell |Pipelines carry objects; modules; runs on Linux |Not run |Modules and a package gallery |Unverified +|*vsh today* |Section 2 |*57* |Nothing: no start-up file, no hooks, no prompt variable, no library interface |Measured +|=== + +What the table says (Inference): + +* *No shell here won users with safety features.* Each offers either + compatibility with existing scripts (bash, zsh, brush, OSH), a clearly better + time at the prompt (fish), or a new model of data (nushell, PowerShell). The + extras this project offers are not in that list, so they will not carry a + shell that is weaker than those on the basics. +* *NGS is a language first.* The project the owner had in mind as a model is a + scripting language for operations work whose own README says its shell has + only just been started. A new language is a long road even for a focused + author, and it is a different product from an interactive shell. +* *The most widely used "shell product" in the table is a collection of + plug-ins.* Oh My Zsh has no interpreter of its own. It reaches its users + inside the shell they already have. +* *A bash-compatible core in Rust already exists, is MIT-licensed, and is built + to be embedded.* brush also tests itself the way gate G0 proposes: the same + script under both shells, comparing output, exit status and effects on the + file system. + +=== 5.2 Modularity + +What "modular like oh-my-zsh" needs from a shell: a start-up file; a way to +read other files; functions and aliases; hook functions that run before and +after each command; a completion interface; a prompt the user can define. A +plug-in is then a file that uses those, and a theme is a prompt definition. + +valence-shell has the second and third (`source`, functions, aliases) and none +of the rest (Measured, section 2.4). So a plug-in system *for* this shell is +not possible today, and building one would mean building most of what makes an +interactive shell first. + +The other direction needs nothing from this shell's interpreter: deliver +`undo`, `redo`, transactions and the engine's file commands *as a plug-in for* +bash, zsh and fish. The host shell supplies the language, the completion, the +prompt and the user's existing configuration. That is stage S2. + +The engine is modular in a different sense that matters more: section 8.4 +splits it into a core, capture, the record, and front ends behind one +interface. + +An NGS-like direction is still open later, without a new parser: a typed +scripting layer over the engine in a language the estate already has +(section 11). + +=== 5.3 Where each extra already exists + +Reported by a research pass on 2026-11-17 from the linked pages, except the +three marked *checked*, which were re-read by hand. + +*Undo, dry run, transactions* + +* `try` runs a command against an overlay of the file system built from Linux + namespaces and overlayfs, then asks "Commit these changes? [y/N]". Linux 5.11 + or later; MIT licence. It calls itself a "semisolate (as opposed to a fully + isolated sandbox)" and says "network calls are all allowed". Its paper won a + best-paper award at OSDI 2026. It has no undo after the commit (*checked*). + https://github.com/binpash/try +* snapper compares and reverts between two snapshots on btrfs or thin LVM + (`undochange`); `zfs rollback` discards everything since a snapshot; NILFS2 + takes checkpoints continuously. All work on whole volumes or by time, not by + command (Inference). + https://man.archlinux.org/man/snapper.8.en, + https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rollback.8.html, + https://docs.kernel.org/filesystems/nilfs2.html +* `trash-cli` and `gio trash` cover deletion only. The author of `trash-cli` + says of aliasing `rm` to it: "You can but you shouldn't". + https://github.com/andreafrancia/trash-cli +* Transactional NTFS: Microsoft "strongly recommends" alternatives and cites + "extremely limited developer interest". The transaction cmdlets of Windows + PowerShell were removed from PowerShell 6 for "very limited usage". + https://learn.microsoft.com/en-us/windows/win32/fileio/deprecation-of-txf, + https://learn.microsoft.com/en-us/powershell/scripting/whats-new/differences-from-windows-powershell +* No mainstream shell has undo (Inference from absence). +* AI coding agents: Claude Code's rewind restores files changed by its own + editing tools; "A file that only Bash commands changed isn't tracked and + stays as it is" (*checked*). Cursor's checkpoints restore "files only"; + Gemini CLI commits to a shadow git repository before its own file tools run; + Copilot CLI restores "the files it changed". No vendor page read claims to + roll back what shell commands did. + https://code.claude.com/docs/en/checkpointing, + https://cursor.com/docs/agent/chat/checkpoints, + https://github.com/google-gemini/gemini-cli/blob/main/docs/cli/checkpointing.md, + https://docs.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/roll-back-changes + +*Containment* + +* Landlock lets any process restrict itself (Linux 5.13 onward); bubblewrap, + firejail, nsjail and systemd's sandbox settings are widely packaged. Agents + use Seatbelt on macOS and bubblewrap, Landlock and seccomp on Linux. + https://docs.kernel.org/userspace-api/landlock.html, + https://github.com/containers/bubblewrap +* `openat2` with `RESOLVE_BENEATH` (Linux 5.6) and the `cap-std` library, which + valence-shell#232 uses, confine a program's own path handling. `cap-std` + states that it "is not a sandbox for untrusted Rust code". + https://man7.org/linux/man-pages/man2/openat2.2.html, + https://github.com/bytecodealliance/cap-std +* Shells with confinement built in exist only as research (Shill, OSDI 2014). + https://www.usenix.org/conference/osdi14/technical-sessions/presentation/moore + +*Record of what was done* + +* `auditd` records system calls. It can record the calls a shell makes for a + built-in command. An event can include a `proctitle` record with the title + and command-line parameters of the process that made the call, so a command + given as `sh -c '...'` can appear there, and a command typed into a running + shell does not. Not every event has that record: an audit rule can leave it + out (corrected three times after review; the field is from the audit field + dictionary, read 2026-11-17, and the rest is not re-read on a primary + page). + https://github.com/linux-audit/audit-documentation/blob/main/specs/fields/field-dictionary.csv, + `sudo` records sessions and `sudo_logsrvd` collects them; Teleport records + terminal sessions and says recordings can be evaded. + https://man7.org/linux/man-pages/man7/audit.rules.7.html, + https://www.sudo.ws/docs/man/sudo_logsrvd.man/, + https://goteleport.com/docs/reference/architecture/session-recording/ +* Sealing exists in general form: journald's Forward Secure Sealing, and + transparency logs such as Rekor. + https://www.freedesktop.org/software/systemd/man/latest/journalctl.html, + https://docs.sigstore.dev/logging/overview/ +* No mature tool that binds shell commands to a sealed chain was found. A chain + alone cannot show that nothing was removed from its end; that needs a witness + outside it. + +*Deletion* + +* NIST SP 800-88 Rev. 2, published September 2025, replaces Rev. 1 of 2014 + (*checked*). It keeps three levels: Clear (ordinary read and write commands), + Purge (device sanitize commands or cryptographic erase) and Destroy. + https://csrc.nist.gov/pubs/sp/800/88/r2/final +* Wei and others (FAST 2011): "none of the existing hard drive-oriented + techniques for individual file sanitization are effective on SSDs". GNU + `shred` lists journaling, snapshots, RAID, compression and flash as defeating + it. + https://www.usenix.org/legacy/events/fast11/tech/full_papers/Wei.pdf, + https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html +* GDPR Article 17 prescribes no technique. The UK regulator asks that backups + be put "beyond use"; a European Data Protection Board report of February 2026 + asks controllers to verify erasure and "be able to demonstrate such erasure". + https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/right-to-erasure/, + https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf + +*Proofs* + +* Smoosh (POPL 2020) is an executable semantics of the POSIX shell validated by + testing: "Our tested semantics". CoLiS proves an interpreter for a core + language correct in Why3; the translation from shell to that language is + trusted. No machine-checked implementation of a full shell was found. + https://arxiv.org/abs/1907.05308, + https://pmc.ncbi.nlm.nih.gov/articles/PMC7480694/ + +''''' + +== 6. Ratings by aspect + +Scale: *0* absent; *1* present, but wrong or unsafe in ordinary use; *2* works +when demonstrated, fails in ordinary use; *3* usable within stated limits; +*4* dependable; *5* as good as the best available. + +"Open PRs" means `main` with valence-shell#230 and #232 merged. "Reaches 4 at" +names the stage of section 8 where the aspect becomes dependable. + +[cols="3,1,1,6,2",options="header"] +|=== +|Aspect |`main` |Open PRs |Evidence |Reaches 4 at + +|Command language |1 |1 |57 of 112; R1 to R6, R9, R11, R12 |S2 by using the user's own shell; S6 for a shell of its own +|File commands |1 |1 |6 of 22 forms; five silent (section 2.3) |S1 +|Running other programs |1 |1 |R1, R2, R10, R14. Pipes and redirection to files work |S2 or S6 +|Interactive use |2 |2 |Editing and history are good; `help`, file completion, Ctrl-Z, prompt and start-up file are missing |S2 or S6 +|Undo: how much it reaches |1 |1 |The shell's own commands only (section 3) |S3 +|Undo: is it right |1 |3 |Fifteen runtime holes; thirteen closed on #232; (b) and (o) open |S1 +|Transactions |2 |3 |Rollback checks first on #230 |S1 +|Durability of its own state |1 |3 |`main` rewrites one file. #230 has the journal, a crash harness and a proved model, for one shell. Two shells on the same root can still lose records: a rewrite by one discards what the other appended since |C3 (the lock) +|Containment |1 |3 |On `main`, `cd` moves the root. #232 confines the built-in commands by handle. Other programs are not confined |S3 and S4 +|Record of what was done |1 |2 |A plain file inside the tree it describes; no sealing. #232 writes the intent before an obliteration |S5 +|Deletion with evidence |1 |2 |Overwrite only, mislabelled; #232 refuses symbolic and hard links |S5, and 3 at most for single files +|Proofs about the model |4 |4 |Agda source checks; Lean and Coq recheck, with the gaps listed in the ledger |Reached +|Link from the proofs to the code |1 |1 |`tested-only` in the ledger; no contracts on the Rust |Phase E +|Tests of the product |0 |1 |No test runs the program on `main`; fifteen do on #232 |S0 +|Modularity and extension |0 |0 |No library interface, start-up file, plug-ins or hooks |S1 and S2 +|Speed |2 |3 |R14; the journal is about thirteen times faster on built-in operations |S0 +|Truth of its own claims |2 |2 |Status pages are honest; feature lists and three in-product texts overstate |S0 +|Privacy |1 |2 |Old file contents in a 0644 file on `main`; 0600 on #230 but still inside the tree, and old variable values are kept |C3, plus a rule for variables +|Releases and packaging |0 |0 |No tags or packages; Rust CI cannot start |S1 +|Portability |1 |1 |Linux only; nothing run elsewhere |Open +|=== + +Taken together: as an engine with proofs it is a 3 and rising; as a shell it is +a 1. + +''''' + +== 7. Routes and what they cost + +=== 7.1 The unit + +One *work package* (WP) is about what valence-shell#230 was: a model, an +implementation, a test harness and its review rounds. That is roughly one long +working session of the kind this month has had. Every figure below is a size +class inferred from this month's work, not a promise (Inference). + +=== 7.2 Five routes for the front end + +*Route A. Grow the present interpreter into a POSIX shell.* + +* Needs: a new word and expansion pipeline in place of the string escapes + (section 9.1), 2 to 3 WP; the missing constructs and built-in commands, up to + dash's level on a public corpus, 3 to 6 WP; options for the file commands, + job control, completion, a start-up file and a prompt, 2 to 4 WP. bash's own + extensions after that have no fixed end. +* Buys: a shell like the others, with the extras attached. +* Others on this road (section 5.1): brush stands on more than 2,500 + comparison tests against bash, and NGS's own README says its shell has only + just been started. +* It is the most expensive route to the least distinctive result. + +*Route B. The engine first, front ends in stages.* Section 8. Recommended. + +*Route C. Put an existing shell core in front now.* brush is the candidate +(Measured): + +* 111 of the 112 lines; MIT licence; `brush-core` 21,732 lines and + `brush-parser` 10,154 lines; version 0.x. +* A program that embeds it can register its own built-in commands + (`register_builtin` in `brush-core` 0.5.0). No hook was found for wrapping + the programs it starts. +* Needs: the engine's commands registered as built-ins, and a decision on how + capture attaches: 1 to 2 WP for a working interactive shell, then the cost of + following a young project whose interface is not yet stable. +* It adds about 32,000 lines that are outside every proof to the path a command + takes. Adding the dependency is the owner's call. + +*Route D. Agents only.* Build the engine, capture and the agent server +(stages S1, S3, S4) and no shell for people. + +*Route E. Research artefact only.* Keep the model, the journal proofs and a +small demonstrator; publish; make no product. + +Rated 1 to 5 on each aspect, higher is better (for cost and risk, higher means +cheaper and safer). Inference. + +[cols="4,1,1,1,1,1",options="header"] +|=== +|Aspect |A own shell |B engine first |C existing core |D agents only |E research only + +|Usable by someone soon |1 |4 |4 |3 |1 +|Delivers what is distinctive |2 |5 |3 |4 |2 +|Cost |1 |3 |3 |4 |5 +|Risk |2 |4 |3 |3 |5 +|Fit with the proofs (small trusted core) |2 |5 |3 |4 |5 +|Who can use it |2 |4 |3 |2 |1 +|A stable product at each step |2 |5 |3 |4 |- +|*Elegance and correctness (E, of 10)* |*4* |*9* |*6* |*7* |*5* +|*Dependability, security, privacy (D, of 10)* |*4* |*8* |*6* |*7* |*8* +|=== + +B and C are not rivals in the long run: C is the likely answer to stage S6 of +route B. + +=== 7.3 Three routes for the engine + +* *One inverse per operation* (today). Eleven operations, each with its own + undo, redo and rollback code. The route plan already asks for a single + inverse table (step B5). +* *Change sets with footprints* (recommended, section 9.2). One type covers the + built-in commands and the captured effects of any other command. +* *Snapshots only*, leaving the work to btrfs or ZFS. No per-command undo + without those file systems, and it needs privileges. Usable as an optional + back end, not as the base. + +=== 7.4 Experiment: capturing any command + +Measured in this session, as root, on Linux 6.18. + +* A command was run in a new user and mount namespace against an overlay of a + directory tree. `sed -i`, an unlink, a new file and a new directory were all + captured in the overlay's upper layer, and the real tree was unchanged. +* `git init`, `git add` and `git commit` over 200 files, then `sed -i` on two + of them, ran normally inside. 377 entries were captured. The real tree's hash + was the same before and after. +* Cost per command: 2.11 ms plain; 4.10 ms with the namespaces; 6.64 ms with + the overlay mounted. About 4.5 ms is added. (R14 adds 50 ms today.) + +Not shown: applying the captured changes to the real tree and undoing them +later, which is the part to build; an ordinary user; nested mounts; large +trees; any system without overlayfs. + +=== 7.5 What the pieces cost + +Inference. "Planned" means already in the route plan. + +[cols="5,2,5",options="header"] +|=== +|Piece |Size |Notes + +|The battery as a CI ratchet; honest wording; `help`; built-ins refuse what they do not understand |0.5 WP |Stage S0 +|R3, R9 (exit status), R14: each a few lines |with S0 |Pure defects +|Rest of Phase B: hole (b), remaining commands on the handle, the ban on ambient file calls, one inverse table |1 WP |Planned +|C3 state outside the tree and a lock; C4 store for old contents; C5 recovery |1 to 2 WP |Planned +|Change-set type, in the model and in Rust |1 to 2 WP |Replaces the inverse table; closes (b) for good +|Library interface, command-line tool with machine-readable output, first tagged release |1 WP |Stage S1; the extraction is planned as Phase E +|File commands with the common options, checked against coreutils |1 WP |Refuse the rest +|Plug-in for bash, zsh and fish |1 WP |Stage S2 +|Capture: run, apply, undo; the experiment first |2 to 3 WP |Stage S3. macOS is a separate, unknown cost +|Agent server |1 to 2 WP |Stage S4 +|Sealed record, verifier, deletion by storage type |1 to 2 WP |Stage S5; needs decisions on keys +|Spec spine: finite model, oracle, trace corpus |2 to 3 WP |Planned, Phase D +|Contracts on the Rust core |2 to 4 WP |Planned, Phase E; depends on tool maturity +|An interactive shell on an existing core |1 to 2 WP, then upkeep |Stage S6, route C +|An interactive shell of our own |8 to 13 WP, then no fixed end |Route A +|=== + +''''' + +== 8. A staged model: a stable product at every step + +=== 8.1 The gap in the present plan + +The route plan's definition of done has eleven gates, all about the engine and +the proofs, and lists "full POSIX" as a non-goal. It does not say what the +shell must be able to do. So a release could meet every gate and still fail +`[ -d / ]`. + +Proposed *gate G0, use*: what the project claims a front end can do equals +what a battery, run in CI against a reference, shows it does. The battery is a +ratchet: a listed failure that starts to pass must be taken off the list, and +an unlisted failure fails the build. valence-shell#232 already uses this +pattern for the runtime holes. + +=== 8.2 The stages + +Each stage ends in a tagged release with a generated report of what passed. + +[cols="1,4,3,4,3,1",options="header"] +|=== +|Stage |Product |Who can use it |Gate |Stop rule |Size + +|*S0* Truthful baseline |The present shell, with claims that match measurements |Nobody new |G0 runs in CI. Built-in commands refuse options and extra operands they do not handle. `help` works. The three texts of section 2.7 are reworded. R3, R9 and R14 fixed |None: always worth doing |0.5 WP +|*S1* The engine as a product |A library and a command-line tool for reversible operations in a contained tree: the operations, `undo`, `redo`, transactions, checkpoints, `explain`, `obliterate`; machine-readable output; real exit codes; state outside the tree |Scripts; other tools; the estate |Every operation's round trip agrees with the executable model on generated sequences. Crash harness clean. All fifteen holes closed. First release |If nobody, the owner included, uses it in a script within a month of release, revisit section 4.4 before going on |4 to 6 WP, mostly planned +|*S2* Undo in the shell you already use |One plug-in file each for bash, zsh and fish: `undo`, `redo`, `txn`, a prompt segment, a per-directory switch, and the engine's file commands in place of the system's inside enabled trees |Anyone on Linux, in their own shell with their own configuration |Same tree as coreutils on generated cases for every supported form. Under 5 ms added to start-up. Four weeks of the owner's daily use with no wrong undo |If the owner switches it off in their own shell, stop and reassess |1 WP +|*S3* Undo for any command |`vsh run -- command`, and the plug-in wrapping command lines in enabled trees: the command runs against an overlay; its changes are applied with the old versions kept; one change set is journalled; `undo` restores exactly |The same people, now for editors, git, builds |On a corpus of twenty real tools: tree hash before equals tree hash after undo, or a clear refusal. A crash while applying recovers to before or after |If capture cannot run for an ordinary user on two mainstream distributions, or exactness fails without a refusal: keep S2 and say so |2 to 3 WP +|*S4* For agents |A tool server offering run, show changes, apply, undo, checkpoint and the record, with the tree confined |People running AI coding agents |Escape tests pass. On a task corpus, rollback restores the tree exactly. The record lists every command with its changes |If agent vendors close the gap themselves, offer the library instead |1 to 2 WP +|*S5* Accountability |A sealed record kept outside the tree, evidence that can be exported and checked by a separate tool, and deletion methods chosen by storage type and named correctly |Operations and compliance work |Edits, deletions, truncation and reordering of the record are detected. The verifier is a second implementation |None set yet |1 to 2 WP +|*S6* A shell of its own |An interactive shell with the engine built in |People who want one program |Decided on evidence from S2 to S4 about what users lack. Battery at dash's level on the POSIX part, then a public corpus |If S2 to S4 users lack nothing a plug-in cannot give, do not build it |Route C or A +|=== + +The assurance work runs beside the stages: the journal model is done; the +finite model, the oracle and the change-set lemmas belong to S1; the apply +protocol to S3; the chain to S5; contracts on the Rust core from S1 onward. + +=== 8.3 How the present phases fit + +Phases A, B, C3 to C5, D and E of the route plan are all stage S1. Phase F is +part of S0. Nothing built this month is lost on routes B, C, D or E. On route +A the interpreter would come first and the engine work would wait. + +What changes under route B: S0 is new and comes first; the POSIX items in +`ROADMAP.adoc` ("v0.10.x") and in the root `CLAUDE.md` ("Immediate priorities") +are parked; the present interpreter stays as an experimental front end with its +claims cut to what the battery shows. + +=== 8.4 Sections that evolve separately + +[cols="2,5,3",options="header"] +|=== +|Section |Holds |Rule + +|Core |Change sets, journal, store for old contents, containment, inverses |No terminal output and no parsing. Everything proved or under contract lives here +|Capture |Running a command against an overlay and producing a change set |Depends on the core only +|Record |The sealed log and its verifier |Depends on the core only +|Front ends |Command-line tool, plug-ins, agent server, interactive shell |May be replaced without touching the core +|Proofs |One model per core module |The Agda tree is the source +|=== + +The core's interface (a change set in, a journal record out) is the stable +boundary. That boundary is what lets each front end ship on its own. + +=== 8.5 Working rules for fast, largely autonomous coding + +. *An oracle before code.* No area is opened for autonomous work without + something executable to check it against and a ratchet: a reference shell for + language behaviour, coreutils for the file commands, the model for the + engine, tree hashes for capture, the crash harness for durability. +. *An experiment before a build*, time-boxed, with GO or NO-GO written down. +. *One stage, one release.* +. *Claims are generated* from what passes, not written by hand. +. *Refuse, never approximate.* Input that is not understood is an error. +. *Mutation checks* on the tests and proofs of each gate, as was done for the + journal model. +. *Exactness stays with the formal side and the owner*: what a theorem says, + licences, identity, keys. +. *A named first user* for each stage, and a stop rule. + +Work that suits this well: S0, the file-command options, the plug-ins, the +agent server. Work that does not: the statements of the theorems, the trust +design of the record, anything about identity or licence. + +''''' + +== 9. Types, logics and proofs that give a more final answer + +=== 9.1 Words that know what was quoted + +R1 to R6 exist because quoting is stored as backslashes in strings. The +structural fix is a word type: a word is a list of segments, each marked quoted +or not. Expansion produces fields. Only unquoted segments are split and matched +against file names. Arguments are built from fields, and no escape character +exists inside the program at all, so the fault of R1 cannot be written. + +One property covers the class: for every byte string `s`, the argument that a +program receives for the quoted form of `s` is `s`. It is a property test +today and a short lemma over the word type later. This matters only if an +interpreter of our own is kept. + +=== 9.2 Change sets with footprints + +An operation's effect is a set of changes; each change gives a path, what was +there before and what is there after (kind, metadata, content hash). The +inverse swaps before and after. A change set may be applied only when every +path it names is in its "before" state. + +A change set's *footprint* is the paths it names together with their parent +directories, because creating or removing `a/b` also changes what `a` holds. + +* *One theorem instead of eleven:* applying the inverse after the change set + returns the tree, given that precondition. `mkdir` and `rmdir`, write and + restore, move and move back are instances. +* *Hole (b) is answered by construction:* an undo whose paths no longer match + what the operation left is not applicable, so it is refused. Nothing needs to + be remembered to check. +* *Paths outside the footprint are untouched* (the frame rule of separation + logic), so changes that other programs make outside it do not invalidate an + undo. +* *Two change sets whose footprints do not overlap commute* (as in the patch + theories of Darcs and Pijul), so an earlier operation can be undone without + undoing the later ones. Different paths are not enough for this: creating + `a/b` depends on creating `a`. Two changes are independent only when neither + path is the other or an ancestor of it. Dependent changes are undone + together, newest first, or the undo is refused. No shell or snapshot tool + offers selective undo. +* The built-in commands and the captured effects of any program become the same + thing, which is what makes stage S3 an extension and not a second engine. + +In the estate's own vocabulary, the "before" half of a change set is the echo +of a lossy map: the witness needed to go back. That reading stays +documentation only, as the bridge rules require. + +=== 9.3 Authority only by handle + +valence-shell#232 gives the built-in commands a handle on the session root and +no other way to reach files. The remaining step in the plan, a ban on ambient +file calls outside one module, turns a convention into something the compiler +checks. For other programs the same idea is Landlock together with the +overlay. + +=== 9.4 A one-use permit for what cannot be undone + +Obliteration is refused inside a transaction by a run-time check. A permit +value that cannot be copied, issued by the confirmation and consumed by the +operation, makes "irreversible operations happen only with a fresh, explicit +permit" something the type checker enforces. Rust's ownership expresses it +directly. It is also the natural meeting point with the estate's affine types. + +=== 9.5 A chained record + +Each entry of the record carries the hash of the one before, and checkpoints +are signed. That makes edits and reordering detectable. It does not show that +entries were not cut from the end; a copy of the latest hash held somewhere +else does. The model-level lemma is small. The real questions are where the +keys and that copy live, and they are the owner's. + +=== 9.6 Ratchets + +The battery, the hole list and the proof manifest are the same device: a list +that may only shrink, checked by a machine. Every stage gate in section 8 has +that form. + +''''' + +== 10. Decisions for the owner + +Rated E (elegance and correctness) and D (dependability, security, privacy), +out of 10. The elegant arm is named in each. + +*Decision 1. Which route.* + +[cols="4,1,1,5",options="header"] +|=== +|Option |E |D |Note + +|B. Engine first, front ends in stages |9 |8 |The elegant arm, and recommended +|A. Own shell first |4 |4 |What the roadmap implies today +|C. An existing shell core in front now |6 |6 |Usable soonest as a shell; a large dependency outside the proofs +|D. Agents only |7 |7 |The out-of-the-box arm: narrow and timely +|E. Research artefact only |5 |8 |Gives up the aim of production +|=== + +*Decision 2. The truthful baseline (S0).* + +[cols="4,1,1,5",options="header"] +|=== +|Option |E |D |Note + +|Now, as its own small pull request from `main` |9 |9 |The elegant arm, and recommended. Needs a fourth branch +|The ratchet and the wording only, no fixes |7 |8 |Leaves `[`, the exit status and the 50 ms wait as they are +|Later, with Phase F |5 |5 |The claims stay ahead of the product meanwhile +|Mark the interpreter experimental and spend nothing more on it |6 |7 |The out-of-the-box arm; fits routes D and E +|=== + +*Decision 3. The first front end after the engine.* + +[cols="4,1,1,5",options="header"] +|=== +|Option |E |D |Note + +|Let the capture experiment decide |9 |8 |The elegant arm, and recommended. It needs only the journal and the store for old contents, so it can run before S1 is finished. GO: capture first, and the plug-in is a thin layer over it. NO-GO: the plug-in with the engine's own commands +|Plug-in first |7 |8 |Daily use soonest; part of it is discarded if capture works +|Agent server first |7 |7 |Follows the clearest demand +|Publish the engine and model first, no front end yet |5 |8 |The out-of-the-box arm +|=== + +*Decision 4. Hole (b): undo destroys what another program has written since.* + +[cols="4,1,1,5",options="header"] +|=== +|Option |E |D |Note + +|Check the footprint: each operation records what it left (kind, size, time, content hash); undo and rollback compare first and refuse on a difference; a forced undo first saves what is there |9 |9 |The elegant arm, and recommended. It is the rule of section 9.2 applied early +|Refuse only, with no way to force |7 |9 |Safe; can leave a history that cannot be unwound +|Always undo, but keep a copy of what was displaced |6 |7 |Recoverable, but newer work is replaced without asking +|Leave it for the change-set engine |8 |5 |The out-of-the-box arm: one fix in the right place, with a data-loss hole open until then +|=== + +*Smaller points, with the default I will use unless told otherwise:* + +* Variables: keep their undo in memory and stop writing old values to disk + (privacy and speed, section 3). This changes what `undo` does to variables + across restarts. +* The three texts of section 2.7 are reworded in S0 or Phase F, whichever comes + first. +* The top-level documents and the implementation directories that do not run + are the owner's to keep or archive. Nothing is deleted. + +''''' + +== 11. Beyond this repository + +For the maintained list. + +* *brush* (MIT, `reubeno/brush`): the candidate core for stage S6. Worth + watching for a stable embedding interface and a hook on program start. +* *`try`* (MIT, `binpash/try`): the closest prior art for stage S3, and a + possible source of test cases and of collaboration. +* *A public conformance corpus:* the Oils project's spec tests run thousands of + cases against several shells. They should replace this review's battery as + the yardstick for any shell language work. Not run here. +* *januskey `reversible-core`:* if `vsh-core` is extracted in S1, the estate + has two reversible engines. Whether they are one is an estate decision (D204 + is a ruling, not an implementation). +* *maa-framework:* stage S5 gives the record a concrete form; its definitions + should come from there. +* *echo-types:* section 9.2's reading of undo data as an echo is documentation + only. A mechanised link would be new work in that repository's terms. +* *AffineScript:* a typed scripting layer over the engine (the NGS-like + direction of section 5.2), and the permit of section 9.4, are natural uses of + affine types. Its readiness was not assessed. +* *`standards`:* PROOF-POLICY grades proofs. Nothing grades whether a product + does what its page says. Gate G0 may be worth proposing estate-wide. +* *The allowed-actions list* (step A1) still stops Rust CI in valence-shell, + maa-framework and absolute-zero. A ratchet that CI cannot run is not a gate. +* *`impl/mcp`:* stale, and written against Deno, which the estate is removing. + The language of the stage S4 server needs a ruling. + +''''' + +== 12. What was not verified + +* Nothing was run as an ordinary user, or on macOS or BSD. +* The capture experiment showed capture only. Applying and undoing are not + built. +* brush was run on the 112 lines, non-interactively. Its interactive behaviour, + stability and upkeep were not assessed. +* Sections 4.2 and 5.3 rest on a research pass that read the linked pages; + three of its items were re-read by hand. Its own list of what it could not + verify stands: among them the status of Plash, how `auditd` relates to a + shell's built-in commands (secondary sources), and whether `try` contains + anything beyond file effects. +* The research pass for section 5.1 stopped on a usage limit. Three projects' + pages (NGS, Oh My Zsh, brush) were read by hand; the other rows of that table + are unverified, and no adoption figures or start-up timings for other shells + were gathered. +* Timings come from one virtual machine. +* The sizes in section 7 are inferred from one month's work. +* Syntax highlighting, the pager and command correction were not examined. +* The Elixir, Zig, OCaml and Idris2 directories were sized, not built, in this + review. +* The battery is home-made and small. + +''''' + +== Appendix A. The battery + +One line per command line tested. Columns: dash, brush 0.4.0, vsh `main`, vsh +containment branch. `ok` means standard output and exit status both match bash +5.2.21; `rc` means the output matches and the exit status does not; `--` means +the output differs. `~` stands for a line break in the last entry. The journal +branch gives the same column as `main`. + +.... +dash brush main cont group command line + ok ok ok ok basic echo hello + ok ok ok ok basic echo "a b" 'c d' + ok ok ok ok basic echo one; echo two + ok ok ok ok vars x=5; echo $x + ok ok ok ok vars x=hello; echo "${x}world" + ok ok ok ok vars x="a b"; echo $x | wc -w + ok ok ok ok vars echo ${UNSET:-default} + ok ok -- -- vars x=abcdef; echo ${x#abc} ${x%def} ${#x} + -- ok -- -- vars x=hello; echo ${x^^} ${x:1:3} + ok ok -- -- vars export FOO=bar; sh -c 'echo $FOO' + ok ok -- -- vars FOO=tmp sh -c 'echo $FOO' + ok ok ok ok vars echo $HOME | grep -c / + ok ok ok ok vars echo $? ; false; echo $? + ok ok -- -- vars echo "$$" | grep -c '[0-9]' + ok ok -- -- arith echo $((1+2*3)) + -- ok -- -- arith x=7; echo $((x % 4)) $((x ** 2)) + ok ok ok ok arith i=0; i=$((i+1)); echo $i + -- ok -- -- arith ((x=3+4)); echo $x + -- ok -- -- arith let y=2*5; echo $y + ok ok -- -- subst echo $(echo nested) + ok ok -- -- subst echo "today: $(date +%Y | cut -c1-2)" + ok ok -- -- subst x=`echo back`; echo $x + ok ok -- -- subst echo $(echo $(echo deep)) + ok ok ok ok pipe echo hello | tr a-z A-Z + ok ok ok ok pipe printf 'b\na\nc\n' | sort | head -2 + ok ok ok ok pipe echo hi | cat | cat | wc -c + ok ok ok ok pipe false | true; echo $? + ok ok -- -- pipe seq 1 5 | awk '{s+=$1} END{print s}' + ok ok ok ok redir echo x > f; cat f + ok ok ok ok redir echo x > f; echo y >> f; wc -l < f + ok ok -- -- redir ls /nonexistent 2>/dev/null; echo rc=$? + ok ok -- -- redir ls /nonexistent > out 2>&1; grep -c . out + -- ok ok ok redir cat <<< "herestring" + ok ok ok ok redir echo err >&2 2>/dev/null; echo ok + ok ok ok ok logic true && echo yes || echo no + ok ok ok ok logic false && echo yes || echo no + ok ok -- -- logic [ -d / ] && echo dir + ok ok -- -- logic [ 3 -gt 2 ] && [ a = a ] && echo ok + -- ok -- -- logic [[ abc == a* ]] && echo match + ok ok ok ok logic test -f /etc/passwd && echo file + ok ok -- -- logic ! false && echo negated + ok ok ok ok if if true; then echo t; else echo f; fi + ok ok -- -- if if [ 1 -eq 2 ]; then echo a; elif [ 1 -eq 1 ]; then echo b; fi + -- ok -- -- if x=3; if (( x > 2 )); then echo big; fi + ok ok ok ok loop for i in 1 2 3; do echo $i; done + ok ok -- -- loop for i in $(seq 3); do echo n$i; done + ok ok -- -- loop for f in /etc/pass*; do echo ${f##*/}; done + ok ok -- -- loop i=0; while [ $i -lt 3 ]; do i=$((i+1)); done; echo $i + -- ok -- -- loop for ((i=0;i<3;i++)); do echo $i; done + ok ok -- -- loop for i in 1 2 3; do [ $i = 2 ] && continue; echo $i; done + ok ok -- -- loop for i in 1 2 3; do [ $i = 2 ] && break; echo $i; done + ok ok -- -- loop until false; do echo once; break; done + -- ok -- -- loop while read l; do echo "got $l"; done <<< "line" + ok ok ok ok case case foo in f*) echo starts;; *) echo other;; esac + ok ok ok ok case x=b; case $x in a|b) echo ab;; esac + ok ok ok ok func f() { echo "in f $1"; }; f arg + ok ok ok ok func f() { return 3; }; f; echo $? + ok ok ok ok func f() { local v=inner; echo $v; }; v=outer; f; echo $v + -- ok ok ok func function g { echo g; }; g + ok ok ok ok func f() { echo $#; }; f a b c + ok ok -- -- func fact() { [ $1 -le 1 ] && echo 1 || echo $(( $1 * $(fact $(($1-1))) )); }; fact 5 + ok ok -- -- glob touch a.txt b.txt c.md; echo *.txt + ok ok -- -- glob touch a1 a2 b1; echo a? + ok ok ok ok glob touch x.rs; echo *.nomatch + ok ok -- -- glob mkdir -p d/e; touch d/e/f; echo d/*/f + -- ok -- -- glob echo {a,b}{1,2} + -- ok -- -- glob echo {1..5} + ok ok ok ok quote echo 'single $x' "double \$x" + -- ok ok ok quote echo "a\tb" 'a\tb' + ok ok -- -- quote x='*'; echo "$x" + -- ok -- -- quote echo $'tab\there' + -- ok -- -- array a=(x y z); echo ${a[1]} ${#a[@]} + -- ok -- -- array a=(1 2 3); for v in "${a[@]}"; do echo $v; done + -- ok -- -- array declare -A m; m[k]=v; echo ${m[k]} + ok ok -- -- sub (cd /tmp; pwd); pwd | grep -c / + ok ok ok ok sub x=1; (x=2); echo $x + ok ok -- -- sub { echo a; echo b; } | wc -l + ok ok -- -- sub echo a & wait; echo done + ok ok ok -- builtin cd /tmp && pwd + ok ok ok ok builtin printf '%s-%d\n' abc 42 + -- ok -- -- builtin read a b <<< "one two"; echo $b + ok ok ok ok builtin set -- p q r; echo $2 $# + ok ok -- -- builtin set -e; false; echo notreached + ok ok -- -- builtin x=1; unset x; echo "[${x}]" + ok ok -- -- builtin type echo | head -1 | grep -c echo + ok -- ok ok builtin alias ll='echo aliased'; ll + ok ok ok ok builtin trap 'echo bye' EXIT; echo hi + ok ok ok ok builtin eval 'echo evaled' + -- ok ok ok builtin shift 2>/dev/null; echo ok + -- ok -- -- builtin pushd /tmp >/dev/null; pwd; popd >/dev/null + ok ok ok ok builtin echo -n nonl; echo + ok ok ok ok builtin echo -e 'a\nb' | wc -l + ok ok rc rc builtin exit 7 + ok ok -- -- builtin command -v ls | grep -c ls + -- ok -- -- builtin true; echo ${PIPESTATUS[0]} + ok ok -- -- fs mkdir -p a/b/c && ls a/b + ok ok ok ok fs mkdir d && rmdir d && echo gone + ok ok ok ok fs echo data > f; cp f g; cat g + ok ok ok ok fs echo data > f; mv f g; cat g + ok ok ok ok fs touch f; chmod 600 f; stat -c %a f + ok ok ok ok fs ln -s target link; readlink link + ok ok ok ok fs touch f; ls -la f | wc -l + ok ok ok ok fs echo a > f; cat f f | wc -l + ok ok ok ok ext ls / | head -1 | wc -l + ok ok ok ok ext env | grep -c PATH + ok ok ok ok ext sh -c 'exit 4'; echo $? + ok ok ok ok ext git --version | grep -c git + ok ok ok ok ext sleep 0.01; echo slept + ok ok ok ok ext echo abc | sed 's/b/X/' + ok ok ok ok ext find . -maxdepth 0 | wc -l + ok ok -- -- ext X=1 Y=2 env | grep -c '^[XY]=' + ok ok -- -- redir cat < `. +The shell under test is called as ` -r -c `; for dash and +brush a two-line wrapper drops the first two arguments. Battery lines have the +form `group|command line`. + +The runner's own error file is kept outside the directories under test (a +correction made after review; all four shells give the same verdicts as before +it). + +[source,bash] +---- +#!/usr/bin/env bash +# Compare everyday shell snippets under bash and vsh. No file is removed. +S="$1"; V="$2"; LIST="$3"; OUT="$4" +export NO_COLOR=1 RUST_BACKTRACE=0 +: > "$OUT" +norm() { sed -e 's/\x1b\[[0-9;]*m//g' -e '/^\[op:[0-9a-f]*\]/d' -e '/^\[txn:[0-9a-f]*\]/d'; } +run_one() { + local cat="$1" snip="$2" d1 d2 d3 bo br vo vr verr st + d1=$(mktemp -d -p "$S/work"); d2=$(mktemp -d -p "$S/work") + bo=$(cd "$d1" && timeout 10 bash -c "$snip" 2>/dev/null "$E" /dev/null 2>&1 > "$OUT" +} +mkdir -p "$S/work" +# Absolute paths: each run changes directory, and a relative path would then +# point somewhere else. +S=$(cd "$S" && pwd) +V=$(cd "$(dirname "$V")" && pwd)/$(basename "$V") +# Error output is kept outside the directories under test. +E="$S/work/stderr.txt" +while IFS= read -r line; do + [ -z "$line" ] && continue + cat="${line%%|*}"; snip="${line#*|}" + run_one "$cat" "$snip" +done < "$LIST" +run_one redir "$(printf 'cat < link`, where `link` is an in-root symlink to an outside file |The outside file is overwritten +|k |`touch lp/new`, where `lp` is an in-root symlink to an outside directory |A file is created outside the root +|m |`obliterate vl --force`, where `vl` is a symlink to an outside file |The outside file's contents are destroyed +|f |`touch f; cd sub; undo` |Deletes the user's `sub/f` and leaves `f` +|h |`chmod 600 s; rm s; undo` |The mode comes back as 0644 +|c |obliterate inside a transaction, then rollback |Changed: a failed rollback is now reported. Obliterate inside a transaction is still allowed +|e |i64 overflow in arithmetic |Inconclusive: the test form hit a parse error first +|=== + +Not re-run: a, b, d, i, l, n. + +*A new hole, found in review on 2026-11-17 (Measured, also on `main`):* +`rollback` reverses the operations on disk but leaves them live in the history. +`begin t; touch f; rollback; undo` ends with "Undo touch failed: No such file or +directory". It joins the list as hole *o*. + +*A second new hole, found on 2026-11-17 while review of the scrub on +valence-shell#232 was answered (Measured on that branch).* `obliterate` finds +the earlier contents of a file in the saved history by path. Content that was +saved while the file had another name is not found: +`echo secret > a; echo x > a; mv a b; obliterate b` leaves `secret` in the +state file, under the name `a`. The same holds for content saved through a +symbolic link that has been removed since, in a history written before +redirections recorded the file they changed. On `main` nothing is scrubbed at +all (hole d), so the same check would fail there too (Inferred; not run on +`main`). It joins the list as hole *p*, and OD-46 rules when it closes. + +*`ShellState::new` discards a failed load* (`state.load().ok()`), on `main` too. +E1d says a corrupt load must fail loudly. + +*CI does not start the Rust or shared workflows* on the open PRs (Measured; +cause Inferred). Seven workflows end in `startup_failure`. Each uses +`dtolnay/rust-toolchain`, `Swatinem/rust-cache` or a `metadatastician/standards` +reusable workflow; every workflow that still starts uses none of them. The +inferred cause is the repository's allowed-actions list. Until the owner changes +that setting, the evidence for Rust changes is local runs only. + +=== 14.2 Why the journal is being redesigned + +The stress-suite repair on #229 added an append log beside the state file, which +is the arm this plan rated lower (OD-5b), with no model and no crash test. Two +review bots then raised seventeen findings of one class, crash consistency and +state-file handling, over several rounds. Nine were fixed one at a time. The +other eight are not being patched; they are listed in valence-shell#230. + +A crash-point harness now measures the design instead +(`impl/rust-cli/tests/journal_crash_tests.rs`). It replays a scripted session, +builds every disk state a crash could leave between two acknowledged steps, and +requires that the shell reopens, that the recovered state is the one before or +after the interrupted step, and that later work survives another reopen. + +Result on the two-file journal, 2,075 crash states (Measured): + +* A journal append cut at any byte, a stray temporary file and a clean shutdown + all recover. +* Two classes fail, both after a power loss, because nothing is synced: the + journal is truncated while the old snapshot is still in place (3 states); and + the first snapshot is cut short, after which the failed load is ignored and + the shell starts empty beside an intact journal (666 states). + +The test pins exactly those two classes in `KNOWN_OPEN` and fails if the list is +wrong in either direction. The list must be empty before #230 can merge. + +=== 14.3 Revised phases + +One PR per phase, signed. Phase C runs on `claude/vsh-journal-redesign`; the +others on `claude/laughing-hawking-pibzji`. Each phase starts with an experiment +and ends at a machine-checked gate. + +*Phase A. Unblock and land what exists.* + +[cols="1,5,3,2",options="header"] +|=== +|Step |What |Gate |Status + +|A1 (owner) |Add `dtolnay/rust-toolchain@*`, `Swatinem/rust-cache@*` and `metadatastician/standards/.github/workflows/*@*` to the allowed-actions list of valence-shell, maa-framework and absolute-zero |The seven workflows start |Open +|A2 (owner) |Mark GitGuardian incident 36740437 a false positive |- |Open +|A3 |Crash-point harness (section 14.2) |Zero failures, or a pinned list of failure classes |Done: two classes pinned +|A4 |Split #229 (OD-35) |#229 green with Rust CI run |Split done; waits on A1 +|A5 |maa-framework#259 and absolute-zero#203 |Both merged |Wait on A1 +|=== + +*Phase C. The journal (this plan's E3, E4 and P8, pulled forward by OD-36).* + +[cols="1,5,3,2",options="header"] +|=== +|Step |What |Gate |Status + +|C1 |Journal model in Agda (OD-30): record framing, event replay, a file and crash model |Checks under the estate invocation, in `All.agda` |Done (section 14.4) +|C2 |One file of length-prefixed, checksummed records behind a small storage interface (append, sync, atomic replace, read), with a real implementation and a simulated one that loses exactly what the model's crash may lose. Compaction writes one snapshot record to a new file, syncs it, renames it over the old one and syncs the directory. A failed load is an error that stops the shell from writing; the files are kept. Rollback writes a record that marks each reversed operation undone. Old `.vsh_state.json` files still load (fixture test) |No failing crash class; the harness also runs against the simulated storage |Done (section 14.4a) +|C3 |State moves out of the command root to an owner-only directory keyed by session root (OD-12a), opened with `O_NOFOLLOW`, checked through the descriptor, and held under an exclusive lock for the life of the shell |The shell's own (built-in) commands cannot delete or edit state, because it lies outside the root they are confined to: a test runs `rm`, `>` and `obliterate` against the state path and finds it unchanged. A second shell on the same root is refused; a two-instance test passes. Not covered: an external command runs as the user and can reach the state by its absolute path, like any other file of theirs. Keeping external commands out needs process isolation, which is not part of this step |Open +|C4 |Deleted-file contents go to a content-addressed blob store, not into the history. A record identifies its file by the names the history gives it over time and by the device and inode read when the content is saved (OD-46) |Removing 100 MB no longer writes about 300 MB of JSON. Hole p is closed: after `mv a b; obliterate b`, nothing that was saved under `a` is left |Open +|C5 |`vsh --recover` (OD-40): for a saved state the shell refuses to load, show how many records are readable and what stops the rest, then keep the readable part or move the file aside, on the user's choice. Nothing is deleted. A journal that ends in zero bytes is one such state (OD-48): the zero bytes are shown and set aside, and the records before them are kept |A damaged journal can be brought back to a state the shell starts on, with the damaged bytes kept |Open +|=== + +By OD-41, C3, C4 and C5 run after Phase B. + +*Phase B. Runtime holes (this plan's Phase 1).* + +[cols="1,5,3",options="header"] +|=== +|Step |What |Gate + +|B0 = E0 |`tests/regressions_parsed_path.rs`: one fail-first test per hole a to p, through the parsed `vsh -c` path. Rewrite `security_no_privilege_escalation` with real assertions |Each test fails on `main` for the documented reason +|B1 |Experiment, one day: port `touch`, `mkdir` and `>` redirection to a root directory handle from the `cap-std` crate; run tests j, k, l |GO or NO-GO written down; decides the containment mechanism (section 14.5) +|B2 = E1b |Session root fixed at start and separate from the working directory; `cd` confined (OD-26a); history paths relative to the session root; content operations refuse symlink escapes and files with more than one hard link; obliterate refuses a symlink |Tests f, g, j, k, l, m, n pass; outside files byte-identical +|B3 = E1c |Obliterate: intent record written and synced before the overwrite; refused inside a transaction (OD-3a); earlier payloads of the target scrubbed |Tests c, d, i pass +|B4 = E1, E1d |Undo restores mode, owner and mtime; unreadable files are refused, not saved empty; redirection records a root-relative path; checked arithmetic |Tests a, e, h pass; an arithmetic never-panics property test +|B5 |One inverse table shared by undo, redo and rollback. This fixes redo losing undo data (`rm f; undo; redo; undo` restores `f` empty) |undo, redo, undo round-trip tests for rm, chmod, cp and `ln -s` +|=== + +B2 comes first inside the phase: it closes the holes that destroy data outside +the sandbox. + +*Phase D. Spec spine* (this plan's Phases 3 and 4 under OD-30): FinFS in Agda +first; oracle v2 with undo, redo, transactions and obliterate, driven by a +state-machine property test against the shell; the golden trace corpus; the four +Lean libraries repaired or visibly quarantined; a proof manifest with grades +derived from CI receipts (OD-34). + +*Phase E. Rust contracts* (OD-31): extract `vsh-core`; Creusot/Pearlite and +Verus contracts on `step` and `inverse`; Kani on FFI seams. + +*Phase F. Honest surface* (one day, any time after B): regenerate the counts in +the root `CLAUDE.md`; fix the remaining contradictions in `ROADMAP.adoc`; +replace the banner line "All filesystem operations are reversible" with a +statement derived from the inverse table. + +=== 14.4 C1 result: the journal model (Measured 2026-11-17) + +Four modules in `proofs/agda/`, commit `13ea833`, checked by +`scripts/check-agda.sh` from a clean build with Agda 2.7.0.1 and agda-stdlib +v2.2. No postulates and no escape pragmas. + +[cols="2,5",options="header"] +|=== +|Module |What it proves + +|`JournalFraming` |A record is a fixed-width length, the payload, then a fixed-width checksum. A record cut short at any point is rejected (`step-torn`). Whatever the decoder accepts is byte for byte a run of well-formed records at the front of the input (`decode-sound`), and re-encoding it gives a clean journal for any input (`decode-normal`). None of this needs any property of the checksum +|`JournalReplay` |Events other than a snapshot or a trim at the history cap never remove or reorder history (`run-ids`). After a rollback no operation of that transaction is live (`rollback-no-live`) +|`JournalModel` |A file is durable bytes plus pending bytes. If a crash keeps the durable bytes and a prefix of the pending ones, loading yields the synced events followed by a prefix of the unsynced events (`crash-recovery`). One pending write is all-or-nothing (`write-atomic`). Compaction by all-or-nothing replacement does not change the recovered state (`compaction-safe`) +|`JournalWitnesses` |One instance of every module parameter, including an event serialisation with a proved round trip, and worked cases closed by `refl` +|=== + +Six mutations were each rejected by the type checker: a wrong left-over tail and +a wrong recovered state in the worked cases; no checksum test; no +canonical-header test; rollback that marks nothing; and `step-torn` without its +premise. + +Limits: + +* These are theorems about the model. The Rust journal does not follow it yet. + The proof ledger records that link as `open`; once C2 lands it can be at most + `tested-only`. +* The crash model is an assumption about storage: synced bytes survive, and the + unsynced part survives as a prefix. Reordered or partly written blocks, and + stale bytes in the unsynced region, are outside it. +* A checksum cannot rule out bytes that were never written as a record but + happen to match. + +[[c2-result]] +*14.4a C2 result (Measured 2026-11-17, local).* The shell state now persists +through `impl/rust-cli/src/journal.rs`: one file, `.vsh_state.journal`, of +records framed as a 4-byte length, the payload and its SHA-256. Each mutation +is one record, appended and synced. The crash harness finds no failing class: +3,079 crash images on the simulated storage (the model's file: synced bytes, +and every prefix of the unsynced ones, after every storage call) and 3,070 +crash states on real files. Four deliberate bugs (no sync, a trim left out of +the record, a rollback written as a commit, a torn tail left in place) were +each caught. `cargo test`: 865 passed, 0 failed. An old `.vsh_state.json` is +converted on first start and kept under a new name. A sync costs about 0.15 ms +here (p95 0.43 ms), under the 2 ms threshold for syncing every record; the +stress suite takes 82 s on disk and 39 s on tmpfs, so its CI step uses tmpfs. + +[[c2-header-check]] +*14.4b Review follow-up: a check on the record's length (Measured 2026-11-17, +local).* A review of C2 found that a changed length field could pass for an +interrupted write. The record then seems to run past the end of the file; the +loader set it and every later record aside, and the shell started with a +shorter history, or an empty one. Each record's header is now the 4-byte +length followed by a 4-byte check of it (the first four bytes of SHA-256 over +a fixed tag and the length), so a record is length, check, payload, SHA-256 +of the payload. A whole header that fails its check is damage, and the shell +refuses to load, as it does for a failed payload checksum. + +The model already had the test. Its decoder accepts a header `h` only if +`hdr (unhdr h) ≡ h`, which says nothing for a bare length and is the header +check once `hdr` carries one. The model gained `classify` (clean, torn or +damaged) with `classify-torn`, `decode-crash-tail` and `crash-tail`: after a +crash that keeps a prefix of what was appended, the rejected bytes are never +classified as damaged, so a crash cannot set off the refusal. Four more +mutations were rejected. The crash harness still finds no failing class: +3,139 crash images on the simulated storage and 3,130 crash states on real +files. `cargo test`: 889 passed, 0 failed, 15 ignored. + +Limits: + +* Damage that leaves a header passing its check, and declaring more bytes than + the file holds, is still read as an interrupted write. For a header changed + at random that is about one case in 2^32. +* A tail of zero bytes is outside the crash model. Some storage can leave one + after a power loss. It is reported as damage once it holds a whole header + (8 bytes; before the check, 36), and the shell then refuses to start until + the file is repaired by hand, because the recovery command (C5) is not + built. Whether such a tail should be read as an interrupted write is an + open decision (section 14.5). +* Journals written by earlier commits of this branch have the old header and + are refused as damaged. The format was never merged or released, so there + is no converter. + +[[c2-open-transaction]] +*14.4c Review follow-up: an open transaction keeps its operations (Measured +2026-11-17, local).* The history size limit (10,000 operations unless set) +dropped the oldest operations whatever they belonged to. A transaction longer +than the limit therefore lost its first operations, and its rollback was +refused: the promise of `begin` ... `rollback` failed at the point where it +was needed, which is most likely for a script or an agent that does many +operations in one transaction. The limit now leaves the operations of the +open transaction alone. The history can be longer than the limit while a +transaction is open, and the first operation after it closes brings it back +under. Rollback still refuses if an operation of the transaction is missing, +which this version can no longer cause itself. The trim count is part of each +record, so a restart replays the same history. `cargo test`: 894 passed, 0 +failed, 15 ignored. + +Two smaller faults from the same review were fixed with it: `redo` read a +file in a way that could change its access time, and after a restart +single-step `undo` rescanned the run of undone operations at the end of the +history on every step. + +A fourth finding followed. When a reversal was done on the disk but the saved +state could not be written (a full disk, for one), rollback printed a warning, +carried on, and reported success, so the saved state could fall behind the +disk by the whole transaction. Rollback now stops at the first reversal it +cannot save and returns an error, with the transaction left open; the next +write that succeeds saves the whole state. The saved state is then at most +one operation behind the disk. Every other command still only warned when its +record could not be saved; what they should do was an open decision (section +14.5), since ruled (OD-49) and done (14.4d). `cargo test`: 896 passed, 0 failed, 15 ignored (898 after the redo fix that followed). + +Not proved: the model's `rollback-no-live` speaks of the operations that are +in the history. That an open transaction's operations all stay there is a +rule of the writer, tested here and not stated in the model. + +Three behaviours changed with it, each following the model. The owner ruled on +them the same day (OD-38 to OD-40): + +* A saved state that cannot be read stops the shell from starting, with the + file left as it is. Before, the shell started with an empty history. A + recovery command follows (OD-40, step C5). +* Rollback marks the operations it reverses as undone, so `redo` can bring + them back one at a time (OD-38). +* Rollback first checks that every operation can be reversed, and refuses + otherwise with nothing changed. If a reversal then fails on the disk, the + transaction stays open, to be rolled back again or committed (OD-39). + Before, it was closed with the operation still in place. + +Still open in Phase C, and now after Phase B (OD-41): the journal lives in the +session root (C3), deleted file contents are still inside the history records +(C4), and there is no recovery command (C5). + +One more for C4, from review: the journal is read whole into memory when the +shell starts, before any limit is checked, so a very large journal in a +directory someone else prepared can exhaust memory. No useful limit can be set +while the journal holds the contents of deleted files, since the shell's own +journal can then be gigabytes, and reading it in pieces would not lower the +memory needed, because replay keeps every record. Once C4 has moved the +contents out, the journal is small and a limit can be enforced. The review +thread is left open for the owner. + +Design choices the model fixes for C2: + +* The history is append-only. Undo and rollback mark entries; redo records a new + operation. +* Everything that must survive a restart, the open transaction included, is a + record in the one file. +* The loader reports the bytes it did not accept. The writer truncates the file + to the accepted part before appending again, and keeps the rejected bytes + aside so that nothing is destroyed silently. +* A record that does not parse is an error, never a skipped line. +* A record and the history trim it causes at the size cap are one event, so a + crash cannot keep one without the other. + +*14.4d A record that cannot be saved is an error (OD-49; Measured 2026-11-18, +local).* The shell does an operation and then writes its record. When the +write failed, every command except rollback printed a warning and reported +success. The functions that write a record (`record_operation`, +`record_redo_operation`, `mark_undone`, and the opening and closing of a +transaction) now return an error, `NotRecorded`, and the commands pass it on: +the file commands, `undo`, `redo`, `begin`, `commit`, `obliterate` and a +redirection. The message says that the change was made and that its record +could not be saved. The record is in the session's history all the same, and +the next record that can be written saves the whole history again, so nothing +is lost unless the shell stops before then. + +Two things stay a warning, because they change no file and a shell on a root +it cannot write to must still work: assigning a variable, and the directory +that `cd -` returns to. The saved state can still be one operation behind the +disk after a crash between a change and its record; the write-ahead record of +Phase E closes that. `cargo test`: 902 passed, 0 failed, 16 ignored, with +`OLDPWD` and `PWD` set and unset. Five single changes, each making one of the +five functions swallow the failure again, each fail a test. + +=== 14.5 Decisions still open + +Ratings are E (elegance and correctness) / D (dependability and security) out of +10. They are judgements, not measurements. + +*Containment mechanism (decided by experiment B1).* + +[cols="5,1,4",options="header"] +|=== +|Option |E / D |Notes + +|Capability handles: all filesystem access goes through a root directory handle (`cap-std`), and ambient `std::fs` calls outside one module are banned by a lint |9 / 8 |Makes a path outside the root unrepresentable instead of checked. New dependency +|A hand-written `openat2` / `O_NOFOLLOW` walk, as E1b describes |6 / 7 |No new dependency; more code to get wrong +|More lexical path checks |3 / 3 |This is what fails today +|=== + +Hard links (hole n) need a link-count check under any option. + +*The secret-scanner review thread on #229* (a history pass that skips `.adoc` +files): fix it in the `standards` reusable workflow (9 / 8, at source), or set +`cancel-in-progress: false` in this one caller (6 / 8). Owner's call; the +recommendation is the first. + +*Deed migration:* nine owner inputs are listed in +`archive/a2ml-v1/MIGRATION-LEDGER.adoc` on #229. + +*A journal that ends in zero bytes* (raised by the header check, section +14.4b). The crash model says a crash keeps a prefix of what was appended. Some +storage can instead leave zero bytes at the end of the file after a power +loss. Today that is damage, and the shell refuses to start (OD-40). + +[cols="5,1,4",options="header"] +|=== +|Option |E / D |Notes + +|Take the zero bytes off the end of the rejected bytes, then classify what is left: empty or a record cut short is an interrupted write. Widen the model's crash to "a prefix, then any number of zero bytes" and prove `crash-tail` for it |8 / 8 |Covers a record cut short and then padded with zeros, which a rule for an all-zero tail alone would report as damage (found in review). The zeros hold no record, so setting them aside loses nothing, and they run to the end of the file, so they cannot hide a good record. Two costs: a record overwritten with zeros is no longer reported as damage, and a damaged last record whose bytes end in a zero byte is read as cut short. The model needs one more assumption, that no record starts with a zero header. The etcd log reader works in a similar way: a zero length ends the log, and a record that fails its check and has a zeroed sector counts as a torn write (https://github.com/etcd-io/etcd/blob/main/server/storage/wal/decoder.go[`decoder.go`], read 2026-11-17) +|Keep it as damage, and build the recovery command (C5) before the first engine release |7 / 9 |Nothing is ever guessed. Until C5 exists, a user must repair the file by hand +|Treat every unreadable tail as the end of the journal |4 / 5 |Always starts. Gives up the difference between a crash and damage, which OD-40 ruled against. SQLite's log is read this way: a frame is valid only if its running checksum matches, so the first bad frame ends the log (https://www.sqlite.org/fileformat2.html#walformat[file format], section 4.1, read 2026-11-17) +|=== + +The recommendation was the first, with C5 still built. *Ruled 2026-11-17 +(OD-48): the second.* The zero bytes stay damage, and C5 handles them. + +*A record that cannot be saved* (raised by review of the rollback, section +14.4c). The shell does an operation and then writes its record. If the write +fails, the operation has happened and the saved state does not know. Today +every command except rollback prints a warning and reports success. + +[cols="5,1,4",options="header"] +|=== +|Option |E / D |Notes + +|Write the intent first: a record of what is about to be done, then the operation, then a record that it was done. If the first record cannot be written, the operation is not done. After a crash, an intent without its completion is settled by looking at the files |9 / 9 |The saved state is never behind the disk. It is the write-ahead rule of databases, and it needs each operation to say what the files look like before and after, which is what the change-set engine (Phase E) is to provide. Two records per operation +|Do the operation, and if its record cannot be saved, return an error that says so. A script or an agent then sees a failure and can stop |7 / 8 |Small. The saved state can still be one operation behind after a crash between the operation and its record +|Keep the warning |3 / 3 |A script does not see it. This is today's behaviour +|=== + +The recommendation was the second now, as part of the first engine release, +and the first with the change-set engine. *Ruled 2026-11-17 (OD-49): as +recommended.* + +=== 14.6 Stop rules added + +The stop rules of section 9 stand. Added: + +* Nothing merges into `main` unless its Rust workflows have run on the head + commit. A local run is not a substitute. +* No persistence change lands unless the crash-point harness passes with an + empty `KNOWN_OPEN`. +* An on-disk format change must still load old files; otherwise stop and ask. +* A review round is fixed as one batch and one push, after a local adversarial + check. + +=== 14.7 Recurring problems and the more final answer + +[cols="3,3,4",options="header"] +|=== +|Problem seen |Fix used so far |More final answer + +|Crash-consistency findings on the journal |Point patches |Checksummed framing, the Agda recovery theorem and the crash-point harness (C1, C2, A3) +|Sandbox escapes through `cd`, symlinks and hard links |Lexical path checks |Capability handles: no ambient paths, enforced by a lint +|Redo loses undo data; rollback leaves operations live |Per-command undo capture |One inverse table shared by undo, redo and rollback (OD-25), following `JournalReplay` +|Obliterate inside a transaction |Report partial failure |Refuse now (OD-3a); later a typed "irrevocable" protocol state (OD-3b) +|Shell and proofs drift apart |Hand-written correspondence tests |A state-machine property test driving the shell against the oracle (Phase D) +|=== + +=== 14.8 Beyond this repo (additions to section 11) + +* *Estate Actions policy:* any repository that calls the + `metadatastician/standards` reusable workflows or the Rust setup actions will + fail to start under the same allowed-actions list (Inferred). +* *standards, secret-scanner reusable:* the history pass skips `.adoc`. +* *standards / rsr-template-repo:* the workflow linter's "SPDX on line 1" rule + disagreed with the governance check's "leading comment block" rule. Fixed in + valence-shell on #229; the template should carry the same fix. +* *Hypatia:* rule CSA001 re-reports code-scanning alerts that the PR head + already fixes. +* *Deed tooling:* the converter the canon names (kyX) was not available; the + migration on #229 was done by hand from the mapping specifications. +* *januskey:* the blob store in C4 overlaps `reversible-core` (OD-6). No + dependency is added; the API frictions are to be recorded. +* *Review bots:* Codex has no credits for security review, and CodeRabbit's free + tier allows about one review an hour. +* *Journal model reuse:* `JournalFraming` is independent of this shell. Any + estate project with an append-only log could reuse it; echo-types or a shared + library would be the place if a second user appears. +* *atoma:* the owner's note of 2026-11-17 (OD-50): the Zig work is in + development in the `atoma` repository. This shell's own Zig FFI layer + (`impl/zig/`) builds and is not connected to the shell, and the `proofs` + command now says so. How the two relate has not been read from `atoma`; it is + to be checked there before anything here is changed or claimed. +* *Obliterate and file identity:* hole p (OD-46) is a case of a general + question, how a saved record identifies its file across renames and links. + januskey's `reversible-core` has the same question for its own store. + +=== 14.9 What was not verified + +* Why the workflows fail to start, and what blocks auto-merge on #229: the + repository settings are not readable from the working session. +* Holes a, b, d, e, i, l and n on the 2026-11-17 build. +* That a state file written before the journal work loads under the journal + branch's code (no fixture test yet; it is part of C2). +* The stress suite on CI, on macOS, or under a release build. +* Anything about the Rust code following the journal model. +* The E / D ratings. + +=== 14.10 Product review (2026-11-17) + +The owner asked for a review of the whole plan as a product before ruling on +hole (b). It is in `VSH-PRODUCT-REVIEW-2026-11-17.adoc`. + +* It measures the shell against bash on everyday use, compares it with other + shells and with existing tools for undo, containment, audit and deletion, + rates each aspect, and sets out routes with their costs. +* Its main finding for this plan: gates G1 to G11 say nothing about use as a + shell. It proposes a gate G0 (what is claimed equals what a battery shows) + and a staged model in which each step is a product. +* The owner ruled on its section 10 the same day: OD-42 to OD-45, in "Owner + rulings (2026-11-17)" above. Hole (b) stays open until the change-set engine + (OD-45). diff --git a/impl/rust-cli/src/commands.rs b/impl/rust-cli/src/commands.rs index 4f91625d..709bc725 100644 --- a/impl/rust-cli/src/commands.rs +++ b/impl/rust-cli/src/commands.rs @@ -97,7 +97,7 @@ pub fn mkdir(state: &mut ShellState, path: &str, verbose: bool) -> Result<()> { // Record in history let op = Operation::new(OperationType::Mkdir, path.to_string(), None); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; // Output println!( @@ -165,7 +165,7 @@ pub fn rmdir(state: &mut ShellState, path: &str, verbose: bool) -> Result<()> { // Record let op = Operation::new(OperationType::Rmdir, path.to_string(), None); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {}", @@ -225,7 +225,7 @@ pub fn touch(state: &mut ShellState, path: &str, verbose: bool) -> Result<()> { let op = Operation::new(OperationType::CreateFile, path.to_string(), None); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {}", @@ -291,7 +291,7 @@ pub fn rm(state: &mut ShellState, path: &str, verbose: bool) -> Result<()> { let op = Operation::new(OperationType::DeleteFile, path.to_string(), None).with_undo_data(content); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {}", @@ -366,7 +366,7 @@ pub fn cp(state: &mut ShellState, src: &str, dst: &str, verbose: bool) -> Result let op = Operation::new(OperationType::CopyFile, dst.to_string(), None) .with_undo_data(src.as_bytes().to_vec()); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {} → {}", @@ -447,7 +447,7 @@ pub fn mv(state: &mut ShellState, src: &str, dst: &str, verbose: bool) -> Result let combined_path = format!("{}\0{}", src, dst); let op = Operation::new(OperationType::Move, combined_path, None); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {} → {}", @@ -518,7 +518,7 @@ pub fn symlink(state: &mut ShellState, target: &str, link: &str, verbose: bool) let op = Operation::new(OperationType::Symlink, link.to_string(), None) .with_undo_data(target.as_bytes().to_vec()); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {} → {}", @@ -580,7 +580,7 @@ pub fn chmod(state: &mut ShellState, mode_str: &str, path: &str, verbose: bool) let op = Operation::new(OperationType::Chmod, path.to_string(), None) .with_undo_data(old_mode.to_le_bytes().to_vec()); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {:o} {}", @@ -725,7 +725,7 @@ pub fn chown(state: &mut ShellState, owner_str: &str, path: &str, verbose: bool) let op = Operation::new(OperationType::Chown, path.to_string(), None) .with_undo_data(undo_str.into_bytes()); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {} {}", @@ -860,11 +860,11 @@ pub fn undo(state: &mut ShellState, count: usize, verbose: bool) -> Result<()> { fs::remove_file(&path).context("Undo touch failed")?; } OperationType::CreateFile => { - let content = op.undo_data.as_ref().cloned().unwrap_or_default(); + let content: &[u8] = op.undo_data.as_deref().unwrap_or_default(); fs::write(&path, content).context("Undo rm failed")?; } OperationType::WriteFile => { - let content = op.undo_data.as_ref().cloned().unwrap_or_default(); + let content: &[u8] = op.undo_data.as_deref().unwrap_or_default(); fs::write(&path, content).context("Undo write failed")?; } OperationType::FileAppended => { @@ -992,7 +992,7 @@ pub fn undo(state: &mut ShellState, count: usize, verbose: bool) -> Result<()> { let undo_id = undo_op.id; let orig_id = op.id; - state.mark_undone(orig_id, undo_id); + state.mark_undone(orig_id, undo_id)?; println!( "{} {} {} {} {}", @@ -1047,126 +1047,170 @@ pub fn redo(state: &mut ShellState, count: usize, verbose: bool) -> Result<()> { } }; - let path = state.resolve_path(&op.path); - - // Re-execute the original operation - match op.op_type { - OperationType::Mkdir => { - fs::create_dir(&path).context("Redo mkdir failed")?; - } - OperationType::Rmdir => { - fs::remove_dir(&path).context("Redo rmdir failed")?; - } - OperationType::CreateFile => { - fs::write(&path, "").context("Redo touch failed")?; - } - OperationType::DeleteFile => { - fs::remove_file(&path).context("Redo rm failed")?; - } - OperationType::WriteFile => { - // Would need new content - anyhow::bail!("WriteFile redo not yet implemented"); - } - OperationType::FileTruncated => { - // Redo truncate: truncate file (original undo_data was the old content) - fs::write(&path, "").context("Redo truncate failed")?; - } - OperationType::FileAppended => { - // Cannot redo append without knowing what was appended - anyhow::bail!( - "FileAppended redo not yet implemented (would need appended content)" - ); - } - OperationType::CopyFile => { - // Redo copy: copy again from src (stored in undo_data) to dst (path) - let src = op - .undo_data - .as_ref() - .map(|d| String::from_utf8_lossy(d).to_string()) - .context("Missing source path for redo cp")?; - let src_path = state.resolve_path(&src); - fs::copy(&src_path, &path).context("Redo cp failed")?; - } - OperationType::Move => { - // Redo move: move src -> dst again - let parts: Vec<&str> = op.path.splitn(2, '\0').collect(); - if parts.len() != 2 { - anyhow::bail!("Invalid move operation record for redo"); - } - let src_path = state.resolve_path(parts[0]); - let dst_path = state.resolve_path(parts[1]); - fs::rename(&src_path, &dst_path).context("Redo mv failed")?; - } - OperationType::Symlink => { - // Redo symlink: create symlink again - let target = op - .undo_data - .as_ref() - .map(|d| String::from_utf8_lossy(d).to_string()) - .context("Missing symlink target for redo")?; - #[cfg(unix)] - std::os::unix::fs::symlink(&target, &path).context("Redo ln -s failed")?; - #[cfg(not(unix))] - anyhow::bail!("Symbolic links not supported on this platform"); - } - OperationType::Unlink => { - // Redo unlink: remove the symlink again - fs::remove_file(&path).context("Redo unlink failed")?; - } - OperationType::SetVariable => { - // Redo set: re-apply the variable value - // The undo_data stores the PREVIOUS value, not the new one. - // For redo, we need the value that was set, but we don't have it. - // Redo of SetVariable is a no-op — variable state has moved on. - // This is correct: variable redo is best-effort; filesystem redo is exact. - println!( - "{}", - "Variable set cannot be exactly redone (state-dependent)".bright_yellow() - ); - } - OperationType::UnsetVariable => { - // Redo unset: unset the variable again - state.variables.remove(&op.path); - state.exported_vars.remove(&op.path); - } - OperationType::Chmod => { - // Redo chmod: cannot exactly redo without storing new mode - println!( - "{}", - "chmod cannot be exactly redone (state-dependent)".bright_yellow() - ); - } - OperationType::Chown => { - // Redo chown: cannot exactly redo without storing new uid:gid - println!( - "{}", - "chown cannot be exactly redone (state-dependent)".bright_yellow() - ); - } - OperationType::HardwareErase | OperationType::Obliterate => { - // Cannot redo - irreversible operations - anyhow::bail!( - "{:?} cannot be redone - operation is irreversible", - op.op_type - ); + if let Err(e) = redo_one(state, &op, verbose) { + // Nothing was redone. Keep the entry, so that the redo can be + // tried again once the cause is fixed. An entry of a kind that + // can never be redone is dropped, or it would block the ones + // behind it for the rest of the session. + if can_be_redone(op.op_type) { + state.redo_stack.push_front(op); } + return Err(e); } + } - let new_op = Operation::new(op.op_type, op.path.clone(), None); - let new_id = new_op.id; - state.record_redo_operation(new_op); + Ok(()) +} - println!( - "{} {} {}", - format!("[op:{}]", &new_id.to_string()[..8]).bright_black(), - "redo".bright_cyan(), - op.path - ); +/// Whether `redo` has a way to do this kind of operation again. +fn can_be_redone(op_type: OperationType) -> bool { + !matches!( + op_type, + OperationType::WriteFile + | OperationType::FileAppended + | OperationType::HardwareErase + | OperationType::Obliterate + ) +} - if verbose { - let proof = op.op_type.proof_reference(); - println!(" {} {}", "Proof:".bright_black(), proof.format_short()); +/// Do one undone operation again and record it. On an error nothing has +/// been recorded. +fn redo_one(state: &mut ShellState, op: &Operation, verbose: bool) -> Result<()> { + let path = state.resolve_path(&op.path); + + // The record this redo writes must be able to undo it. A removal or a + // truncation needs the contents as they are now, read before the file + // is touched. A copy, a link and an unlink need what the first record + // carried (the source path or the link target). + let undo_data: Option> = match op.op_type { + // Read as `rm` and a truncating redirection do: without changing + // the file's access time where the system allows it. + OperationType::DeleteFile | OperationType::FileTruncated => Some( + crate::fs_pure::read_to_end(&path) + .context("Redo: could not read the file to keep its contents")?, + ), + OperationType::CopyFile | OperationType::Symlink | OperationType::Unlink => { + op.undo_data.clone() + } + _ => None, + }; + + // Re-execute the original operation + match op.op_type { + OperationType::Mkdir => { + fs::create_dir(&path).context("Redo mkdir failed")?; + } + OperationType::Rmdir => { + fs::remove_dir(&path).context("Redo rmdir failed")?; + } + OperationType::CreateFile => { + fs::write(&path, "").context("Redo touch failed")?; + } + OperationType::DeleteFile => { + fs::remove_file(&path).context("Redo rm failed")?; + } + OperationType::WriteFile => { + // Would need new content + anyhow::bail!("WriteFile redo not yet implemented"); + } + OperationType::FileTruncated => { + // Redo truncate: truncate file (original undo_data was the old content) + fs::write(&path, "").context("Redo truncate failed")?; + } + OperationType::FileAppended => { + // Cannot redo append without knowing what was appended + anyhow::bail!("FileAppended redo not yet implemented (would need appended content)"); + } + OperationType::CopyFile => { + // Redo copy: copy again from src (stored in undo_data) to dst (path) + let src = op + .undo_data + .as_ref() + .map(|d| String::from_utf8_lossy(d).to_string()) + .context("Missing source path for redo cp")?; + let src_path = state.resolve_path(&src); + fs::copy(&src_path, &path).context("Redo cp failed")?; + } + OperationType::Move => { + // Redo move: move src -> dst again + let parts: Vec<&str> = op.path.splitn(2, '\0').collect(); + if parts.len() != 2 { + anyhow::bail!("Invalid move operation record for redo"); + } + let src_path = state.resolve_path(parts[0]); + let dst_path = state.resolve_path(parts[1]); + fs::rename(&src_path, &dst_path).context("Redo mv failed")?; + } + OperationType::Symlink => { + // Redo symlink: create symlink again + let target = op + .undo_data + .as_ref() + .map(|d| String::from_utf8_lossy(d).to_string()) + .context("Missing symlink target for redo")?; + #[cfg(unix)] + std::os::unix::fs::symlink(&target, &path).context("Redo ln -s failed")?; + #[cfg(not(unix))] + anyhow::bail!("Symbolic links not supported on this platform"); + } + OperationType::Unlink => { + // Redo unlink: remove the symlink again + fs::remove_file(&path).context("Redo unlink failed")?; + } + OperationType::SetVariable => { + // Redo set: re-apply the variable value + // The undo_data stores the PREVIOUS value, not the new one. + // For redo, we need the value that was set, but we don't have it. + // Redo of SetVariable is a no-op — variable state has moved on. + // This is correct: variable redo is best-effort; filesystem redo is exact. + println!( + "{}", + "Variable set cannot be exactly redone (state-dependent)".bright_yellow() + ); + } + OperationType::UnsetVariable => { + // Redo unset: unset the variable again + state.variables.remove(&op.path); + state.exported_vars.remove(&op.path); } + OperationType::Chmod => { + // Redo chmod: cannot exactly redo without storing new mode + println!( + "{}", + "chmod cannot be exactly redone (state-dependent)".bright_yellow() + ); + } + OperationType::Chown => { + // Redo chown: cannot exactly redo without storing new uid:gid + println!( + "{}", + "chown cannot be exactly redone (state-dependent)".bright_yellow() + ); + } + OperationType::HardwareErase | OperationType::Obliterate => { + // Cannot redo - irreversible operations + anyhow::bail!( + "{:?} cannot be redone - operation is irreversible", + op.op_type + ); + } + } + + let mut new_op = Operation::new(op.op_type, op.path.clone(), None); + new_op.undo_data = undo_data; + let new_id = new_op.id; + state.record_redo_operation(new_op)?; + + println!( + "{} {} {}", + format!("[op:{}]", &new_id.to_string()[..8]).bright_black(), + "redo".bright_cyan(), + op.path + ); + + if verbose { + let proof = op.op_type.proof_reference(); + println!(" {} {}", "Proof:".bright_black(), proof.format_short()); } Ok(()) @@ -1313,18 +1357,69 @@ pub fn commit_transaction(state: &mut ShellState) -> Result<()> { Ok(()) } -/// Rollback the current transaction, reversing all operations atomically. +/// Why `op` cannot be reversed, as far as its record alone shows: it has no +/// inverse, or the data its inverse needs was not recorded. +/// +/// This does not look at the disk. A reversal can still fail for a reason +/// only the filesystem knows (a file removed by another program, say); +/// [`rollback_transaction`] then leaves the transaction open. +fn rollback_blocker(op: &crate::state::Operation) -> Option { + let Some(inverse) = op.op_type.inverse() else { + return Some(format!("{} cannot be reversed", op.op_type)); + }; + let undo = op.undo_data.as_deref(); + let blocked = match inverse { + OperationType::FileAppended => undo.is_none_or(|data| data.len() < 8), + OperationType::Chmod => undo.is_none_or(|data| data.len() < 4), + OperationType::Chown => undo.is_none_or(|data| { + let text = String::from_utf8_lossy(data); + let mut parts = text.splitn(2, ':'); + let valid = |part: Option<&str>| part.is_some_and(|p| p.parse::().is_ok()); + !(valid(parts.next()) && valid(parts.next())) + }), + OperationType::Symlink => undo.is_none(), + OperationType::Move => !op.path.contains('\0'), + // Restoring a removed file, or a file's earlier content, needs that + // content. Without it the reversal would write an empty file. + OperationType::CreateFile | OperationType::WriteFile => undo.is_none(), + // Restoring a variable needs its earlier value, in the form the + // shell saved it. Anything else would be read as "it was unset". + OperationType::SetVariable => undo.is_none_or(|data| { + serde_json::from_slice::>(data).is_err() + }), + _ => false, + }; + blocked.then(|| { + format!( + "the data needed to reverse this {} was not recorded", + op.op_type + ) + }) +} + +/// Rollback the current transaction, reversing its operations. +/// +/// Undoes the operations of the active transaction started with +/// [`begin_transaction`], newest first. Each operation is marked undone as +/// soon as it has been reversed, so a later `undo` does not try to reverse +/// it again and an interrupted rollback can be resumed. Operations already +/// undone inside the transaction are left alone. +/// +/// Before anything is reversed, every operation is checked: if one has no +/// inverse (`obliterate`, for example) or lacks the data its inverse needs, +/// the rollback is refused and nothing is changed. That check reads only +/// the operation records, not the disk. /// -/// Undoes all operations in the active transaction started with [`begin_transaction`]. -/// Operations are reversed in LIFO order. Reports partial failures if any rollback fails. +/// The transaction is closed only when every operation has been reversed. +/// If a reversal fails on the disk, the transaction stays open: run +/// `rollback` again once the cause is fixed, or `commit` to keep what +/// remains. /// /// # Arguments /// * `state` - Mutable shell state for transaction management /// /// # Errors -/// Returns error if: -/// - No active transaction -/// - Rollback operations fail (reports which operations failed) +/// Returns error if an operation could not be reversed (reports which). /// /// # Examples /// ```no_run @@ -1338,19 +1433,88 @@ pub fn commit_transaction(state: &mut ShellState) -> Result<()> { /// # Ok::<(), anyhow::Error>(()) /// ``` pub fn rollback_transaction(state: &mut ShellState) -> Result<()> { + let Some(txn_id) = state.active_transaction.as_ref().map(|txn| txn.id) else { + println!("{}", "Nothing to rollback".bright_yellow()); + return Ok(()); + }; + + // Every operation of the transaction must still be in the history. + // The history size limit does not drop the operations of an open + // transaction, so this holds for a state this version wrote. It is + // checked all the same, for a state that came from somewhere else: + // reversing only what is left and then closing the transaction would + // leave the missing ones in place behind a transaction recorded as + // rolled back. + // + // The history is indexed once. Looking each operation up by a scan of + // the history took time that grew with the square of the transaction, + // and an open transaction can now be longer than the history limit. + let dropped = state.active_transaction.as_ref().map_or(0, |txn| { + let present: std::collections::HashSet = + state.history.iter().map(|op| op.id).collect(); + txn.operations + .iter() + .filter(|id| !present.contains(id)) + .count() + }); + if dropped > 0 { + eprintln!( + "{} {} of this transaction's operations are no longer in the history, \ + so it cannot be rolled back. Nothing was changed. Commit to close \ + the transaction.", + "Rollback refused:".bright_red(), + dropped + ); + anyhow::bail!( + "Transaction rollback refused: {dropped} operations are no longer in the history" + ) + } + let ops: Vec<_> = state .current_transaction_ops() .iter() + .filter(|o| !o.undone) .map(|o| (*o).clone()) .collect(); if ops.is_empty() { + // Nothing to reverse, but the transaction must not stay open. + close_rolled_back(state)?; println!("{}", "Nothing to rollback".bright_yellow()); return Ok(()); } - // Collect rollback failures + // Check first: if any operation is known not to be reversible, refuse + // the whole rollback before touching the disk, so a transaction is + // never left half rolled back for a reason that was knowable. + let blockers: Vec<(&str, String)> = ops + .iter() + .filter_map(|op| rollback_blocker(op).map(|why| (op.path.as_str(), why))) + .collect(); + if !blockers.is_empty() { + eprintln!( + "{} {} of {} operations cannot be reversed. Nothing was changed.", + "Rollback refused:".bright_red(), + blockers.len(), + ops.len() + ); + for (path, why) in &blockers { + eprintln!(" {} {}: {}", "Blocked:".bright_red(), path, why); + } + eprintln!("The transaction is still open. Commit to keep its operations."); + anyhow::bail!( + "Transaction rollback refused: {}/{} operations cannot be reversed", + blockers.len(), + ops.len() + ) + } + + // The failure that stopped the rollback, if any, and how many + // operations were reversed before it. let mut failed_rollbacks: Vec<(String, String)> = Vec::new(); + let mut reversed = 0usize; + // Set when a reversal was done on the disk but could not be saved. + let mut unsaved = false; // Undo all operations in reverse order for op in ops.iter().rev() { @@ -1364,11 +1528,11 @@ pub fn rollback_transaction(state: &mut ShellState) -> Result<()> { fs::remove_file(&path).context("Failed to remove file") } OperationType::CreateFile => { - let content = op.undo_data.as_ref().cloned().unwrap_or_default(); + let content: &[u8] = op.undo_data.as_deref().unwrap_or_default(); fs::write(&path, content).context("Failed to restore file") } OperationType::WriteFile => { - let content = op.undo_data.as_ref().cloned().unwrap_or_default(); + let content: &[u8] = op.undo_data.as_deref().unwrap_or_default(); fs::write(&path, content).context("Failed to restore file content") } OperationType::FileAppended => { @@ -1514,18 +1678,59 @@ pub fn rollback_transaction(state: &mut ShellState) -> Result<()> { } }; - // Collect failures - if let Err(e) = result { - failed_rollbacks.push((op.path.clone(), e.to_string())); + // Record each reversal as it happens, so the history never + // shows a reversed operation as live. + match result { + Ok(()) => { + unsaved = state.mark_undone(op.id, txn_id).is_err(); + reversed += 1; + } + Err(e) => failed_rollbacks.push((op.path.clone(), e.to_string())), } + } else { + failed_rollbacks.push(( + op.path.clone(), + format!("{} cannot be reversed", op.op_type), + )); + } + + // Stop at the first failure. Older operations may depend on the one + // that could not be reversed (a file renamed, then written to), so + // reversing them out of order could put the wrong content in the + // wrong place. + // + // Stop as well when a reversal could not be saved. Going on would + // put the saved state further behind the disk with each operation; + // stopping keeps it at most one operation behind. + if !failed_rollbacks.is_empty() || unsaved { + break; } } - // Clear the transaction - state.active_transaction = None; + if unsaved { + eprintln!( + "{} {} of {} operations were reversed, but the saved state could not be \ + written, so the last reversal is not recorded in it.", + "Rollback stopped:".bright_red(), + reversed, + ops.len() + ); + eprintln!( + "The transaction is still open. Fix the cause and run rollback again before \ + closing the shell: the next write that succeeds saves the whole state." + ); + anyhow::bail!( + "Transaction rollback stopped: the saved state could not be written \ + after {} of {} operations were reversed", + reversed, + ops.len() + ) + } // Report results if failed_rollbacks.is_empty() { + // Everything is reversed: close the transaction as rolled back. + close_rolled_back(state)?; println!( "{} {} operations", "Transaction rolled back:".bright_green(), @@ -1533,26 +1738,49 @@ pub fn rollback_transaction(state: &mut ShellState) -> Result<()> { ); Ok(()) } else { - let succeeded = ops.len() - failed_rollbacks.len(); eprintln!( - "{} {} succeeded, {} failed", + "{} {} reversed, {} failed, {} not attempted", "Partial rollback:".bright_yellow(), - succeeded, - failed_rollbacks.len() + reversed, + failed_rollbacks.len(), + ops.len() - reversed - failed_rollbacks.len() ); for (path, err) in &failed_rollbacks { eprintln!(" {} {}: {}", "Failed:".bright_red(), path, err); } + eprintln!( + "The transaction is still open. Run rollback again once the cause is fixed, \ + or commit to keep the remaining operations." + ); anyhow::bail!( - "Transaction rollback incomplete: {}/{} operations failed", - failed_rollbacks.len(), + "Transaction rollback incomplete: {} of {} operations reversed", + reversed, ops.len() ) } } +/// Close the open transaction as rolled back, and report it if that could +/// not be saved. +fn close_rolled_back(state: &mut ShellState) -> Result<()> { + state.abort_transaction(); + if state.is_saved() { + return Ok(()); + } + eprintln!( + "{} every operation was reversed and the transaction is closed, but the \ + saved state could not be written.", + "Rollback not saved:".bright_red() + ); + eprintln!( + "The next write that succeeds saves the whole state. If the shell is closed \ + before that, the transaction will be shown as open when it starts again." + ); + anyhow::bail!("Transaction rolled back, but the saved state could not be written") +} + /// Display the operation dependency graph (DAG). /// /// Shows the current state in the operation history as a directed acyclic graph. @@ -2051,7 +2279,7 @@ pub fn hardware_erase(state: &mut ShellState, device: &str, method: Option<&str> None, // Not part of a transaction ); let op_id = op.id; - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {} (method: {}, type: {:?})", diff --git a/impl/rust-cli/src/commands/secure_deletion.rs b/impl/rust-cli/src/commands/secure_deletion.rs index 37373908..d71e2e4a 100644 --- a/impl/rust-cli/src/commands/secure_deletion.rs +++ b/impl/rust-cli/src/commands/secure_deletion.rs @@ -337,7 +337,7 @@ pub fn obliterate(state: &mut ShellState, path: &str, verbose: bool, force: bool let op_id = op.id; // RMO residue: persist the append-only attestation BEFORE moving `op`. append_rmo_residue(state, &op, "obliterated"); - state.record_operation(op); + state.record_operation(op)?; // Output println!( @@ -471,7 +471,7 @@ pub fn obliterate_dir( let op_id = op.id; // RMO residue: persist the append-only attestation BEFORE moving `op`. append_rmo_residue(state, &op, "obliterated-dir"); - state.record_operation(op); + state.record_operation(op)?; println!( "{} {} {}", diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index aeb72a25..91adf4dc 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -401,14 +401,16 @@ impl ExecutableCommand for Command { anyhow::bail!("cd: {}: Not a directory", target.display()); } - // Store current directory as previous before changing - state.previous_dir = Some(state.root.clone()); - - // Change the shell's working directory - std::env::set_current_dir(&target)?; + // Change first. The directory that was left is recorded only + // once the change has succeeded: a `cd` that fails must not + // alter what `cd -` returns to, now or after a restart. + let previous = state.root.clone(); + let canonical = std::fs::canonicalize(&target)?; + std::env::set_current_dir(&canonical)?; // Update state root to track current directory - state.root = std::fs::canonicalize(target)?; + state.root = canonical; + state.set_previous_dir(previous); Ok(ExecutionResult::Success) } diff --git a/impl/rust-cli/src/journal.rs b/impl/rust-cli/src/journal.rs new file mode 100644 index 00000000..b4745717 --- /dev/null +++ b/impl/rust-cli/src/journal.rs @@ -0,0 +1,1002 @@ +// SPDX-License-Identifier: MPL-2.0 +//! Journal storage: record framing and the storage interface. +//! +//! The shell's persisted state is one append-only file of records. This +//! module is the byte level of it: how a record is framed, how a file is +//! decoded, and the small interface through which bytes reach the disk. +//! +//! It follows the Agda model in `proofs/agda/JournalFraming.agda` and +//! `proofs/agda/JournalModel.agda`. The model is not machine-linked to this +//! code; the names are kept the same so the two can be read side by side, +//! and [`SimStorage`] implements the model's file and crash semantics so the +//! crash tests exercise the same assumptions the proofs make. +//! +//! # Record format +//! +//! ```text +//! | length: u32, little-endian | check of the length: 4 bytes | payload: `length` bytes | SHA-256 of payload: 32 bytes | +//! ``` +//! +//! The first two fields are the header. The check is the first four bytes of +//! SHA-256 over a fixed tag followed by the length field. It lets the length +//! be tested before it is used to find the end of the record. In the model +//! the header is the function `hdr`, and the decoder's test +//! `hdr (unhdr h) ≡ h` is this check. +//! +//! * A record cut short at any point is rejected (model: `step-torn`), and is +//! reported as an interrupted append, never as damage (model: +//! `classify-torn`). +//! * A whole header that fails its check is damage. An interrupted append +//! cannot leave one, because it leaves a prefix of what was written. +//! * Whatever [`decode`] accepts is, byte for byte, a run of well-formed +//! records at the front of the input (model: `decode-sound`). +//! * The checksum makes it unlikely that bytes never written as a record are +//! accepted as one. It is an integrity check against accidental damage, +//! not an authentication code: anyone who can write the file can write a +//! valid record. +//! +//! # Durability +//! +//! [`Storage::append`] is not durable until [`Storage::sync`] returns. +//! [`Storage::replace`] is all-or-nothing and durable when it returns. The +//! assumption about a crash, shared with the model: synced bytes survive, +//! and bytes appended since the last sync survive as a prefix. + +use sha2::{Digest, Sha256}; +use std::fs; +use std::io::{self, Read, Write}; +use std::path::{Path, PathBuf}; + +/// Width of the length field. +pub const LENGTH_FIELD_LEN: usize = 4; +/// Width of the check on the length field. +pub const HEADER_CHECK_LEN: usize = 4; +/// Width of the header: the length field, then its check. +pub const HEADER_LEN: usize = LENGTH_FIELD_LEN + HEADER_CHECK_LEN; +/// Width of the checksum. +pub const SUM_LEN: usize = 32; +/// Longest payload the length field can carry (model: `Frameable`). +pub const MAX_PAYLOAD: usize = u32::MAX as usize; + +/// A payload too long for the length field. +#[derive(Debug, thiserror::Error)] +#[error("journal record of {0} bytes exceeds the {MAX_PAYLOAD}-byte limit")] +pub struct PayloadTooLong(pub usize); + +/// Hashed in front of the length field in every header check. It names this +/// record format, so a journal written in another format is rejected at its +/// first record and not misread. +const HEADER_TAG: &[u8] = b"vsh-journal-header-v1"; + +fn checksum(payload: &[u8]) -> [u8; SUM_LEN] { + Sha256::digest(payload).into() +} + +/// The check stored after a length field. +fn header_check(length_field: &[u8; LENGTH_FIELD_LEN]) -> [u8; HEADER_CHECK_LEN] { + let digest = Sha256::new() + .chain_update(HEADER_TAG) + .chain_update(length_field) + .finalize(); + let mut check = [0; HEADER_CHECK_LEN]; + check.copy_from_slice(&digest[..HEADER_CHECK_LEN]); + check +} + +/// Append the frame for `payload` to `out` (model: `frame`). +pub fn write_frame(out: &mut Vec, payload: &[u8]) -> Result<(), PayloadTooLong> { + let len = u32::try_from(payload.len()).map_err(|_| PayloadTooLong(payload.len()))?; + let length_field = len.to_le_bytes(); + out.reserve(HEADER_LEN + payload.len() + SUM_LEN); + out.extend_from_slice(&length_field); + out.extend_from_slice(&header_check(&length_field)); + out.extend_from_slice(payload); + out.extend_from_slice(&checksum(payload)); + Ok(()) +} + +/// The frame for `payload`. +pub fn frame(payload: &[u8]) -> Result, PayloadTooLong> { + let mut out = Vec::new(); + write_frame(&mut out, payload)?; + Ok(out) +} + +/// What stands where a record's header should be. +enum Header { + /// Fewer bytes than a header has. + Incomplete, + /// A whole header whose check does not match its length field. + Bad, + /// A whole header that passes its check, with the payload length it + /// declares. + Good(usize), +} + +/// Read a header from the front of `bytes`. +fn read_header(bytes: &[u8]) -> Header { + let Some((length_field, rest)) = bytes.split_first_chunk::() else { + return Header::Incomplete; + }; + let Some(check) = rest.first_chunk::() else { + return Header::Incomplete; + }; + if *check != header_check(length_field) { + return Header::Bad; + } + Header::Good(u32::from_le_bytes(*length_field) as usize) +} + +/// Read one record from the front of `bytes` (model: `step`). Returns the +/// payload and the number of bytes the record occupies, or `None` if `bytes` +/// does not start with a well-formed record. +fn step(bytes: &[u8]) -> Option<(&[u8], usize)> { + let Header::Good(len) = read_header(bytes) else { + return None; + }; + let payload_end = HEADER_LEN.checked_add(len)?; + let frame_end = payload_end.checked_add(SUM_LEN)?; + let payload = bytes.get(HEADER_LEN..payload_end)?; + let sum = bytes.get(payload_end..frame_end)?; + (sum == checksum(payload)).then_some((payload, frame_end)) +} + +/// What follows the accepted records in a journal file (model: `Tail`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Tail { + /// Nothing. The journal is clean. + Clean, + /// An incomplete record: the file ends before the record's header, + /// payload or checksum does, and the header, if all of it is there, + /// passes its check. This is what an interrupted append leaves, and all + /// it can leave if a crash keeps a prefix of what was written (model: + /// `classify-torn`). + Torn, + /// A whole header that fails its check, or a complete record whose + /// checksum does not match its payload. An interrupted append cannot + /// produce either; the file has been damaged. + Damaged, +} + +/// The result of decoding a journal file (model: `Decoded`). +#[derive(Debug, PartialEq, Eq)] +pub struct Decoded<'a> { + /// The payloads of the records accepted, in order. + pub records: Vec<&'a [u8]>, + /// How many bytes those records occupy. The bytes from here to the end + /// of the input were not accepted. + pub accepted: usize, + /// What the bytes that were not accepted are. + /// + /// The distinction has one blind spot. Damage to a header that still + /// passes the header check, and that makes the record appear to run past + /// the end of the file, looks the same as an interrupted append and is + /// reported as [`Tail::Torn`]. The check is four bytes, so a header + /// changed at random passes it about once in 2^32 times. + pub tail: Tail, +} + +/// Classify the bytes after the last accepted record (model: `classify`). +fn classify_tail(rest: &[u8]) -> Tail { + if rest.is_empty() { + return Tail::Clean; + } + let len = match read_header(rest) { + Header::Incomplete => return Tail::Torn, + Header::Bad => return Tail::Damaged, + Header::Good(len) => len, + }; + let complete = HEADER_LEN + .checked_add(len) + .and_then(|end| end.checked_add(SUM_LEN)) + .is_some_and(|frame_end| frame_end <= rest.len()); + if complete { + Tail::Damaged + } else { + Tail::Torn + } +} + +/// Decode as many records as the front of `bytes` holds (model: `decode`). +pub fn decode(bytes: &[u8]) -> Decoded<'_> { + let mut records = Vec::new(); + let mut accepted = 0; + while let Some((payload, used)) = step(&bytes[accepted..]) { + records.push(payload); + accepted += used; + } + let tail = classify_tail(&bytes[accepted..]); + Decoded { + records, + accepted, + tail, + } +} + +/// Where the journal's bytes live (model: `File`, with `append`, `sync` and +/// atomic replacement). +pub trait Storage { + /// The whole contents, or `None` if nothing has been stored yet. + fn read(&mut self) -> io::Result>>; + + /// Add `bytes` at the end. Not durable until [`Storage::sync`]. + fn append(&mut self, bytes: &[u8]) -> io::Result<()>; + + /// Make everything appended so far durable. + fn sync(&mut self) -> io::Result<()>; + + /// Replace the whole contents with `bytes`. All-or-nothing, and durable + /// when it returns: after a crash the contents are the old ones (as any + /// crash of them) or exactly `bytes`. + fn replace(&mut self, bytes: &[u8]) -> io::Result<()>; + + /// Keep `bytes` that the journal did not accept, so that dropping them + /// from the journal destroys nothing. Returns a description of where + /// they were put. + fn set_aside(&mut self, bytes: &[u8]) -> io::Result; +} + +impl Storage for Box { + fn read(&mut self) -> io::Result>> { + (**self).read() + } + fn append(&mut self, bytes: &[u8]) -> io::Result<()> { + (**self).append(bytes) + } + fn sync(&mut self) -> io::Result<()> { + (**self).sync() + } + fn replace(&mut self, bytes: &[u8]) -> io::Result<()> { + (**self).replace(bytes) + } + fn set_aside(&mut self, bytes: &[u8]) -> io::Result { + (**self).set_aside(bytes) + } +} + +/// [`Storage`] in one file on disk. +/// +/// The file is created readable and writable by its owner only, is never +/// opened through a symlink, and must be a regular file owned by the +/// current user; group and other permission bits on an existing file are +/// removed before it is used. +/// +/// # Single owner +/// +/// One `FileStorage` must own its path. Nothing here stops a second one, +/// in this process or another, from using the same path, and if it does, +/// writes are lost: after one of them replaces the file, the other keeps +/// appending to the file that was replaced. Keeping a second owner out +/// (an exclusive lock held for the life of the shell) is the caller's job +/// and is not done yet; see route plan section 14.3, step C3. +#[derive(Debug)] +pub struct FileStorage { + path: PathBuf, + file: Option, +} + +impl FileStorage { + /// Storage at `path`. Nothing is opened or created until first use. + pub fn new(path: impl Into) -> Self { + Self { + path: path.into(), + file: None, + } + } + + /// The file this storage reads and writes. + pub fn path(&self) -> &Path { + &self.path + } + + fn options() -> fs::OpenOptions { + let mut options = fs::OpenOptions::new(); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + // O_NONBLOCK: a FIFO in place of the file must fail or open at + // once, not block before it can be checked. It has no effect on + // reads and writes of a regular file. + options + .mode(0o600) + .custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC | libc::O_NONBLOCK); + } + options + } + + /// Check, through the open descriptor, that `file` is ours to use, and + /// remove group and other access. + #[cfg(unix)] + fn vet(&self, file: &fs::File) -> io::Result<()> { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + let meta = file.metadata()?; + if !meta.is_file() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("{} is not a regular file", self.path.display()), + )); + } + let euid = nix::unistd::geteuid().as_raw(); + if meta.uid() != euid { + return Err(io::Error::new( + io::ErrorKind::PermissionDenied, + format!("{} is owned by another user", self.path.display()), + )); + } + // A second name for the file means writes here would land in + // whatever that other name is, which may not be ours to change. + if meta.nlink() != 1 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!( + "{} has {} hard links; refusing to use it", + self.path.display(), + meta.nlink() + ), + )); + } + if meta.mode() & 0o077 != 0 { + file.set_permissions(fs::Permissions::from_mode(0o600))?; + } + Ok(()) + } + + #[cfg(not(unix))] + fn vet(&self, _file: &fs::File) -> io::Result<()> { + Ok(()) + } + + fn parent(&self) -> &Path { + match self.path.parent() { + Some(dir) if !dir.as_os_str().is_empty() => dir, + _ => Path::new("."), + } + } + + /// Make a created or renamed directory entry durable. + fn sync_dir(&self) -> io::Result<()> { + #[cfg(unix)] + { + fs::File::open(self.parent())?.sync_all()?; + } + Ok(()) + } + + /// The open append handle, creating the file on first use. + fn handle(&mut self) -> io::Result<&mut fs::File> { + if self.file.is_none() { + let file = Self::options().append(true).create(true).open(&self.path)?; + self.vet(&file)?; + // The open may have created the file, now or on an earlier + // attempt whose directory sync failed. Sync the directory every + // time a handle is established, so no append is acknowledged + // while the file's directory entry might not be durable. + self.sync_dir()?; + self.file = Some(file); + } + // The branch above filled `self.file`. + self.file + .as_mut() + .ok_or_else(|| io::Error::other("journal file handle missing")) + } +} + +impl Storage for FileStorage { + fn read(&mut self) -> io::Result>> { + let mut file = match Self::options().read(true).open(&self.path) { + Ok(file) => file, + Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(None), + Err(e) => return Err(e), + }; + self.vet(&file)?; + let mut bytes = Vec::new(); + file.read_to_end(&mut bytes)?; + Ok(Some(bytes)) + } + + fn append(&mut self, bytes: &[u8]) -> io::Result<()> { + self.handle()?.write_all(bytes) + } + + fn sync(&mut self) -> io::Result<()> { + self.handle()?.sync_data() + } + + fn replace(&mut self, bytes: &[u8]) -> io::Result<()> { + let name = self + .path + .file_name() + .ok_or_else(|| { + io::Error::new(io::ErrorKind::InvalidInput, "journal path has no file name") + })? + .to_string_lossy() + .into_owned(); + let temp = self.parent().join(format!( + "{name}.{}.{:016x}.tmp", + std::process::id(), + rand_suffix() + )); + let written = (|| { + let mut file = Self::options().write(true).create_new(true).open(&temp)?; + file.write_all(bytes)?; + file.sync_all()?; + fs::rename(&temp, &self.path) + })(); + if let Err(e) = written { + // Best effort: a leftover temporary file is ignored by `read`. + let _ = fs::remove_file(&temp); + return Err(e); + } + // The old handle points at the replaced file. + self.file = None; + self.sync_dir() + } + + fn set_aside(&mut self, bytes: &[u8]) -> io::Result { + let mut name = self.path.clone().into_os_string(); + name.push(format!(".rejected.{:016x}", rand_suffix())); + let aside = PathBuf::from(name); + let mut file = Self::options().write(true).create_new(true).open(&aside)?; + file.write_all(bytes)?; + file.sync_all()?; + self.sync_dir()?; + Ok(aside.display().to_string()) + } +} + +/// A value that differs between calls, for temporary file names. Uniqueness +/// is enforced by `create_new`; this only makes collisions unlikely. +fn rand_suffix() -> u64 { + use std::collections::hash_map::RandomState; + use std::hash::{BuildHasher, Hasher}; + RandomState::new().build_hasher().finish() +} + +/// [`Storage`] in memory, with the model's crash semantics. +/// +/// It keeps the bytes that would be on disk (`durable`) apart from the bytes +/// appended since the last sync (`pending`), exactly as the model's `File` +/// does. [`SimStorage::crash_images`] lists every content a crash could +/// leave (model: `Crash`). Cloning shares the state, so a test can keep a +/// handle to storage it has given away. +#[derive(Debug, Clone, Default)] +pub struct SimStorage { + state: std::sync::Arc>, +} + +#[derive(Debug, Default)] +struct SimState { + exists: bool, + durable: Vec, + pending: Vec, + /// One entry per storage call that changed something: the state after + /// it. + trace: Vec, + /// Byte runs handed to `set_aside`. + aside: Vec>, + /// Fail the next calls of these kinds (to test error handling). + fail_append: bool, + /// Fail appends after writing this many bytes of them. + fail_append_after: Option, + fail_sync: bool, + fail_replace: bool, +} + +/// The simulated file after one storage call. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SimPoint { + /// Bytes that would survive any crash. + pub durable: Vec, + /// Bytes appended and not yet synced. + pub pending: Vec, +} + +impl SimPoint { + /// Every content a crash at this point could leave: the durable bytes + /// followed by each prefix of the pending bytes. + pub fn crash_images(&self) -> Vec> { + (0..=self.pending.len()) + .map(|cut| { + let mut image = self.durable.clone(); + image.extend_from_slice(&self.pending[..cut]); + image + }) + .collect() + } +} + +impl SimStorage { + fn lock(&self) -> std::sync::MutexGuard<'_, SimState> { + // A panic while the lock is held cannot leave the plain data in an + // unusable state, so a poisoned lock is still safe to read. + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + } + + /// Empty storage: nothing has been stored yet. + pub fn new() -> Self { + Self::default() + } + + /// Storage whose durable contents are `bytes`, as after a restart. + pub fn with_contents(bytes: Vec) -> Self { + let storage = Self::default(); + { + let mut state = storage.lock(); + state.exists = true; + state.durable = bytes; + } + storage + } + + /// The current state of the simulated file. + pub fn point(&self) -> SimPoint { + let state = self.lock(); + SimPoint { + durable: state.durable.clone(), + pending: state.pending.clone(), + } + } + + /// The state after each storage call so far that changed something. + pub fn trace(&self) -> Vec { + self.lock().trace.clone() + } + + /// Every content a crash now could leave. + pub fn crash_images(&self) -> Vec> { + self.point().crash_images() + } + + /// Byte runs set aside so far. + pub fn set_aside_bytes(&self) -> Vec> { + self.lock().aside.clone() + } + + /// Make appends fail until turned off. + pub fn fail_append(&self, fail: bool) { + self.lock().fail_append = fail; + } + + /// Make appends fail after writing their first `bytes` bytes, as a real + /// write can; `None` turns it off. + pub fn fail_append_after(&self, bytes: Option) { + self.lock().fail_append_after = bytes; + } + + /// Make syncs fail until turned off. + pub fn fail_sync(&self, fail: bool) { + self.lock().fail_sync = fail; + } + + /// Make replacements fail until turned off. + pub fn fail_replace(&self, fail: bool) { + self.lock().fail_replace = fail; + } +} + +impl SimState { + fn record(&mut self) { + self.trace.push(SimPoint { + durable: self.durable.clone(), + pending: self.pending.clone(), + }); + } +} + +fn injected(what: &str) -> io::Error { + io::Error::other(format!("simulated {what} failure")) +} + +impl Storage for SimStorage { + fn read(&mut self) -> io::Result>> { + let state = self.lock(); + if !state.exists { + return Ok(None); + } + let mut bytes = state.durable.clone(); + bytes.extend_from_slice(&state.pending); + Ok(Some(bytes)) + } + + fn append(&mut self, bytes: &[u8]) -> io::Result<()> { + let mut state = self.lock(); + if state.fail_append { + return Err(injected("append")); + } + if let Some(limit) = state.fail_append_after { + state.exists = true; + let written = limit.min(bytes.len()); + state.pending.extend_from_slice(&bytes[..written]); + state.record(); + return Err(injected("partial append")); + } + state.exists = true; + state.pending.extend_from_slice(bytes); + state.record(); + Ok(()) + } + + fn sync(&mut self) -> io::Result<()> { + let mut state = self.lock(); + if state.fail_sync { + return Err(injected("sync")); + } + let pending = std::mem::take(&mut state.pending); + state.durable.extend_from_slice(&pending); + state.record(); + Ok(()) + } + + fn replace(&mut self, bytes: &[u8]) -> io::Result<()> { + let mut state = self.lock(); + if state.fail_replace { + return Err(injected("replace")); + } + state.exists = true; + state.durable = bytes.to_vec(); + state.pending.clear(); + state.record(); + Ok(()) + } + + fn set_aside(&mut self, bytes: &[u8]) -> io::Result { + self.lock().aside.push(bytes.to_vec()); + Ok("memory".to_string()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn frames(payloads: &[&[u8]]) -> Vec { + let mut out = Vec::new(); + for payload in payloads { + write_frame(&mut out, payload).unwrap(); + } + out + } + + #[test] + fn frame_layout_is_length_check_payload_checksum() { + let bytes = frame(b"ab").unwrap(); + let length_field = 2u32.to_le_bytes(); + assert_eq!(HEADER_LEN, 8); + assert_eq!(bytes.len(), HEADER_LEN + 2 + SUM_LEN); + assert_eq!(&bytes[..LENGTH_FIELD_LEN], &length_field); + assert_eq!( + &bytes[LENGTH_FIELD_LEN..HEADER_LEN], + &header_check(&length_field) + ); + assert_eq!(&bytes[HEADER_LEN..HEADER_LEN + 2], b"ab"); + assert_eq!(&bytes[HEADER_LEN + 2..], &checksum(b"ab")); + } + + /// The header check is written down here so that a change to it, or to + /// the tag hashed with it, is a change to this test: the bytes on disk + /// are a format, and old journals must not be misread. + #[test] + fn header_check_is_the_start_of_sha256_over_the_tag_and_the_length() { + let length_field = 5u32.to_le_bytes(); + let mut hashed = b"vsh-journal-header-v1".to_vec(); + hashed.extend_from_slice(&length_field); + assert_eq!(header_check(&length_field), Sha256::digest(&hashed)[..4]); + // Different lengths give different checks, or the check says nothing. + assert_ne!( + header_check(&5u32.to_le_bytes()), + header_check(&6u32.to_le_bytes()) + ); + } + + /// Model: `decode-clean`. + #[test] + fn frames_decode_to_their_payloads() { + let payloads: [&[u8]; 4] = [b"one", b"", b"three", &[0u8; 300]]; + let bytes = frames(&payloads); + let decoded = decode(&bytes); + assert_eq!(decoded.records, payloads); + assert_eq!(decoded.accepted, bytes.len()); + assert_eq!(decoded.tail, Tail::Clean); + } + + /// A cut anywhere inside a record is an interrupted append (model: + /// `classify-torn`); a changed bit anywhere in a whole record, its header + /// included, is damage. + #[test] + fn a_cut_record_is_torn_and_a_changed_record_is_damaged() { + let first = frame(b"first").unwrap(); + let bytes = frames(&[b"first", b"second", b"third"]); + + // Every cut that is not on a record boundary. + let boundaries = [ + 0, + first.len(), + first.len() + frame(b"second").unwrap().len(), + ]; + for cut in 0..bytes.len() { + let expected = if boundaries.contains(&cut) { + Tail::Clean + } else { + Tail::Torn + }; + assert_eq!(decode(&bytes[..cut]).tail, expected, "cut at {cut}"); + } + + // Every single flipped bit in the second record, with the third + // record still after it. + for i in first.len()..first.len() + HEADER_LEN + 6 + SUM_LEN { + for bit in 0..8 { + let mut damaged = bytes.clone(); + damaged[i] ^= 1 << bit; + let decoded = decode(&damaged); + assert_eq!(decoded.records, [b"first"], "byte {i} bit {bit}"); + assert_eq!(decoded.tail, Tail::Damaged, "byte {i} bit {bit}"); + } + } + } + + /// A length field changed so that the record seems to run past the end + /// of the file must not pass for an interrupted append. Before the + /// header had a check it did, and the records after it were dropped. + #[test] + fn a_length_changed_to_run_past_the_end_is_damage_not_a_torn_tail() { + let bytes = frames(&[b"first", b"second", b"third"]); + let second = frame(b"first").unwrap().len(); + let third = second + frame(b"second").unwrap().len(); + for (start, kept) in [(0, 0), (second, 1), (third, 2)] { + // Every bit of the length field, then every bit of its check. + for i in start..start + HEADER_LEN { + for bit in 0..8 { + let mut damaged = bytes.clone(); + damaged[i] ^= 1 << bit; + let decoded = decode(&damaged); + assert_eq!(decoded.records.len(), kept, "byte {i} bit {bit}"); + assert_eq!(decoded.accepted, start, "byte {i} bit {bit}"); + assert_eq!(decoded.tail, Tail::Damaged, "byte {i} bit {bit}"); + } + } + // The case from the review: the length made as large as it goes. + let mut damaged = bytes.clone(); + damaged[start..start + LENGTH_FIELD_LEN].copy_from_slice(&u32::MAX.to_le_bytes()); + assert_eq!(decode(&damaged).tail, Tail::Damaged, "record at {start}"); + } + } + + /// A tail of zero bytes is not a record cut short: its header fails the + /// check. Some file systems can leave zeros after a crash; the journal + /// reports that as damage and leaves the decision to a person. + #[test] + fn a_tail_of_zero_bytes_is_damage_once_it_holds_a_whole_header() { + let good = frames(&[b"first"]); + for zeros in 1..=HEADER_LEN + 40 { + let mut bytes = good.clone(); + bytes.extend(std::iter::repeat_n(0u8, zeros)); + let decoded = decode(&bytes); + assert_eq!(decoded.records, [b"first"], "{zeros} zero bytes"); + let expected = if zeros < HEADER_LEN { + Tail::Torn + } else { + Tail::Damaged + }; + assert_eq!(decoded.tail, expected, "{zeros} zero bytes"); + } + } + + /// Model: `decode-crash` with nothing synced. Every prefix of a run of + /// frames decodes to the whole records it contains, and nothing else. + #[test] + fn every_prefix_decodes_to_whole_records_only() { + let payloads: [&[u8]; 3] = [b"alpha", b"", b"gamma-gamma"]; + let bytes = frames(&payloads); + let mut boundaries = vec![0]; + for payload in payloads { + boundaries.push(boundaries.last().unwrap() + HEADER_LEN + payload.len() + SUM_LEN); + } + for cut in 0..=bytes.len() { + let whole = boundaries.iter().filter(|&&b| b != 0 && b <= cut).count(); + let decoded = decode(&bytes[..cut]); + assert_eq!(decoded.records, &payloads[..whole], "cut at {cut}"); + assert_eq!(decoded.accepted, boundaries[whole], "cut at {cut}"); + } + } + + /// Model: `decode-sound`. For arbitrary bytes, the accepted records + /// re-encode to exactly the accepted front of the input. + #[test] + fn accepted_records_reencode_to_the_accepted_bytes() { + let good = frames(&[b"x", b"yy"]); + let mut inputs: Vec> = vec![ + Vec::new(), + vec![0; 3], + vec![0xff; 64], + good.clone(), + [good.as_slice(), &[1, 2, 3]].concat(), + ]; + // A flipped bit anywhere in a good journal. + for i in 0..good.len() { + let mut damaged = good.clone(); + damaged[i] ^= 0x01; + inputs.push(damaged); + } + for input in inputs { + let decoded = decode(&input); + assert_eq!(frames(&decoded.records), &input[..decoded.accepted]); + } + } + + /// A changed byte in a record rejects that record and everything after + /// it; earlier records are kept. + #[test] + fn damage_rejects_the_record_and_what_follows() { + let first = frame(b"first").unwrap(); + let bytes = frames(&[b"first", b"second", b"third"]); + for i in first.len()..first.len() + HEADER_LEN + 6 + SUM_LEN { + let mut damaged = bytes.clone(); + damaged[i] ^= 0x80; + let decoded = decode(&damaged); + assert_eq!(decoded.records, [b"first"], "byte {i}"); + assert_eq!(decoded.accepted, first.len(), "byte {i}"); + } + } + + /// A good header that declares the longest payload there is, with the + /// file ending soon after it. This is what the start of a very large + /// append looks like, so it is torn, and the arithmetic must not wrap. + #[test] + fn a_good_header_for_the_longest_payload_is_torn_without_overflow() { + let length_field = u32::MAX.to_le_bytes(); + let mut bytes = length_field.to_vec(); + bytes.extend_from_slice(&header_check(&length_field)); + bytes.extend_from_slice(&[7; 40]); + let decoded = decode(&bytes); + assert_eq!(decoded.accepted, 0); + assert_eq!(decoded.tail, Tail::Torn); + } + + #[test] + fn sim_storage_keeps_pending_apart_until_sync() { + let mut storage = SimStorage::new(); + assert_eq!(storage.read().unwrap(), None); + storage.append(b"ab").unwrap(); + assert_eq!(storage.read().unwrap().unwrap(), b"ab"); + assert_eq!( + storage.crash_images(), + vec![b"".to_vec(), b"a".to_vec(), b"ab".to_vec()] + ); + storage.sync().unwrap(); + assert_eq!(storage.crash_images(), vec![b"ab".to_vec()]); + storage.append(b"c").unwrap(); + storage.replace(b"xyz").unwrap(); + assert_eq!(storage.crash_images(), vec![b"xyz".to_vec()]); + assert_eq!(storage.trace().len(), 4); + } + + #[test] + fn sim_storage_can_fail_an_append_part_way() { + let mut storage = SimStorage::new(); + storage.fail_append_after(Some(2)); + assert!(storage.append(b"abcd").is_err()); + assert_eq!(storage.point().pending, b"ab"); + storage.fail_append_after(None); + storage.append(b"cd").unwrap(); + assert_eq!(storage.point().pending, b"abcd"); + } + + #[cfg(unix)] + mod file { + use super::super::*; + use std::os::unix::fs::PermissionsExt; + + fn storage() -> (tempfile::TempDir, FileStorage) { + let dir = tempfile::tempdir().unwrap(); + let storage = FileStorage::new(dir.path().join("journal")); + (dir, storage) + } + + fn mode(path: &Path) -> u32 { + fs::metadata(path).unwrap().permissions().mode() & 0o777 + } + + #[test] + fn read_of_a_missing_file_is_none() { + let (_dir, mut storage) = storage(); + assert_eq!(storage.read().unwrap(), None); + } + + #[test] + fn append_sync_read_round_trip() { + let (_dir, mut storage) = storage(); + storage.append(b"one").unwrap(); + storage.append(b"two").unwrap(); + storage.sync().unwrap(); + assert_eq!(storage.read().unwrap().unwrap(), b"onetwo"); + assert_eq!(mode(storage.path()), 0o600); + } + + #[test] + fn replace_swaps_contents_and_later_appends_follow_it() { + let (dir, mut storage) = storage(); + storage.append(b"old").unwrap(); + storage.replace(b"new").unwrap(); + storage.append(b"+more").unwrap(); + storage.sync().unwrap(); + assert_eq!(storage.read().unwrap().unwrap(), b"new+more"); + assert_eq!(mode(storage.path()), 0o600); + let names: Vec<_> = fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name()) + .collect(); + assert_eq!(names, ["journal"], "no temporary file is left behind"); + } + + #[test] + fn an_existing_file_with_loose_permissions_is_tightened() { + let (_dir, mut storage) = storage(); + fs::write(storage.path(), b"").unwrap(); + fs::set_permissions(storage.path(), fs::Permissions::from_mode(0o644)).unwrap(); + storage.append(b"secret").unwrap(); + assert_eq!(mode(storage.path()), 0o600); + } + + #[test] + fn a_symlink_is_refused_and_its_target_untouched() { + let (dir, mut storage) = storage(); + let target = dir.path().join("target"); + fs::write(&target, b"keep me").unwrap(); + std::os::unix::fs::symlink(&target, storage.path()).unwrap(); + + assert!(storage.append(b"x").is_err()); + assert!(storage.read().is_err()); + assert_eq!(fs::read(&target).unwrap(), b"keep me"); + + // Replacement renames over the link itself, never through it. + storage.replace(b"fresh").unwrap(); + assert_eq!(fs::read(&target).unwrap(), b"keep me"); + assert_eq!(storage.read().unwrap().unwrap(), b"fresh"); + } + + /// A FIFO with no peer blocks a plain `open`. The storage must + /// refuse it without waiting. + #[test] + fn a_fifo_in_place_of_the_file_is_refused_without_blocking() { + let (dir, storage) = storage(); + nix::unistd::mkfifo(storage.path(), nix::sys::stat::Mode::S_IRWXU).unwrap(); + + let (done, finished) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + let mut storage = storage; + let outcome = (storage.read().is_err(), storage.append(b"x").is_err()); + let _ = done.send(outcome); + drop(dir); + }); + let outcome = finished + .recv_timeout(std::time::Duration::from_secs(10)) + .expect("storage calls on a FIFO must not block"); + assert_eq!(outcome, (true, true)); + } + + #[test] + fn a_file_with_a_second_hard_link_is_refused_and_left_alone() { + let (dir, mut storage) = storage(); + let other = dir.path().join("someone-elses"); + fs::write(&other, b"theirs").unwrap(); + fs::set_permissions(&other, fs::Permissions::from_mode(0o644)).unwrap(); + fs::hard_link(&other, storage.path()).unwrap(); + + assert!(storage.append(b"x").is_err()); + assert!(storage.read().is_err()); + assert_eq!(fs::read(&other).unwrap(), b"theirs"); + assert_eq!(mode(&other), 0o644, "its permissions are not changed"); + } + + #[test] + fn a_directory_in_place_of_the_file_is_refused() { + let (_dir, mut storage) = storage(); + fs::create_dir(storage.path()).unwrap(); + assert!(storage.append(b"x").is_err()); + assert!(storage.read().is_err()); + } + } +} diff --git a/impl/rust-cli/src/lib.rs b/impl/rust-cli/src/lib.rs index 2323f8db..fb9bfcd1 100644 --- a/impl/rust-cli/src/lib.rs +++ b/impl/rust-cli/src/lib.rs @@ -25,6 +25,7 @@ pub mod help; pub mod highlighter; pub mod history; pub mod job; +pub mod journal; pub mod pager; pub mod parser; pub mod posix_builtins; diff --git a/impl/rust-cli/src/redirection.rs b/impl/rust-cli/src/redirection.rs index 8c8c1558..21f253d7 100644 --- a/impl/rust-cli/src/redirection.rs +++ b/impl/rust-cli/src/redirection.rs @@ -441,7 +441,7 @@ impl RedirectSetup { op = op.with_undo_data(data); } - state.record_operation(op); + state.record_operation(op)?; } Ok(()) diff --git a/impl/rust-cli/src/state.rs b/impl/rust-cli/src/state.rs index 7f75389b..09d11d71 100644 --- a/impl/rust-cli/src/state.rs +++ b/impl/rust-cli/src/state.rs @@ -11,10 +11,12 @@ use colored::Colorize; use serde::{Deserialize, Serialize}; use std::collections::{BTreeMap, HashMap, HashSet, VecDeque}; use std::fs; +use std::io; use std::path::{Path, PathBuf}; use uuid::Uuid; use crate::functions::FunctionTable; +use crate::journal::{self, FileStorage, Storage}; use crate::posix_builtins::{AliasTable, TrapTable}; use crate::proof_refs::ProofReference; @@ -245,7 +247,8 @@ pub struct Transaction { /// Main shell state managing operation history, undo/redo stacks, and transactions. /// -/// The state is persisted to `~/.cache/vsh/state.json` and restored on shell startup. +/// The state is persisted to `.vsh_state.journal` in the session root and restored on +/// shell startup. /// All filesystem operations are relative to the `root` path. /// /// # Invariants @@ -278,8 +281,29 @@ pub struct ShellState { /// Completed transactions pub transactions: Vec, - /// State file path for persistence - state_file: PathBuf, + /// Where the state is persisted: one journal file of framed records + /// (see [`crate::journal`]). Every mutation that must survive a restart + /// is one record appended and synced, so a crash leaves the state + /// before or after it. The model is `proofs/agda/JournalModel.agda`. + storage: Box, + + /// Records in the journal since its last snapshot record. The journal + /// is rewritten as one snapshot once this outgrows the history, which + /// keeps persistence amortised O(1) per operation. + journal_records: usize, + + /// A journal write failed, so the journal may be behind this state or + /// end in a partial record. The next persist rewrites it whole. + rewrite_pending: bool, + + /// Indices of a contiguous block of `history` known to be all undone. + /// + /// Repeated single-step undo used to rescan every already-undone + /// operation at the end of the history (twice: to pick the next target + /// and to mark it), so undoing n operations one at a time cost O(n²). + /// Undo skips this block instead. It is a hint: it is checked before + /// use and an empty range means "no hint". + undone_run: std::ops::Range, /// Last exit code from external command (for $? support) pub last_exit_code: i32, @@ -296,6 +320,9 @@ pub struct ShellState { /// Maximum history size (None = unlimited, Some(n) = limit to n operations) /// When limit is reached, oldest operations are removed /// Default: 10,000 operations + /// + /// The operations of an open transaction are never removed, so the + /// history can be longer than this while a transaction is open. pub max_history_size: Option, /// Directory for archiving old history entries @@ -324,9 +351,75 @@ pub struct ShellState { pub aliases: AliasTable, } +/// A change was made, and its record could not be saved. +/// +/// The change itself went through, and this session's history holds its +/// record. The saved history does not. The next record that can be +/// written saves the whole history again, which puts that right; if the +/// shell stops before then, the saved history will not know of the change. +/// +/// Every command that changes something returns this error (owner ruling +/// OD-49). Before, the shell printed a warning and reported success, so a +/// script or an agent could not see that the saved history had fallen +/// behind the disk. +#[derive(Debug)] +pub struct NotRecorded { + /// Why the record could not be written. + pub cause: String, +} + +impl NotRecorded { + fn new(cause: anyhow::Error) -> Self { + Self { + cause: format!("{cause:#}"), + } + } +} + +impl std::fmt::Display for NotRecorded { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + f, + "the change was made, but its record could not be saved ({}). \ + The next record that can be written will save it", + self.cause + ) + } +} + +impl std::error::Error for NotRecorded {} + impl ShellState { - /// Create new shell state with given root + /// Create new shell state with given root. + /// + /// State is kept in `/.vsh_state.journal`. A `.vsh_state.json` + /// written by an earlier version is read once, converted, and renamed + /// to `.vsh_state.json.migrated`. + /// + /// # Errors + /// + /// Fails if the saved state cannot be read or understood. Nothing is + /// overwritten in that case: the files are left as they are. pub fn new(root: &str) -> Result { + let root_path = Self::prepare_root(root)?; + let storage = FileStorage::new(root_path.join(JOURNAL_FILE_NAME)); + let legacy = root_path.join(LEGACY_STATE_FILE_NAME); + Self::open(root_path, Box::new(storage), Some(&legacy)) + } + + /// Create shell state that persists to `storage` instead of a file + /// under the root. Used by tests to run the shell against simulated + /// storage. + /// + /// # Errors + /// + /// As [`Self::new`]. + pub fn with_storage(root: &str, storage: Box) -> Result { + let root_path = Self::prepare_root(root)?; + Self::open(root_path, storage, None) + } + + fn prepare_root(root: &str) -> Result { let root_path = PathBuf::from(root); // Ensure root exists @@ -334,17 +427,24 @@ impl ShellState { fs::create_dir_all(&root_path).context("Failed to create sandbox root")?; } - let root_path = - fs::canonicalize(&root_path).context("Failed to canonicalize sandbox root")?; - let state_file = root_path.join(".vsh_state.json"); + fs::canonicalize(&root_path).context("Failed to canonicalize sandbox root") + } + fn open( + root_path: PathBuf, + storage: Box, + legacy: Option<&Path>, + ) -> Result { let mut state = Self { root: root_path, history: Vec::new(), redo_stack: VecDeque::new(), active_transaction: None, transactions: Vec::new(), - state_file, + storage, + journal_records: 0, + rewrite_pending: false, + undone_run: 0..0, last_exit_code: 0, previous_dir: None, variables: HashMap::new(), @@ -360,14 +460,20 @@ impl ShellState { aliases: AliasTable::new(), }; - // Try to load existing state - state.load().ok(); + // A saved state that cannot be loaded is an error, not an empty + // history: starting empty would let the next write replace it. + state.load(legacy)?; Ok(state) } - /// Record an operation - pub fn record_operation(&mut self, mut op: Operation) { + /// Record an operation. + /// + /// # Errors + /// + /// Returns [`NotRecorded`] if the record could not be saved. The + /// operation is in this session's history all the same. + pub fn record_operation(&mut self, mut op: Operation) -> Result<(), NotRecorded> { // Associate with active transaction if any if let Some(ref mut txn) = self.active_transaction { op.transaction_id = Some(txn.id); @@ -377,46 +483,55 @@ impl ShellState { self.history.push(op); // Enforce history size limit - self.enforce_history_limit(); + let trimmed = self.enforce_history_limit(); // Clear redo stack when new operation is performed self.redo_stack.clear(); - // Persist state - warn on failure but don't abort - if let Err(e) = self.save() { - eprintln!("{} Failed to save state: {}", "Warning:".bright_yellow(), e); - eprintln!("Operation succeeded but may not persist across restarts"); - } + self.persist(Mutation::Recorded { trim: trimmed }) + .map_err(NotRecorded::new) } /// Record an operation from a redo without clearing the redo stack. /// /// Unlike [`Self::record_operation`], this preserves remaining redo entries /// so that multiple sequential redos work correctly. - pub fn record_redo_operation(&mut self, mut op: Operation) { + /// + /// # Errors + /// + /// Returns [`NotRecorded`] if the record could not be saved. + pub fn record_redo_operation(&mut self, mut op: Operation) -> Result<(), NotRecorded> { if let Some(ref mut txn) = self.active_transaction { op.transaction_id = Some(txn.id); txn.operations.push(op.id); } self.history.push(op); - self.enforce_history_limit(); + let trimmed = self.enforce_history_limit(); - // Persist state - warn on failure but don't abort - if let Err(e) = self.save() { - eprintln!("{} Failed to save state: {}", "Warning:".bright_yellow(), e); - eprintln!("Redo succeeded but may not persist across restarts"); - } + self.persist(Mutation::Recorded { trim: trimmed }) + .map_err(NotRecorded::new) } /// Enforce history size limit by removing oldest operations /// /// When history exceeds max_history_size, removes the oldest operations. /// If history_archive_path is set, saves removed operations to archive. - fn enforce_history_limit(&mut self) { + /// Returns the number of operations removed. + /// + /// An open transaction keeps all of its operations, however many there + /// are: rollback has to reverse every one of them, and it cannot reverse + /// one whose record is gone. The history can therefore be longer than + /// the limit while a transaction is open. The first operation recorded + /// after the transaction closes brings it back under the limit. + fn enforce_history_limit(&mut self) -> usize { if let Some(limit) = self.max_history_size { if self.history.len() > limit { - let excess = self.history.len() - limit; + let over = self.history.len() - limit; + let excess = over.min(self.droppable_prefix()); + if excess == 0 { + return 0; + } // Archive old operations if path is set if let Some(ref archive_path) = self.history_archive_path { @@ -432,11 +547,14 @@ impl ShellState { // Remove oldest operations self.history.drain(0..excess); + self.shift_undone_run(excess); // Note: This means undo can only go back `limit` operations // Users should be aware via configuration + return excess; } } + 0 } /// Archive operations to disk for later reference @@ -464,34 +582,142 @@ impl ShellState { Ok(()) } - /// Get the last N undoable operations - pub fn last_n_undoable(&self, n: usize) -> Vec<&Operation> { + /// Whether the saved state holds everything this state has done. + /// + /// It does not after a write to it has failed. The next write that + /// succeeds saves the whole state again and puts that right. + pub fn is_saved(&self) -> bool { + !self.rewrite_pending + } + + /// How many operations at the front of the history may be dropped: all + /// of them, or those before the first operation of the open transaction. + fn droppable_prefix(&self) -> usize { + let Some(txn) = self.active_transaction.as_ref() else { + return self.history.len(); + }; self.history .iter() - .rev() - .filter(|op| !op.undone) + .position(|op| op.transaction_id == Some(txn.id)) + .unwrap_or(self.history.len()) + } + + /// Set the undone-block hint to the run of undone operations at the end + /// of the history. Used when the history was replaced or marked in bulk, + /// so that the next single-step undo does not rescan that run. + fn rebuild_undone_run(&mut self) { + let end = self.history.len(); + let start = self + .history + .iter() + .rposition(|op| !op.undone) + .map_or(0, |last_live| last_live + 1); + self.undone_run = start..end; + } + + /// Get the last N undoable operations + pub fn last_n_undoable(&self, n: usize) -> Vec<&Operation> { + self.undoable_indices_rev() .take(n) + .map(|i| &self.history[i]) .collect() } - /// Mark an operation as undone - pub fn mark_undone(&mut self, op_id: Uuid, undone_by: Uuid) { - if let Some(op) = self.history.iter_mut().find(|o| o.id == op_id) { - op.undone = true; - op.undone_by = Some(undone_by); + /// Indices of operations not yet undone, newest first, skipping the + /// known-undone block instead of rescanning it. + fn undoable_indices_rev(&self) -> impl Iterator + '_ { + let run = self.checked_undone_run(); + let mut next = self.history.len(); + std::iter::from_fn(move || { + while next > 0 { + next -= 1; + if next + 1 == run.end { + next = run.start; + continue; + } + if !self.history[next].undone { + return Some(next); + } + } + None + }) + } - // Push to redo stack - self.redo_stack.push_front(op.clone()); + /// The undone-block hint, or an empty range if it does not hold. + fn checked_undone_run(&self) -> std::ops::Range { + let run = self.undone_run.clone(); + let holds = !run.is_empty() + && run.end <= self.history.len() + && self.history[run.start].undone + && self.history[run.end - 1].undone; + if holds { + run + } else { + 0..0 } + } - // Persist state - warn on failure - if let Err(e) = self.save() { - eprintln!( - "{} Failed to save undo state: {}", - "Warning:".bright_yellow(), - e - ); + /// Record that `history[index]` is now undone in the block hint. + fn extend_undone_run(&mut self, index: usize) { + let run = self.checked_undone_run(); + self.undone_run = if run.is_empty() { + index..index + 1 + } else if index + 1 == run.start { + index..run.end + } else if index == run.end { + run.start..index + 1 + } else if run.contains(&index) { + run + } else { + index..index + 1 + }; + } + + /// Keep the block hint aligned after the oldest `removed` operations + /// were dropped from the front of the history. + fn shift_undone_run(&mut self, removed: usize) { + let start = self.undone_run.start.saturating_sub(removed); + let end = self.undone_run.end.saturating_sub(removed); + self.undone_run = start..end; + } + + /// Mark an operation as undone. + /// + /// # Errors + /// + /// Returns [`NotRecorded`] if the mark could not be saved. The + /// operation is marked in this session's history all the same. + pub fn mark_undone(&mut self, op_id: Uuid, undone_by: Uuid) -> Result<(), NotRecorded> { + // Undo targets the newest operations that are not yet undone, so look + // there first; fall back to a full scan for any other id. + let index = self + .undoable_indices_rev() + .find(|&i| self.history[i].id == op_id) + .or_else(|| self.history.iter().rposition(|o| o.id == op_id)); + + // An operation already undone keeps its first mark, as in the model. + let found = match index { + Some(i) if !self.history[i].undone => { + let op = &mut self.history[i]; + op.undone = true; + op.undone_by = Some(undone_by); + + // Push to redo stack + self.redo_stack.push_front(op.clone()); + self.extend_undone_run(i); + true + } + _ => false, + }; + + if !found { + return Ok(()); } + self.persist(Mutation::Undone { + id: op_id, + undone_by, + }) + .map_err(NotRecorded::new) } /// Begin a new transaction @@ -511,14 +737,9 @@ impl ShellState { let id = txn.id; self.active_transaction = Some(txn); - // Persist state - warn on failure - if let Err(e) = self.save() { - eprintln!( - "{} Failed to save transaction start: {}", - "Warning:".bright_yellow(), - e - ); - } + // The transaction is open in this session whether or not that + // could be saved. + self.persist(Mutation::Begun).map_err(NotRecorded::new)?; Ok(id) } @@ -534,26 +755,66 @@ impl ShellState { committed.committed = true; self.transactions.push(committed); - // Persist state - warn on failure - if let Err(e) = self.save() { + // The transaction is closed in this session whether or not that + // could be saved. + self.persist(Mutation::Committed) + .map_err(NotRecorded::new)?; + + Ok(()) + } + + /// Set the directory `cd -` returns to, and journal it so it survives a + /// restart without waiting for the next snapshot. + /// + /// A failure to save it is only a warning: `cd` changes no file, and a + /// shell on a root it cannot write to must still be able to move about. + pub fn set_previous_dir(&mut self, dir: PathBuf) { + self.previous_dir = Some(dir); + if let Err(e) = self.persist(Mutation::PreviousDir) { eprintln!( - "{} Failed to save transaction commit: {}", + "{} Failed to save previous directory: {}", "Warning:".bright_yellow(), e ); } + } - Ok(()) + /// Close the active transaction as rolled back, and persist that, so a + /// restart does not resurrect it. + /// + /// Every operation of the transaction that is not yet marked undone is + /// marked undone by the transaction, as the model's `rolledBack` event + /// does. Call this only once those operations have been reversed on + /// disk; [`crate::commands::rollback_transaction`] does that. + pub fn abort_transaction(&mut self) { + let Some(txn) = self.active_transaction.take() else { + return; + }; + Self::mark_rolled_back(&mut self.history, &txn); + self.rebuild_undone_run(); + + // Persist state - warn on failure + if let Err(e) = self.persist(Mutation::RolledBack) { + eprintln!( + "{} Failed to save transaction rollback: {}", + "Warning:".bright_yellow(), + e + ); + } } /// Get operations in current transaction pub fn current_transaction_ops(&self) -> Vec<&Operation> { match &self.active_transaction { - Some(txn) => self - .history - .iter() - .filter(|op| txn.operations.contains(&op.id)) - .collect(), + Some(txn) => { + // Indexed once: looking each operation up in the list took + // time that grew with the square of the transaction. + let members: HashSet = txn.operations.iter().copied().collect(); + self.history + .iter() + .filter(|op| members.contains(&op.id)) + .collect() + } None => Vec::new(), } } @@ -584,35 +845,283 @@ impl ShellState { self.redo_stack.pop_front() } - /// Save state to file - fn save(&self) -> Result<()> { - let state = SerializableState { - history: self.history.clone(), - transactions: self.transactions.clone(), - active_transaction: self.active_transaction.clone(), - previous_dir: self.previous_dir.clone(), + /// Mark as undone, by the transaction, each of its operations that is + /// not undone yet. + fn mark_rolled_back(history: &mut [Operation], txn: &Transaction) { + let members: HashSet = txn.operations.iter().copied().collect(); + for op in history.iter_mut() { + if !op.undone && members.contains(&op.id) { + op.undone = true; + op.undone_by = Some(txn.id); + } + } + } + + /// Persist one mutation that has just been applied to this state, as + /// one journal record, appended and synced. + fn persist(&mut self, mutation: Mutation) -> Result<()> { + if self.journal_due_for_rewrite() { + return self.rewrite_journal(); + } + let event = match mutation { + Mutation::Recorded { trim } => match self.history.last() { + Some(op) => EventRef::Recorded { op, trim }, + // With `max_history_size == Some(0)` the operation is trimmed + // at once and there is nothing to append. + None => return self.rewrite_journal(), + }, + Mutation::Undone { id, undone_by } => EventRef::Undone { id, undone_by }, + Mutation::Begun => match self.active_transaction.as_ref() { + Some(txn) => EventRef::Begun(txn), + None => return Ok(()), + }, + Mutation::Committed => EventRef::Committed, + Mutation::RolledBack => EventRef::RolledBack, + Mutation::PreviousDir => EventRef::PreviousDir(self.previous_dir.as_ref()), }; + let appended = append_event(self.storage.as_mut(), &event); + match appended { + Ok(()) => self.journal_records += 1, + // The journal may now end in a partial record, and it lacks this + // mutation. Rewrite it whole before appending again. + Err(_) => self.rewrite_pending = true, + } + appended + } - let json = serde_json::to_string_pretty(&state)?; - fs::write(&self.state_file, json)?; + /// The journal is rewritten as one snapshot record once it holds at + /// least half as many records as the history has operations (and at + /// least `JOURNAL_MIN_RECORDS`). A rewrite costs O(history length), + /// which is then at most twice the number of records it absorbs, so + /// persistence stays amortised O(1) per operation. + fn journal_due_for_rewrite(&self) -> bool { + self.rewrite_pending + || (self.journal_records >= JOURNAL_MIN_RECORDS + && 2 * self.journal_records >= self.history.len()) + } - Ok(()) + /// Replace the journal with one snapshot record of the current state. + /// + /// The storage replaces the file all-or-nothing, so a crash leaves the + /// old journal or the new one, and both load to this state (model: + /// `compaction-safe`). + fn rewrite_journal(&mut self) -> Result<()> { + let snapshot = EventRef::Snapshot(SnapshotRef { + version: SNAPSHOT_VERSION, + history: &self.history, + transactions: &self.transactions, + active_transaction: self.active_transaction.as_ref(), + previous_dir: self.previous_dir.as_ref(), + }); + let replaced = encode_event(&snapshot) + .and_then(|bytes| self.storage.replace(&bytes).map_err(anyhow::Error::from)); + match replaced { + Ok(()) => { + self.journal_records = 0; + self.rewrite_pending = false; + } + Err(_) => self.rewrite_pending = true, + } + replaced } - /// Load state from file - fn load(&mut self) -> Result<()> { - if !self.state_file.exists() { - return Ok(()); + /// Rewrite the journal as one snapshot now. Normal operation does this + /// by itself; tests call it to exercise the rewrite. + #[doc(hidden)] + pub fn compact_journal(&mut self) -> Result<()> { + self.rewrite_journal() + } + + /// Load the saved state by replaying the journal. + /// + /// Bytes at the end of the journal that are not a complete record are + /// what an interrupted write leaves. They are set aside, reported, and + /// removed from the journal, so that later records are not appended + /// behind them. A complete record that is not a known event is an + /// error: nothing is skipped. + fn load(&mut self, legacy: Option<&Path>) -> Result<()> { + let stored = self + .storage + .read() + .context("Failed to read the saved shell state")?; + match stored { + // An empty journal holds nothing, so it does not stand in the + // way of converting an older state file. + Some(bytes) if !bytes.is_empty() => { + let decoded = journal::decode(&bytes); + // A record whose header fails its check, or a whole record + // that fails its checksum, is damage, not an interrupted + // write. Stop before anything is changed: the records after + // it may be good, and only a person can decide what to keep. + if decoded.tail == journal::Tail::Damaged { + anyhow::bail!( + "Saved shell state is damaged: record {} fails its integrity check \ + ({} of {} bytes are readable). Nothing was changed.", + decoded.records.len() + 1, + decoded.accepted, + bytes.len() + ); + } + for (index, payload) in decoded.records.iter().enumerate() { + let event: Event = serde_json::from_slice(payload).with_context(|| { + format!( + "Saved shell state is damaged: record {} is not a known event", + index + 1 + ) + })?; + self.apply_event(event)?; + } + if decoded.accepted < bytes.len() { + let rejected = &bytes[decoded.accepted..]; + let kept = self + .storage + .set_aside(rejected) + .context("Failed to set aside an incomplete record of the saved state")?; + eprintln!( + "{} the saved shell state ended in {} bytes that are not a complete \ + record (an interrupted write). They were set aside in {}.", + "Warning:".bright_yellow(), + rejected.len(), + kept + ); + self.storage + .replace(&bytes[..decoded.accepted]) + .context("Failed to repair the end of the saved shell state")?; + } + } + _ => { + if let Some(legacy) = legacy.filter(|path| path.exists()) { + self.migrate_legacy(legacy)?; + } + } } - let json = crate::fs_pure::read_to_string(&self.state_file)?; - let state: SerializableState = serde_json::from_str(&json)?; + // The history was replaced wholesale. Point the undone-block hint at + // the run of undone operations at its end, so that single-step undo + // after a restart does not rescan that run every time. + self.rebuild_undone_run(); + + Ok(()) + } - self.history = state.history; - self.transactions = state.transactions; - self.active_transaction = state.active_transaction; - self.previous_dir = state.previous_dir; + /// Read a `.vsh_state.json` written before the journal existed, write + /// it out as a journal, and rename the old file out of the way. + fn migrate_legacy(&mut self, legacy: &Path) -> Result<()> { + // Read it under the same rules as the journal: a regular file, owned + // by this user, with one name, never through a symbolic link, and + // readable by its owner only from now on. It holds the same data. + let json = FileStorage::new(legacy) + .read() + .with_context(|| format!("Failed to read {}", legacy.display()))? + .with_context(|| format!("{} disappeared while it was read", legacy.display()))?; + let snapshot: Snapshot = serde_json::from_slice(&json) + .with_context(|| format!("{} is not a valid saved shell state", legacy.display()))?; + self.apply_event(Event::Snapshot(snapshot))?; + self.rewrite_journal() + .context("Failed to convert the saved shell state to a journal")?; + + if let Err(e) = Self::set_legacy_aside(legacy) { + eprintln!( + "{} could not rename {} after converting it: {}", + "Warning:".bright_yellow(), + legacy.display(), + e + ); + } + Ok(()) + } + /// Give the converted old state file a name that is not in use: + /// `.migrated`, or `.migrated.1`, `.2` and so on when an + /// earlier backup is already there. A backup is never replaced. + /// + /// The new name is made as a second link to the file and the old name is + /// then removed. Making a link fails when the name exists, which a rename + /// does not, so nothing can be overwritten between a check and the move. + fn set_legacy_aside(legacy: &Path) -> io::Result { + const ATTEMPTS: u32 = 1000; + for attempt in 0..ATTEMPTS { + let mut name = legacy.as_os_str().to_owned(); + name.push(".migrated"); + if attempt > 0 { + name.push(format!(".{attempt}")); + } + let candidate = PathBuf::from(name); + match fs::hard_link(legacy, &candidate) { + Ok(()) => { + fs::remove_file(legacy)?; + return Ok(candidate); + } + Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue, + Err(e) => return Err(e), + } + } + Err(io::Error::new( + io::ErrorKind::AlreadyExists, + format!("{ATTEMPTS} earlier backups are in the way"), + )) + } + + /// Apply one journal event to the in-memory state (model: `apply`). + /// Mirrors what the live mutation did, minus side effects (archiving, + /// redo stack, I/O); the redo stack is not persisted. + fn apply_event(&mut self, event: Event) -> Result<()> { + match event { + Event::Snapshot(snapshot) => { + if snapshot.version > SNAPSHOT_VERSION { + anyhow::bail!( + "Saved shell state has format version {}, newer than this vsh \ + understands ({})", + snapshot.version, + SNAPSHOT_VERSION + ); + } + self.history = snapshot.history; + self.transactions = snapshot.transactions; + self.active_transaction = snapshot.active_transaction; + self.previous_dir = snapshot.previous_dir; + self.journal_records = 0; + return Ok(()); + } + Event::Recorded { op, trim } => { + if let Some(txn) = self.active_transaction.as_mut() { + if op.transaction_id == Some(txn.id) { + txn.operations.push(op.id); + } + } + self.history.push(op); + let trim = trim.min(self.history.len()); + self.history.drain(0..trim); + } + Event::Undone { id, undone_by } => { + if let Some(op) = self.history.iter_mut().rev().find(|o| o.id == id) { + if !op.undone { + op.undone = true; + op.undone_by = Some(undone_by); + } + } + } + Event::Begun(txn) => { + if self.active_transaction.is_none() { + self.active_transaction = Some(txn); + } + } + Event::Committed => { + if let Some(mut txn) = self.active_transaction.take() { + txn.committed = true; + self.transactions.push(txn); + } + } + Event::RolledBack => { + if let Some(txn) = self.active_transaction.take() { + Self::mark_rolled_back(&mut self.history, &txn); + } + } + Event::PreviousDir(dir) => { + self.previous_dir = dir; + } + } + self.journal_records += 1; Ok(()) } @@ -643,7 +1152,17 @@ impl ShellState { // Record operation let op = Operation::new(OperationType::SetVariable, name, None).with_undo_data(undo_data); - self.record_operation(op); + self.record_variable_operation(op); + } + + /// Record an operation on a variable. If the record cannot be saved + /// that is only a warning: an assignment changes no file, and a script + /// on a root the shell cannot write to must still be able to set its + /// variables. + fn record_variable_operation(&mut self, op: Operation) { + if let Err(e) = self.record_operation(op) { + eprintln!("{} {}", "Warning:".bright_yellow(), e); + } } /// Unset a shell variable with undo tracking. @@ -663,7 +1182,7 @@ impl ShellState { // Record operation let op = Operation::new(OperationType::UnsetVariable, name.to_string(), None) .with_undo_data(undo_data); - self.record_operation(op); + self.record_variable_operation(op); } /// Get a shell variable value (scalar only) @@ -909,8 +1428,26 @@ impl ShellState { } } -#[derive(Serialize, Deserialize)] -struct SerializableState { +/// The journal file, inside the session root. +const JOURNAL_FILE_NAME: &str = ".vsh_state.journal"; + +/// The single-file state written before the journal existed. +const LEGACY_STATE_FILE_NAME: &str = ".vsh_state.json"; + +/// Layout version written into every snapshot record. +const SNAPSHOT_VERSION: u32 = 1; + +/// Minimum number of journal records before the journal is rewritten as one +/// snapshot. +const JOURNAL_MIN_RECORDS: usize = 1024; + +/// The whole persisted state (model: the `State` in a `snapshot` event). +/// A `.vsh_state.json` from before the journal has the same fields, without +/// `version`. +#[derive(Deserialize)] +struct Snapshot { + #[serde(default)] + version: u32, history: Vec, transactions: Vec, active_transaction: Option, @@ -918,6 +1455,80 @@ struct SerializableState { previous_dir: Option, } +/// Borrowing twin of [`Snapshot`], so a rewrite does not clone the whole +/// history first. +#[derive(Serialize)] +struct SnapshotRef<'a> { + version: u32, + history: &'a [Operation], + transactions: &'a [Transaction], + active_transaction: Option<&'a Transaction>, + previous_dir: Option<&'a PathBuf>, +} + +/// One journal record (model: `Event` in `proofs/agda/JournalReplay.agda`). +#[derive(Deserialize)] +enum Event { + /// The whole state; what a journal rewrite leaves as its only record. + Snapshot(Snapshot), + /// An operation appended to the history, then the oldest `trim` + /// operations dropped by the history size limit (0 when under it). One + /// record, so a crash cannot keep the operation without its trim. + Recorded { + op: Operation, + #[serde(default)] + trim: usize, + }, + /// An operation marked undone. + Undone { id: Uuid, undone_by: Uuid }, + /// A transaction opened. + Begun(Transaction), + /// The open transaction committed. + Committed, + /// The open transaction rolled back: each of its operations not yet + /// undone is marked undone by it, and it is closed. + RolledBack, + /// The directory `cd -` returns to, as set by `cd`. + PreviousDir(Option), +} + +/// Borrowing twin of [`Event`]; serialises to the same shape. +#[derive(Serialize)] +enum EventRef<'a> { + Snapshot(SnapshotRef<'a>), + Recorded { op: &'a Operation, trim: usize }, + Undone { id: Uuid, undone_by: Uuid }, + Begun(&'a Transaction), + Committed, + RolledBack, + PreviousDir(Option<&'a PathBuf>), +} + +/// A mutation already applied to the state, to be persisted. It names the +/// event; [`ShellState::persist`] reads the data from the state. +enum Mutation { + Recorded { trim: usize }, + Undone { id: Uuid, undone_by: Uuid }, + Begun, + Committed, + RolledBack, + PreviousDir, +} + +/// The framed journal record for `event`. +fn encode_event(event: &EventRef<'_>) -> Result> { + let payload = serde_json::to_vec(event)?; + Ok(journal::frame(&payload)?) +} + +/// Append `event` to the journal and make it durable. +fn append_event(storage: &mut dyn Storage, event: &EventRef<'_>) -> Result<()> { + let bytes = encode_event(event)?; + storage.append(&bytes)?; + storage.sync()?; + Ok(()) +} + /// Resolve `path` against the sandbox `root`, clamping any `..` traversal so /// the result never escapes `root`. /// @@ -1225,7 +1836,9 @@ mod tests { #[test] fn test_set_variable_tracked_records_operation() { - let mut state = ShellState::new("/tmp/vsh_tracked_var_1").unwrap(); + // A fresh directory: these tests record an operation, which is saved. + let root = persisted_root(); + let mut state = reopen(&root); state.set_variable_tracked("FOO", "bar"); assert_eq!(state.get_variable("FOO"), Some("bar")); @@ -1238,7 +1851,9 @@ mod tests { #[test] fn test_set_variable_tracked_captures_previous() { - let mut state = ShellState::new("/tmp/vsh_tracked_var_2").unwrap(); + // A fresh directory: these tests record an operation, which is saved. + let root = persisted_root(); + let mut state = reopen(&root); // Set initial value state.set_variable("FOO", "old_value"); // Now track-set a new value @@ -1257,7 +1872,9 @@ mod tests { #[test] fn test_set_variable_tracked_new_var_captures_none() { - let mut state = ShellState::new("/tmp/vsh_tracked_var_3").unwrap(); + // A fresh directory: these tests record an operation, which is saved. + let root = persisted_root(); + let mut state = reopen(&root); state.set_variable_tracked("NEW_VAR", "value"); let last_op = state.history.last().unwrap(); @@ -1268,7 +1885,9 @@ mod tests { #[test] fn test_unset_variable_tracked_records_operation() { - let mut state = ShellState::new("/tmp/vsh_tracked_var_4").unwrap(); + // A fresh directory: these tests record an operation, which is saved. + let root = persisted_root(); + let mut state = reopen(&root); state.set_variable("FOO", "bar"); state.unset_variable_tracked("FOO"); @@ -1323,4 +1942,743 @@ mod tests { fn test_chown_display() { assert_eq!(format!("{}", OperationType::Chown), "chown"); } + + // ======================================================================== + // Persistence: the journal + // ======================================================================== + + use crate::journal::SimStorage; + + fn persisted_root() -> tempfile::TempDir { + tempfile::TempDir::new().unwrap() + } + + fn reopen(root: &tempfile::TempDir) -> ShellState { + ShellState::new(root.path().to_str().unwrap()).unwrap() + } + + fn journal_path(root: &tempfile::TempDir) -> PathBuf { + root.path().join(JOURNAL_FILE_NAME) + } + + /// The payloads of the records in the journal file, as text. + fn journal_records(root: &tempfile::TempDir) -> Vec { + let bytes = fs::read(journal_path(root)).unwrap(); + let decoded = journal::decode(&bytes); + assert_eq!(decoded.accepted, bytes.len(), "journal should be clean"); + decoded + .records + .iter() + .map(|payload| String::from_utf8(payload.to_vec()).unwrap()) + .collect() + } + + /// A state on simulated storage, and a handle to that storage. + fn sim_state(root: &tempfile::TempDir) -> (ShellState, SimStorage) { + let storage = SimStorage::new(); + let state = + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())) + .unwrap(); + (state, storage) + } + + /// A state loaded from simulated storage holding `bytes`. + fn sim_reopen(root: &tempfile::TempDir, bytes: Vec) -> Result { + ShellState::with_storage( + root.path().to_str().unwrap(), + Box::new(SimStorage::with_contents(bytes)), + ) + } + + /// Record an operation on `path`. Some tests make the storage refuse + /// the write and then check what the next write does, so a record that + /// cannot be saved is not a failure here. + fn record(state: &mut ShellState, path: &str) -> Uuid { + let op = Operation::new(OperationType::CreateFile, path.to_string(), None); + let id = op.id; + let _ = state.record_operation(op); + id + } + + fn summary(state: &ShellState) -> Vec<(Uuid, bool, Option)> { + state + .history + .iter() + .map(|op| (op.id, op.undone, op.undone_by)) + .collect() + } + + #[test] + fn test_journal_replay_restores_history_and_undo_marks() { + let root = persisted_root(); + let mut state = reopen(&root); + record(&mut state, "a"); + let b = record(&mut state, "b"); + record(&mut state, "c"); + state.mark_undone(b, Uuid::new_v4()).unwrap(); + + // One record per mutation, and no other state file. + assert_eq!(journal_records(&root).len(), 4); + assert!(!root.path().join(LEGACY_STATE_FILE_NAME).exists()); + + let reloaded = reopen(&root); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_journal_is_rewritten_as_one_snapshot_and_reloads() { + let root = persisted_root(); + let mut state = reopen(&root); + for i in 0..(JOURNAL_MIN_RECORDS + 10) { + record(&mut state, &format!("f{i}")); + } + assert!(state.journal_records < JOURNAL_MIN_RECORDS); + let records = journal_records(&root); + assert!( + records.len() < JOURNAL_MIN_RECORDS, + "journal should be compact" + ); + assert!(records[0].starts_with("{\"Snapshot\"")); + + let reloaded = reopen(&root); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_journal_replays_history_trim() { + let root = persisted_root(); + let mut state = reopen(&root); + state.max_history_size = Some(3); + for i in 0..5 { + record(&mut state, &format!("f{i}")); + } + assert_eq!(state.history.len(), 3); + + // The reopened state has the default (larger) limit, so the trim must + // come from the journal, not from re-applying a limit. + let reloaded = reopen(&root); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_record_and_trim_share_one_record() { + let root = persisted_root(); + let mut state = reopen(&root); + state.max_history_size = Some(3); + for i in 0..5 { + record(&mut state, &format!("f{i}")); + } + // One record per operation: no trim can be split from its record. + let records = journal_records(&root); + assert_eq!(records.len(), 5); + assert!(records[4].contains("\"trim\":1")); + } + + /// An interrupted write leaves part of a record at the end of the file. + fn append_torn_record(root: &tempfile::TempDir) -> Vec { + use std::io::Write; + let whole = journal::frame(b"{\"Recorded\":{\"op\":{\"path\":\"caf\xC3\xA9\"}}}").unwrap(); + let torn = whole[..whole.len() - 5].to_vec(); + let mut file = fs::OpenOptions::new() + .append(true) + .open(journal_path(root)) + .unwrap(); + file.write_all(&torn).unwrap(); + torn + } + + #[test] + fn test_torn_tail_is_set_aside_and_ignored() { + let root = persisted_root(); + let mut state = reopen(&root); + record(&mut state, "a"); + record(&mut state, "b"); + let torn = append_torn_record(&root); + + let reloaded = reopen(&root); + assert_eq!(summary(&reloaded), summary(&state)); + + // The journal is clean again, and the torn bytes were kept. + assert_eq!(journal_records(&root).len(), 2); + let aside: Vec = fs::read_dir(root.path()) + .unwrap() + .map(|entry| entry.unwrap().path()) + .filter(|path| path.to_string_lossy().contains(".rejected.")) + .collect(); + assert_eq!(aside.len(), 1); + assert_eq!(fs::read(&aside[0]).unwrap(), torn); + } + + #[test] + fn test_operations_after_torn_tail_survive_reload() { + let root = persisted_root(); + let mut state = reopen(&root); + record(&mut state, "a"); + append_torn_record(&root); + + // The restarted shell keeps working after the crash... + let mut restarted = reopen(&root); + record(&mut restarted, "b"); + record(&mut restarted, "c"); + + // ...and those later operations must not hide behind the torn record. + let reloaded = reopen(&root); + assert_eq!(reloaded.history.len(), 3); + assert_eq!(summary(&reloaded), summary(&restarted)); + } + + #[cfg(unix)] + #[test] + fn test_journal_file_is_private() { + use std::os::unix::fs::PermissionsExt; + let root = persisted_root(); + let mut state = reopen(&root); + record(&mut state, "a"); + state.compact_journal().unwrap(); + record(&mut state, "b"); + let mode = fs::metadata(journal_path(&root)) + .unwrap() + .permissions() + .mode() + & 0o777; + assert_eq!(mode, 0o600); + } + + #[test] + fn test_previous_dir_survives_reload() { + let root = persisted_root(); + let mut state = reopen(&root); + record(&mut state, "a"); + state.set_previous_dir(PathBuf::from("/tmp/elsewhere")); + + let reloaded = reopen(&root); + assert_eq!( + reloaded.previous_dir.as_deref(), + Some(Path::new("/tmp/elsewhere")) + ); + } + + #[test] + fn test_transaction_boundaries_are_journal_records() { + let root = persisted_root(); + let mut state = reopen(&root); + let txn = state.begin_transaction("t").unwrap(); + let a = record(&mut state, "a"); + state.commit_transaction().unwrap(); + + // Begin, record and commit are three appended records: no rewrite. + let records = journal_records(&root); + assert_eq!(records.len(), 3); + assert!(records[0].starts_with("{\"Begun\"")); + assert_eq!(records[2], "\"Committed\""); + + let reloaded = reopen(&root); + assert!(reloaded.active_transaction.is_none()); + assert_eq!(reloaded.transactions.len(), 1); + assert_eq!(reloaded.transactions[0].id, txn); + assert!(reloaded.transactions[0].committed); + assert_eq!(reloaded.transactions[0].operations, vec![a]); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_rolled_back_transaction_is_closed_and_its_operations_undone() { + let root = persisted_root(); + let mut state = reopen(&root); + let before = record(&mut state, "before"); + let txn = state.begin_transaction("t").unwrap(); + let a = record(&mut state, "a"); + let b = record(&mut state, "b"); + let undo = Uuid::new_v4(); + state.mark_undone(b, undo).unwrap(); + state.abort_transaction(); + + assert!(state.active_transaction.is_none()); + assert_eq!( + summary(&state), + vec![ + (before, false, None), + (a, true, Some(txn)), + // Already undone inside the transaction: keeps its first mark. + (b, true, Some(undo)), + ] + ); + assert!(state + .last_n_undoable(usize::MAX) + .iter() + .all(|op| op.id == before)); + + let reloaded = reopen(&root); + assert!(reloaded.active_transaction.is_none()); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_unknown_record_is_an_error_and_nothing_is_overwritten() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + let mut bytes = storage.point().durable; + journal::write_frame(&mut bytes, b"{\"FromTheFuture\":1}").unwrap(); + + let storage = SimStorage::with_contents(bytes.clone()); + let opened = + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())); + let message = format!("{:#}", opened.err().expect("load must fail")); + assert!( + message.contains("record 2 is not a known event"), + "{message}" + ); + assert_eq!(storage.point().durable, bytes, "storage must be untouched"); + assert!(storage.trace().is_empty()); + } + + #[test] + fn test_damaged_record_is_an_error_and_nothing_is_overwritten() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + record(&mut state, "b"); + record(&mut state, "c"); + let good = storage.point().durable; + + // Flip a bit in the middle record's payload. The third record is + // still intact after it. + let first_len = + journal::decode(&good).records[0].len() + journal::HEADER_LEN + journal::SUM_LEN; + let mut damaged = good.clone(); + damaged[first_len + journal::HEADER_LEN + 3] ^= 0x01; + + let storage = SimStorage::with_contents(damaged.clone()); + let opened = + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())); + let message = format!("{:#}", opened.err().expect("load must fail")); + assert!( + message.contains("record 2 fails its integrity check"), + "{message}" + ); + assert_eq!( + storage.point().durable, + damaged, + "storage must be untouched" + ); + assert!(storage.trace().is_empty()); + assert!(storage.set_aside_bytes().is_empty()); + } + + /// A length field changed so that the record seems to run past the end + /// of the file. Without a check on the length this looked like an + /// interrupted write: every record from there on was set aside and the + /// shell started with a shorter history, or an empty one. + #[test] + fn test_a_changed_length_is_damage_and_not_an_interrupted_write() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + record(&mut state, "b"); + record(&mut state, "c"); + let good = storage.point().durable; + let first_len = + journal::decode(&good).records[0].len() + journal::HEADER_LEN + journal::SUM_LEN; + + // The first record and the second, each with one high bit of its + // length set, so the declared end is far past the end of the file. + for (start, record_number) in [(0, 1), (first_len, 2)] { + let mut damaged = good.clone(); + damaged[start + journal::LENGTH_FIELD_LEN - 1] ^= 0x40; + + let storage = SimStorage::with_contents(damaged.clone()); + let opened = + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())); + let message = format!("{:#}", opened.err().expect("load must fail")); + assert!( + message.contains(&format!("record {record_number} fails its integrity check")), + "{message}" + ); + assert_eq!( + storage.point().durable, + damaged, + "storage must be untouched" + ); + assert!(storage.trace().is_empty()); + assert!(storage.set_aside_bytes().is_empty()); + } + } + + #[cfg(unix)] + #[test] + fn test_legacy_state_file_that_is_a_symlink_or_fifo_is_refused() { + let root = persisted_root(); + let legacy = root.path().join(LEGACY_STATE_FILE_NAME); + + // A symlink to a real state file elsewhere is not followed. + let elsewhere = root.path().join("elsewhere.json"); + fs::write(&elsewhere, LEGACY_STATE).unwrap(); + std::os::unix::fs::symlink(&elsewhere, &legacy).unwrap(); + assert!(ShellState::new(root.path().to_str().unwrap()).is_err()); + assert!(!journal_path(&root).exists()); + fs::remove_file(&legacy).unwrap(); + + // A FIFO is refused without waiting for a writer. + nix::unistd::mkfifo(&legacy, nix::sys::stat::Mode::S_IRWXU).unwrap(); + let root_path = root.path().to_str().unwrap().to_string(); + let (done, finished) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + let _ = done.send(ShellState::new(&root_path).is_err()); + }); + let refused = finished + .recv_timeout(std::time::Duration::from_secs(10)) + .expect("opening the shell must not block on a FIFO"); + assert!(refused); + } + + #[cfg(unix)] + #[test] + fn test_legacy_state_file_is_made_private_before_it_is_kept() { + use std::os::unix::fs::PermissionsExt; + let root = persisted_root(); + let legacy = root.path().join(LEGACY_STATE_FILE_NAME); + fs::write(&legacy, LEGACY_STATE).unwrap(); + fs::set_permissions(&legacy, fs::Permissions::from_mode(0o644)).unwrap(); + + reopen(&root); + let migrated = root.path().join(".vsh_state.json.migrated"); + let mode = fs::metadata(migrated).unwrap().permissions().mode() & 0o777; + assert_eq!(mode, 0o600); + } + + #[test] + fn test_newer_snapshot_version_is_refused() { + let root = persisted_root(); + let payload = format!( + "{{\"Snapshot\":{{\"version\":{},\"history\":[],\"transactions\":[],\"active_transaction\":null}}}}", + SNAPSHOT_VERSION + 1 + ); + let bytes = journal::frame(payload.as_bytes()).unwrap(); + let message = format!( + "{:#}", + sim_reopen(&root, bytes).err().expect("load must fail") + ); + assert!(message.contains("newer than this vsh"), "{message}"); + } + + /// `.vsh_state.json` exactly as `main` wrote it before the journal: + /// pretty-printed, no version, undo data as a number array. + const LEGACY_STATE: &str = r#"{ + "history": [ + { + "id": "11111111-1111-4111-8111-111111111111", + "op_type": "Mkdir", + "path": "project", + "timestamp": "2026-10-01T09:30:00.123456789Z", + "transaction_id": null, + "undo_data": null, + "undone": false, + "undone_by": null + }, + { + "id": "22222222-2222-4222-8222-222222222222", + "op_type": "DeleteFile", + "path": "project/notes.txt", + "timestamp": "2026-10-01T09:31:00Z", + "transaction_id": null, + "undo_data": [104, 105, 10], + "undone": true, + "undone_by": "33333333-3333-4333-8333-333333333333" + }, + { + "id": "44444444-4444-4444-8444-444444444444", + "op_type": "CreateFile", + "path": "project/draft.txt", + "timestamp": "2026-10-01T09:32:00Z", + "transaction_id": "55555555-5555-4555-8555-555555555555", + "undo_data": null, + "undone": false, + "undone_by": null + } + ], + "transactions": [ + { + "id": "66666666-6666-4666-8666-666666666666", + "name": "earlier", + "started": "2026-10-01T09:00:00Z", + "operations": [], + "committed": true + } + ], + "active_transaction": { + "id": "55555555-5555-4555-8555-555555555555", + "name": "draft", + "started": "2026-10-01T09:31:30Z", + "operations": ["44444444-4444-4444-8444-444444444444"], + "committed": false + }, + "previous_dir": "/home/someone" +}"#; + + #[test] + fn test_legacy_state_file_is_converted_and_kept() { + let root = persisted_root(); + let legacy = root.path().join(LEGACY_STATE_FILE_NAME); + fs::write(&legacy, LEGACY_STATE).unwrap(); + + let state = reopen(&root); + assert_eq!(state.history.len(), 3); + assert_eq!(state.history[0].op_type, OperationType::Mkdir); + assert_eq!(state.history[1].undo_data.as_deref(), Some(&b"hi\n"[..])); + assert!(state.history[1].undone); + assert_eq!(state.transactions.len(), 1); + let txn = state.active_transaction.as_ref().unwrap(); + assert_eq!(txn.name, "draft"); + assert_eq!(txn.operations, vec![state.history[2].id]); + assert_eq!( + state.previous_dir.as_deref(), + Some(Path::new("/home/someone")) + ); + + // The journal now holds the state; the old file is renamed, not lost. + assert!(!legacy.exists()); + let migrated = root.path().join(".vsh_state.json.migrated"); + assert_eq!(fs::read_to_string(migrated).unwrap(), LEGACY_STATE); + assert_eq!(journal_records(&root).len(), 1); + + let reloaded = reopen(&root); + assert_eq!(summary(&reloaded), summary(&state)); + assert_eq!( + reloaded.active_transaction.as_ref().unwrap().operations, + txn.operations + ); + } + + #[test] + fn test_an_earlier_backup_of_the_legacy_state_is_not_replaced() { + let root = persisted_root(); + let legacy = root.path().join(LEGACY_STATE_FILE_NAME); + fs::write(&legacy, LEGACY_STATE).unwrap(); + let earlier = root.path().join(".vsh_state.json.migrated"); + fs::write(&earlier, "an earlier backup").unwrap(); + let earlier_still = root.path().join(".vsh_state.json.migrated.1"); + fs::write(&earlier_still, "one before that").unwrap(); + + let state = reopen(&root); + assert_eq!(state.history.len(), 3); + + assert!(!legacy.exists()); + assert_eq!(fs::read_to_string(&earlier).unwrap(), "an earlier backup"); + assert_eq!( + fs::read_to_string(&earlier_still).unwrap(), + "one before that" + ); + let fresh = root.path().join(".vsh_state.json.migrated.2"); + assert_eq!(fs::read_to_string(fresh).unwrap(), LEGACY_STATE); + } + + #[test] + fn test_legacy_state_file_is_converted_beside_an_empty_journal() { + let root = persisted_root(); + fs::write(root.path().join(LEGACY_STATE_FILE_NAME), LEGACY_STATE).unwrap(); + fs::write(journal_path(&root), b"").unwrap(); + + let state = reopen(&root); + assert_eq!(state.history.len(), 3); + assert_eq!(journal_records(&root).len(), 1); + } + + #[test] + fn test_damaged_legacy_state_file_is_an_error_and_is_kept() { + let root = persisted_root(); + let legacy = root.path().join(LEGACY_STATE_FILE_NAME); + let damaged = &LEGACY_STATE[..LEGACY_STATE.len() / 2]; + fs::write(&legacy, damaged).unwrap(); + + assert!(ShellState::new(root.path().to_str().unwrap()).is_err()); + assert_eq!(fs::read_to_string(&legacy).unwrap(), damaged); + assert!(!journal_path(&root).exists()); + } + + #[test] + fn test_failed_append_is_repaired_by_the_next_persist() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + + storage.fail_append(true); + record(&mut state, "b"); // an error; the journal lacks "b" + assert!(state.rewrite_pending); + storage.fail_append(false); + + record(&mut state, "c"); + assert!(!state.rewrite_pending); + let reloaded = sim_reopen(&root, storage.point().durable).unwrap(); + assert_eq!(reloaded.history.len(), 3); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_append_torn_by_a_failure_is_repaired_by_the_next_persist() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + + // The write stops part of the way through the record. + storage.fail_append_after(Some(7)); + record(&mut state, "b"); + assert!(state.rewrite_pending); + assert_eq!(storage.point().pending.len(), 7); + storage.fail_append_after(None); + + record(&mut state, "c"); + assert!(!state.rewrite_pending); + let point = storage.point(); + assert!(point.pending.is_empty()); + let reloaded = sim_reopen(&root, point.durable).unwrap(); + assert_eq!(reloaded.history.len(), 3); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_failed_sync_is_repaired_by_the_next_persist() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + + storage.fail_sync(true); + record(&mut state, "b"); // appended, never made durable + assert!(state.rewrite_pending); + storage.fail_sync(false); + + state + .mark_undone(state.history[0].id, Uuid::new_v4()) + .unwrap(); + assert!(!state.rewrite_pending); + let point = storage.point(); + assert!(point.pending.is_empty()); + let reloaded = sim_reopen(&root, point.durable).unwrap(); + assert_eq!(summary(&reloaded), summary(&state)); + } + + #[test] + fn test_failed_rewrite_is_retried() { + let root = persisted_root(); + let (mut state, storage) = sim_state(&root); + record(&mut state, "a"); + + storage.fail_replace(true); + assert!(state.compact_journal().is_err()); + assert!(state.rewrite_pending); + storage.fail_replace(false); + + record(&mut state, "b"); + assert!(!state.rewrite_pending); + let reloaded = sim_reopen(&root, storage.point().durable).unwrap(); + assert_eq!(summary(&reloaded), summary(&state)); + } + + /// `last_n_undoable` must agree with a plain newest-first scan for + /// non-undone operations, whatever the undone-block hint holds. + fn assert_undoable_matches_scan(state: &ShellState) { + let naive: Vec = state + .history + .iter() + .rev() + .filter(|op| !op.undone) + .map(|op| op.id) + .collect(); + let fast: Vec = state + .last_n_undoable(usize::MAX) + .iter() + .map(|op| op.id) + .collect(); + assert_eq!(fast, naive); + } + + fn undo_one(state: &mut ShellState) { + let id = state.last_n_undoable(1)[0].id; + state.mark_undone(id, Uuid::new_v4()).unwrap(); + } + + #[test] + fn test_undone_block_hint_agrees_with_full_scan() { + let root = persisted_root(); + let mut state = reopen(&root); + state.max_history_size = Some(12); + for i in 0..10 { + record(&mut state, &format!("a{i}")); + } + for _ in 0..3 { + undo_one(&mut state); + assert_undoable_matches_scan(&state); + } + // New operations after an undone block, then undo across it. + for i in 0..4 { + record(&mut state, &format!("b{i}")); + assert_undoable_matches_scan(&state); + } + for _ in 0..6 { + undo_one(&mut state); + assert_undoable_matches_scan(&state); + } + // Out-of-order mark of an older operation. + let oldest_live = state.last_n_undoable(usize::MAX).last().unwrap().id; + state.mark_undone(oldest_live, Uuid::new_v4()).unwrap(); + assert_undoable_matches_scan(&state); + // Trims from the front shift the block. + for i in 0..8 { + record(&mut state, &format!("c{i}")); + assert_undoable_matches_scan(&state); + } + while !state.last_n_undoable(1).is_empty() { + undo_one(&mut state); + assert_undoable_matches_scan(&state); + } + } + + /// After a restart the hint covers the run of undone operations at the + /// end of the history, so single-step undo does not rescan that run + /// every time. The same after a rollback, which marks in bulk. + #[test] + fn test_undone_block_hint_is_rebuilt_after_reload_and_rollback() { + let root = persisted_root(); + let mut state = reopen(&root); + for i in 0..6 { + record(&mut state, &format!("a{i}")); + } + for _ in 0..3 { + undo_one(&mut state); + } + + let mut reloaded = reopen(&root); + assert_eq!(reloaded.checked_undone_run(), 3..6); + assert_undoable_matches_scan(&reloaded); + undo_one(&mut reloaded); + assert_eq!(reloaded.checked_undone_run(), 2..6); + assert_undoable_matches_scan(&reloaded); + + // A history that ends in a live operation has no such run. + record(&mut reloaded, "b"); + let mut again = reopen(&root); + assert_eq!(again.checked_undone_run(), 0..0); + assert_undoable_matches_scan(&again); + + // A rollback marks its operations in one go; they are the newest. + again.begin_transaction("t").unwrap(); + record(&mut again, "c0"); + record(&mut again, "c1"); + again.abort_transaction(); + let len = again.history.len(); + assert_eq!(again.checked_undone_run(), len - 2..len); + assert_undoable_matches_scan(&again); + } + + #[test] + fn test_transaction_membership_survives_journal_replay() { + let root = persisted_root(); + let mut state = reopen(&root); + state.begin_transaction("t").unwrap(); + let a = record(&mut state, "a"); + + let reloaded = reopen(&root); + let txn = reloaded.active_transaction.as_ref().unwrap(); + assert_eq!(txn.operations, vec![a]); + } } diff --git a/impl/rust-cli/tests/cd_previous_dir_tests.rs b/impl/rust-cli/tests/cd_previous_dir_tests.rs new file mode 100644 index 00000000..771af768 --- /dev/null +++ b/impl/rust-cli/tests/cd_previous_dir_tests.rs @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! A `cd` that fails must not change what `cd -` returns to. +//! +//! `cd` used to record the directory it was leaving before it tried to +//! change, so a change that failed after its checks (here: no search +//! permission) still set the previous directory, and that was written to the +//! journal and survived a restart. +//! +//! The test runs the built shell. Root may enter any directory, so when the +//! test itself runs as root the shell is started as the unprivileged user +//! `nobody`. + +#![cfg(unix)] + +use std::fs::{self, Permissions}; +use std::os::unix::fs::PermissionsExt; +use std::os::unix::process::CommandExt; +use std::path::Path; +use std::process::{Command, Output, Stdio}; + +const NOBODY: u32 = 65534; + +fn vsh(root: &Path, script: &str) -> Output { + let built = Path::new(env!("CARGO_BIN_EXE_vsh")); + let as_root = nix::unistd::geteuid().is_root(); + + // As root the shell is started as another user, who may not be able to + // reach the build directory (a checkout under a private home directory, + // for one). Give that user a copy it can run. The directory is removed + // when this function returns, after the shell has finished. + let staged = as_root.then(|| { + let dir = tempfile::tempdir().expect("directory for a copy of the shell"); + fs::set_permissions(dir.path(), Permissions::from_mode(0o755)).expect("open it"); + let copy = dir.path().join("vsh"); + fs::copy(built, ©).expect("copy the built shell"); + fs::set_permissions(©, Permissions::from_mode(0o755)).expect("make it runnable"); + (dir, copy) + }); + let program = staged.as_ref().map_or(built, |(_, copy)| copy.as_path()); + + let mut command = Command::new(program); + command + .arg("-r") + .arg(root) + .arg("-c") + .arg(script) + .current_dir(root) + .env("HOME", root) + .env_remove("RUST_BACKTRACE") + .stdin(Stdio::null()); + if as_root { + command.uid(NOBODY).gid(NOBODY); + } + command.output().expect("start the built shell") +} + +#[test] +fn a_cd_that_fails_does_not_record_a_previous_directory() { + let root = tempfile::tempdir().expect("temporary directory"); + // The shell may run as another user, so it must be able to write here. + fs::set_permissions(root.path(), Permissions::from_mode(0o777)).expect("open the root"); + let locked = root.path().join("locked"); + fs::create_dir(&locked).expect("create the directory"); + fs::set_permissions(&locked, Permissions::from_mode(0o000)).expect("lock the directory"); + + let failed = vsh(root.path(), "cd locked"); + let back = vsh(root.path(), "cd -"); + + // Let the temporary directory be removed whatever the assertions find. + fs::set_permissions(&locked, Permissions::from_mode(0o755)).expect("unlock the directory"); + + assert!( + !failed.status.success(), + "cd into a directory without search permission should fail; stderr: {}", + String::from_utf8_lossy(&failed.stderr) + ); + let stderr = String::from_utf8_lossy(&back.stderr); + assert!( + !back.status.success() && stderr.contains("OLDPWD not set"), + "after a failed cd there is no previous directory to return to; \ + status {:?}, stderr: {stderr}", + back.status.code() + ); +} + +#[test] +fn a_cd_that_succeeds_still_records_where_it_came_from() { + let root = tempfile::tempdir().expect("temporary directory"); + fs::set_permissions(root.path(), Permissions::from_mode(0o777)).expect("open the root"); + fs::create_dir(root.path().join("inner")).expect("create the directory"); + fs::set_permissions(root.path().join("inner"), Permissions::from_mode(0o777)) + .expect("open the directory"); + + let run = vsh(root.path(), "cd inner; cd -; pwd"); + + assert!( + run.status.success(), + "stderr: {}", + String::from_utf8_lossy(&run.stderr) + ); + let stdout = String::from_utf8_lossy(&run.stdout); + let last = stdout.lines().last().unwrap_or(""); + let expected = fs::canonicalize(root.path()).expect("canonical root"); + assert_eq!( + Path::new(last), + expected, + "`cd -` should return to the root" + ); +} diff --git a/impl/rust-cli/tests/journal_crash_tests.rs b/impl/rust-cli/tests/journal_crash_tests.rs new file mode 100644 index 00000000..c322227d --- /dev/null +++ b/impl/rust-cli/tests/journal_crash_tests.rs @@ -0,0 +1,442 @@ +// SPDX-License-Identifier: MPL-2.0 +//! Crash-point tests for shell-state persistence. +//! +//! Two tests, from two directions. +//! +//! # Black box, on real files +//! +//! [`crash_at_every_point_recovers_a_consistent_state`] knows only that state +//! lives in files named `.vsh_state*` in the session root, so it applies +//! unchanged to any on-disk format. +//! +//! 1. Run a scripted session. After every acknowledged mutation, capture the +//! bytes of every state file and the observable state. +//! 2. For each consecutive pair of captures, enumerate the disk states a +//! crash could leave: +//! * *clean*: every file old, or every file new; +//! * *torn append*: a file that grew by appending, cut at every byte +//! offset between its old and new length; +//! * *torn create*: a file that did not exist before, cut at every length; +//! * *mixed files*: files that changed independently at old or new, in +//! every combination; +//! * *stray temp*: the old state plus a half-written temporary copy of a +//! rewritten file. +//! 3. For each crash state, reopen the shell and require: +//! * **opens**: `ShellState::new` succeeds; +//! * **atomic**: the recovered state equals the state before or after the +//! interrupted mutation, never anything else; +//! * **stable**: after recording one more operation and reopening, the +//! state is the recovered state plus that operation. +//! +//! # Model based, on simulated storage +//! +//! [`crash_at_every_storage_point_recovers_before_or_after`] runs the same +//! session on [`SimStorage`], which keeps synced bytes apart from unsynced +//! ones exactly as the Agda model's file does +//! (`proofs/agda/JournalModel.agda`). After every storage call it takes each +//! content the model says a crash could leave (the synced bytes and every +//! prefix of the unsynced ones) and requires the same three properties, plus +//! **durable**: once a mutation is acknowledged, every crash recovers the +//! state after it. +//! +//! The model's theorems (`crash-recovery`, `write-atomic`, +//! `compaction-safe`) say these properties hold for the model. This test is +//! evidence, not proof, that the Rust code behaves like it. +//! +//! No failure class is tolerated. The two-file format that preceded this +//! one failed two of them (measured 2026-11-17: a journal truncated while the +//! old snapshot was still in place, and a first snapshot cut short). + +mod fixtures; + +use fixtures::tempfile::TempDir; +use std::collections::{BTreeMap, BTreeSet}; +use std::fs; +use std::path::{Path, PathBuf}; +use uuid::Uuid; +use vsh::journal::SimStorage; +use vsh::state::{Operation, OperationType, ShellState}; + +type Files = BTreeMap>; + +/// What a user can observe of the persisted state. +#[derive(Clone, Debug, PartialEq, Eq)] +struct Observed { + history: Vec<(Uuid, bool, Option)>, + /// The open transaction and the operations in it. + active: Option<(Uuid, Vec)>, + /// Committed transactions. + committed: Vec, + previous_dir: Option, +} + +fn observe(state: &ShellState) -> Observed { + Observed { + history: state + .history + .iter() + .map(|op| (op.id, op.undone, op.undone_by)) + .collect(), + active: state + .active_transaction + .as_ref() + .map(|txn| (txn.id, txn.operations.clone())), + committed: state.transactions.iter().map(|txn| txn.id).collect(), + previous_dir: state.previous_dir.clone(), + } +} + +fn open(root: &Path) -> anyhow::Result { + ShellState::new(root.to_str().unwrap()) +} + +fn state_files(root: &Path) -> Files { + let mut files = Files::new(); + for entry in fs::read_dir(root).unwrap() { + let entry = entry.unwrap(); + let name = entry.file_name().to_string_lossy().into_owned(); + if name.starts_with(".vsh_state") && entry.path().is_file() { + files.insert(name, fs::read(entry.path()).unwrap()); + } + } + files +} + +fn materialise(files: &Files) -> TempDir { + let dir = TempDir::new().unwrap(); + for (name, bytes) in files { + fs::write(dir.path().join(name), bytes).unwrap(); + } + dir +} + +fn record(state: &mut ShellState, path: &str) -> Uuid { + let op = Operation::new(OperationType::CreateFile, path.to_string(), None); + let id = op.id; + state.record_operation(op).unwrap(); + id +} + +/// A scripted session touching every kind of persisted mutation: plain +/// records, an undo mark, a transaction that commits, one that is rolled +/// back with operations in it, a journal rewrite, the previous directory, +/// and records that trim the history at its size limit. `after` is called +/// once at the start and once after each acknowledged step. +fn run_session(state: &mut ShellState, mut after: impl FnMut(&'static str, &ShellState)) { + after("start", state); + record(state, "a"); + after("record a", state); + let b = record(state, "b-with-non-ascii-é"); + after("record b", state); + state.mark_undone(b, Uuid::new_v4()).unwrap(); + after("undo b", state); + state.begin_transaction("t1").unwrap(); + after("begin t1", state); + record(state, "c"); + after("record c in t1", state); + state.commit_transaction().unwrap(); + after("commit t1", state); + record(state, "d"); + after("record d", state); + state.begin_transaction("t2").unwrap(); + after("begin t2", state); + record(state, "e"); + after("record e in t2", state); + state.abort_transaction(); + after("rollback t2", state); + state.compact_journal().unwrap(); + after("rewrite journal", state); + record(state, "f"); + after("record f", state); + state.set_previous_dir(PathBuf::from("/somewhere/else")); + after("set previous dir", state); + state.max_history_size = Some(state.history.len()); + record(state, "g"); + after("record g with trim", state); + state.begin_transaction("t3").unwrap(); + after("begin t3", state); + state.compact_journal().unwrap(); + after("rewrite journal in t3", state); + record(state, "h"); + after("record h in t3 with trim", state); +} + +/// One acknowledged point of the session: what is on disk and what the user +/// was told. +struct Capture { + label: &'static str, + files: Files, + observed: Observed, +} + +fn run_file_session() -> Vec { + let root = TempDir::new().unwrap(); + let mut state = open(root.path()).unwrap(); + let mut captures = Vec::new(); + run_session(&mut state, |label, state| { + captures.push(Capture { + label, + files: state_files(root.path()), + observed: observe(state), + }); + }); + captures +} + +/// Every disk state a crash between `old` and `new` could leave, tagged with +/// its kind. +fn crash_states(old: &Files, new: &Files) -> Vec<(&'static str, Files)> { + let names: BTreeSet<&String> = old.keys().chain(new.keys()).collect(); + let changed: Vec<&String> = names + .into_iter() + .filter(|n| old.get(*n) != new.get(*n)) + .collect(); + + let mut states = vec![("clean", old.clone()), ("clean", new.clone())]; + for name in &changed { + states.extend(torn_states(old, new, name)); + states.extend(stray_temp_state(old, new, name)); + } + states.extend(mixed_file_states(old, new, &changed)); + states +} + +/// `old` with one file replaced by `bytes`. +fn with_file(old: &Files, name: &str, bytes: &[u8]) -> Files { + let mut files = old.clone(); + files.insert(name.to_string(), bytes.to_vec()); + files +} + +/// One file cut short, the rest old. +/// +/// * *Torn append*: the file grew by appending; cut between its old and new +/// length. +/// * *Torn create*: the file did not exist; cut at every length. A process +/// crash leaves this if the file is created by writing in place; if it is +/// created by rename, only a power loss before the data was synced can. +fn torn_states(old: &Files, new: &Files, name: &str) -> Vec<(&'static str, Files)> { + let Some(after) = new.get(name) else { + return Vec::new(); + }; + let (kind, lengths) = match old.get(name) { + None => ("torn-create", 0..after.len()), + Some(before) if after.len() > before.len() && after.starts_with(before) => { + ("torn-append", before.len() + 1..after.len()) + } + Some(_) => return Vec::new(), + }; + lengths + .map(|len| (kind, with_file(old, name, &after[..len]))) + .collect() +} + +/// Each changed file independently old or new: what a crash between two +/// writes, or a power loss without `fsync` ordering, can leave. +fn mixed_file_states(old: &Files, new: &Files, changed: &[&String]) -> Vec<(&'static str, Files)> { + if changed.len() < 2 { + return Vec::new(); + } + (1..(1u32 << changed.len()) - 1) + .map(|mask| { + let mut files = old.clone(); + for (bit, name) in changed.iter().enumerate() { + if mask & (1 << bit) == 0 { + continue; + } + match new.get(*name) { + Some(bytes) => files.insert((*name).clone(), bytes.clone()), + None => files.remove(*name), + }; + } + ("mixed-files", files) + }) + .collect() +} + +/// A rewritten (not appended) file half-copied beside the old state, as a +/// crash during write-then-rename leaves it. +fn stray_temp_state(old: &Files, new: &Files, name: &str) -> Option<(&'static str, Files)> { + let after = new.get(name)?; + let appended = old + .get(name) + .is_some_and(|before| after.starts_with(before)); + if appended || after.is_empty() { + return None; + } + let temp = format!("{name}.0123456789abcdef.tmp"); + Some(( + "stray-temp", + with_file(old, &temp, &after[..after.len() / 2]), + )) +} + +/// Check the three properties for one crash state. Returns the property that +/// failed, if any. +fn check(files: &Files, before: &Observed, after: &Observed) -> Option<&'static str> { + let dir = materialise(files); + let Ok(mut state) = open(dir.path()) else { + return Some("opens"); + }; + let recovered = observe(&state); + if recovered != *before && recovered != *after { + return Some("atomic"); + } + + // The oracle is built from the recovered state alone, so a regression + // that rewrites an existing entry while recording a new one is caught. + let extra = record(&mut state, "after-crash"); + drop(state); + let expected = plus_operation(recovered, extra); + match open(dir.path()) { + Ok(reopened) if observe(&reopened) == expected => {} + _ => return Some("stable"), + } + None +} + +/// `observed` after one more operation has been recorded. +fn plus_operation(mut observed: Observed, id: Uuid) -> Observed { + observed.history.push((id, false, None)); + if let Some((_, operations)) = observed.active.as_mut() { + operations.push(id); + } + observed +} + +#[test] +fn crash_at_every_point_recovers_a_consistent_state() { + let captures = run_file_session(); + let mut failures: BTreeMap> = BTreeMap::new(); + let mut checked = 0usize; + + for pair in captures.windows(2) { + let (old, new) = (&pair[0], &pair[1]); + for (kind, files) in crash_states(&old.files, &new.files) { + checked += 1; + if let Some(property) = check(&files, &old.observed, &new.observed) { + let sizes: Vec = files + .iter() + .map(|(name, bytes)| format!("{name}={}", bytes.len())) + .collect(); + failures + .entry(format!("{kind}/{property}")) + .or_default() + .push(format!("during '{}': {}", new.label, sizes.join(" "))); + } + } + } + + println!("checked {checked} crash states"); + for (class, cases) in &failures { + let steps: BTreeSet<&str> = cases.iter().filter_map(|c| c.split('\'').nth(1)).collect(); + println!( + "FAIL {class}: {} case(s) during {steps:?}; first: {}", + cases.len(), + cases[0] + ); + } + + assert!( + failures.is_empty(), + "crash states that did not recover: see the FAIL lines above" + ); +} + +/// Open a shell on simulated storage holding `bytes`, as after a restart. +fn open_sim(root: &Path, bytes: Vec) -> anyhow::Result<(ShellState, SimStorage)> { + let storage = SimStorage::with_contents(bytes); + let state = ShellState::with_storage(root.to_str().unwrap(), Box::new(storage.clone()))?; + Ok((state, storage)) +} + +/// Check one crash image against the model's properties. Returns what +/// failed, if anything. +fn check_image( + root: &Path, + image: &[u8], + before: &Observed, + after: &Observed, + acknowledged: bool, +) -> Option<&'static str> { + let Ok((mut state, storage)) = open_sim(root, image.to_vec()) else { + return Some("opens"); + }; + let recovered = observe(&state); + if acknowledged && recovered != *after { + return Some("durable"); + } + if recovered != *before && recovered != *after { + return Some("atomic"); + } + + let extra = record(&mut state, "after-crash"); + drop(state); + let point = storage.point(); + if !point.pending.is_empty() { + return Some("stable (the record was not synced)"); + } + match open_sim(root, point.durable) { + Ok((reopened, _)) if observe(&reopened) == plus_operation(recovered, extra) => None, + _ => Some("stable"), + } +} + +#[test] +fn crash_at_every_storage_point_recovers_before_or_after() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())).unwrap(); + + // After each step: its label, what was observed, and how many storage + // calls had been made by then. + let mut steps: Vec<(&'static str, Observed, usize)> = Vec::new(); + run_session(&mut state, |label, state| { + steps.push((label, observe(state), storage.trace().len())); + }); + let trace = storage.trace(); + + let mut checked = 0usize; + let mut failures: Vec = Vec::new(); + for pair in steps.windows(2) { + let (_, before, first_call) = &pair[0]; + let (label, after, end_call) = &pair[1]; + assert!(end_call > first_call, "step '{label}' wrote nothing"); + for (call, point) in trace.iter().enumerate().take(*end_call).skip(*first_call) { + // The step is acknowledged once its last storage call returns. + let acknowledged = call + 1 == *end_call; + if acknowledged { + assert!( + point.pending.is_empty(), + "step '{label}' was acknowledged with unsynced bytes" + ); + } + for image in point.crash_images() { + checked += 1; + if let Some(property) = + check_image(root.path(), &image, before, after, acknowledged) + { + failures.push(format!( + "{property}: during '{label}', storage call {call}, {} of {} bytes", + image.len(), + point.durable.len() + point.pending.len() + )); + } + } + } + } + + println!( + "checked {checked} crash images over {} storage calls", + trace.len() + ); + for failure in failures.iter().take(20) { + println!("FAIL {failure}"); + } + assert!( + failures.is_empty(), + "{} crash images did not recover", + failures.len() + ); +} diff --git a/impl/rust-cli/tests/rollback_tests.rs b/impl/rust-cli/tests/rollback_tests.rs new file mode 100644 index 00000000..036d98a4 --- /dev/null +++ b/impl/rust-cli/tests/rollback_tests.rs @@ -0,0 +1,582 @@ +// SPDX-License-Identifier: MPL-2.0 +//! Transaction rollback: what it reverses, what it marks, and when it +//! refuses. +//! +//! The rules under test (owner rulings 2026-11-17, route plan section 14): +//! +//! * Rollback marks each operation it reverses as undone, so a later `undo` +//! does not try to reverse it again, and `redo` can bring it back. +//! * Before reversing anything, rollback checks every operation's record. +//! If one cannot be reversed, the rollback is refused and nothing changes. +//! * If a reversal fails on the disk, the transaction stays open, and the +//! rollback can be run again. + +mod fixtures; + +use fixtures::tempfile::TempDir; +use std::fs; +use uuid::Uuid; +use vsh::commands; +use vsh::journal::SimStorage; +use vsh::state::{Operation, OperationType, ShellState}; + +fn open(root: &TempDir) -> ShellState { + ShellState::new(root.path().to_str().unwrap()).unwrap() +} + +fn live(state: &ShellState) -> Vec { + state + .history + .iter() + .filter(|op| !op.undone) + .map(|op| op.id) + .collect() +} + +#[test] +fn rollback_reverses_and_marks_every_operation() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::touch(&mut state, "kept", false).unwrap(); + let kept = state.history[0].id; + + commands::begin_transaction(&mut state, "t").unwrap(); + let txn = state.active_transaction.as_ref().unwrap().id; + commands::mkdir(&mut state, "dir", false).unwrap(); + commands::touch(&mut state, "dir/file", false).unwrap(); + commands::rollback_transaction(&mut state).unwrap(); + + assert!(!root.path().join("dir").exists()); + assert!(root.path().join("kept").exists()); + assert!(state.active_transaction.is_none()); + assert_eq!(live(&state), vec![kept]); + for op in &state.history[1..] { + assert_eq!(op.undone_by, Some(txn)); + } + + // The reported bug: `undo` used to pick the rolled-back `touch` and fail + // with "No such file or directory". It now undoes the operation before + // the transaction. + commands::undo(&mut state, 1, false).unwrap(); + assert!(!root.path().join("kept").exists()); + + // The same after a restart. + let reloaded = open(&root); + assert!(reloaded.active_transaction.is_none()); + assert!(live(&reloaded).is_empty()); +} + +#[test] +fn redo_brings_rolled_back_operations_back_in_order() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::mkdir(&mut state, "dir", false).unwrap(); + commands::touch(&mut state, "dir/file", false).unwrap(); + commands::rollback_transaction(&mut state).unwrap(); + assert!(!root.path().join("dir").exists()); + + commands::redo(&mut state, 1, false).unwrap(); + assert!(root.path().join("dir").is_dir()); + assert!(!root.path().join("dir/file").exists()); + commands::redo(&mut state, 1, false).unwrap(); + assert!(root.path().join("dir/file").is_file()); +} + +#[test] +fn rollback_is_refused_whole_when_an_operation_has_no_inverse() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + // An irreversible operation inside the transaction. + state + .record_operation(Operation::new( + OperationType::Obliterate, + "gone".to_string(), + None, + )) + .unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + let before = live(&state); + + let refused = commands::rollback_transaction(&mut state).unwrap_err(); + assert!(refused.to_string().contains("refused"), "{refused}"); + + // Nothing was reversed, nothing was marked, and the transaction is open. + assert!(root.path().join("a").exists()); + assert!(root.path().join("b").exists()); + assert_eq!(live(&state), before); + assert!(state.active_transaction.is_some()); + + // Commit is the way out. + commands::commit_transaction(&mut state).unwrap(); + assert!(state.active_transaction.is_none()); + assert_eq!(live(&state), before); +} + +#[test] +fn rollback_is_refused_when_undo_data_is_missing() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + // A chmod record without the previous mode cannot be reversed. Before + // the check, this record made rollback panic on a short slice. + state + .record_operation( + Operation::new(OperationType::Chmod, "a".to_string(), None).with_undo_data(vec![1, 2]), + ) + .unwrap(); + + assert!(commands::rollback_transaction(&mut state).is_err()); + assert!(root.path().join("a").exists()); + assert!(state.active_transaction.is_some()); +} + +#[test] +fn a_reversal_that_fails_on_disk_leaves_the_transaction_open_to_retry() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + let (a, b) = (state.history[0].id, state.history[1].id); + + // Something outside the shell removes `a`, so its reversal will fail. + fs::remove_file(root.path().join("a")).unwrap(); + assert!(commands::rollback_transaction(&mut state).is_err()); + + // `b` was reversed and is marked; `a` is still live; the transaction + // is still open. + assert!(!root.path().join("b").exists()); + assert_eq!(live(&state), vec![a]); + assert!(state.history.iter().any(|op| op.id == b && op.undone)); + assert!(state.active_transaction.is_some()); + + // The same after a restart: the rollback is resumable. + drop(state); + let mut state = open(&root); + assert!(state.active_transaction.is_some()); + assert_eq!(live(&state), vec![a]); + + // Once the cause is fixed, a second rollback finishes the job without + // touching `b` again. + fs::write(root.path().join("a"), b"").unwrap(); + commands::rollback_transaction(&mut state).unwrap(); + assert!(!root.path().join("a").exists()); + assert!(state.active_transaction.is_none()); + assert!(live(&state).is_empty()); +} + +/// After a failed reversal, rollback stops. Older operations may depend on +/// the one that failed, so they are not reversed out of order. +#[test] +fn rollback_stops_at_the_first_failed_reversal() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "older", false).unwrap(); + commands::touch(&mut state, "newer", false).unwrap(); + let older = state.history[0].id; + + // The newest operation is reversed first, and its reversal will fail. + fs::remove_file(root.path().join("newer")).unwrap(); + assert!(commands::rollback_transaction(&mut state).is_err()); + + // The older operation was not touched. + assert!(root.path().join("older").exists()); + assert!(live(&state).contains(&older)); + assert!(state.active_transaction.is_some()); +} + +/// The history size limit does not drop the operations of an open +/// transaction: rollback has to reverse every one of them. Before, a +/// transaction longer than the limit could not be rolled back at all. +#[test] +fn an_open_transaction_keeps_its_operations_past_the_history_limit() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + state.max_history_size = Some(1); + commands::touch(&mut state, "before", false).unwrap(); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + commands::touch(&mut state, "c", false).unwrap(); + assert_eq!( + state.history.len(), + 3, + "the operation before the transaction was dropped; its own were kept" + ); + + commands::rollback_transaction(&mut state).unwrap(); + for name in ["a", "b", "c"] { + assert!(!root.path().join(name).exists(), "{name} was reversed"); + } + assert!(root.path().join("before").exists()); + assert!(state.active_transaction.is_none()); + + // With the transaction closed, the next operation brings the history + // back under the limit, and a restart agrees with that. + commands::touch(&mut state, "after", false).unwrap(); + assert_eq!(state.history.len(), 1); + let reloaded = open(&root); + assert_eq!(reloaded.history.len(), 1); + assert_eq!(reloaded.history[0].id, state.history[0].id); +} + +/// The same after a restart: the kept operations are in the journal, and the +/// reloaded transaction can still be rolled back. +#[test] +fn a_transaction_longer_than_the_limit_can_be_rolled_back_after_a_restart() { + let root = TempDir::new().unwrap(); + { + let mut state = open(&root); + state.max_history_size = Some(2); + commands::begin_transaction(&mut state, "t").unwrap(); + for name in ["a", "b", "c", "d"] { + commands::touch(&mut state, name, false).unwrap(); + } + assert_eq!(state.history.len(), 4); + } + + let mut state = open(&root); + state.max_history_size = Some(2); + assert_eq!(state.history.len(), 4); + assert_eq!( + state.active_transaction.as_ref().unwrap().operations.len(), + 4 + ); + commands::rollback_transaction(&mut state).unwrap(); + for name in ["a", "b", "c", "d"] { + assert!(!root.path().join(name).exists(), "{name} was reversed"); + } + assert!(state.active_transaction.is_none()); +} + +/// An operation undone inside the transaction needs no reversal, so it never +/// stands in the way of a rollback, whatever the history limit is. +#[test] +fn an_operation_undone_inside_a_transaction_does_not_block_its_rollback() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + state.max_history_size = Some(2); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + commands::undo(&mut state, 1, false).unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + commands::touch(&mut state, "c", false).unwrap(); + + commands::rollback_transaction(&mut state).unwrap(); + for name in ["a", "b", "c"] { + assert!(!root.path().join(name).exists(), "{name} is gone"); + } + assert!(state.active_transaction.is_none()); +} + +/// A saved state from somewhere else may still lack an operation of its open +/// transaction. What is left cannot stand for the whole transaction, so +/// rollback is refused and nothing is changed. +#[test] +fn rollback_is_refused_when_an_operation_of_the_transaction_is_missing() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + // This version never does this itself. + state.history.remove(0); + + let refused = commands::rollback_transaction(&mut state).unwrap_err(); + assert!( + refused.to_string().contains("no longer in the history"), + "{refused}" + ); + assert!(root.path().join("a").exists()); + assert!(root.path().join("b").exists()); + assert!(state.active_transaction.is_some()); + + commands::commit_transaction(&mut state).unwrap(); + assert!(state.active_transaction.is_none()); +} + +/// A redo that fails has redone nothing, so its entry stays on the redo +/// stack and the redo can be tried again once the cause is fixed. Before, +/// the entry was dropped and the operation could not be redone in that +/// session. +#[test] +fn a_redo_that_fails_keeps_its_entry_to_try_again() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + let file = root.path().join("notes"); + fs::write(&file, "contents\n").unwrap(); + commands::rm(&mut state, "notes", false).unwrap(); + commands::undo(&mut state, 1, false).unwrap(); + let entry = state.get_redo_stack().front().unwrap().id; + + // The redo has to read the file first. Put a directory in its place, so + // that the read fails whoever runs the test. + fs::remove_file(&file).unwrap(); + fs::create_dir(&file).unwrap(); + let failed = commands::redo(&mut state, 1, false).unwrap_err(); + assert!(failed.to_string().contains("could not read"), "{failed}"); + assert_eq!(state.get_redo_stack().len(), 1); + assert_eq!(state.get_redo_stack().front().unwrap().id, entry); + let records = state.history.len(); + + // With the file back, the same redo goes through. + fs::remove_dir(&file).unwrap(); + fs::write(&file, "contents\n").unwrap(); + commands::redo(&mut state, 1, false).unwrap(); + assert!(!file.exists()); + assert!(state.get_redo_stack().is_empty()); + assert_eq!(state.history.len(), records + 1); +} + +/// An entry of a kind that can never be redone is dropped when it fails, so +/// that it does not block the entries behind it. +#[test] +fn an_entry_that_can_never_be_redone_does_not_block_the_redo_stack() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::touch(&mut state, "a", false).unwrap(); + let mut appended = Operation::new(OperationType::FileAppended, "a".to_string(), None); + appended.undo_data = Some(0u64.to_le_bytes().to_vec()); + state.record_operation(appended).unwrap(); + commands::undo(&mut state, 2, false).unwrap(); + assert_eq!(state.get_redo_stack().len(), 2); + + // The entries come back oldest first: `touch a`, then the append. + commands::redo(&mut state, 1, false).unwrap(); + assert!(root.path().join("a").exists()); + assert!(commands::redo(&mut state, 1, false).is_err()); + assert!(state.get_redo_stack().is_empty()); +} + +/// A state whose saved form goes to simulated storage, which can be made to +/// refuse writes. +fn open_on(root: &TempDir, storage: &SimStorage) -> ShellState { + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())).unwrap() +} + +/// A reversal that is done on the disk but cannot be saved stops the +/// rollback with an error, so the saved state is never more than one +/// operation behind the disk. Before, rollback went on and reported success. +#[test] +fn rollback_stops_when_a_reversal_cannot_be_saved() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = open_on(&root, &storage); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + assert!(state.is_saved()); + + // The storage stops taking writes. + storage.fail_append(true); + storage.fail_replace(true); + let stopped = commands::rollback_transaction(&mut state).unwrap_err(); + assert!( + stopped.to_string().contains("could not be written"), + "{stopped}" + ); + assert!(!root.path().join("b").exists(), "the newest was reversed"); + assert!(root.path().join("a").exists(), "the next was not attempted"); + assert!(state.active_transaction.is_some()); + assert!(!state.is_saved()); + + // Writes work again. The rollback finishes and the saved state agrees. + storage.fail_append(false); + storage.fail_replace(false); + commands::rollback_transaction(&mut state).unwrap(); + assert!(!root.path().join("a").exists()); + assert!(state.active_transaction.is_none()); + assert!(state.is_saved()); + + let reloaded = open_on(&root, &storage); + assert!(reloaded.active_transaction.is_none()); + assert!(live(&reloaded).is_empty()); +} + +/// Closing the transaction is a write too. If it cannot be saved, rollback +/// says so; the next write that succeeds saves the whole state. +#[test] +fn rollback_reports_a_close_that_cannot_be_saved() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = open_on(&root, &storage); + commands::begin_transaction(&mut state, "t").unwrap(); + + storage.fail_append(true); + storage.fail_replace(true); + let unsaved = commands::rollback_transaction(&mut state).unwrap_err(); + assert!( + unsaved.to_string().contains("could not be written"), + "{unsaved}" + ); + assert!(state.active_transaction.is_none()); + assert!(!state.is_saved()); + // A restart now would still show the transaction as open. + assert!(open_on(&root, &storage).active_transaction.is_some()); + + storage.fail_append(false); + storage.fail_replace(false); + commands::touch(&mut state, "later", false).unwrap(); + assert!(state.is_saved()); + assert!(open_on(&root, &storage).active_transaction.is_none()); +} + +/// Redo reads a file to keep its contents before it removes or truncates +/// it. Like `rm` and a truncating redirection, it must not change the +/// file's access time by doing so. +#[cfg(target_os = "linux")] +#[test] +fn redoing_a_truncation_does_not_change_the_access_time() { + use std::time::{Duration, SystemTime}; + + /// An access time long ago, so that a plain read would update it under + /// the usual `relatime` mount option. + fn set_old_access_time(path: &std::path::Path) -> SystemTime { + let old = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000_000); + let file = fs::OpenOptions::new().write(true).open(path).unwrap(); + file.set_times(fs::FileTimes::new().set_accessed(old)) + .unwrap(); + old + } + + let root = TempDir::new().unwrap(); + + // Does this file system update access times on a plain read at all? If + // it is mounted `noatime` the fault cannot show here. + let control = root.path().join("control"); + fs::write(&control, "x").unwrap(); + let old = set_old_access_time(&control); + fs::read(&control).unwrap(); + if fs::metadata(&control).unwrap().accessed().unwrap() == old { + eprintln!("this file system does not update access times; nothing to check"); + return; + } + + let mut state = open(&root); + let file = root.path().join("log"); + fs::write(&file, "old contents\n").unwrap(); + + // A truncation as the redirection code records it, then undone. + let mut op = Operation::new(OperationType::FileTruncated, "log".to_string(), None); + op.undo_data = Some(b"old contents\n".to_vec()); + fs::write(&file, "").unwrap(); + state.record_operation(op).unwrap(); + commands::undo(&mut state, 1, false).unwrap(); + assert_eq!(fs::read_to_string(&file).unwrap(), "old contents\n"); + + let old = set_old_access_time(&file); + commands::redo(&mut state, 1, false).unwrap(); + // Taken before this test reads the file itself. + let accessed = fs::metadata(&file).unwrap().accessed().unwrap(); + assert_eq!(accessed, old, "redo changed the file's access time"); + assert_eq!(fs::read_to_string(&file).unwrap(), ""); +} + +/// Records that lack the data their reversal needs are found before +/// anything is reversed, whatever the operation type. +#[test] +fn rollback_checks_every_kind_of_undo_data_first() { + for (kind, undo_data) in [ + (OperationType::DeleteFile, None), + (OperationType::WriteFile, None), + (OperationType::FileTruncated, None), + (OperationType::SetVariable, None), + (OperationType::SetVariable, Some(b"not json".to_vec())), + (OperationType::UnsetVariable, None), + ] { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "kept", false).unwrap(); + let mut op = Operation::new(kind, "x".to_string(), None); + op.undo_data = undo_data.clone(); + state.record_operation(op).unwrap(); + + let refused = commands::rollback_transaction(&mut state).unwrap_err(); + assert!( + refused.to_string().contains("refused"), + "{kind:?} with {undo_data:?}: {refused}" + ); + assert!( + root.path().join("kept").exists(), + "{kind:?}: nothing reversed" + ); + assert!(!root.path().join("x").exists(), "{kind:?}: nothing written"); + } +} + +#[test] +fn rollback_skips_operations_already_undone_inside_the_transaction() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::touch(&mut state, "a", false).unwrap(); + commands::touch(&mut state, "b", false).unwrap(); + commands::undo(&mut state, 1, false).unwrap(); + assert!(!root.path().join("b").exists()); + + // Before, rollback tried to remove `b` a second time and failed. + commands::rollback_transaction(&mut state).unwrap(); + assert!(!root.path().join("a").exists()); + assert!(state.active_transaction.is_none()); +} + +#[test] +fn rollback_of_an_empty_transaction_closes_it() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + commands::begin_transaction(&mut state, "t").unwrap(); + commands::rollback_transaction(&mut state).unwrap(); + assert!(state.active_transaction.is_none()); + + // A new transaction can begin, also after a restart. + drop(state); + let mut state = open(&root); + assert!(state.active_transaction.is_none()); + commands::begin_transaction(&mut state, "next").unwrap(); +} + +#[test] +fn a_redone_removal_can_be_undone_with_its_contents() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + let file = root.path().join("notes"); + fs::write(&file, "kept across redo\n").unwrap(); + + commands::rm(&mut state, "notes", false).unwrap(); + commands::undo(&mut state, 1, false).unwrap(); + assert_eq!(fs::read_to_string(&file).unwrap(), "kept across redo\n"); + + commands::redo(&mut state, 1, false).unwrap(); + assert!(!file.exists()); + + // The redo wrote a new record. Undoing it must bring the contents back, + // not an empty file. + commands::undo(&mut state, 1, false).unwrap(); + assert_eq!(fs::read_to_string(&file).unwrap(), "kept across redo\n"); +} + +#[test] +fn a_removal_redone_inside_a_transaction_can_be_rolled_back() { + let root = TempDir::new().unwrap(); + let mut state = open(&root); + let file = root.path().join("notes"); + fs::write(&file, "kept across rollback\n").unwrap(); + + commands::rm(&mut state, "notes", false).unwrap(); + commands::undo(&mut state, 1, false).unwrap(); + + commands::begin_transaction(&mut state, "t").unwrap(); + commands::redo(&mut state, 1, false).unwrap(); + assert!(!file.exists()); + + commands::rollback_transaction(&mut state).unwrap(); + + assert!(state.active_transaction.is_none()); + assert_eq!(fs::read_to_string(&file).unwrap(), "kept across rollback\n"); +} diff --git a/impl/rust-cli/tests/stress_tests.rs b/impl/rust-cli/tests/stress_tests.rs index 34628f05..5c4b5d17 100644 --- a/impl/rust-cli/tests/stress_tests.rs +++ b/impl/rust-cli/tests/stress_tests.rs @@ -10,17 +10,54 @@ //! //! These tests validate that the shell remains stable, performant, //! and correct even under stress. +//! +//! All tests are `#[ignore]`d; CI runs them on Linux with +//! `cargo test --test stress_tests -- --ignored --test-threads=1` +//! (see `.github/workflows/compilation_tests.yml`). +//! +//! The shell keeps its persisted state in `.vsh_state.journal` inside the +//! sandbox root, so directory listings below account for files with the +//! `.vsh_state` prefix explicitly. mod fixtures; use fixtures::tempfile::TempDir; use std::fs; use std::io::Write; +use std::path::Path; use std::sync::{Arc, Mutex}; use std::thread; use std::time::Instant; use vsh::state::ShellState; +/// Prefix shared by the shell's own state files in the sandbox root. +const STATE_FILE_PREFIX: &str = ".vsh_state"; + +/// Names of the entries in `dir` that are not the shell's own state files. +fn user_entries(dir: &Path) -> Vec { + let mut names: Vec = fs::read_dir(dir) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .filter(|name| !name.starts_with(STATE_FILE_PREFIX)) + .collect(); + names.sort(); + names +} + +/// Total size in bytes of the shell's own state files in `dir`. +fn state_files_size(dir: &Path) -> u64 { + fs::read_dir(dir) + .unwrap() + .map(|e| e.unwrap()) + .filter(|e| { + e.file_name() + .to_string_lossy() + .starts_with(STATE_FILE_PREFIX) + }) + .map(|e| e.metadata().unwrap().len()) + .sum() +} + // ============================================================ // Stress Test 1: Deep Directory Nesting // ============================================================ @@ -28,46 +65,59 @@ use vsh::state::ShellState; #[test] #[ignore] // Run with: cargo test --test stress_tests -- --ignored fn stress_deep_nesting_1000_levels() { - let temp = TempDir::new().unwrap(); - let start = Instant::now(); + const DEPTH: usize = 1000; - // Create 1000-level deep directory structure - let mut current_path = temp.path().to_path_buf(); - for i in 0..1000 { - current_path.push(format!("level_{:04}", i)); - } + let temp = TempDir::new().unwrap(); + let mut state = ShellState::new(temp.path().to_str().unwrap()).unwrap(); - // Create the entire nested structure - fs::create_dir_all(¤t_path).unwrap(); + // The shell resolves every path to an absolute one, so the deepest path + // must fit in PATH_MAX. One-character components ("d/d/.../d") cost two + // bytes per level: about 2,000 bytes plus the root, which fits Linux's + // 4096. (Ten-character names would need ~11,000 bytes, which no + // absolute-path API accepts; macOS's PATH_MAX of 1024 cannot hold 1000 + // levels at all.) + let deepest_len = state.root.as_os_str().len() + 2 * DEPTH; + let path_max = libc::PATH_MAX as usize; + assert!( + deepest_len < path_max, + "{DEPTH}-level nesting needs a {deepest_len}-byte absolute path; \ + this platform's PATH_MAX is {path_max}" + ); + // Create the 1000-level structure through the shell, one level per op. + let start = Instant::now(); + let mut rel = String::from("d"); + for level in 0..DEPTH { + if level > 0 { + rel.push_str("/d"); + } + vsh::commands::mkdir(&mut state, &rel, false).unwrap(); + } let create_time = start.elapsed(); println!( "✓ Created 1000-level nested directories in {:?}", create_time ); - // Verify it exists - assert!(current_path.exists()); - assert!(current_path.is_dir()); + // Verify it exists and every level was recorded + let deepest = state.resolve_path(&rel); + assert!(deepest.exists()); + assert!(deepest.is_dir()); + assert_eq!(state.history.len(), DEPTH, "Each mkdir should be recorded"); - // Test undo (remove all levels) + // Undo all 1000 operations (innermost first) let undo_start = Instant::now(); - let mut state = ShellState::new(temp.path().to_str().unwrap()).unwrap(); - - // Record the creation in state - for i in 0..1000 { - let path = format!("level_{:04}", i); - vsh::commands::mkdir(&mut state, &path, true).unwrap(); - } - - // Undo all 1000 operations - for _ in 0..1000 { + for _ in 0..DEPTH { vsh::commands::undo(&mut state, 1, false).unwrap(); } - let undo_time = undo_start.elapsed(); println!("✓ Undid 1000 operations in {:?}", undo_time); + assert!( + user_entries(temp.path()).is_empty(), + "Undo should remove every level" + ); + // Verify requirements assert!(create_time.as_secs() < 5, "Should create in <5s"); assert!(undo_time.as_secs() < 2, "Should undo in <2s"); @@ -76,30 +126,69 @@ fn stress_deep_nesting_1000_levels() { #[test] #[ignore] fn stress_deep_nesting_no_stack_overflow() { + use nix::fcntl::{openat, OFlag}; + use nix::sys::stat::{mkdirat, Mode}; + use std::os::fd::OwnedFd; + + const DEPTH: usize = 5000; + let temp = TempDir::new().unwrap(); + let mut state = ShellState::new(temp.path().to_str().unwrap()).unwrap(); - // Create even deeper nesting (5000 levels) to test for stack overflow - let mut current_path = temp.path().to_path_buf(); - for i in 0..5000 { - current_path.push(format!("d{}", i)); + // Even deeper nesting (5000 levels) to test for stack overflow. Its + // absolute path is far longer than PATH_MAX, so it cannot be created + // with path-based calls (`fs::create_dir_all` fails with ENAMETOOLONG). + // Build it one level at a time relative to the parent's descriptor. + let deep_rel = vec!["d"; DEPTH].join("/"); + assert!( + state.root.join(&deep_rel).as_os_str().len() > libc::PATH_MAX as usize, + "this test is meant to go beyond PATH_MAX" + ); + let mut dir: OwnedFd = fs::File::open(&state.root).unwrap().into(); + for _ in 0..DEPTH { + mkdirat(&dir, "d", Mode::from_bits_truncate(0o755)).unwrap(); + dir = openat( + &dir, + "d", + OFlag::O_RDONLY | OFlag::O_DIRECTORY, + Mode::empty(), + ) + .unwrap(); } - - // Should not cause stack overflow - let result = fs::create_dir_all(¤t_path); + drop(dir); + + // The shell must stay stable when handed a path this deep: resolving it + // is pure normalisation, and filesystem operations on it fail with an + // error instead of overflowing the stack or panicking. + assert_eq!(state.resolve_path(&deep_rel), state.root.join(&deep_rel)); + let before = state.history.len(); + let mkdir_result = vsh::commands::mkdir(&mut state, &format!("{deep_rel}/x"), false); + assert!( + mkdir_result.is_err(), + "mkdir beyond PATH_MAX should fail cleanly" + ); + let touch_result = vsh::commands::touch(&mut state, &format!("{deep_rel}/f"), false); assert!( - result.is_ok(), - "Should handle 5000 levels without stack overflow" + touch_result.is_err(), + "touch beyond PATH_MAX should fail cleanly" + ); + assert_eq!( + state.history.len(), + before, + "Failed operations must not be recorded" ); // Cleanup should also not overflow - let cleanup_result = fs::remove_dir_all(temp.path().join("d0")); + let cleanup_result = fs::remove_dir_all(temp.path().join("d")); assert!(cleanup_result.is_ok(), "Cleanup should not overflow"); + assert!(user_entries(temp.path()).is_empty()); } // ============================================================ // Stress Test 2: Large Files // ============================================================ +/// Writes 1 GB of temporary data to disk (about 1.5 s on a local SSD). #[test] #[ignore] fn stress_large_file_1gb() { @@ -119,44 +208,69 @@ fn stress_large_file_1gb() { file.write_all(&chunk).unwrap(); } file.sync_all().unwrap(); + drop(file); let create_time = start.elapsed(); println!("✓ Created 1GB file in {:?}", create_time); // Verify size + const SIZE: u64 = 1024 * 1024 * 1024; let metadata = fs::metadata(&large_file).unwrap(); - assert_eq!( - metadata.len(), - 1024 * 1024 * 1024, - "File should be exactly 1GB" - ); + assert_eq!(metadata.len(), SIZE, "File should be exactly 1GB"); // Test operations on large file let mut state = ShellState::new(temp.path().to_str().unwrap()).unwrap(); - // Touch shouldn't fail on large file (just updates metadata) - let touch_start = Instant::now(); - vsh::commands::touch(&mut state, "large.bin", true).unwrap(); - let touch_time = touch_start.elapsed(); - println!("✓ Touch on 1GB file in {:?}", touch_time); - - assert!( - touch_time.as_millis() < 100, - "Touch should be <100ms even for large files" + // Metadata-only operations must not depend on the file size. (`touch` + // in vsh only creates files, so it cannot be applied to this one; the + // reversible metadata operations are `chmod` and `mv`.) Each operation + // and its undo should be <100ms even for a 1GB file. + let timed = |label: &str, op: &mut dyn FnMut()| { + let t = Instant::now(); + op(); + let elapsed = t.elapsed(); + println!("✓ {label} on 1GB file in {elapsed:?}"); + assert!( + elapsed.as_millis() < 100, + "{label} should be <100ms even for large files (was {elapsed:?})" + ); + }; + + timed("chmod", &mut || { + vsh::commands::chmod(&mut state, "600", "large.bin", false).unwrap() + }); + timed("undo chmod", &mut || { + vsh::commands::undo(&mut state, 1, false).unwrap() + }); + timed("mv", &mut || { + vsh::commands::mv(&mut state, "large.bin", "moved.bin", false).unwrap() + }); + assert_eq!( + fs::metadata(temp.path().join("moved.bin")).unwrap().len(), + SIZE ); + timed("undo mv", &mut || { + vsh::commands::undo(&mut state, 1, false).unwrap() + }); - // Memory usage check: should not load entire file into memory - // This test would fail if implementation naively reads entire file + // Memory usage check: should not load entire file into memory. + // These operations never read file content: the file is intact. + assert_eq!(fs::metadata(&large_file).unwrap().len(), SIZE); + assert_eq!( + metadata.permissions(), + fs::metadata(&large_file).unwrap().permissions() + ); } #[test] #[ignore] fn stress_large_file_undo_streaming() { + const SIZE: usize = 100 * 1024 * 1024; + let temp = TempDir::new().unwrap(); let file_path = temp.path().join("large.txt"); // Create 100MB file - let _size = 100 * 1024 * 1024; let mut file = fs::File::create(&file_path).unwrap(); let chunk = vec![b'A'; 1024 * 1024]; // 1MB chunks @@ -165,19 +279,24 @@ fn stress_large_file_undo_streaming() { } drop(file); - // Verify we don't OOM when saving undo data for large file let mut state = ShellState::new(temp.path().to_str().unwrap()).unwrap(); - // This should use streaming, not load all 100MB into memory + // Removing a file is the operation that saves the file's content as undo + // data. (`touch` cannot be used here: in vsh it only creates files.) + // + // NOT streaming: `rm` currently holds the whole content in memory as + // `undo_data` and persists it in the state journal as JSON. This test + // checks that a 100MB rm/undo round trip completes, restores the content + // byte for byte, and that the undo itself stays fast. Out-of-line + // (streamed) undo storage is open work, recorded in + // docs/audits/2026-11-16-hypatia-remediation.adoc. let undo_save_start = Instant::now(); - vsh::commands::touch(&mut state, "large.txt", true).unwrap(); + vsh::commands::rm(&mut state, "large.txt", false).unwrap(); let undo_save_time = undo_save_start.elapsed(); + assert!(!file_path.exists(), "rm should remove the file"); println!("✓ Saved undo data for 100MB file in {:?}", undo_save_time); - // Check memory didn't spike (we can't easily test this in Rust, - // but in practice this would be monitored externally) - // Undo should also be fast let undo_start = Instant::now(); vsh::commands::undo(&mut state, 1, false).unwrap(); @@ -185,6 +304,14 @@ fn stress_large_file_undo_streaming() { println!("✓ Undid large file operation in {:?}", undo_time); assert!(undo_time.as_millis() < 500, "Undo should be <500ms"); + + // The restored file must be the original, byte for byte. + let restored = fs::read(&file_path).unwrap(); + assert_eq!(restored.len(), SIZE, "Restored file should be 100MB"); + assert!( + restored.iter().all(|&b| b == b'A'), + "Restored content should match the original" + ); } // ============================================================ @@ -225,9 +352,14 @@ fn stress_many_operations_10000() { assert!(create_time.as_secs() < 30, "Should complete in <30s"); assert!(undo_time.as_secs() < 30, "Should undo in <30s"); - // Verify all files removed - let entries: Vec<_> = fs::read_dir(temp.path()).unwrap().collect(); - assert_eq!(entries.len(), 0, "All files should be undone"); + // Verify all files removed (only the shell's own state files remain) + let entries = user_entries(temp.path()); + assert!( + entries.is_empty(), + "All files should be undone, found {} (first: {:?})", + entries.len(), + entries.first() + ); } #[test] @@ -273,13 +405,22 @@ fn stress_undo_redo_efficiency() { vsh::commands::undo(&mut state, 1, false).unwrap(); } let undo_time = start.elapsed(); + assert!( + user_entries(temp.path()).is_empty(), + "Undo should remove all files" + ); // Redo should also be O(1) per operation let redo_start = Instant::now(); for _ in 0..100 { - state.pop_redo().unwrap(); + vsh::commands::redo(&mut state, 1, false).unwrap(); } let redo_time = redo_start.elapsed(); + assert_eq!( + user_entries(temp.path()).len(), + 100, + "Redo should recreate all files" + ); println!("✓ Undo 100: {:?}, Redo 100: {:?}", undo_time, redo_time); @@ -342,9 +483,16 @@ fn stress_concurrent_multiple_instances() { handle.join().unwrap(); } - // Verify total files created (10 threads * 100 files = 1000) - let entries: Vec<_> = fs::read_dir(&temp_path).unwrap().collect(); + // Verify total files created (10 threads * 100 files = 1000). The + // instances share one sandbox root, so its state files are there too; + // nothing else may be. + let entries = user_entries(&temp_path); assert_eq!(entries.len(), 1000, "Should have 1000 total files"); + let mut expected: Vec = (0..10) + .flat_map(|t| (0..100).map(move |i| format!("thread_{}_file_{}.txt", t, i))) + .collect(); + expected.sort(); + assert_eq!(entries, expected, "Exactly the threads' files should exist"); println!("✓ Concurrent access: 10 threads, 100 ops each = 1000 files"); } @@ -402,8 +550,11 @@ fn stress_resource_limits_max_history() { let temp = TempDir::new().unwrap(); let mut state = ShellState::new(temp.path().to_str().unwrap()).unwrap(); - // Create operations until we hit a reasonable limit - // In practice, there should be a configurable max history size + // ShellState bounds its history (`max_history_size`, 10,000 by default); + // a user-facing setting for it remains tracked by #190. + let limit = state + .max_history_size + .expect("ShellState should bound its history by default"); for i in 0..100_000 { let path = format!("file_{}.txt", i); @@ -412,17 +563,30 @@ fn stress_resource_limits_max_history() { // Every 10k operations, check if we have a limit if i % 10_000 == 0 { println!(" {} operations...", i); + assert!(state.history.len() <= limit); } } - // Even with 100k operations, should not OOM - // (This test would be better with actual memory monitoring) - + // Even with 100k operations, should not OOM: the in-memory history is + // capped at the limit... let history_size = state.history.len(); println!("✓ Completed {} operations without OOM", history_size); + assert_eq!( + history_size, limit, + "History should be capped at max_history_size" + ); - // A configurable production history bound remains tracked by #190. - // assert!(history_size <= MAX_HISTORY_SIZE); + // ...and so is the persisted state, which must not grow with the total + // number of operations. A touch op serialises to ~200 bytes, so a + // snapshot plus journal of a capped history stays within a few MB; + // 32 MiB leaves ample headroom while catching unbounded growth (100k + // uncapped ops alone would exceed it). + let persisted = state_files_size(temp.path()); + println!("✓ Persisted state: {} bytes", persisted); + assert!( + persisted < 32 * 1024 * 1024, + "Persisted state should be bounded by the history limit (was {persisted} bytes)" + ); } #[test] @@ -456,3 +620,82 @@ fn stress_many_small_files() { println!("✓ Cleaned up 10,000 files in {:?}", cleanup_time); } + +/// Storage that keeps nothing, for a test that times the shell's own work. +struct NoStorage; + +impl vsh::journal::Storage for NoStorage { + fn read(&mut self) -> std::io::Result>> { + Ok(None) + } + fn append(&mut self, _bytes: &[u8]) -> std::io::Result<()> { + Ok(()) + } + fn sync(&mut self) -> std::io::Result<()> { + Ok(()) + } + fn replace(&mut self, _bytes: &[u8]) -> std::io::Result<()> { + Ok(()) + } + fn set_aside(&mut self, _bytes: &[u8]) -> std::io::Result { + Ok("nowhere".to_string()) + } +} + +/// How long it takes to roll back a transaction of `operations` tracked +/// variable assignments, with nothing stored. +fn time_to_roll_back(operations: usize) -> std::time::Duration { + let temp = TempDir::new().unwrap(); + let mut state = + ShellState::with_storage(temp.path().to_str().unwrap(), Box::new(NoStorage)).unwrap(); + state.max_history_size = Some(1_000); + + vsh::commands::begin_transaction(&mut state, "long").unwrap(); + for i in 0..operations { + state.set_variable_tracked("counter", i.to_string()); + } + assert_eq!( + state.history.len(), + operations, + "an open transaction keeps its operations past the history limit" + ); + + let start = Instant::now(); + vsh::commands::rollback_transaction(&mut state).unwrap(); + let taken = start.elapsed(); + + assert!(state.active_transaction.is_none()); + assert_eq!(state.get_variable("counter"), None); + taken +} + +/// Rolling back a long transaction takes time in proportion to its length. +/// +/// The check that every operation of the transaction is still in the +/// history, and the list of its operations, each looked one operation up by +/// scanning the other list, so the time grew with the square of the +/// transaction. An open transaction keeps all of its operations, however +/// many, so nothing bounded that. +/// +/// The test compares two lengths, so it does not depend on how fast the +/// machine is. Twice the operations should take about twice the time; with +/// the scans it took three times as long (measured 2026-11-17, 30,000 and +/// 60,000 operations: a ratio of 3.06 with the scans and 1.73 without). +#[test] +#[ignore] +fn stress_rollback_of_a_long_transaction() { + const SHORT: usize = 30_000; + let short = time_to_roll_back(SHORT); + let long = time_to_roll_back(2 * SHORT); + let ratio = long.as_secs_f64() / short.as_secs_f64(); + + println!( + "✓ Rollback of {SHORT} operations: {short:?}; of {}: {long:?}; ratio {ratio:.2}", + 2 * SHORT + ); + assert!( + ratio < 2.7, + "rollback time should grow in proportion to the transaction: twice the operations \ + took {ratio:.2} times as long ({short:?} and {long:?})" + ); +} diff --git a/impl/rust-cli/tests/unsaved_record_tests.rs b/impl/rust-cli/tests/unsaved_record_tests.rs new file mode 100644 index 00000000..594241db --- /dev/null +++ b/impl/rust-cli/tests/unsaved_record_tests.rs @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! A command whose record cannot be saved returns an error (owner ruling +//! OD-49, route plan section 14.5). +//! +//! The shell does an operation and then writes its record. If the write +//! fails, the operation has happened and the saved history does not know. +//! Every command except rollback used to print a warning and report +//! success, so a script or an agent could not see it. +//! +//! The rules under test: +//! +//! * The command returns a [`NotRecorded`] error. The change was made, and +//! this session's history holds its record. +//! * The next record that can be written saves the whole history again, so +//! nothing is lost if the shell keeps running until then. +//! * Assigning a variable and `cd` change no file. For them a record that +//! cannot be saved stays a warning. + +mod fixtures; + +use fixtures::tempfile::TempDir; +use std::fs; +use vsh::commands; +use vsh::journal::SimStorage; +use vsh::state::{NotRecorded, ShellState}; + +/// A state whose saved form goes to simulated storage, which can be made to +/// refuse writes. +fn open_on(root: &TempDir, storage: &SimStorage) -> ShellState { + ShellState::with_storage(root.path().to_str().unwrap(), Box::new(storage.clone())).unwrap() +} + +fn refuse_writes(storage: &SimStorage, refuse: bool) { + storage.fail_append(refuse); + storage.fail_replace(refuse); +} + +/// Whether `outcome` is the error that says: done, but not recorded. +fn not_recorded(outcome: anyhow::Result) -> bool { + match outcome { + Ok(_) => false, + Err(error) => error.downcast_ref::().is_some(), + } +} + +#[test] +fn a_command_whose_record_cannot_be_saved_returns_an_error() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = open_on(&root, &storage); + + refuse_writes(&storage, true); + let outcome = commands::mkdir(&mut state, "d", false); + + let error = outcome.unwrap_err(); + assert!(error.downcast_ref::().is_some(), "{error:#}"); + assert!( + error.to_string().contains("the change was made"), + "{error:#}" + ); + // The change was made, and this session knows of it. + assert!(root.path().join("d").is_dir()); + assert_eq!(state.history.len(), 1); + assert!(!state.is_saved()); +} + +#[test] +fn the_next_record_that_can_be_written_saves_the_one_that_could_not() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = open_on(&root, &storage); + refuse_writes(&storage, true); + assert!(not_recorded(commands::mkdir(&mut state, "d", false))); + + refuse_writes(&storage, false); + commands::touch(&mut state, "f", false).unwrap(); + + assert!(state.is_saved()); + let reloaded = open_on(&root, &storage); + let recorded: Vec<&str> = reloaded.history.iter().map(|op| op.path.as_str()).collect(); + assert_eq!(recorded, ["d", "f"]); +} + +#[test] +fn every_kind_of_change_reports_a_record_that_cannot_be_saved() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = open_on(&root, &storage); + fs::write(root.path().join("old"), "content").unwrap(); + commands::touch(&mut state, "made", false).unwrap(); + + refuse_writes(&storage, true); + + // Each change is made, and each reports that it is not recorded. + assert!(not_recorded(commands::touch(&mut state, "f", false))); + assert!(root.path().join("f").exists()); + assert!(not_recorded(commands::rm(&mut state, "old", false))); + assert!(!root.path().join("old").exists()); + assert!(not_recorded(commands::undo(&mut state, 1, false))); + assert!(root.path().join("old").exists(), "the removal was undone"); + assert!(not_recorded(commands::redo(&mut state, 1, false))); + assert!(!root.path().join("old").exists(), "the removal was redone"); + assert!(not_recorded(commands::begin_transaction(&mut state, "t"))); + assert!(state.active_transaction.is_some()); + assert!(not_recorded(commands::commit_transaction(&mut state))); + assert!(state.active_transaction.is_none()); +} + +#[test] +fn a_variable_whose_record_cannot_be_saved_is_still_set() { + let root = TempDir::new().unwrap(); + let storage = SimStorage::new(); + let mut state = open_on(&root, &storage); + + refuse_writes(&storage, true); + state.set_variable_tracked("name".to_string(), "value".to_string()); + + assert_eq!(state.get_variable("name"), Some("value")); +} diff --git a/proofs/agda/All.agda b/proofs/agda/All.agda index f521f376..4a1c3bde 100644 --- a/proofs/agda/All.agda +++ b/proofs/agda/All.agda @@ -14,3 +14,7 @@ import CopyMoveOperations import FileContentOperations import RMOOperations import PreconditionWitnesses +import JournalFraming +import JournalReplay +import JournalModel +import JournalWitnesses diff --git a/proofs/agda/JournalFraming.agda b/proofs/agda/JournalFraming.agda new file mode 100644 index 00000000..8f85711c --- /dev/null +++ b/proofs/agda/JournalFraming.agda @@ -0,0 +1,522 @@ +{-# OPTIONS --safe --without-K --cubical-compatible #-} +-- SPDX-License-Identifier: MPL-2.0 +{- Valence Shell - journal record framing (Agda) + + The on-disk journal is one file of records. Each record is a *frame*: + + header (fixed width, carries the payload length) + payload + checksum of the payload (fixed width) + + This module defines the frame, a decoder, and proves: + + * `step-frame` a frame at the front of the input is read back exactly; + * `step-sound` whatever the decoder accepts is byte-for-byte a frame + at the front of the input (nothing is misread); + * `step-torn` a frame cut short at any point is rejected; + * `classify-torn` and it is classified as torn, never as damaged; + * `decode-sound` the accepted records, re-encoded, followed by the + rejected tail, are exactly the input; + * `decode-frames` a sequence of frames followed by a rejected tail + decodes to exactly those records and that tail; + * `decode-crash` after a crash that keeps the synced frames and any + prefix of the unsynced ones, the decoder returns the + synced records followed by a prefix of the unsynced + records; + * `decode-crash-tail` after such a crash the bytes the decoder rejects + are never classified as damaged; + * `decode-normal` re-encoding what was accepted gives a clean journal, + for any input at all. + + What is assumed, as module parameters (no postulates): + + * the header has a fixed width of at least one byte (`hdr-length`); + * the checksum has a fixed width (`sum-length`). + + Nothing is assumed about the checksum's strength. The torn-tail results + need none: truncation is caught by the length in the header. A checksum + only makes it unlikely that bytes which were never written as a frame are + accepted as one; `step-sound` states exactly what acceptance means, and it + holds for any checksum function. + + The header may carry more than the length. The decoder accepts a header + `h` only if it is canonical, `hdr (unhdr h) ≡ h`. If `hdr n` is the length + followed by a check of the length, as in the Rust format, that test is the + header check. With a header that is the bare length it holds for every + input and says nothing. + + `classify` names what the rejected tail is: `clean`, `torn` (the input + ends inside a record) or `damaged` (a whole header that is not canonical, + or a whole record that was still rejected). One direction is proved: + a crash never yields `damaged` (`classify-torn`, `decode-crash-tail`). + The other direction is not a theorem and cannot be one: damage that + leaves a canonical header declaring more bytes than the file holds is + classified `torn`. How unlikely that is depends on the strength of the + header check, about which nothing is assumed here. + + `Frameable p` says the header can carry the length of `p`. With a 4-byte + length field this is `length p < 2^32`; the writer must refuse longer + payloads. + + Added 2026-11-17 for route-plan Phase C (journal redesign), step C1. + `classify` and its theorems were added the same day, with the header + check in the Rust format. +-} + +open import Data.Nat.Base using (ℕ; zero; suc; _+_; _≤_; z≤n; s≤s) +open import Data.List.Base using (List; []; _∷_; _++_; length) +open import Relation.Binary.Definitions using (DecidableEquality) +open import Relation.Binary.PropositionalEquality + using (_≡_; _≢_; refl; sym; trans; cong; cong₂; subst) + +module JournalFraming + (Byte : Set) + (_≟_ : DecidableEquality Byte) + (w c : ℕ) -- header width is `suc w` + (hdr : ℕ → List Byte) + (hdr-length : ∀ n → length (hdr n) ≡ suc w) + (unhdr : List Byte → ℕ) + (sum : List Byte → List Byte) + (sum-length : ∀ p → length (sum p) ≡ c) + where + +open import Data.Empty using (⊥; ⊥-elim) +open import Data.Unit.Base using (⊤; tt) +open import Data.Maybe.Base using (Maybe; just; nothing; _>>=_) +import Data.Maybe.Base as Maybe +open import Data.Product.Base using (_×_; _,_; proj₁; proj₂; ∃) +import Data.Product.Base as Prod +open import Data.Sum.Base using (_⊎_; inj₁; inj₂) +open import Data.Nat.Properties using (≤-trans; m≤n+m; suc-injective) +open import Data.List.Properties + using (++-assoc; ++-identityʳ; ++-identityʳ-unique; ++-conicalˡ; ++-conicalʳ; + length-++; ∷-injective; ≡-dec) +open import Data.List.Relation.Unary.All using (All; []; _∷_) +import Data.List.Relation.Unary.All.Properties as All +open import Relation.Nullary using (Dec; yes; no; ¬_) + +Payload : Set +Payload = List Byte + +_≟ᴸ_ : DecidableEquality (List Byte) +_≟ᴸ_ = ≡-dec _≟_ + +------------------------------------------------------------------------ +-- List lemmas + +-- Two ways of splitting one list overlap in one of two ways. +++-split : ∀ {A : Set} (xs : List A) {ys} t {u} → t ++ u ≡ xs ++ ys → + (∃ λ v → xs ≡ t ++ v × u ≡ v ++ ys) ⊎ + (∃ λ v → t ≡ xs ++ v × ys ≡ v ++ u) +++-split [] t eq = inj₂ (t , refl , sym eq) +++-split (x ∷ xs) [] eq = inj₁ (x ∷ xs , refl , eq) +++-split (x ∷ xs) (y ∷ t) eq with ∷-injective eq +... | y≡x , eq′ with ++-split xs t eq′ +... | inj₁ (v , xs≡ , u≡) = inj₁ (v , cong₂ _∷_ (sym y≡x) xs≡ , u≡) +... | inj₂ (v , t≡ , ys≡) = inj₂ (v , cong₂ _∷_ y≡x t≡ , ys≡) + +-- Equal appends with equally long fronts have equal fronts and backs. +++-cancel-length : ∀ {A : Set} (xs xs′ : List A) {ys ys′} → + length xs ≡ length xs′ → xs ++ ys ≡ xs′ ++ ys′ → + xs ≡ xs′ × ys ≡ ys′ +++-cancel-length [] [] _ eq = refl , eq +++-cancel-length [] (_ ∷ _) () +++-cancel-length (_ ∷ _) [] () +++-cancel-length (x ∷ xs) (x′ ∷ xs′) len eq with ∷-injective eq +... | x≡ , eq′ with ++-cancel-length xs xs′ (suc-injective len) eq′ +... | xs≡ , ys≡ = cong₂ _∷_ x≡ xs≡ , ys≡ + +------------------------------------------------------------------------ +-- Taking exactly n bytes + +splitExact : ℕ → List Byte → Maybe (List Byte × List Byte) +splitExact zero bs = just ([] , bs) +splitExact (suc n) [] = nothing +splitExact (suc n) (b ∷ bs) = Maybe.map (Prod.map₁ (b ∷_)) (splitExact n bs) + +splitExact-++ : ∀ xs ys → splitExact (length xs) (xs ++ ys) ≡ just (xs , ys) +splitExact-++ [] ys = refl +splitExact-++ (x ∷ xs) ys rewrite splitExact-++ xs ys = refl + +splitExact-len : ∀ {n} xs ys → length xs ≡ n → + splitExact n (xs ++ ys) ≡ just (xs , ys) +splitExact-len xs ys refl = splitExact-++ xs ys + +splitExact-sound : ∀ n bs {xs ys} → splitExact n bs ≡ just (xs , ys) → + (xs ++ ys ≡ bs) × (length xs ≡ n) +splitExact-sound zero bs refl = refl , refl +splitExact-sound (suc n) (b ∷ bs) eq with splitExact n bs in eq′ +splitExact-sound (suc n) (b ∷ bs) refl | just (xs′ , ys′) = + let ++≡ , len≡ = splitExact-sound n bs eq′ in cong (b ∷_) ++≡ , cong suc len≡ + +------------------------------------------------------------------------ +-- Frames + +frame : Payload → List Byte +frame p = hdr (length p) ++ (p ++ sum p) + +frames : List Payload → List Byte +frames [] = [] +frames (p ∷ ps) = frame p ++ frames ps + +-- The header can carry this payload's length. +Frameable : Payload → Set +Frameable p = unhdr (hdr (length p)) ≡ length p + +frame-++ : ∀ p rest → frame p ++ rest ≡ hdr (length p) ++ (p ++ (sum p ++ rest)) +frame-++ p rest = + trans (++-assoc (hdr (length p)) (p ++ sum p) rest) + (cong (hdr (length p) ++_) (++-assoc p (sum p) rest)) + +frames-++ : ∀ ps qs → frames (ps ++ qs) ≡ frames ps ++ frames qs +frames-++ [] qs = refl +frames-++ (p ∷ ps) qs = + trans (cong (frame p ++_) (frames-++ ps qs)) + (sym (++-assoc (frame p) (frames ps) (frames qs))) + +------------------------------------------------------------------------ +-- Reading one frame + +guard : ∀ {P : Set} → Dec P → Maybe ⊤ +guard (yes _) = just tt +guard (no _) = nothing + +guard-yes : ∀ {P : Set} (d : Dec P) → P → guard d ≡ just tt +guard-yes (yes _) _ = refl +guard-yes (no ¬p) p = ⊥-elim (¬p p) + +afterSum : Payload → List Byte × List Byte → Maybe (Payload × List Byte) +afterSum p (s , r) = guard (s ≟ᴸ sum p) >>= λ _ → just (p , r) + +afterPayload : List Byte × List Byte → Maybe (Payload × List Byte) +afterPayload (p , r) = splitExact c r >>= afterSum p + +afterHeader : List Byte × List Byte → Maybe (Payload × List Byte) +afterHeader (h , r) = + guard (hdr (unhdr h) ≟ᴸ h) >>= λ _ → splitExact (unhdr h) r >>= afterPayload + +-- `just (p , rest)` when the input starts with a well-formed frame carrying +-- `p`; `nothing` otherwise. +step : List Byte → Maybe (Payload × List Byte) +step bs = splitExact (suc w) bs >>= afterHeader + +step-frame : ∀ p rest → Frameable p → step (frame p ++ rest) ≡ just (p , rest) +step-frame p rest fr + rewrite frame-++ p rest + | splitExact-len (hdr (length p)) (p ++ (sum p ++ rest)) (hdr-length (length p)) + | fr + | guard-yes (hdr (length p) ≟ᴸ hdr (length p)) refl + | splitExact-++ p (sum p ++ rest) + | splitExact-len (sum p) rest (sum-length p) + | guard-yes (sum p ≟ᴸ sum p) refl + = refl + +step-sound : ∀ bs {p r} → step bs ≡ just (p , r) → + (frame p ++ r ≡ bs) × Frameable p +step-sound bs eq with splitExact (suc w) bs in e₁ +... | just (h , r₁) with hdr (unhdr h) ≟ᴸ h +... | yes canon with splitExact (unhdr h) r₁ in e₂ +... | just (p′ , r₂) with splitExact c r₂ in e₃ +... | just (s , r₃) with s ≟ᴸ sum p′ +step-sound bs refl | just (h , r₁) | yes canon | just (p′ , r₂) | just (s , r₃) | yes s≡ = + let h++ , _ = splitExact-sound (suc w) bs e₁ + p++ , plen = splitExact-sound (unhdr h) r₁ e₂ + s++ , _ = splitExact-sound c r₂ e₃ + hdr≡ : hdr (length p′) ≡ h + hdr≡ = trans (cong hdr plen) canon + in trans (frame-++ p′ r₃) + (trans (cong₂ _++_ hdr≡ + (cong (p′ ++_) (trans (cong (_++ r₃) (sym s≡)) s++))) + (trans (cong (h ++_) p++) h++)) + , trans (cong unhdr hdr≡) (sym plen) + +step-[] : step [] ≡ nothing +step-[] = refl + +-- A frame cut short is rejected. +step-torn : ∀ p t u → Frameable p → u ≢ [] → t ++ u ≡ frame p → step t ≡ nothing +step-torn p t u fr u≢[] eq with step t in st +... | nothing = refl +... | just (p′ , r) = ⊥-elim (u≢[] (++-conicalʳ r u r++u≡[])) + where + sound = step-sound t st + fr′ : Frameable p′ + fr′ = proj₂ sound + + whole : hdr (length p′) ++ (p′ ++ (sum p′ ++ (r ++ u))) ≡ hdr (length p) ++ (p ++ sum p) + whole = + trans (cong (hdr (length p′) ++_) + (trans (cong (p′ ++_) (sym (++-assoc (sum p′) r u))) + (sym (++-assoc p′ (sum p′ ++ r) u)))) + (trans (sym (++-assoc (hdr (length p′)) ((p′ ++ (sum p′ ++ r))) u)) + (trans (cong (_++ u) (sym (frame-++ p′ r))) + (trans (cong (_++ u) (proj₁ sound)) eq))) + + hdrs = ++-cancel-length (hdr (length p′)) (hdr (length p)) + (trans (hdr-length (length p′)) (sym (hdr-length (length p)))) whole + + len≡ : length p′ ≡ length p + len≡ = trans (sym fr′) (trans (cong unhdr (proj₁ hdrs)) fr) + + bodies = ++-cancel-length p′ p len≡ (proj₂ hdrs) + + r++u≡[] : r ++ u ≡ [] + r++u≡[] = ++-identityʳ-unique (sum p) + (sym (trans (cong (λ q → sum q ++ (r ++ u)) (sym (proj₁ bodies))) + (proj₂ bodies))) + +------------------------------------------------------------------------ +-- Reading the whole journal + +-- Result: the records accepted, and the bytes left over. The journal is +-- clean exactly when nothing is left over. +Decoded : Set +Decoded = List Payload × List Byte + +continue : (List Byte → Decoded) → List Byte → Maybe (Payload × List Byte) → Decoded +continue rec bs nothing = [] , bs +continue rec bs (just (p , r)) = Prod.map₁ (p ∷_) (rec r) + +decodeFuel : ℕ → List Byte → Decoded +decodeFuel zero bs = [] , bs +decodeFuel (suc k) bs = continue (decodeFuel k) bs (step bs) + +decode : List Byte → Decoded +decode bs = decodeFuel (length bs) bs + +decodeFuel-sound : ∀ k bs → + frames (proj₁ (decodeFuel k bs)) ++ proj₂ (decodeFuel k bs) ≡ bs +decodeFuel-sound zero bs = refl +decodeFuel-sound (suc k) bs with step bs in st +... | nothing = refl +... | just (p , r) = + trans (++-assoc (frame p) _ _) + (trans (cong (frame p ++_) (decodeFuel-sound k r)) (proj₁ (step-sound bs st))) + +-- Nothing is misread: the accepted records are, byte for byte, the front of +-- the input. +decode-sound : ∀ bs → frames (proj₁ (decode bs)) ++ proj₂ (decode bs) ≡ bs +decode-sound bs = decodeFuel-sound (length bs) bs + +decodeFuel-frameable : ∀ k bs → All Frameable (proj₁ (decodeFuel k bs)) +decodeFuel-frameable zero bs = [] +decodeFuel-frameable (suc k) bs with step bs in st +... | nothing = [] +... | just (p , r) = proj₂ (step-sound bs st) ∷ decodeFuel-frameable k r + +decode-frameable : ∀ bs → All Frameable (proj₁ (decode bs)) +decode-frameable bs = decodeFuel-frameable (length bs) bs + +decodeFuel-frames : ∀ ps t → All Frameable ps → step t ≡ nothing → + ∀ k → length ps ≤ k → decodeFuel k (frames ps ++ t) ≡ (ps , t) +decodeFuel-frames [] t [] st zero _ = refl +decodeFuel-frames [] t [] st (suc k) _ rewrite st = refl +decodeFuel-frames (p ∷ ps) t (fr ∷ frs) st (suc k) (s≤s le) + rewrite ++-assoc (frame p) (frames ps) t + | step-frame p (frames ps ++ t) fr + | decodeFuel-frames ps t frs st k le + = refl + +frame-longer : ∀ p rest → suc (length rest) ≤ length (frame p ++ rest) +frame-longer p rest = subst (suc (length rest) ≤_) (sym len≡) + (s≤s (m≤n+m (length rest) (w + length (p ++ sum p)))) + where + len≡ : length (frame p ++ rest) ≡ suc (w + length (p ++ sum p)) + length rest + len≡ = trans (length-++ (frame p)) + (cong (_+ length rest) + (trans (length-++ (hdr (length p))) + (cong (_+ length (p ++ sum p)) (hdr-length (length p))))) + +frames-longer : ∀ ps t → length ps ≤ length (frames ps ++ t) +frames-longer [] t = z≤n +frames-longer (p ∷ ps) t = + subst (suc (length ps) ≤_) (cong length (sym (++-assoc (frame p) (frames ps) t))) + (≤-trans (s≤s (frames-longer ps t)) (frame-longer p (frames ps ++ t))) + +-- Frames followed by a rejected tail decode to exactly those records and +-- that tail. +decode-frames : ∀ ps t → All Frameable ps → step t ≡ nothing → + decode (frames ps ++ t) ≡ (ps , t) +decode-frames ps t frs st = + decodeFuel-frames ps t frs st (length (frames ps ++ t)) (frames-longer ps t) + +decode-clean : ∀ ps → All Frameable ps → decode (frames ps) ≡ (ps , []) +decode-clean ps frs = + trans (cong decode (sym (++-identityʳ (frames ps)))) (decode-frames ps [] frs step-[]) + +-- Re-encoding what was accepted gives a clean journal, whatever the input. +decode-normal : ∀ bs → decode (frames (proj₁ (decode bs))) ≡ (proj₁ (decode bs) , []) +decode-normal bs = decode-clean (proj₁ (decode bs)) (decode-frameable bs) + +------------------------------------------------------------------------ +-- What the rejected tail is + +-- Why the decoder stopped. `clean`: nothing is left. `torn`: the input ends +-- inside a record, and its header, if all of it is there, is canonical. +-- `damaged`: a whole header that is not canonical, or a whole record. The +-- last is damage when the decoder rejected that record, which is the only +-- case `classify` is meant for: the bytes left over by `decode`. +data Tail : Set where + clean torn damaged : Tail + +clean≢damaged : clean ≢ damaged +clean≢damaged () + +torn≢damaged : torn ≢ damaged +torn≢damaged () + +-- The part after the header: is all of it there? +classifyBody : Maybe (List Byte × List Byte) → Tail +classifyBody nothing = torn +classifyBody (just _) = damaged + +whenCanon : ∀ {P : Set} → Dec P → Tail → Tail +whenCanon (yes _) t = t +whenCanon (no _) _ = damaged + +whenCanon-yes : ∀ {P : Set} (d : Dec P) → P → ∀ t → whenCanon d t ≡ t +whenCanon-yes (yes _) _ t = refl +whenCanon-yes (no ¬p) p t = ⊥-elim (¬p p) + +classifyHeader : Maybe (List Byte × List Byte) → Tail +classifyHeader nothing = torn +classifyHeader (just (h , r)) = + whenCanon (hdr (unhdr h) ≟ᴸ h) (classifyBody (splitExact (unhdr h + c) r)) + +classify : List Byte → Tail +classify [] = clean +classify (b ∷ bs) = classifyHeader (splitExact (suc w) (b ∷ bs)) + +classify-∷ : ∀ t → t ≢ [] → classify t ≡ classifyHeader (splitExact (suc w) t) +classify-∷ [] t≢ = ⊥-elim (t≢ refl) +classify-∷ (_ ∷ _) _ = refl + +-- A list cannot give as many bytes as a longer list has. +splitExact-short : ∀ xs y ys → splitExact (length (xs ++ y ∷ ys)) xs ≡ nothing +splitExact-short [] y ys = refl +splitExact-short (x ∷ xs) y ys rewrite splitExact-short xs y ys = refl + +-- A whole header written by `frame`, followed by a body cut short, is torn. +torn-body : ∀ p v x u → Frameable p → p ++ sum p ≡ v ++ x ∷ u → + classifyHeader (splitExact (suc w) (hdr (length p) ++ v)) ≡ torn +torn-body p v x u fr body = + trans (cong classifyHeader (splitExact-len H v (hdr-length (length p)))) + (trans (whenCanon-yes (hdr (unhdr H) ≟ᴸ H) (cong hdr fr) + (classifyBody (splitExact (unhdr H + c) v))) + (cong classifyBody + (trans (cong (λ n → splitExact n v) len≡) (splitExact-short v x u)))) + where + H = hdr (length p) + + len≡ : unhdr H + c ≡ length (v ++ x ∷ u) + len≡ = trans (cong (_+ c) fr) + (trans (cong (length p +_) (sym (sum-length p))) + (trans (sym (length-++ p)) (cong length body))) + +-- An interrupted append is never reported as damage: a frame cut short, +-- with at least one byte kept, is torn. +classify-torn : ∀ p t u → Frameable p → t ≢ [] → u ≢ [] → t ++ u ≡ frame p → + classify t ≡ torn +classify-torn p t [] fr t≢ u≢ eq = ⊥-elim (u≢ refl) +classify-torn p t (x ∷ u) fr t≢ u≢ eq with ++-split (hdr (length p)) t eq +-- The cut is at the end of the header. +... | inj₁ ([] , H≡ , u≡) = + trans (classify-∷ t t≢) + (trans (cong (λ s → classifyHeader (splitExact (suc w) s)) + (trans (sym (++-identityʳ t)) + (trans (sym H≡) (sym (++-identityʳ (hdr (length p))))))) + (torn-body p [] x u fr (sym u≡))) +-- The cut is inside the header. +... | inj₁ (y ∷ v , H≡ , _) = + trans (classify-∷ t t≢) + (cong classifyHeader + (trans (cong (λ n → splitExact n t) + (trans (sym (hdr-length (length p))) (cong length H≡))) + (splitExact-short t y v))) +-- The cut is after the header. +... | inj₂ (v , t≡ , body) = + trans (classify-∷ t t≢) + (trans (cong (λ s → classifyHeader (splitExact (suc w) s)) t≡) + (torn-body p v x u fr body)) + +-- The same without asking that a byte was kept: a frame cut short at any +-- point is not classified as damage. +classify-cut : ∀ p t u → Frameable p → u ≢ [] → t ++ u ≡ frame p → + classify t ≢ damaged +classify-cut p [] u fr u≢ eq = clean≢damaged +classify-cut p (b ∷ t) u fr u≢ eq = + subst (_≢ damaged) (sym (classify-torn p (b ∷ t) u fr (λ ()) u≢ eq)) torn≢damaged + +------------------------------------------------------------------------ +-- A prefix of a run of frames + +-- Any prefix of `frames qs` is some whole frames of `qs` followed by a tail +-- that the decoder rejects (an empty tail, or a frame cut short), and that +-- tail is not classified as damage. +frames-split⁺ : ∀ qs → All Frameable qs → ∀ t u → t ++ u ≡ frames qs → + ∃ λ qs₁ → ∃ λ qs₂ → ∃ λ t′ → + (qs₁ ++ qs₂ ≡ qs) × (t ≡ frames qs₁ ++ t′) × (step t′ ≡ nothing) + × (classify t′ ≢ damaged) +frames-split⁺ [] [] t u eq = + [] , [] , [] , refl , ++-conicalˡ t u eq , step-[] , clean≢damaged +frames-split⁺ (q ∷ qs) (fr ∷ frs) t u eq with ++-split (frame q) t eq +... | inj₁ ([] , fq≡ , _) = + (q ∷ []) , qs , [] , refl + , trans (sym (++-identityʳ t)) + (trans (sym fq≡) + (sym (trans (++-identityʳ (frame q ++ [])) (++-identityʳ (frame q))))) + , step-[] , clean≢damaged +... | inj₁ (x ∷ v , fq≡ , _) = + [] , (q ∷ qs) , t , refl , refl , step-torn q t (x ∷ v) fr (λ ()) (sym fq≡) + , classify-cut q t (x ∷ v) fr (λ ()) (sym fq≡) +... | inj₂ (v , t≡ , fqs≡) with frames-split⁺ qs frs v u (sym fqs≡) +... | qs₁ , qs₂ , t′ , split , v≡ , st , cl = + (q ∷ qs₁) , qs₂ , t′ , cong (q ∷_) split + , trans t≡ (trans (cong (frame q ++_) v≡) (sym (++-assoc (frame q) (frames qs₁) t′))) + , st , cl + +frames-split : ∀ qs → All Frameable qs → ∀ t u → t ++ u ≡ frames qs → + ∃ λ qs₁ → ∃ λ qs₂ → ∃ λ t′ → + (qs₁ ++ qs₂ ≡ qs) × (t ≡ frames qs₁ ++ t′) × (step t′ ≡ nothing) +frames-split qs frs t u eq with frames-split⁺ qs frs t u eq +... | qs₁ , qs₂ , t′ , split , t≡ , st , _ = qs₁ , qs₂ , t′ , split , t≡ , st + +-- Crash recovery at the level of records. `es` are the synced records and +-- `qs` the unsynced ones; the surviving file is the synced frames followed +-- by any prefix `t` of the unsynced frames. The decoder returns `es` +-- followed by a prefix `qs₁` of `qs`, and the tail it rejects is not +-- classified as damage. +decode-crash⁺ : ∀ es qs → All Frameable es → All Frameable qs → + ∀ t u → t ++ u ≡ frames qs → + ∃ λ qs₁ → ∃ λ qs₂ → ∃ λ t′ → + (qs₁ ++ qs₂ ≡ qs) × (decode (frames es ++ t) ≡ (es ++ qs₁ , t′)) + × (classify t′ ≢ damaged) +decode-crash⁺ es qs fes fqs t u eq with frames-split⁺ qs fqs t u eq +... | qs₁ , qs₂ , t′ , split , t≡ , st , cl = + qs₁ , qs₂ , t′ , split + , trans (cong decode bytes≡) + (decode-frames (es ++ qs₁) t′ + (All.++⁺ fes (All.++⁻ˡ qs₁ (subst (All Frameable) (sym split) fqs))) st) + , cl + where + bytes≡ : frames es ++ t ≡ frames (es ++ qs₁) ++ t′ + bytes≡ = trans (cong (frames es ++_) t≡) + (trans (sym (++-assoc (frames es) (frames qs₁) t′)) + (cong (_++ t′) (sym (frames-++ es qs₁)))) + +decode-crash : ∀ es qs → All Frameable es → All Frameable qs → + ∀ t u → t ++ u ≡ frames qs → + ∃ λ qs₁ → ∃ λ qs₂ → ∃ λ t′ → + (qs₁ ++ qs₂ ≡ qs) × (decode (frames es ++ t) ≡ (es ++ qs₁ , t′)) +decode-crash es qs fes fqs t u eq with decode-crash⁺ es qs fes fqs t u eq +... | qs₁ , qs₂ , t′ , split , dec≡ , _ = qs₁ , qs₂ , t′ , split , dec≡ + +-- After a crash, what the decoder rejects is never classified as damage. +-- A reader that refuses a damaged journal is therefore never stopped by a +-- crash, as long as the crash keeps a prefix of what was appended. +decode-crash-tail : ∀ es qs → All Frameable es → All Frameable qs → + ∀ t u → t ++ u ≡ frames qs → + classify (proj₂ (decode (frames es ++ t))) ≢ damaged +decode-crash-tail es qs fes fqs t u eq with decode-crash⁺ es qs fes fqs t u eq +... | qs₁ , qs₂ , t′ , split , dec≡ , cl = + subst (λ d → classify (proj₂ d) ≢ damaged) (sym dec≡) cl diff --git a/proofs/agda/JournalModel.agda b/proofs/agda/JournalModel.agda new file mode 100644 index 00000000..4334996a --- /dev/null +++ b/proofs/agda/JournalModel.agda @@ -0,0 +1,248 @@ +{-# OPTIONS --safe --without-K --cubical-compatible #-} +-- SPDX-License-Identifier: MPL-2.0 +{- Valence Shell - journal model: storage, crashes and recovery (Agda) + + This module joins the record framing (`JournalFraming`) and the event + replay (`JournalReplay`) through a model of the file the journal lives + in, and proves what a restart recovers after a crash. + + The file model. A file is the bytes known to be on disk (`durable`) and + the bytes appended since the last sync (`pending`). `sync` moves pending + to durable. A crash leaves the durable bytes followed by *some prefix* of + the pending bytes (`Crash`). + + Main results, for a journal whose synced events are `es` and whose + appended, not yet synced events are `qs`: + + * `crash-recovery` after any crash, loading the file succeeds and yields + the replay of `es` followed by a prefix of `qs`; + * `crash-state` so the recovered state is one the shell actually + passed through after its last sync; + * `crash-tail` and the bytes the loader does not accept are never + classified as damage, so a loader that refuses a + damaged journal is not stopped by a crash; + * `synced-survives` with nothing pending, the recovered state is exactly + the replay of `es`; + * `write-atomic` with one event pending, the recovered state is the + one before that event or the one after it; + * `compaction-safe` replacing the file by one snapshot record, when the + replacement is all-or-nothing, never changes the + recovered state; + * `append-shape`, `sync-shape` appending a record then syncing keeps the + file in the shape these theorems are about. + + What this does NOT prove. These are statements about the model: + + * The crash model is an assumption about the storage: synced bytes + survive, and the unsynced part survives as a prefix. It does not cover + reordered or partly written blocks, or stale bytes in the unsynced + region. For arbitrary file contents only `JournalFraming.decode-sound` + and `decode-normal` apply: what is accepted is byte for byte a run of + well-formed frames at the front of the file. Bytes that were never + written as a frame but carry a matching checksum would be accepted. + * `compaction-safe` assumes the replacement is all-or-nothing (write a + new file, sync it, rename it over the old one, sync the directory). + * The Rust code is not derived from this model. The link is the storage + interface in the Rust code and the crash-point harness + (`impl/rust-cli/tests/journal_crash_tests.rs`), which is test evidence. + + `enc`/`dec` are the event serialisation (JSON in the Rust code); the only + property used is that decoding an encoded event gives it back. + + Added 2026-11-17 for route-plan Phase C (journal redesign), step C1. +-} + +open import Data.Nat.Base using (ℕ; suc) +open import Data.List.Base using (List; []; _∷_; _++_; length; map) +open import Data.Maybe.Base using (Maybe; just; nothing) +open import Relation.Binary.Definitions using (DecidableEquality) +open import Relation.Binary.PropositionalEquality + using (_≡_; _≢_; refl; sym; trans; cong; cong₂; subst) + +import JournalFraming +import JournalReplay + +module JournalModel + -- framing + (Byte : Set) + (_≟_ : DecidableEquality Byte) + (w c : ℕ) + (hdr : ℕ → List Byte) + (hdr-length : ∀ n → length (hdr n) ≡ suc w) + (unhdr : List Byte → ℕ) + (sum : List Byte → List Byte) + (sum-length : ∀ p → length (sum p) ≡ c) + -- events + (Op : Set) (opId : Op → ℕ) (Aux : Set) + -- serialisation + (enc : JournalReplay.Event Op opId Aux → List Byte) + (dec : List Byte → Maybe (JournalReplay.Event Op opId Aux)) + (dec-enc : ∀ e → dec (enc e) ≡ just e) + where + +open JournalFraming Byte _≟_ w c hdr hdr-length unhdr sum sum-length public +open JournalReplay Op opId Aux public + +import Data.Maybe.Base as Maybe +open import Data.Product.Base using (_×_; _,_; proj₁; proj₂; ∃) +open import Data.Sum.Base using (_⊎_; inj₁; inj₂) +open import Data.List.Properties + using (++-identityʳ; ++-conicalˡ; map-++; ∷-injective) +open import Data.List.Relation.Unary.All using (All; []; _∷_) + +------------------------------------------------------------------------ +-- The file + +record File : Set where + constructor file + field + durable : List Byte -- on disk + pending : List Byte -- appended since the last sync + +open File public + +append : List Byte → File → File +append bs f = file (durable f) (pending f ++ bs) + +sync : File → File +sync f = file (durable f ++ pending f) [] + +-- What a crash can leave: the durable bytes and a prefix of the pending +-- bytes. +data Crash (f : File) : List Byte → Set where + crashed : ∀ t u → pending f ≡ t ++ u → Crash f (durable f ++ t) + +------------------------------------------------------------------------ +-- The journal in the file + +log : List Event → List Byte +log es = frames (map enc es) + +-- The events can be framed (each serialised event fits the length field). +Fits : List Event → Set +Fits es = All Frameable (map enc es) + +journal : List Event → List Event → File +journal es qs = file (log es) (log qs) + +log-++ : ∀ es qs → log (es ++ qs) ≡ log es ++ log qs +log-++ es qs = trans (cong frames (map-++ enc es qs)) (frames-++ (map enc es) (map enc qs)) + +-- Writing one event appends its frame. +append-shape : ∀ es qs e → + append (frame (enc e)) (journal es qs) ≡ journal es (qs ++ e ∷ []) +append-shape es qs e = + cong (file (log es)) + (sym (trans (log-++ qs (e ∷ [])) + (cong (log qs ++_) (++-identityʳ (frame (enc e)))))) + +sync-shape : ∀ es qs → sync (journal es qs) ≡ journal (es ++ qs) [] +sync-shape es qs = cong (λ d → file d []) (sym (log-++ es qs)) + +------------------------------------------------------------------------ +-- Loading + +consM : ∀ {A : Set} → Maybe A → Maybe (List A) → Maybe (List A) +consM (just x) (just xs) = just (x ∷ xs) +consM (just _) nothing = nothing +consM nothing _ = nothing + +-- Every accepted record must parse; one that does not is an error, never +-- skipped. +decodeAll : List Payload → Maybe (List Event) +decodeAll [] = just [] +decodeAll (p ∷ ps) = consM (dec p) (decodeAll ps) + +decodeAll-enc : ∀ es → decodeAll (map enc es) ≡ just es +decodeAll-enc [] = refl +decodeAll-enc (e ∷ es) rewrite dec-enc e | decodeAll-enc es = refl + +-- The state recovered from a file's bytes, or `nothing` if a record does +-- not parse. +load : List Byte → Maybe State +load bs = Maybe.map replay (decodeAll (proj₁ (decode bs))) + +-- The bytes the loader did not accept. The writer truncates the file to the +-- accepted part before appending again (`JournalFraming.decode-normal`). +discarded : List Byte → List Byte +discarded bs = proj₂ (decode bs) + +------------------------------------------------------------------------ +-- Crash recovery + +map-split : ∀ (ps₁ : List Payload) {ps₂} (xs : List Event) → ps₁ ++ ps₂ ≡ map enc xs → + ∃ λ xs₁ → ∃ λ xs₂ → (xs₁ ++ xs₂ ≡ xs) × (ps₁ ≡ map enc xs₁) +map-split [] xs eq = [] , xs , refl , refl +map-split (p ∷ ps₁) (x ∷ xs) eq with ∷-injective eq +... | p≡ , eq′ with map-split ps₁ xs eq′ +... | xs₁ , xs₂ , split , ps₁≡ = + x ∷ xs₁ , xs₂ , cong (x ∷_) split , cong₂ _∷_ p≡ ps₁≡ + +crash-recovery : ∀ es qs → Fits es → Fits qs → + ∀ {bs} → Crash (journal es qs) bs → + ∃ λ qs₁ → ∃ λ qs₂ → (qs₁ ++ qs₂ ≡ qs) × (load bs ≡ just (replay (es ++ qs₁))) +crash-recovery es qs fes fqs (crashed t u pend) + with decode-crash (map enc es) (map enc qs) fes fqs t u (sym pend) +... | ps₁ , ps₂ , t′ , psplit , dec≡ with map-split ps₁ qs psplit +... | qs₁ , qs₂ , split , ps₁≡ = + qs₁ , qs₂ , split + , trans (cong (λ d → Maybe.map replay (decodeAll (proj₁ d))) dec≡) + (trans (cong (λ ps → Maybe.map replay (decodeAll (map enc es ++ ps))) ps₁≡) + (trans (cong (λ ps → Maybe.map replay (decodeAll ps)) (sym (map-++ enc es qs₁))) + (cong (Maybe.map replay) (decodeAll-enc (es ++ qs₁))))) + +-- The recovered state is one the shell passed through after its last sync. +crash-state : ∀ es qs → Fits es → Fits qs → + ∀ {bs} → Crash (journal es qs) bs → + ∃ λ qs₁ → ∃ λ qs₂ → (qs₁ ++ qs₂ ≡ qs) × (load bs ≡ just (run (replay es) qs₁)) +crash-state es qs fes fqs cr with crash-recovery es qs fes fqs cr +... | qs₁ , qs₂ , split , ld = + qs₁ , qs₂ , split , trans ld (cong just (replay-++ es qs₁)) + +-- What the loader does not accept after a crash is empty or a record cut +-- short. It is never classified as damage, so a loader that refuses a +-- damaged journal still starts after any crash. +crash-tail : ∀ es qs → Fits es → Fits qs → + ∀ {bs} → Crash (journal es qs) bs → classify (discarded bs) ≢ damaged +crash-tail es qs fes fqs (crashed t u pend) = + decode-crash-tail (map enc es) (map enc qs) fes fqs t u (sym pend) + +-- With nothing pending, a crash loses nothing. +synced-survives : ∀ es → Fits es → + ∀ {bs} → Crash (journal es []) bs → load bs ≡ just (replay es) +synced-survives es fes cr with crash-recovery es [] fes [] cr +... | qs₁ , qs₂ , split , ld = + trans ld (cong (λ q → just (replay q)) + (trans (cong (es ++_) (++-conicalˡ qs₁ qs₂ split)) (++-identityʳ es))) + +-- With one event pending, the recovered state is the one before it or the +-- one after it. +write-atomic : ∀ es q → Fits es → Fits (q ∷ []) → + ∀ {bs} → Crash (journal es (q ∷ [])) bs → + (load bs ≡ just (replay es)) ⊎ (load bs ≡ just (replay (es ++ q ∷ []))) +write-atomic es q fes fq cr with crash-recovery es (q ∷ []) fes fq cr +... | [] , _ , _ , ld = + inj₁ (trans ld (cong (λ x → just (replay x)) (++-identityʳ es))) +... | x ∷ qs₁ , qs₂ , split , ld with ∷-injective split +... | x≡q , rest = + inj₂ (trans ld (cong (λ x → just (replay (es ++ x))) + (cong₂ _∷_ x≡q (++-conicalˡ qs₁ qs₂ rest)))) + +------------------------------------------------------------------------ +-- Compaction + +-- Compaction writes a new file holding one snapshot of the current state +-- and puts it in place of the old one. If a crash leaves either the old +-- file (as any crash of it) or the complete new file, the recovered state +-- is the same. +compacted : List Event → List Byte +compacted es = log (snapshot (replay es) ∷ []) + +compaction-safe : ∀ es → Fits es → Fits (snapshot (replay es) ∷ []) → + ∀ {bs} → Crash (journal es []) bs ⊎ (bs ≡ compacted es) → + load bs ≡ just (replay es) +compaction-safe es fes fsnap (inj₁ cr) = synced-survives es fes cr +compaction-safe es fes fsnap (inj₂ refl) = + trans (cong (λ d → Maybe.map replay (decodeAll (proj₁ d))) + (decode-clean (map enc (snapshot (replay es) ∷ [])) fsnap)) + (cong (Maybe.map replay) (decodeAll-enc (snapshot (replay es) ∷ []))) diff --git a/proofs/agda/JournalReplay.agda b/proofs/agda/JournalReplay.agda new file mode 100644 index 00000000..bae45865 --- /dev/null +++ b/proofs/agda/JournalReplay.agda @@ -0,0 +1,275 @@ +{-# OPTIONS --safe --without-K --cubical-compatible #-} +-- SPDX-License-Identifier: MPL-2.0 +{- Valence Shell - journal events and replay (Agda) + + The shell's persisted state is rebuilt by replaying a log of events. This + module defines the state, the events, and `replay`, and proves: + + * `replay-++` replaying a log then more events is replaying the + longer log; + * `run-ids` events other than a snapshot or a trim never + remove or reorder history: the operation ids + before are a prefix of the operation ids after; + * `rollback-closes` after `rolledBack` no transaction is open; + * `rollback-no-live` after `rolledBack` no operation of the rolled-back + transaction is still live (not undone); + * `recorded-joins` an operation recorded inside a transaction belongs + to it; + * `snapshot-replay` a log consisting of one snapshot of a state replays + to that state (this is what compaction writes). + + Modelling choices, stated so the Rust code can be held to them: + + * The history is append-only. Undo, redo and rollback never delete an + entry; undo and rollback mark it, and redo records a new operation. + * Everything that must survive a restart is an event in the one log, + including the open transaction and the auxiliary state (`Aux`, for + example the previous directory). There is no second file. + * Rollback marks every operation of the open transaction as undone by the + transaction. On `main` as of 2026-11-17 the Rust code does not do this + (`begin; touch f; rollback; undo` fails); the code is to follow the + model. + * The history size cap is part of the record event: `recorded n o` + appends `o` and drops the `n` oldest entries, as one event, so a crash + cannot keep the new operation without the trim it caused. With `n = 0` + it only appends. An event that drops entries is not `Incremental`. + + `Op` is the operation record (type, path, undo data); only its id matters + here. + + Added 2026-11-17 for route-plan Phase C (journal redesign), step C1. +-} + +open import Data.Nat.Base using (ℕ; zero; suc) + +module JournalReplay (Op : Set) (opId : Op → ℕ) (Aux : Set) where + +open import Data.Empty using (⊥) +open import Data.Unit.Base using (⊤; tt) +open import Data.Maybe.Base using (Maybe; just; nothing) +open import Data.Product.Base using (_×_; _,_; proj₁; proj₂; ∃) +open import Data.Nat.Properties using () renaming (_≟_ to _≟ℕ_) +open import Data.List.Base using (List; []; _∷_; _++_; map; foldl; drop) +open import Data.List.Properties using (++-assoc; ++-identityʳ; map-++) +open import Data.List.Membership.Propositional using (_∈_) +open import Data.List.Membership.DecPropositional _≟ℕ_ using (_∈?_) +open import Data.List.Relation.Unary.Any using (here) +open import Data.List.Relation.Unary.All using (All; []; _∷_) +open import Relation.Nullary using (Dec; yes; no; ¬_) +open import Relation.Binary.PropositionalEquality + using (_≡_; refl; sym; trans; cong; subst) + +------------------------------------------------------------------------ +-- State + +record Entry : Set where + constructor entry + field + op : Op + undoneBy : Maybe ℕ -- id of the undo operation or transaction + +open Entry public + +Live : Entry → Set +Live e = undoneBy e ≡ nothing + +record Txn : Set where + constructor txn + field + txnId : ℕ + ops : List ℕ -- ids of the operations recorded inside it + +open Txn public + +record State : Set where + constructor state + field + history : List Entry + active : Maybe Txn -- the open transaction, if any + closed : List ℕ -- ids of committed transactions + aux : Maybe Aux + +open State public + +empty : State +empty = state [] nothing [] nothing + +ids : List Entry → List ℕ +ids = map (λ e → opId (op e)) + +------------------------------------------------------------------------ +-- Events + +data Event : Set where + recorded : ℕ → Op → Event -- an operation was performed; the + -- ℕ oldest entries are then dropped + undone : ℕ → ℕ → Event -- target id, id of the undo operation + begun : ℕ → Event -- a transaction was opened + committed : Event + rolledBack : Event + auxSet : Maybe Aux → Event + snapshot : State → Event -- the whole state (written by compaction) + +-- Events that only ever extend the history. +Incremental : Event → Set +Incremental (recorded zero _) = ⊤ +Incremental (recorded (suc _) _) = ⊥ +Incremental (undone _ _) = ⊤ +Incremental (begun _) = ⊤ +Incremental committed = ⊤ +Incremental rolledBack = ⊤ +Incremental (auxSet _) = ⊤ +Incremental (snapshot _) = ⊥ + +-- Mark an entry as undone when `d` holds. An entry already undone keeps its +-- first mark. +firstMark : ℕ → Maybe ℕ → Maybe ℕ +firstMark by nothing = just by +firstMark by (just b) = just b + +markWhen : ∀ {P : Set} → Dec P → ℕ → Entry → Entry +markWhen (yes _) by e = entry (op e) (firstMark by (undoneBy e)) +markWhen (no _) by e = e + +markId : ℕ → ℕ → Entry → Entry +markId target by e = markWhen (opId (op e) ≟ℕ target) by e + +markIn : List ℕ → ℕ → Entry → Entry +markIn targets by e = markWhen (opId (op e) ∈? targets) by e + +joinTxn : ℕ → Maybe Txn → Maybe Txn +joinTxn i nothing = nothing +joinTxn i (just (txn t os)) = just (txn t (i ∷ os)) + +beginTxn : ℕ → Maybe Txn → Maybe Txn +beginTxn t nothing = just (txn t []) +beginTxn t (just x) = just x -- already open: the event is ignored + +commitTxn : Maybe Txn → List ℕ → List ℕ +commitTxn nothing cs = cs +commitTxn (just (txn t _)) cs = t ∷ cs + +rollbackHistory : Maybe Txn → List Entry → List Entry +rollbackHistory nothing h = h +rollbackHistory (just (txn t os)) h = map (markIn os t) h + +apply : Event → State → State +apply (recorded n o) s = record s { history = drop n (history s ++ entry o nothing ∷ []) + ; active = joinTxn (opId o) (active s) } +apply (undone i by) s = record s { history = map (markId i by) (history s) } +apply (begun t) s = record s { active = beginTxn t (active s) } +apply committed s = record s { active = nothing + ; closed = commitTxn (active s) (closed s) } +apply rolledBack s = record s { history = rollbackHistory (active s) (history s) + ; active = nothing } +apply (auxSet a) s = record s { aux = a } +apply (snapshot s′) _ = s′ + +run : State → List Event → State +run = foldl (λ s e → apply e s) + +replay : List Event → State +replay = run empty + +------------------------------------------------------------------------ +-- Replay of a longer log + +run-++ : ∀ s es fs → run s (es ++ fs) ≡ run (run s es) fs +run-++ s [] fs = refl +run-++ s (e ∷ es) fs = run-++ (apply e s) es fs + +replay-++ : ∀ es fs → replay (es ++ fs) ≡ run (replay es) fs +replay-++ = run-++ empty + +------------------------------------------------------------------------ +-- History is append-only + +markWhen-id : ∀ {P : Set} (d : Dec P) by e → opId (op (markWhen d by e)) ≡ opId (op e) +markWhen-id (yes _) by e = refl +markWhen-id (no _) by e = refl + +ids-map : ∀ (f : Entry → Entry) → (∀ e → opId (op (f e)) ≡ opId (op e)) → + ∀ h → ids (map f h) ≡ ids h +ids-map f same [] = refl +ids-map f same (e ∷ h) rewrite same e | ids-map f same h = refl + +-- `xs ⊑ ys`: `xs` is a prefix of `ys`. +_⊑_ : List ℕ → List ℕ → Set +xs ⊑ ys = ∃ λ zs → xs ++ zs ≡ ys + +⊑-refl : ∀ {xs} → xs ⊑ xs +⊑-refl {xs} = [] , ++-identityʳ xs + +⊑-reflexive : ∀ {xs ys} → xs ≡ ys → xs ⊑ ys +⊑-reflexive {xs} eq = [] , trans (++-identityʳ xs) eq + +⊑-trans : ∀ {xs ys zs} → xs ⊑ ys → ys ⊑ zs → xs ⊑ zs +⊑-trans {xs} (as , refl) (bs , refl) = as ++ bs , sym (++-assoc xs as bs) + +rollbackHistory-ids : ∀ t h → ids (rollbackHistory t h) ≡ ids h +rollbackHistory-ids nothing h = refl +rollbackHistory-ids (just (txn t os)) h = + ids-map (markIn os t) (λ e → markWhen-id (opId (op e) ∈? os) t e) h + +apply-ids : ∀ e s → Incremental e → ids (history s) ⊑ ids (history (apply e s)) +apply-ids (recorded zero o) s _ = + opId o ∷ [] , sym (map-++ (λ e → opId (op e)) (history s) (entry o nothing ∷ [])) +apply-ids (undone i by) s _ = + ⊑-reflexive (sym (ids-map (markId i by) + (λ e → markWhen-id (opId (op e) ≟ℕ i) by e) (history s))) +apply-ids (begun t) s _ = ⊑-refl +apply-ids committed s _ = ⊑-refl +apply-ids rolledBack s _ = + ⊑-reflexive (sym (rollbackHistory-ids (active s) (history s))) +apply-ids (auxSet a) s _ = ⊑-refl + +-- Incremental events only ever extend the sequence of operation +-- ids: replaying a longer log yields a history that the shorter log's +-- history is a prefix of. +run-ids : ∀ s fs → All Incremental fs → ids (history s) ⊑ ids (history (run s fs)) +run-ids s [] [] = ⊑-refl +run-ids s (e ∷ fs) (inc ∷ incs) = + ⊑-trans (apply-ids e s inc) (run-ids (apply e s) fs incs) + +------------------------------------------------------------------------ +-- Transactions + +recorded-joins : ∀ n o s t os → active s ≡ just (txn t os) → + ∃ λ os′ → active (apply (recorded n o) s) ≡ just (txn t os′) × opId o ∈ os′ +recorded-joins n o s t os eq rewrite eq = opId o ∷ os , refl , here refl + +rollback-closes : ∀ s → active (apply rolledBack s) ≡ nothing +rollback-closes s = refl + +firstMark-just : ∀ by m → ¬ (firstMark by m ≡ nothing) +firstMark-just by nothing () +firstMark-just by (just _) () + +markWhen-live : ∀ {P : Set} (d : Dec P) by e → Live (markWhen d by e) → ¬ P +markWhen-live (yes _) by e live _ = firstMark-just by (undoneBy e) live +markWhen-live (no ¬p) by e _ = ¬p + +markIn-no-live : ∀ os t h → + All (λ e → Live e → ¬ (opId (op e) ∈ os)) (map (markIn os t) h) +markIn-no-live os t [] = [] +markIn-no-live os t (e ∷ h) = + (λ live → subst (λ i → ¬ (i ∈ os)) + (sym (markWhen-id (opId (op e) ∈? os) t e)) + (markWhen-live (opId (op e) ∈? os) t e live)) + ∷ markIn-no-live os t h + +-- After a rollback, no operation of the rolled-back transaction is live, so +-- a later `undo` cannot select one. +rollback-no-live : ∀ s t os → active s ≡ just (txn t os) → + All (λ e → Live e → ¬ (opId (op e) ∈ os)) (history (apply rolledBack s)) +rollback-no-live s t os eq rewrite eq = markIn-no-live os t (history s) + +------------------------------------------------------------------------ +-- Compaction + +-- Compaction replaces the log with one snapshot of the current state. +snapshot-replay : ∀ s → replay (snapshot s ∷ []) ≡ s +snapshot-replay s = refl + +snapshot-then : ∀ s fs → replay (snapshot s ∷ fs) ≡ run s fs +snapshot-then s fs = refl diff --git a/proofs/agda/JournalWitnesses.agda b/proofs/agda/JournalWitnesses.agda new file mode 100644 index 00000000..0cfc85bb --- /dev/null +++ b/proofs/agda/JournalWitnesses.agda @@ -0,0 +1,416 @@ +{-# OPTIONS --safe --without-K --cubical-compatible #-} +-- SPDX-License-Identifier: MPL-2.0 +{- Valence Shell - journal model: an instance and worked cases (Agda) + + `JournalModel` is stated over parameters (a byte type, a header, a + checksum, an event serialisation). If those parameters could not all be + met at once, its theorems would say nothing. This module builds one + instance and runs the definitions on concrete data, so that: + + * the parameters are shown to be jointly satisfiable (`dec-enc` included: + a serialisation of every event, snapshots too, with a proved round + trip); + * the decoder and loader are shown to compute (each case below is closed + by `refl`); + * the hypotheses of the crash theorems are shown to be inhabited (each + theorem is applied to a concrete crash). + + The instance is deliberately small: a "byte" is a natural number, the + header is one cell holding the payload length, and the checksum is one + cell holding the sum of the payload. It is not the Rust format. + + A second framing instance (`C`) has a header of two cells: the length, + then a check of the length. It shows what the decoder's canonical-header + test does once the header carries a check, which is the shape of the Rust + format: a changed length is damage, even when the changed length runs + past the end of the input. + + Added 2026-11-17 for route-plan Phase C (journal redesign), step C1. +-} + +module JournalWitnesses where + +open import Data.Nat.Base using (ℕ; zero; suc; _+_) +open import Data.Nat.Properties using (_≟_) +open import Data.List.Base using (List; []; _∷_; _++_; length; foldr) +open import Data.List.Properties using (++-assoc; ++-identityʳ) +open import Data.List.Relation.Unary.All using (All; universal) +open import Data.Maybe.Base using (Maybe; just; nothing; _>>=_) +import Data.Maybe.Base as Maybe +open import Data.Product.Base using (_×_; _,_; proj₁; proj₂; ∃) +import Data.Product.Base as Prod +open import Data.Sum.Base using (_⊎_; inj₂) +open import Relation.Binary.PropositionalEquality + using (_≡_; _≢_; refl; sym; trans; cong; subst) + +import JournalFraming +import JournalReplay +import JournalModel + +open JournalReplay ℕ (λ o → o) ℕ + +------------------------------------------------------------------------ +-- Framing parameters + +hdr : ℕ → List ℕ +hdr n = n ∷ [] + +unhdr : List ℕ → ℕ +unhdr [] = 0 +unhdr (n ∷ _) = n + +chk : List ℕ → List ℕ +chk p = foldr _+_ 0 p ∷ [] + +------------------------------------------------------------------------ +-- A serialisation of events with a proved round trip + +record Codec (A : Set) : Set where + constructor codec + field + put : A → List ℕ + get : List ℕ → Maybe (A × List ℕ) + get-put : ∀ a rest → get (put a ++ rest) ≡ just (a , rest) + +open Codec + +getNat : List ℕ → Maybe (ℕ × List ℕ) +getNat [] = nothing +getNat (n ∷ r) = just (n , r) + +natC : Codec ℕ +natC = codec (λ n → n ∷ []) getNat (λ _ _ → refl) + +module _ {A B : Set} (ca : Codec A) (cb : Codec B) where + getPair : List ℕ → Maybe ((A × B) × List ℕ) + getPair bs = get ca bs >>= λ ar → get cb (proj₂ ar) >>= λ br → + just ((proj₁ ar , proj₁ br) , proj₂ br) + + get-putPair : ∀ ab rest → getPair ((put ca (proj₁ ab) ++ put cb (proj₂ ab)) ++ rest) + ≡ just (ab , rest) + get-putPair (a , b) rest + rewrite ++-assoc (put ca a) (put cb b) rest + | get-put ca a (put cb b ++ rest) + | get-put cb b rest + = refl + + pairC : Codec (A × B) + pairC = codec (λ ab → put ca (proj₁ ab) ++ put cb (proj₂ ab)) getPair get-putPair + +module _ {A : Set} (ca : Codec A) where + putMaybe : Maybe A → List ℕ + putMaybe nothing = 0 ∷ [] + putMaybe (just a) = 1 ∷ put ca a + + getMaybe : List ℕ → Maybe (Maybe A × List ℕ) + getMaybe (0 ∷ r) = just (nothing , r) + getMaybe (1 ∷ r) = Maybe.map (Prod.map₁ just) (get ca r) + getMaybe _ = nothing + + get-putMaybe : ∀ m rest → getMaybe (putMaybe m ++ rest) ≡ just (m , rest) + get-putMaybe nothing rest = refl + get-putMaybe (just a) rest rewrite get-put ca a rest = refl + + maybeC : Codec (Maybe A) + maybeC = codec putMaybe getMaybe get-putMaybe + + puts : List A → List ℕ + puts [] = [] + puts (x ∷ xs) = put ca x ++ puts xs + + gets : ℕ → List ℕ → Maybe (List A × List ℕ) + gets zero r = just ([] , r) + gets (suc n) r = get ca r >>= λ xr → gets n (proj₂ xr) >>= λ xsr → + just (proj₁ xr ∷ proj₁ xsr , proj₂ xsr) + + gets-puts : ∀ xs rest → gets (length xs) (puts xs ++ rest) ≡ just (xs , rest) + gets-puts [] rest = refl + gets-puts (x ∷ xs) rest + rewrite ++-assoc (put ca x) (puts xs) rest + | get-put ca x (puts xs ++ rest) + | gets-puts xs rest + = refl + + getList : List ℕ → Maybe (List A × List ℕ) + getList [] = nothing + getList (n ∷ r) = gets n r + + listC : Codec (List A) + listC = codec (λ xs → length xs ∷ puts xs) getList gets-puts + +module _ {A B : Set} (f : A → B) (g : B → A) (fg : ∀ b → f (g b) ≡ b) (ca : Codec A) where + get-putIso : ∀ b rest → Maybe.map (Prod.map₁ f) (get ca (put ca (g b) ++ rest)) ≡ just (b , rest) + get-putIso b rest rewrite get-put ca (g b) rest | fg b = refl + + isoC : Codec B + isoC = codec (λ b → put ca (g b)) (λ bs → Maybe.map (Prod.map₁ f) (get ca bs)) get-putIso + +entryC : Codec Entry +entryC = isoC (λ p → entry (proj₁ p) (proj₂ p)) (λ e → op e , undoneBy e) (λ _ → refl) + (pairC natC (maybeC natC)) + +txnC : Codec Txn +txnC = isoC (λ p → txn (proj₁ p) (proj₂ p)) (λ t → txnId t , ops t) (λ _ → refl) + (pairC natC (listC natC)) + +stateC : Codec State +stateC = isoC (λ p → state (proj₁ p) (proj₁ (proj₂ p)) + (proj₁ (proj₂ (proj₂ p))) (proj₂ (proj₂ (proj₂ p)))) + (λ s → history s , active s , closed s , aux s) (λ _ → refl) + (pairC (listC entryC) + (pairC (maybeC txnC) (pairC (listC natC) (maybeC natC)))) + +-- A parse that must use up its whole input. +whole : ∀ {A : Set} → Maybe (A × List ℕ) → Maybe A +whole (just (a , [])) = just a +whole (just (_ , _ ∷ _)) = nothing +whole nothing = nothing + +whole-put : ∀ {A : Set} (ca : Codec A) a → whole (get ca (put ca a)) ≡ just a +whole-put ca a = + trans (cong (λ bs → whole (get ca bs)) (sym (++-identityʳ (put ca a)))) + (cong whole (get-put ca a [])) + +enc : Event → List ℕ +enc (recorded n o) = 0 ∷ n ∷ o ∷ [] +enc (undone i by) = 1 ∷ i ∷ by ∷ [] +enc (begun t) = 2 ∷ t ∷ [] +enc committed = 3 ∷ [] +enc rolledBack = 4 ∷ [] +enc (auxSet a) = 5 ∷ put (maybeC natC) a +enc (snapshot s) = 6 ∷ put stateC s + +dec : List ℕ → Maybe Event +dec (0 ∷ n ∷ o ∷ []) = just (recorded n o) +dec (1 ∷ i ∷ by ∷ []) = just (undone i by) +dec (2 ∷ t ∷ []) = just (begun t) +dec (3 ∷ []) = just committed +dec (4 ∷ []) = just rolledBack +dec (5 ∷ r) = Maybe.map auxSet (whole (get (maybeC natC) r)) +dec (6 ∷ r) = Maybe.map snapshot (whole (get stateC r)) +dec _ = nothing + +dec-enc : ∀ e → dec (enc e) ≡ just e +dec-enc (recorded n o) = refl +dec-enc (undone i by) = refl +dec-enc (begun t) = refl +dec-enc committed = refl +dec-enc rolledBack = refl +dec-enc (auxSet a) = cong (Maybe.map auxSet) (whole-put (maybeC natC) a) +dec-enc (snapshot s) = cong (Maybe.map snapshot) (whole-put stateC s) + +------------------------------------------------------------------------ +-- The instance + +module M = JournalModel ℕ _≟_ 0 1 hdr (λ _ → refl) unhdr chk (λ _ → refl) + ℕ (λ o → o) ℕ enc dec dec-enc + +-- In this instance every payload can be framed. +fits : ∀ es → M.Fits es +fits es = universal (λ _ → refl) _ + +------------------------------------------------------------------------ +-- Framing: worked cases + +-- A frame is length, payload, sum. +frame-example : M.frame (5 ∷ 6 ∷ []) ≡ 2 ∷ 5 ∷ 6 ∷ 11 ∷ [] +frame-example = refl + +-- Two frames decode to two records, nothing left over. +decode-example : + M.decode (2 ∷ 5 ∷ 6 ∷ 11 ∷ 0 ∷ 0 ∷ []) ≡ ((5 ∷ 6 ∷ []) ∷ [] ∷ [] , []) +decode-example = refl + +-- The second frame cut short at each point: the first record is kept, and +-- the cut bytes are reported as left over. +torn-header-only : M.decode (1 ∷ 9 ∷ 9 ∷ 2 ∷ []) ≡ ((9 ∷ []) ∷ [] , 2 ∷ []) +torn-header-only = refl + +torn-in-payload : M.decode (1 ∷ 9 ∷ 9 ∷ 2 ∷ 5 ∷ []) ≡ ((9 ∷ []) ∷ [] , 2 ∷ 5 ∷ []) +torn-in-payload = refl + +torn-before-sum : M.decode (1 ∷ 9 ∷ 9 ∷ 2 ∷ 5 ∷ 6 ∷ []) ≡ ((9 ∷ []) ∷ [] , 2 ∷ 5 ∷ 6 ∷ []) +torn-before-sum = refl + +-- A wrong checksum: the record is rejected, and so is everything after it. +bad-sum : M.decode (1 ∷ 9 ∷ 9 ∷ 2 ∷ 5 ∷ 6 ∷ 12 ∷ 0 ∷ 0 ∷ []) + ≡ ((9 ∷ []) ∷ [] , 2 ∷ 5 ∷ 6 ∷ 12 ∷ 0 ∷ 0 ∷ []) +bad-sum = refl + +------------------------------------------------------------------------ +-- The rejected tail: worked cases + +-- The tails of the four cases above. A record cut short is torn; a whole +-- record with a wrong checksum is damaged. +torn-header-only-tail : M.classify (2 ∷ []) ≡ M.torn +torn-header-only-tail = refl + +torn-before-sum-tail : M.classify (2 ∷ 5 ∷ 6 ∷ []) ≡ M.torn +torn-before-sum-tail = refl + +bad-sum-tail : M.classify (2 ∷ 5 ∷ 6 ∷ 12 ∷ 0 ∷ 0 ∷ []) ≡ M.damaged +bad-sum-tail = refl + +clean-tail : M.classify [] ≡ M.clean +clean-tail = refl + +-- With a header that is the bare length, a changed length that runs past +-- the end of the input cannot be told from a record cut short. The frame +-- of 5 ∷ 6 ∷ [] with its length changed from 2 to 9: +changed-length-looks-torn : M.classify (9 ∷ 5 ∷ 6 ∷ 11 ∷ []) ≡ M.torn +changed-length-looks-torn = refl + +------------------------------------------------------------------------ +-- A header that carries a check of the length + +-- The header is the length followed by a check cell. The check here is the +-- successor of the length; the Rust format uses part of a hash. +hdrC : ℕ → List ℕ +hdrC n = n ∷ suc n ∷ [] + +module C = JournalFraming ℕ _≟_ 1 1 hdrC (λ _ → refl) unhdr chk (λ _ → refl) + +-- Every payload can be framed. +fitsC : ∀ p → C.Frameable p +fitsC p = refl + +-- A frame is length, check of the length, payload, sum. +frameC-example : C.frame (5 ∷ 6 ∷ []) ≡ 2 ∷ 3 ∷ 5 ∷ 6 ∷ 11 ∷ [] +frameC-example = refl + +decodeC-example : C.decode (2 ∷ 3 ∷ 5 ∷ 6 ∷ 11 ∷ 0 ∷ 1 ∷ 0 ∷ []) + ≡ ((5 ∷ 6 ∷ []) ∷ [] ∷ [] , []) +decodeC-example = refl + +-- The same change as above, length 2 to 9 with the check cell untouched. +-- The header is no longer canonical, so this is damage. Nothing is +-- accepted, and the record after it is not dropped as if it were torn. +changed-length-is-damage : + C.classify (9 ∷ 3 ∷ 5 ∷ 6 ∷ 11 ∷ 0 ∷ 1 ∷ 0 ∷ []) ≡ C.damaged +changed-length-is-damage = refl + +changed-length-accepts-nothing : + C.decode (9 ∷ 3 ∷ 5 ∷ 6 ∷ 11 ∷ 0 ∷ 1 ∷ 0 ∷ []) + ≡ ([] , 9 ∷ 3 ∷ 5 ∷ 6 ∷ 11 ∷ 0 ∷ 1 ∷ 0 ∷ []) +changed-length-accepts-nothing = refl + +-- A changed check cell is damage too. +changed-check-is-damage : C.classify (2 ∷ 4 ∷ 5 ∷ 6 ∷ 11 ∷ []) ≡ C.damaged +changed-check-is-damage = refl + +-- A record cut short is still torn, at every cut: inside the header, at +-- its end, inside the payload, before the sum. +cut-in-header : C.classify (2 ∷ []) ≡ C.torn +cut-in-header = refl + +cut-after-header : C.classify (2 ∷ 3 ∷ []) ≡ C.torn +cut-after-header = refl + +cut-in-payload : C.classify (2 ∷ 3 ∷ 5 ∷ []) ≡ C.torn +cut-in-payload = refl + +cut-before-sum : C.classify (2 ∷ 3 ∷ 5 ∷ 6 ∷ []) ≡ C.torn +cut-before-sum = refl + +-- The theorem applies: the hypotheses of `classify-torn` are inhabited. +classify-torn-applies : C.classify (2 ∷ 3 ∷ 5 ∷ []) ≡ C.torn +classify-torn-applies = + C.classify-torn (5 ∷ 6 ∷ []) (2 ∷ 3 ∷ 5 ∷ []) (6 ∷ 11 ∷ []) (fitsC (5 ∷ 6 ∷ [])) + (λ ()) (λ ()) refl + +-- What is not caught: damage that leaves a canonical header declaring more +-- than the input holds. Length and check cell changed together, 2 ∷ 3 to +-- 9 ∷ 10. This is the blind spot of any header check, and how unlikely it +-- is depends on the check's strength. +changed-header-with-matching-check-looks-torn : + C.classify (9 ∷ 10 ∷ 5 ∷ 6 ∷ 11 ∷ []) ≡ C.torn +changed-header-with-matching-check-looks-torn = refl + +------------------------------------------------------------------------ +-- Replay: worked cases + +-- Rollback marks the operations it reverses and closes the transaction, so +-- nothing is left for a later `undo` to select. +session : List Event +session = begun 1 ∷ recorded 0 7 ∷ recorded 0 8 ∷ rolledBack ∷ [] + +rollback-example : + replay session ≡ state (entry 7 (just 1) ∷ entry 8 (just 1) ∷ []) nothing [] nothing +rollback-example = refl + +-- Commit leaves the operations live and records the transaction. +commit-example : + replay (begun 1 ∷ recorded 0 7 ∷ committed ∷ []) + ≡ state (entry 7 nothing ∷ []) nothing (1 ∷ []) nothing +commit-example = refl + +-- An undo mark, then a redo recorded as a new operation. +undo-redo-example : + replay (recorded 0 7 ∷ undone 7 8 ∷ recorded 0 9 ∷ []) + ≡ state (entry 7 (just 8) ∷ entry 9 nothing ∷ []) nothing [] nothing +undo-redo-example = refl + +-- A record at the history cap drops the oldest entry in the same event. +trim-example : + replay (recorded 0 7 ∷ recorded 0 8 ∷ recorded 1 9 ∷ []) + ≡ state (entry 8 nothing ∷ entry 9 nothing ∷ []) nothing [] nothing +trim-example = refl + +------------------------------------------------------------------------ +-- Crashes: worked cases and applied theorems + +synced : List Event +synced = begun 1 ∷ recorded 0 7 ∷ [] + +-- The synced part of the file. +synced-bytes : M.log synced ≡ 2 ∷ 2 ∷ 1 ∷ 3 ∷ 3 ∷ 0 ∷ 0 ∷ 7 ∷ 7 ∷ [] +synced-bytes = refl + +-- `rolledBack` is appended but not yet synced; its frame is 1 ∷ 4 ∷ 4 ∷ []. +-- A crash keeps one cell of it. Loading gives the state before it. +crash-mid-record : M.load (M.log synced ++ 1 ∷ []) ≡ just (replay synced) +crash-mid-record = refl + +-- A crash keeps all of it. Loading gives the state after it. +crash-after-record : + M.load (M.log synced ++ 1 ∷ 4 ∷ 4 ∷ []) ≡ just (replay (synced ++ rolledBack ∷ [])) +crash-after-record = refl + +a-crash : M.Crash (M.journal synced (rolledBack ∷ [])) (M.log synced ++ 1 ∷ 4 ∷ []) +a-crash = M.crashed (1 ∷ 4 ∷ []) (4 ∷ []) refl + +-- The theorems apply to it: their hypotheses are inhabited. +write-atomic-applies : + (M.load (M.log synced ++ 1 ∷ 4 ∷ []) ≡ just (replay synced)) + ⊎ (M.load (M.log synced ++ 1 ∷ 4 ∷ []) ≡ just (replay (synced ++ rolledBack ∷ []))) +write-atomic-applies = + M.write-atomic synced rolledBack (fits synced) (fits (rolledBack ∷ [])) a-crash + +crash-recovery-applies : + ∃ λ qs₁ → ∃ λ qs₂ → (qs₁ ++ qs₂ ≡ rolledBack ∷ []) + × (M.load (M.log synced ++ 1 ∷ 4 ∷ []) ≡ just (replay (synced ++ qs₁))) +crash-recovery-applies = + M.crash-recovery synced (rolledBack ∷ []) (fits synced) (fits (rolledBack ∷ [])) a-crash + +-- What the loader leaves over after that crash is not classified as damage. +crash-tail-applies : + M.classify (M.discarded (M.log synced ++ 1 ∷ 4 ∷ [])) ≢ M.damaged +crash-tail-applies = + M.crash-tail synced (rolledBack ∷ []) (fits synced) (fits (rolledBack ∷ [])) a-crash + +crash-tail-example : M.classify (M.discarded (M.log synced ++ 1 ∷ 4 ∷ [])) ≡ M.torn +crash-tail-example = refl + +-- Compaction: the snapshot record alone loads to the same state. +compaction-example : M.load (M.compacted synced) ≡ just (replay synced) +compaction-example = refl + +compaction-applies : M.load (M.compacted synced) ≡ just (replay synced) +compaction-applies = + M.compaction-safe synced (fits synced) (fits (snapshot (replay synced) ∷ [])) + (inj₂ refl) + +-- A record that does not parse is an error, not a skipped line: tag 9 is no +-- event, and its frame is well formed. +unknown-record : M.load (1 ∷ 9 ∷ 9 ∷ []) ≡ nothing +unknown-record = refl diff --git a/proofs/agda/README.adoc b/proofs/agda/README.adoc index 4b425c6c..8e1f71a0 100644 --- a/proofs/agda/README.adoc +++ b/proofs/agda/README.adoc @@ -22,6 +22,10 @@ theorems (owner ruling 2026-11-09, per `standards` PROOF-POLICY §1). The Lean 4 | `SymlinkOperations.agda` | Symlink creation + traversal. | `RMOOperations.agda` | Remove-Match-Obliterate (GDPR-relevant). | `PreconditionWitnesses.agda` | One inhabitant per operation precondition (non-vacuity). +| `JournalFraming.agda` | Journal record framing and decoder: a record cut short is rejected, and whatever is accepted is byte for byte a run of well-formed records. The rejected bytes are classified as clean, torn or damaged; a record cut short is torn, never damaged. A changed record is caught only as far as its header check and its checksum can tell. +| `JournalReplay.agda` | Journal events and replay: history is append-only; rollback leaves no operation of the rolled-back transaction live. +| `JournalModel.agda` | File and crash model; recovery after a crash is the synced events plus a prefix of the unsynced ones. +| `JournalWitnesses.agda` | One instance of the journal model's parameters, and worked cases closed by `refl`. | `All.agda` | Imports every module; the file CI checks. |=== @@ -48,7 +52,7 @@ cd proofs/agda agda --safe --without-K --cubical-compatible All.agda ``` -This is the estate invocation (PROOF-POLICY §2.1). It checks all ten modules +This is the estate invocation (PROOF-POLICY §2.1). It checks all fourteen modules under Agda 2.7.0.1 + agda-stdlib v2.2; CI runs the same command (`.github/workflows/agda.yml`). See `VERIFICATION.adoc` for the receipt. diff --git a/proofs/agda/VERIFICATION.adoc b/proofs/agda/VERIFICATION.adoc index b620c7e1..ee656c45 100644 --- a/proofs/agda/VERIFICATION.adoc +++ b/proofs/agda/VERIFICATION.adoc @@ -28,6 +28,94 @@ Toolchain digest: Agda release archive sha256 installed from the upstream archive, not Guix: an open §6 `deviation`, recorded in the workflow and in `PROOF-NEEDS.adoc`. +== What was run (2026-11-17): journal model added + +[cols="3,2",options="header"] +|=== +| Invocation | Agda 2.7.0.1 + agda-stdlib v2.2 (`6cd04e46`) + +| `bash scripts/check-agda.sh` (the invocation above, plus the import-graph + check), from a clean `_build` +| exit 0; all 14 modules checked +|=== + +Four modules joined the cone: `JournalFraming`, `JournalReplay`, +`JournalModel`, `JournalWitnesses` (route-plan Phase C, step C1). They carry +the same options, no `postulate` and no escape pragma. Their assumptions are +module parameters, and `JournalWitnesses` builds one instance of all of them, +so the theorems are not vacuous. + +What they state, and for what: + +* `JournalFraming`: a record is a header that carries the length, then the + payload, then a checksum. A record cut short + at any point is rejected (`step-torn`); what the decoder accepts is byte + for byte a run of well-formed records at the front of the input + (`decode-sound`). Neither needs any property of the checksum. +* `JournalReplay`: events other than a snapshot or a trim at the history cap + never remove or reorder history (`run-ids`); after a rollback no operation of that transaction is + live (`rollback-no-live`). +* `JournalModel`: if a crash keeps the synced bytes and a prefix of the + unsynced bytes, loading yields the synced events followed by a prefix of + the unsynced events (`crash-recovery`), so one pending write is + all-or-nothing (`write-atomic`). + +Limits, which the module headers repeat: + +* These are theorems about the model. The Rust journal + (`impl/rust-cli/src/journal.rs`, `state.rs`) was rewritten the same day to + mirror it, but is not derived from it or proved against it. The link is + test evidence: the crash-point harness runs the shell on simulated storage + that implements the model's file and checks every crash image the model + allows. +* The crash model is an assumption about the storage. It does not cover + reordered or partly written blocks, or stale bytes in the unsynced region. +* A checksum cannot rule out bytes that were never written as a record but + happen to match; `decode-sound` says exactly what acceptance means. + +Mutation check (2026-11-17, local): six edits were each rejected by the type +checker. They were: a wrong left-over tail and a wrong recovered state in +the worked cases; dropping the checksum test; dropping the canonical-header +test; rollback that marks nothing; and `step-torn` without its "something +was cut off" premise. The unedited tree passed as the control. + +=== Added the same day: what the rejected bytes are + +A review of the Rust journal found that a changed length field could pass +for an interrupted write. The Rust format gained a check on the length (the +header is now the length followed by four bytes of a hash of it), and the +model gained the matching statements. `bash scripts/check-agda.sh` exits 0 +with them; the module count is unchanged at 14. + +* The model needed no new test in the decoder. It already accepts a header + `h` only if it is canonical, `hdr (unhdr h) ≡ h`. For a header that is the + bare length that holds for every input. For a header that carries a check + of the length it is the header check. +* `JournalFraming.classify` names the bytes the decoder rejects: `clean`, + `torn` (the input ends inside a record) or `damaged` (a whole header that + is not canonical, or a whole record that was still rejected). +* `classify-torn`: a record cut short, with at least one byte kept, is torn. +* `decode-crash-tail`, and `crash-tail` in `JournalModel`: after a crash that + keeps a prefix of what was appended, the rejected bytes are never + classified as damaged. A reader that refuses a damaged journal is + therefore not stopped by a crash. +* `JournalWitnesses` has a second framing instance, `C`, whose header is the + length and a check cell. There a changed length is damaged + (`changed-length-is-damage`); with the bare-length header the same change + is classified as torn (`changed-length-looks-torn`). + +Not proved, and not provable: that damage is always classified as damaged. +Damage that leaves a canonical header declaring more bytes than the input +holds is classified as torn (`changed-header-with-matching-check-looks-torn`). +How unlikely that is depends on the strength of the header check, and the +model assumes nothing about it. + +Mutation check (2026-11-17, local): four more edits were each rejected. They +were: a body cut short classified as damaged; a header cut short classified +as damaged; a header that is not canonical passed on to the body test; and +the decoder without its canonical-header test. The unedited tree passed as +the control. + == What changed to get there * *`funext` removed.* Whole-filesystem equalities are now pointwise