Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
0e96fb8
fix(state): repair the stress suite and gate it in CI again
claude Nov 17, 2026
a79a9f3
fix(state): recover from a torn journal; journal cd's previous_dir
claude Nov 17, 2026
5cf57b7
fix(state): treat invalid UTF-8 in the journal tail as a torn line
claude Nov 17, 2026
bb73e18
fix(state): repair newline-less journal tail; atomic record+trim; 060…
claude Nov 17, 2026
cd763ad
fix(state): rebuild instead of gapping when state files vanish; retry…
claude Nov 17, 2026
aa859ac
test(state): crash-point harness for shell-state persistence
claude Nov 17, 2026
87a28cc
test(state): split crash_states into one helper per crash kind
claude Nov 17, 2026
756ac81
test(state): build the crash harness's stability oracle from the reco…
claude Nov 17, 2026
13ea833
proof(agda): journal model for the state-store redesign (C1)
claude Nov 17, 2026
f3cb941
docs(plan): amend the route plan with the 2026-11-17 rulings and review
claude Nov 17, 2026
6b552bb
feat(journal): record framing and a storage interface (C2, byte layer)
claude Nov 17, 2026
67d36e7
proof(agda): a record and its history trim are one journal event
claude Nov 17, 2026
24171e3
feat(state): persist through one checksummed journal (C2)
claude Nov 17, 2026
930fbc6
fix(journal): refuse a FIFO without blocking, always sync the directory
claude Nov 17, 2026
88e0d62
feat(rollback): check first, and record the 2026-11-17 rulings
claude Nov 17, 2026
4f7fa6a
fix(journal): tell damage from an interrupted write; six review findings
claude Nov 17, 2026
49b0ae7
docs(plans): product review: the shell measured against bash, routes,…
claude Nov 17, 2026
c2357ed
docs(plans): six corrections to the product review, from review
claude Nov 17, 2026
d52c74c
docs(plans): record owner rulings OD-42 to OD-45 after the product re…
claude Nov 17, 2026
6af3a27
fix(journal): three review findings: redo, the old-state backup, a fa…
claude Nov 17, 2026
6aef809
fix(journal): a check on each record's length, and the proof that goe…
claude Nov 17, 2026
17f3b62
fix(journal): an open transaction keeps its operations; two smaller r…
claude Nov 17, 2026
baafe40
fix(journal): rollback stops when a reversal cannot be saved
claude Nov 17, 2026
4c82ac9
fix(journal): five review findings: a failed redo keeps its entry, an…
claude Nov 17, 2026
c3dafae
perf(rollback): index the history once when checking a transaction
claude Nov 17, 2026
9c451f9
docs(review): an audit event can include a proctitle record, not always
claude Nov 17, 2026
8d6d0c8
docs(plan): owner rulings OD-46 to OD-49, and hole p
claude Nov 17, 2026
8f50cb7
docs(plan): typed words lands with the front end; behaviour changes c…
claude Nov 18, 2026
b46a338
fix(state): a command whose record cannot be saved returns an error (…
claude Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions .github/workflows/compilation_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,12 +85,23 @@ jobs:
working-directory: impl/rust-cli
run: cargo test --verbose

# The ignored stress suite (tests/stress_tests.rs) is not run here. It ran
# under `continue-on-error: true`, which hid its state (Hypatia RE005):
# on 2026-11-16, 5 of its tests fail outright and 2 run for more than two
# minutes each, so this step only ever burned the job's 30-minute timeout.
# Run it by hand with `cargo test --test stress_tests -- --ignored` until
# the suite is repaired and can gate honestly.
# The ignored stress suite (tests/stress_tests.rs) gates the build. It was
# repaired on 2026-11-17 (see docs/audits/2026-11-16-hypatia-remediation.adoc);
# there is deliberately no continue-on-error (Hypatia RE005).
# The shell syncs its journal after every operation, so on a real disk
# this suite mostly measures the disk (82 s on ext4 against 39 s on tmpfs
# locally). TMPDIR points at tmpfs so that the step measures the shell,
# and 5 minutes leaves wide margin.
# Linux only: stress_deep_nesting_1000_levels needs 1000 directory levels
# in one absolute path (~2,000 bytes), which fits Linux's PATH_MAX of 4096
# but not macOS's 1024.
- name: Run ignored tests (stress)
if: runner.os == 'Linux'
working-directory: impl/rust-cli
timeout-minutes: 5
env:
TMPDIR: /dev/shm
run: cargo test --test stress_tests -- --ignored --test-threads=1

# ============================================================
# MSRV (Minimum Supported Rust Version)
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,9 @@ htmlcov/
ai-cli-crash-capture/
.cache/
.vsh_state.json
.vsh_state.json.migrated
.vsh_state.journal
.vsh_state.journal.*

# ============
# Strategic
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ valence-shell/
proofs/
lean4/ # Lean recheck leg + executable model oracle (101 theorems)
coq/ # CIC proofs + extraction (131 theorems, 0 admits — re-verified 2026-07-16, Coq 8.18.0)
agda/ # Theory source of truth (owner ruling 2026-11-09); 10 modules, 0 postulates, --safe --without-K --cubical-compatible
agda/ # Theory source of truth (owner ruling 2026-11-09); 14 modules (10 filesystem, 4 journal model), 0 postulates, --safe --without-K --cubical-compatible
z3/ # SMT checks (ASSUMED; 2 files definitional, 3 axiomatic)
idris2/ # ABI carrier (0 holes, 0 partials, 2 primitive-eq axioms)
docs/ # Design docs, roadmaps, PROOF_HOLES_AUDIT.adoc
Expand Down
16 changes: 14 additions & 2 deletions EXPLAINME.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,22 @@ reproduce. Absence of a tool is an open obligation, not a passing leg

| Same theorems in Agda (the theory source since the 2026-11-09 owner ruling)
| proven (2026-11-09)
| `agda --safe --without-K --cubical-compatible All.agda`, Agda 2.7.0.1 + stdlib v2.2: 10/10 modules; CI `.github/workflows/agda.yml`
| `proofs/agda/` (ten modules via `All.agda`)
| `agda --safe --without-K --cubical-compatible All.agda`, Agda 2.7.0.1 + stdlib v2.2: all modules in `All.agda` (ten for these theorems; fourteen with the journal model below); CI `.github/workflows/agda.yml`
| `proofs/agda/` (via `All.agda`)
| No postulates, no holes, the estate invocation. Filesystem equality is pointwise (`_≗_`). Grade `PWG-3` is derived from the CI receipt on the merge head, not asserted here. See `docs/audits/2026-11-09-proof-ground-truth.adoc`

| Journal model: after a crash, loading yields the synced events followed by a prefix of the unsynced ones; a cut-short record is rejected, and is classified as torn, never as damaged; rollback leaves no operation of the rolled-back transaction live
| proven (2026-11-17)
| `bash scripts/check-agda.sh`, Agda 2.7.0.1 + stdlib v2.2; ten mutations rejected locally (`proofs/agda/VERIFICATION.adoc`)
| `proofs/agda/JournalModel.agda` `crash-recovery`, `write-atomic`, `crash-tail`; `JournalFraming.agda` `step-torn`, `decode-sound`, `classify-torn`; `JournalReplay.agda` `run-ids`, `rollback-no-live`
| About the model only. The crash model (synced bytes survive; the unsynced part survives as a prefix) is an assumption about storage, and reordered or partly written blocks are outside it. No property of the checksum or of the header check is proved or needed. The converse is not proved and does not hold: damage that leaves a header passing its check and declaring more bytes than the file holds is classified as torn

| The Rust journal follows the journal model
| tested-only
| `cargo test --test journal_crash_tests`: 3,139 crash images on simulated storage that implements the model's file, and 3,130 crash states on real files, none failing (2026-11-17, local, with the header check); four deliberate bugs were each caught. Unit tests flip every bit of every header and of a whole record and cut a journal at every byte
| `impl/rust-cli/src/journal.rs`, `impl/rust-cli/src/state.rs`, `impl/rust-cli/tests/journal_crash_tests.rs`
| Written to mirror the model, not derived from it or proved against it. One scripted session, not all sessions. The journal file still lives in the session root, so the shell's own commands can delete or alter it; the model does not cover that

| Same theorems restated in Z3 (Isabelle and Mizar archived 2026-11-09)
| argued (the ledger word for PROOF-POLICY §5 `ASSUMED`: a direct SMT run with no proof certificate)
| `scripts/verify-z3.sh`, exact result sequence per file
Expand Down
6 changes: 3 additions & 3 deletions docs/ARCHITECTURE_DIAGRAMS.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ System Documentation
│ │ │ │ │ │ │
│ │ ┌──────▼──────────────────▼──────────────────▼───────┐ │ │
│ │ │ State Persistence (JSON) │ │ │
│ │ │ ~/.vsh_state.json │ │ │
│ │ │ <root>/.vsh_state.journal │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
Expand Down Expand Up @@ -214,7 +214,7 @@ USER INPUT: echo "Hello $USER" > output.txt
┌─────────────────┐
│ 7. UPDATE │ Update shell state
│ STATE │ last_exit_code = 0
│ (state.rs) │ Persist to ~/.vsh_state.json
│ (state.rs) │ Persist to <root>/.vsh_state.journal
└────────┬────────┘
│
▼
Expand Down Expand Up @@ -391,7 +391,7 @@ USER: undo

┌────────────┐
│ START │ Initialize shell
│ (clean) │ Load ~/.vsh_state.json (if exists)
│ (clean) │ Load <root>/.vsh_state.journal (if exists)
└──────┬─────┘
│
▼
Expand Down
133 changes: 132 additions & 1 deletion docs/audits/2026-11-16-hypatia-remediation.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ They close when the underlying alert closes.
| `npx_in_workflow` (2) | fixed | `bunx --no-install --bun rescript build`, which never fetches an unpinned package.
| `missing_timeout_minutes`, WH006 | fixed | `labels.yml`, `label-triage.yml` (echo-types versions).
| RE001 (3) | fixed | `harden-runner`: `block` to `api.github.com` in `instant-sync.yml`, `block` to the SMTP endpoint in `push-email-notify.yml`, `audit` in `hypatia-scan.yml`.
| RE005 | fixed | The ignored stress suite no longer runs under `continue-on-error` in `compilation_tests.yml`; see <<stress>>.
| RE005 | fixed | The ignored stress suite no longer runs under `continue-on-error` in `compilation_tests.yml`; after its repair it gates the Linux test jobs again. See <<stress>>.
| WH008 (3) | fixed | See the Secret Scanner section.
| SD022 (2) | fixed | Both paths named another tree (`echidna/src/rust/`, the Agda tarball's `src/data/lib`); now qualified.
| SD009 | left for the owner | `ffi/zig/src/main.zig` has no SPDX header. Licence edits are owner-only (estate doctrine 6).
Expand Down Expand Up @@ -94,3 +94,134 @@ Run one test at a time on 2026-11-16, 120 s limit each: 5 fail
(`stress_many_operations_10000`, `stress_resource_limits_max_history`), 3 pass;
`stress_large_file_1gb` was not run. The CI step only ever reached the
30-minute job timeout. Repairing the suite is separate work.

=== Repair (2026-11-17)

All 11 tests now pass. The step is back in `compilation_tests.yml` as
`cargo test --test stress_tests -- --ignored --test-threads=1`, Linux only,
with `timeout-minutes: 5` and no `continue-on-error`. No test was deleted or
skipped, and no assertion was loosened.

Shell bugs fixed (`src/state.rs`, `src/commands.rs`):

* *O(n²) persistence.* Every recorded or undone operation rewrote the whole
history to `.vsh_state.json` (cloned, then pretty-printed). 1,000 touches
took 5.9 s; 10,000 and 100,000 never finished. The hot path now appends
one line to `.vsh_state.journal`; the snapshot is rewritten only when the
journal reaches half the history length (at least 1,024 entries), which
keeps the cost amortised O(1). Snapshots are written to a temporary file
and renamed, and record the last journal sequence number they contain, so
a crash cannot tear the snapshot or replay an entry twice. A torn journal
line or a sequence gap ends replay and forces the next save to be a full
snapshot, so operations made after a crash are not stranded behind the
bad line (Codex review on #229). After the fix, 1,000 touches take
0.04 s. Seven new unit tests in `state.rs` cover replay, compaction, trim,
a torn tail, crash-after-rename and transactions; two more cover
operations after a torn tail and `cd -` surviving a restart (`cd` now
journals `previous_dir`).
+
*Superseded later on 2026-11-17.* Review kept finding crash-consistency
defects in that two-file design (a snapshot beside a journal), and a
crash-point harness confirmed two failing classes. It was replaced by one
file of length-prefixed, checksummed records, each appended and synced,
following an Agda model (`proofs/agda/JournalModel.agda`). The journal is
rewritten as a single snapshot record, all-or-nothing, at the same threshold.
A failed load is now an error instead of an empty history, and an old
`.vsh_state.json` is converted on first start and kept as
`.vsh_state.json.migrated`. The crash harness
(`impl/rust-cli/tests/journal_crash_tests.rs`) finds no failing class on the
new format. The amortised cost is unchanged; each operation now also pays
one sync.
* *O(n²) single-step undo.* Each `undo 1` walked past every already-undone
operation to pick its target, then searched for that target again from the
oldest end of the history to mark it. A checked hint for the block of undone operations removes the rescan:
undoing 10,000 operations one at a time went from 1.8 s to 0.34 s. A unit
test compares it against a full scan through undo, record, trim and an
out-of-order mark.
* *Rollback not persisted.* `rollback` cleared the active transaction
without saving. That had been masked by the next operation's full rewrite,
and the journal would have exposed it. It now goes through
`ShellState::abort_transaction`, which saves.
* *Rollback left reversed operations live* (found in review, also on
`main`): `begin t; touch f; rollback; undo` failed with "No such file or
directory". Rollback now marks each operation undone as it reverses it,
skips operations already undone inside the transaction, and closes the
transaction only when every operation has been reversed. It first checks
each operation's record and refuses the whole rollback, changing nothing,
if one has no inverse or lacks its undo data (owner ruling OD-39). If a
reversal then fails on the disk, the transaction stays open so the
rollback can be run again.

Test expectations corrected (`tests/stress_tests.rs`):

* `stress_concurrent_multiple_instances`, `stress_many_operations_10000`:
the listings counted the shell's own `.vsh_state*` files. They now exclude
only those files, and the concurrent test checks the exact set of 1,000
names.
* `stress_deep_nesting_1000_levels`: `level_NNNN` names give an
11,000-byte path, past `PATH_MAX`, and the test then ran `mkdir` on paths
that already existed. It now builds the 1,000 levels through the shell
with one-character names (about 2,000 bytes, asserted against `PATH_MAX`)
and undoes them all.
* `stress_deep_nesting_no_stack_overflow`: it only called
`fs::create_dir_all`, which fails with ENAMETOOLONG at 5,000 levels. It
now builds the tree with `mkdirat`/`openat`, checks that shell `mkdir` and
`touch` at that depth fail cleanly without recording anything, and removes
the tree.
* `stress_large_file_undo_streaming`, `stress_large_file_1gb`: both called
`touch` on an existing file, but vsh's `touch` only creates files
(EEXIST). The 100 MB test now runs `rm` (the operation that saves file
content as undo data) plus `undo` and checks the restored bytes. The 1 GB
test times `chmod`, `mv` and their undos.
* `stress_undo_redo_efficiency`: "redo" popped the redo stack without
redoing. It now calls `commands::redo` and checks the files return.
* `stress_resource_limits_max_history`: it now asserts the history is capped
at `max_history_size` and that the persisted state stays under 32 MiB
(measured: 2.3 MB).

Measured (debug build, 4 cores, each test run alone with the CI flags):
whole suite 29-51 s across three runs. These are the timings of the two-file
design. With the single-file journal and a sync per record, the whole suite
takes 82 s on this machine's ext4 disk and 39 s on tmpfs (one run each). The
CI step therefore puts its temporary directories on tmpfs, so that it
measures the shell's own cost and not the runner's disk.

[cols="3,1",options="header"]
|===
| Test | Wall time
| `stress_concurrent_multiple_instances` | 0.6 s
| `stress_concurrent_no_corruption` | 0.2 s
| `stress_deep_nesting_1000_levels` | 0.5 s
| `stress_deep_nesting_no_stack_overflow` | 0.7 s
| `stress_history_memory_efficiency` | 0.5 s
| `stress_large_file_1gb` | 1.8 s
| `stress_large_file_undo_streaming` | 12.6 s
| `stress_many_operations_10000` | 2.3 s
| `stress_many_small_files` | 1.2 s
| `stress_resource_limits_max_history` | 10-32 s
| `stress_undo_redo_efficiency` | 0.2 s
|===

Still open:

* *Undo data is not streamed.* `rm` reads the whole file into
`undo_data`, and the journal stores those bytes as a JSON number array
(about 3 bytes per byte). Removing a 100 MB file takes about 11.5 s in a
debug build and writes about 300 MB of state. The test name promises
streaming; the test comment now says it is not implemented. The fix is
out-of-line undo storage (for example, renaming the file into a store
under the root), which changes the reversibility implementation, so it is
left for the owner.
* *History trimming is O(limit) per operation.* At the cap, each new
operation drains the oldest entry from the front of a `Vec`, moving about
1.2 MB at the default 10,000 limit. That is about 7 s of the 100,000-op
test. A `VecDeque` would remove it but changes the public
`ShellState::history` type that tests and `CLAUDE.md` document as a `Vec`.
* *macOS.* The suite is not run there: `PATH_MAX` is 1024, so the
1,000-level test cannot pass with absolute paths, and the rest was not
measured on macOS.
* The journal still lives in the session root, so shell commands can delete
or edit it, and two shells on one root are not kept apart. It is created
with mode 0600, never opened through a symlink, and synced on every
record. Moving it to an owner-only directory under an exclusive lock is
the next step (route plan section 14.3, C3).
Loading
Loading