[lukemelia]

Stacked on #6991, which records realm writes as revisions through a history-first write transaction. Uses cardstack/deck#7 (deckd /tree and /read, and a revision's parent in /commit and /outcome); DECKD_REF in ci.yaml pins that PR's commit.

What this does

A realm that records history checks itself against its history at mount (before the boot index decision), on an hourly schedule, and whenever a write finds disk behind HEAD. Each run:

1. Settles writes whose answer was lost. A write is now journaled pending before deckd is asked to record it, in a statement of its own. A row still pending after five minutes has its outcome asked of deckd, which fences an absent operation: it becomes aborted, and a retry under that ID gets 409 operation-fenced.
2. Completes revisions a crash cut short. A committed revision whose bytes aren't all on disk, or whose index pass was never queued, has its paths brought to what HEAD holds, under those paths' write locks, through the realm's own commit. The commit runs in a mode that records nothing and rewrites even files already holding the right bytes, so every path is announced and indexed.
3. Compares disk with HEAD. The scan uses deckd /tree blob IDs against a git blob hash of each file. Whatever still differs is re-checked under its file lock, then recorded as one system revision:
   • Unrecorded changes (recording was disabled) (kind = gap, Boxel-Gap: recording-disabled) when the realm last ran without recording;
   • Unattributed external changes (kind = drift) otherwise, with an alert.

If recovery fails, the realm goes read-only:

• writes answer 423 realm-read-only with X-Boxel-Realm-Read-Only;
• reads keep working, and _readiness-check answers 200 with the same header;
• it stays read-only until an operator calls /_grafana-clear-realm-read-only, which re-runs recovery first (force=true skips it).

The runbook is docs/realm-history-recovery-runbook.md.

Key decisions

• Recovery is excluded by a lease row, not the history-commit lock. Recovery takes pool connections while it runs, and an advisory lock held across those while other replicas' waiters each pin a connection is how a pool runs dry. Each Record still takes the history-commit lock briefly, as any write does. A replica that loses the lease at mount waits for the holder, with a bound.
• Catch-up writes HEAD's bytes, not each revision in turn. Under the paths' locks, HEAD is the newest state of each path, so no write that landed after the interrupted one is rolled back.
• A write to a path whose revision hasn't reached disk is refused with 503 realm-recovering until recovery completes it. A writer holds its file locks until its bytes are on disk, so such a revision's writer is gone. Staging an append from older bytes would drop that revision's content.
• A write first settles any pending write to its paths. It asks deckd for that write's outcome without waiting out the grace period. Without that, it would stage over disk that lacks a change deckd may already hold, and record its files back as they were. An outcome of absent fences the pending write. One that committed is a revision disk lacks, so the guard above refuses the write. If deckd can't answer, the write gets 503 and nothing is recorded.
• History order comes from parents, not journal order. A replayed revision, or a pending one recovery settles, is journaled after revisions history put on top of it. Each row records the parent deckd reports. materializedRevision() walks back from the revision no other row names as its parent, until it has passed every revision not yet on disk. A replay or outcome for an operation that changed nothing is answered as unchanged, not journaled.
• A retry under a pending operation ID goes ahead as the write. deckd answers a commit under an already-committed ID with its revision, so the retry either finishes the first attempt or makes the write.
• A realm's first recording mount takes in what it already holds as one gap revision. realm-history-write-test therefore expects fixture files as modified rather than added, and one more revision.
• Recovery state lives in a new realm_history_state table, off realm_metadata, so the archived filters never have to tell a read-only realm from an archived one.
• The schedule runs in the realm-server that serves each realm. The worker job asks POST <realm>/_history-recovery, as the realm's owner, because only that process holds the realm's files, history and broadcaster.
• Writes that don't record yet show up as drift. /_atomic, source DELETE, create-realm and publish don't record yet, so on a recording realm their changes are taken in as drift and alert. That's accepted while the flag is test-only, and the runbook says so.

Schema

1793129747681_realm-history-recovery.js (additive):

• on realm_history_revisions: state, paths, kind and post_committed_at, with revision and parent made nullable for pending rows, and an index on (realm_url, parent);
• a new realm_history_state table.

Both tables stay out of the host's SQLite schema, whose file is renamed to the new migration's timestamp with its content unchanged.

Telemetry

One boxel:operations line per recovery (kind: "realm-recovery"). It carries the outcome (clean | recovered | read-only | leased), resolved operations, completed revisions, drift and gap paths and revisions, and the scan's file count, bytes and timings. realm-history-group.json alerts on read-only and on driftPaths > 0.

Testing

Run with a deckd built from cardstack/deck#7. The four newest recovery tests haven't run locally yet, so CI is their first run:

• TEST_FILES=realm-history-recovery-test pnpm test (19 tests)
• TEST_FILES=realm-history-write-test pnpm test (17 tests)

The recovery tests remount one directory and database to cover:

• a crash after Record, mid-Materialize, after Materialize, and after post-commit;
• two replicas mounting at once;
• the stale-disk guard;
• a lost commit answer that did commit, and one that didn't, which ends fenced;
• a later write to a lost write's paths, for each of the three outcomes;
• a retry that lands after another write, which is journaled on its real parent;
• drift at mount and through the scheduled task;
• a gap after recording was off;
• read-only with history unreachable and with a corrupt repo, then the operator clear;
• the flag off.

The deckd and revision-reading test helpers moved to tests/helpers/realm-history.ts, so both files share them.

Drift-detection cost was measured locally and recorded on the ticket for Gate 2. On a warm local SSD it's 56 ms for a copy of experiments-realm (621 files, 27 MB), and 0.44 s and 2.7 s for 10k and 50k synthetic files.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnysKbB8Y5FgNf4nQuAd6M

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files      1 suites   1h 23m 20s ⏱️
4 006 tests 3 830 ✅ 9 💤   0 ❌ 167 🔥
4 016 runs  3 673 ✅ 9 💤 167 ❌ 167 🔥

Results for commit ec7b5ce.

For more details on these errors, see this check.

Realm Server Test Results

    1 files    368 suites   2h 14m 13s ⏱️
6 165 tests 6 164 ✅ 0 💤 1 ❌
6 228 runs  6 227 ✅ 0 💤 1 ❌

Results for commit ec7b5ce.

For more details on these errors, see this check.

[lukemelia]

[Claude Code 🤖] Closed without merging. The revision-history design has moved away from recording a revision inside each write. A separate service will now record changes after they land on disk, so this approach is superseded.