Repository navigation
feat: record local command history with Recall - #1155
Merged
Merged
Conversation
kvlonge
enabled auto-merge (squash)
October 19, 2026 08:24
kvlonge
force-pushed
the
feat/recall-core
branch
from
October 19, 2026 08:53
7db97b5 to
4070f4e
Compare
…e local SQLite store
… query, schema and output
…ut per-value copies
…ems before output Startup never waits on the store: setup runs on the writer thread and retries until the close deadline. The shutdown timeout bounds the whole exit, including busy waits and maintenance, and a second interrupt stops waiting at once. A store file that is not a database is moved aside once and a fresh store started. Closing notices end with the command's own status, and a notice that cannot be written never changes the exit code. Output is forwarded to Recall under the compute-log storage lock, dropped output is counted once, heartbeats and retention use wall-clock time so a suspended run is kept, and oversized SQL is dropped before it is hashed or copied.
…h BLAKE3 The command thread now only enqueues each lifecycle event and executed SQL; rows, artifact digests and content hashes are built on the delivery thread, which also deduplicates SQL per invocation. Events that cannot be built are counted as failed. Row content is hashed with BLAKE3 over one encoded buffer, and the column is renamed to content_hash.
…e found corrupt Moving a corrupt store aside now takes an exclusive lock on recall.sqlite3.lock, waiting within the startup or shutdown budget, and under it re-checks that the path still names the same file (device and inode) that was found corrupt. A store another process already replaced is used as it is, so concurrent commands move the corrupt file and its companions aside exactly once.
…ream Large writes were split into chunks by repeatedly copying the remaining buffer, quadratic in the write size: a warm dense compile printing 25 MB of JSON spent most of its time there. Chunks are now cut straight from the written bytes. Each stream also records at most 1 MiB per invocation; later output takes one filtered sequence per contiguous run, so output_invocations reports a filtered gap, and the views count trailing gaps from the delivery row.
…l] max_output_bytes max_output_bytes (also under targets.<name>.recall) bounds the bytes of each output stream Recall records: the first half live and the last half from a ring flushed at close, each under sequences reserved in write order. The local SQLite store defaults to 2 MiB and the default is chosen per store; 0 records everything and invalid values are configuration errors. The left-out middle is one gap row with its byte count: output_invocations reports gap_reason 'truncated' and truncated_bytes, and sqb recall output prints a note in its place. Writers no longer checkpoint as they commit or close: each session runs a passive checkpoint when it starts, beside the command, so exits do not wait on WAL fsyncs.
kvlonge
force-pushed
the
feat/recall-core
branch
from
October 19, 2026 12:08
2cd0dae to
2dc9fa2
Compare
…tcome The invocation_started payload extracted --select as UTF-8 strings, so a selector holding a lone surrogate raised UnicodeEncodeError before the command ran, replacing the error the command reports. selection_count now counts the selectors without decoding them. Recall records such text with \xNN and \uXXXX escapes instead of replacement characters, and its lifecycle and SQL subscriber drops any encoding failure rather than raising it into the command.
The delivery thread signalled completion before closing its store and exiting, so it could outlive the command for a moment. It now closes the store first and close() joins it once it has finished; a thread still busy at the shutdown deadline stays detached. Helpers that list child processes through /proc skip threads that exit during the walk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Investigating a run today means digging through compute-log files. Recall keeps a local, queryable history of every command: what ran, what it did and exactly what it printed (CHI-644, CHI-645, CHI-648 and the local part of CHI-646; epic CHI-577).
Changes
sqlbuild-recallcrate: append-only records with deterministic IDs, SQLite and ClickHouse DDL and views, retention,[recall]settings, a bundled-SQLite local store and a background writer with a bounded queue, heartbeats and a hard shutdown budget.logs/recall.sqlite3: every observed command, its lifecycle events, executed SQL (deduplicated) and its exact output bytes. Large output is kept as its first and last half under[recall] max_output_bytes(2 MiB per stream locally), with the omitted middle recorded as atruncatedgap. Unwritable directories print the setup SQL; a corrupt store is moved aside once under a lock.sqb recall query | schema | output(C970–C975), read-only.sqb recall.Verification
compile --json+1.6% instructions; small-project compile +1.0%; build +1.7% on the command thread.recalltarget key golden. Independent review plus one follow-up; fensu, clippy, ruff, ty, check-ci.