Repository navigation
feat!: split Recall read and write connections, faster recall queries, richer failure records - #1210
Merged
Merged
Conversation
kvlonge
enabled auto-merge (squash)
October 22, 2026 01:12
kvlonge
force-pushed
the
feat/recall-read-write-connections
branch
from
October 22, 2026 01:20
07a8d7b to
d901683
Compare
`[recall]` and `[targets.<name>.recall]` now name `write_connection`, used by every command to record history, and `read_connection`, used by `sqb recall query`, `schema` and `output`. Readers never run DDL, so a SELECT-only user can investigate shared history. `read_connection` alone reads a shared store while the machine records locally. A writer that may not upgrade a store behind this version keeps writing the columns that exist and prints the statements an administrator should run once. `sqb recall schema --ddl` prints the DDL that creates the store or brings it up to date. BREAKING CHANGE: the Recall `connection` key is removed. A configuration that still sets it fails with an error naming `write_connection` and `read_connection`; set both to the old connection to keep the previous behaviour.
With --json or --format json, a failing recall query or schema still exits 1 and prints its coded error on stderr, and now also prints {"error": {"code", "message", "help"}} on stdout so callers that parse the output read the error.
A failed invocation now records error_type, error_code and a one-line error_message on its invocation_failed event and in the invocations view, also when the command reports the failure only through its exit code. sqb debug records C980 (connection), C981 (query) or C982 (other check). The Recall schema version rises to 2 for the changed view.
SQLBUILD_GIT_REVISION and SQLBUILD_GIT_DIRTY set the recorded revision and dirty flag where the deployment has no .git. A .git file's gitdir pointer is followed, and a worktree reads its refs from the common directory.
--format table still renders in this release but prints a deprecation warning first; the default long format is for reading, --json for programs and --format csv for export. Docs examples now use the long format.
sqb recall only reads history, so it no longer publishes lifecycle events, loads or runs project sinks, or keeps compute logs. The Recall store connects through a light ClickHouse client helper instead of importing the whole adapter, a query no longer inspects the store before running (it checks only when the query fails), and clickhouse-connect >= 0.15 keeps dataframe libraries out of the import. Same machine, 5 runs each of `sqb recall query --json "SELECT 1"`: SQLite store 568M -> 400M instructions, about 0.38 s -> 0.13 s; ClickHouse store 1,120M -> 845M instructions, about 0.68 s -> 0.29 s.
A plan that fails before it can be produced now prints the same {"error": {...}} document as sqb recall --json, so callers parsing the output read the error.
The ClickHouse `invocations` and `statements` views now fold each invocation by `invocation_id` and `invocation_started_at`, which its rows share, so a filter on either column is pushed into the table scans instead of folding the whole retention window first. `statements` gains `invocation_started_at` in both dialects. The views stay additive; no summary tables are written. Synthetic store of 12,004 invocations and 804,268 events, rows read: statements for one invocation 1,773,957 -> 49,014 (381 -> 56 ms); invocations started in the last hour 2,705,033 -> 908,070; invocations for one invocation 121,525 -> 121,537 (already bounded).
A query that ran while a concurrent command was still creating the local store's views failed with C974 (no such table), which made the concurrent-builds test flaky under load. It now reports C971, like a store that does not exist yet.
references/investigating-runs.md describes how the store works, values worth knowing, traps, output-format guidance and worked investigations checked against SQLite and ClickHouse stores, and ends with every view and column. That section is generated from `sqb recall schema` by the docs-skill generator, and check-docs-skill fails when it drifts. SKILL.md points to the guide and now names read_connection. The bundled docs pages are regenerated.
The schema version row expires with retention like any other row. A store reading as version 0 now has its views replaced by a writer allowed to, before the current version is recorded. A writer that may not replace them stays quiet as before and records no version, so sqb recall schema --ddl still lists the upgrade.
Typed test cases and shared test helpers for the new Recall tests, the version row and DDL helpers split into single-entry modules, the shared Recall connection registry as a class attribute, and no literal comparisons or lossy result probes in the new native code.
Recording resolves and expands only write_connection, and sqb recall only read_connection, so a machine holding only the writer's or only the reader's credentials works. The rule that write_connection needs read_connection now applies to the merged settings, so sqlbuild_local.toml or a target may supply either side; breaking it still fails every command with C975. Other unresolvable recording settings record nothing, as before.
…Recall query A query naming a relation the complete store lacks, such as a misspelled view, reported 'not recorded yet' (C971). It is now retried once, in case the views were created since it ran, and otherwise fails as an invalid query (C974) with the server's message. A store that is not created yet still reports C971.
…pires The schema version row expires with retention. A writer that finds none now compares the stored tables' and views' columns with its own, both additive: it replaces the views only on an older store, records its version on an older or current one, and leaves a newer store untouched. Writers append the highest version they see again once the newest row is older than half the retention, so it does not expire while the store is in use. A writer that may not upgrade an older store without a version row prints the daily upgrade notice instead of failing silently.
A command's exit-code failure cause is now kept per thread, so concurrent invocations never take each other's. A raised error whose type, code or message cannot be read still publishes invocation_failed and reaches its caller. sqb recall schema --ddl with --json or --format is a usage error. sqb recall routes diagnostics again, so --debug shows them and a failure prints only its coded error, while still keeping no compute log. Documents the merged connection rule, per-side resolution and the schema version row.
An unknown write connection, a non-string adapter, a SQLite connection without a path, an empty ClickHouse database or an unterminated environment reference now fails the command with C975, like a write connection without a read connection; an unset environment variable still only skips recording. A ClickHouse writer that sees none of the views, as when granted only the tables, issues no view DDL, warns of no view upgrade and records no version it cannot vouch for. A test checks that both dialects' views expose the same columns.
…or document
plan --json now prints {"error": ...} on stdout when it fails. The golden builder read that document as a report with no diagnostics; it now records the coded stderr for it, so the goldens keep checking the error.
kvlonge
force-pushed
the
feat/recall-read-write-connections
branch
from
October 22, 2026 03:03
0b3ba46 to
6652e74
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Recall readers held the writer's credentials, failures lacked structured causes, every
sqb recallcall paid 0.4-0.7 s of start-up, and filtered queries folded the whole retention window.Changes
[recall] connectionis replaced bywrite_connectionandread_connection. A write connection needs a read connection somewhere in the merged settings. Each side resolves only its own connection. Readers never run DDL. Admins: after upgrading, runsqb recall schema --ddl(schema version 2) for SELECT/INSERT-only writers, which warn daily until then.sqb recallruns no sinks, so project sinks no longer see its output. It publishes no lifecycle events and keeps no compute logs.--jsonfailures ofrecall query/schemaandplanprint{"error": ...};--ddl --jsonis a usage error.error_type,error_codeanderror_message, including exit-code failures (sqb debug: C980-C982).SQLBUILD_GIT_REVISION/SQLBUILD_GIT_DIRTY; worktree.gitfiles are followed.--format tableis deprecated.invocations/statementsfilters reach table reads;statementsgainsinvocation_started_at.references/investigating-runs.mdwith a generated schema section.Verification
SELECT 1: SQLite 568M to 504M instructions; ClickHouse 1,120M to 891M.uv sync --all-extrasandmake check-ci.