Skip to content

feat: add a SQLite adapter - #1150

Merged
kvlonge merged 8 commits into
mainfrom
feat/recall-sqlite
Oct 19, 2026
Merged

kvlonge merged 8 commits into
mainfrom
feat/recall-sqlite

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Why

Recall needs a local, file-based adapter, and SQLite is a useful lightweight warehouse in its own right (CHI-642, CHI-643; part of the Recall epic CHI-577).

Changes

  • New built-in sqlite adapter: one database file; schema relations stored as schema__name (main unprefixed) with a relation registry and compile-time collision refusal (A706); WAL, busy_timeout, BEGIN IMMEDIATE, in-process writer queue, read-only mode; SQLite 3.44+ (A701).
  • Declared column types from source columns, compile-time inferred types (new inferred_columns plan-entry field) or contracts; contract DECIMAL(p,s) stored as exact fixed-scale text.
  • Tables, views, incremental append/merge/delete+insert/cursor, sequential microbatch, snapshots, test, seed, query, debug, diff, clone, scenarios (including capture and local replay).
  • Cursor and microbatch windows normalise ISO timestamps (T separator, fractions, offsets) to UTC.
  • Documented limits with coded errors: no SQL functions (A705), Python UDFs, grants or concurrent microbatch writes.
  • Adapter docs page.

Verification

  • Integration tests on real files (contract suite, all strategies, killed writers, two-process locking, interrupts) and CLI E2E, including a DuckDB oracle on the same project and hourly windows with ISO timestamps.
  • Planner differential: only the new inferred_columns field; compiler differential: sqlite in the unknown-adapter golden.
  • Independent review plus one follow-up; fensu, clippy, ruff, ty, check-ci.

One database file per connection: SQLBuild schemas are stored as
schema__name relations, main unprefixed, with a schema registry.
Connections use WAL, a busy timeout and legacy_alter_table, require
SQLite 3.44 and queue in-process writers per file. Tables keep declared
and compile-time inferred column types; enforced decimals are stored as
canonical text. Promotion, merge, delete_insert and cursor windows run
in single transactions.
Cursor bounds render in SQLite's 'YYYY-MM-DD HH:MM:SS' text form,
including runtime batch bounds, so text comparisons order correctly.
Row counts come from COUNT(*) so full diffs can size tables. CLI E2E
covers build, test, seed, query, debug, clone, diff, every incremental
strategy against a DuckDB oracle, sequential microbatches without state
tables, snapshots, decimal contracts, coded limits and lock waits.
…acks

- Report DECIMAL(p,s) only for canonical decimal text and every other decimal as NUMERIC, so
  type enforcement always stores contract decimals as text; added columns use the same storage.
- Record each schema relation's schema and name in _sqlbuild_relations as DDL runs; listings
  and relation_exists read it, and two relations sharing one table name fail with A706 at
  compile time.
- Compare timestamp cursors as canonical UTC text on both sides in source filters, window
  deletes, watermarks, target envelopes and diff coverage, through an optional
  render_cursor_operand adapter hook that leaves other adapters unchanged.
- Roll back only an open transaction, so an interrupted statement keeps its own error.
- Restore the janitor's render_rename_view path; SQLite runs ALTER VIEW by recreating the view.
- Count updated and inserted merge rows, URI-encode read-only paths and recreate indexes after a
  column type rebuild.
@kvlonge
kvlonge enabled auto-merge (squash) October 19, 2026 04:23
@kvlonge
kvlonge merged commit e86f3a0 into main Oct 19, 2026
45 checks passed
@kvlonge
kvlonge deleted the feat/recall-sqlite branch October 19, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant