feat(insights): the store indexes the words (M19c)

PLAN.md

@@ -5478,6 +5478,38 @@ never looks like a fact.
   fixture the plan goes from 102 786 to 114 483 prompt tokens (+11%) and
   from 36 650 to 41 670 completion tokens (+14%).
 
+**As built — the store (2026-10-27).** `INSIGHTS_DB_VERSION = 2`. Migration 1
+creates `insight_word (id, entry)`, indexed on `word` and on `id`. `entry` is
+the word entry as JSON text, without its `meaning`; `word`, `concern`, `sense`
+and `confidence` are generated columns over it. The decisions:
+
+- **JSON text with generated columns, as `insight` has.** These strings come
+  from a model or a side-car, and a plain column cannot hold U+0000 (the binding
+  reads it back cut short). The round-trip property's words are binary strings,
+  and they import whole. `in` rides in `entry`: it is a reserved word in SQL.
+- **No `meaning`.** A sentence per entry was a third of the table at Broadleaf's
+  scale. An answer that shows it reads the block, by the M16b rule this
+  section's table cites.
+- **Backfilled after all.** This section said there was nothing to backfill,
+  but a build that asked prompt 2 while its store was still at version 1 (#35)
+  writes words into version-1 blocks. The migration projects them.
+- **Written per record**, in the unit's transaction: first the replaced
+  record's rows are deleted, then the record is written, then its rows. An
+  import deletes the whole table and rebuilds it from the blocks. There is one
+  row per entry, a repeated word included: the projection says what the block
+  says.
+- **A reader still refuses an older store** (M16c). The refusal now names the
+  free way out: any `explain` on the store migrates it, `--dry-run` included. So
+  `serve`'s panel shows nothing on a version-1 store until then.
+- **Tested from a real file.** `packages/insights/test/fixtures/store-v1.sql`
+  is the `.dump` of a store the version-1 build wrote (6 records, 2 failures, a
+  run, 2 blocks with words), kept with the export that build wrote.
+- **Scale, on a synthetic side-car** with Broadleaf's counts and this section's
+  word estimates (28 206 records, 131 190 entries): the import takes 1.5 s and
+  the export is byte-identical. The projection is 22 MB of a 91 MB store,
+  `WHERE word = 'order'` takes 0.5 ms, and the migration's backfill 0.6 s. The
+  real Broadleaf side-car is owed.
+
 ### 20.4 Milestone split
 
 **M19a — the lexicon, the split, the artifact** (closes the profile half of #19)
@@ -5540,10 +5572,11 @@ never looks like a fact.
       `@codegraph/vocabulary` (types only, through the env); the module
       prompt quoting its types' entries through a `RecordSource.words(id)`
       projection; the word list in the fingerprint inputs.
-- [ ] The store at version 2: migration 1 creating `insight_word`, written
+- [x] The store at version 2: migration 1 creating `insight_word`, written
       per record in the unit's transaction and rebuilt on `--import`; the
       migration tests of §20.3; the round-trip property unchanged and green
-      on every existing side-car.
+      on every existing side-car. *(The real Broadleaf side-car is owed;
+      measured on a synthetic one at its scale.)*
 - [ ] `--terms N`; `--dry-run`/`--estimate` reflecting the longer
       completions (`COMPLETION_TOKENS_PER_BLOCK` re-measured); a reader's
       answer for a record explained under prompt `"1"`; on the fake client

docs/cli.md

@@ -336,7 +336,9 @@ invocation:
 side-car carries it). The command writes nothing: the store is opened as a
 reader, a missing one is a usage error rather than a file that appears, and one
 written by a newer codegraph — or a SQLite file that is not an insights store —
-is named and left alone.
+is named and left alone. One an older build wrote is refused and left alone too:
+any `codegraph explain` on it migrates it, `--dry-run` included, which calls no
+model.
 
 ### `scm`
 Mines a repository's git history into a deterministic `history.jsonl`:

docs/insights.md

@@ -188,6 +188,23 @@ id would come back shorter); and the tables are **rowid tables** (`WITHOUT
 ROWID` keeps kilobyte-wide rows in interior pages: 96 MB against 57 MB on
 Broadleaf, for no faster read).
 
+**Version 2: the words, projected (M19c).** `insight_word` holds one row per
+entry of a block's `words`: the entry as JSON text, without its `meaning` (a
+sentence, read from the block when an answer shows it), with `word`, `concern`,
+`sense` and `confidence` as generated columns over it, indexed on `word` and
+on `id`. "Every entity that defines `order`" is then an index lookup, where it
+would otherwise scan every block. A record's rows are written with it, in the
+same transaction, after the rows of the record it replaces are deleted, and
+`--import` rebuilds the table from the blocks. Nothing is exported from it: the
+block stays the one copy that counts, so `decode → import → export` is
+untouched. Migration 1 creates the table and projects the words a version-1
+block already holds. A build that asked prompt 2 while its store was still at
+version 1 wrote some. It is the first rung of the ladder to run on a real file,
+and its tests start from one the version-1 build wrote
+(`packages/insights/test/fixtures/store-v1.sql`): the records, failures, ledger
+and export stamp come through intact, the export is the same bytes, a reader
+refuses the file untouched, and a migration that fails leaves it at version 1.
+
 A `run` table is the one thing the store holds that the side-car does not: one
 row per run — when, which models and provider, counts, usage, why it aborted.
 The trailer only ever knew the last run. It is declared losable: delete the
@@ -233,7 +250,9 @@ Broadleaf, the graph the rest.
 
 **The store's readers (M16c).** A reader gets the store with `{ readOnly:
 true }`: no migration, and SQLite itself refuses every write (`PRAGMA
-query_only`). It is handed an ordinary connection rather than a read-only file
+query_only`). So a store an older build wrote is refused rather than read; the
+refusal says that any `codegraph explain` on it migrates it, `--dry-run`
+included, which calls no model. It is handed an ordinary connection rather than a read-only file
 handle, because the latter — the first attempt — may not clean up after itself
 and stranded `-wal`/`-shm` beside the model. What a reader asks never reads a
 block: `query()` returns ROWS (envelope, description, and the same

packages/cli/src/args.ts

@@ -515,7 +515,7 @@ export const SERVE_SPEC: CommandSpec = {
       type: "string",
       describe:
         "The insights store behind the page's explanation panel (what 'codegraph explain' wrote). " +
-        "Default: <model>.insights.db beside the first model; with none, the page simply shows no explanations.",
+        "Default: <model>.insights.db beside the first model; with none, or one an older codegraph wrote that no 'codegraph explain' has migrated yet, the page shows no explanations.",
       placeholder: "FILE",
     },
   ],

packages/cli/src/insight-route.ts

@@ -43,7 +43,8 @@ export function insightLookup(storePath: string): InsightLookup {
     try {
       store = sqliteInsightsStore(db, { readOnly: true });
     } catch {
-      // Not ours, or from a newer build: nothing this page can show. `codegraph insights` says why.
+      // Not ours, from a newer build, or from an older one no explain has migrated yet: nothing this page can show.
+      // `codegraph insights` says why.
       db.close();
     }
     return store;

packages/cli/test/explain.test.ts

@@ -172,6 +172,25 @@ describe("explain --dry-run", () => {
     expect(codeB).toBe(EXIT.OK);
   });
 
+  it("migrates a store an older build wrote, as a reader's refusal says it does, and calls nothing", async () => {
+    const dir = mkdtempSync(join(tmpdir(), "explain-v1-"));
+    try {
+      const path = join(dir, "model.insights.db");
+      const v1 = loadSqlite().open(path);
+      v1.exec(readFileSync(fileURLToPath(new URL("../../insights/test/fixtures/store-v1.sql", import.meta.url)), "utf8"));
+      v1.close();
+      const seam: ExplainSeam = { ...realSeam(), env: {}, clientFor: () => { throw new Error("a dry run asks for no client"); } };
+      expect(await explainCommand(options(["--out", join(dir, "model.insights.jsonl"), "--dry-run"]), captureIo(), seam)).toBe(EXIT.OK);
+      const after = loadSqlite().open(path);
+      expect(after.prepare("PRAGMA user_version").get()).toEqual({ user_version: 2 });
+      expect(after.prepare("SELECT count(*) AS n FROM insight").get()).toEqual({ n: 6 });
+      after.close();
+      expect(readdirSync(dir)).toEqual(["model.insights.db"]);
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+
   it("--json prints one machine-readable plan whose operations precede their type", async () => {
     const io = captureIo();
     await explainCommand(options(["--dry-run", "--json"]), io, seamWith({}).seam);

packages/cli/test/insights.test.ts

@@ -1,6 +1,7 @@
 import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
+import { fileURLToPath } from "node:url";
 import { afterAll, beforeAll, describe, expect, it } from "vitest";
 import { loadSqlite } from "@codegraph/analyzer";
 import {
@@ -218,4 +219,21 @@ describe("codegraph insights: no store, or not ours", () => {
       rmSync(other);
     }
   });
+
+  it("a store an older build wrote is refused, with the way to migrate it that calls no model, and left alone", () => {
+    const older = join(dir, "older.insights.db");
+    const v1 = loadSqlite().open(older);
+    v1.exec(readFileSync(fileURLToPath(new URL("../../insights/test/fixtures/store-v1.sql", import.meta.url)), "utf8"));
+    v1.close();
+    const before = readFileSync(older);
+    const io = captureIo();
+    try {
+      expect(runSync(["insights", model, "--store", older], io)).toBe(EXIT.USAGE);
+      expect(io.stderr()).toMatch(/at version 1 and this build reads 2/u);
+      expect(io.stderr()).toMatch(/'codegraph explain' on it does, '--dry-run' included/u);
+      expect(readFileSync(older).equals(before)).toBe(true);
+    } finally {
+      rmSync(older);
+    }
+  });
 });

packages/insights/src/store-sqlite.ts

@@ -52,6 +52,14 @@ import {
  * would come back SHORTER, silently. So a column string holding U+0000 is
  * refused at write; everything a model or a provider worded (block, reason,
  * metadata) is stored as JSON text, where both are escapes and survive.
+ *
+ * A RECORD'S WORDS ARE A PROJECTION (version 2, M19c). `insight_word` holds one
+ * row per entry of a block's `words`: the entry as JSON text, less its
+ * `meaning` (a sentence, read from the block when an answer shows it), with
+ * generated columns over it, so "every entity that defines `order`" is an
+ * index lookup rather than a scan of every block. It is written with its
+ * record, in the same transaction, and rebuilt by an import: the block stays
+ * the one copy that counts, and nothing is exported from the projection.
  */
 
 /** Index i takes a store from version i to i+1. Append; never edit a shipped entry. */
@@ -108,6 +116,23 @@ CREATE TABLE failure (
   prompt_tokens INTEGER, completion_tokens INTEGER, cost REAL,
   run_id INTEGER REFERENCES run(id)
 );
+`,
+  // A version-1 block can hold words already (a build that asked prompt 2 before its store was version 2): projected here.
+  `
+CREATE TABLE insight_word (
+  id TEXT NOT NULL REFERENCES insight(id),
+  entry TEXT NOT NULL,
+  word TEXT GENERATED ALWAYS AS (entry ->> '$.word') VIRTUAL,
+  concern TEXT GENERATED ALWAYS AS (entry ->> '$.concern') VIRTUAL,
+  sense TEXT GENERATED ALWAYS AS (entry ->> '$.sense') VIRTUAL,
+  confidence REAL GENERATED ALWAYS AS (entry ->> '$.confidence') VIRTUAL
+);
+CREATE INDEX insight_word_id ON insight_word(id);
+CREATE INDEX insight_word_word ON insight_word(word);
+
+INSERT INTO insight_word (id, entry)
+  SELECT insight.id, json_remove(entry.value, '$.meaning') FROM insight, json_each(insight.block, '$.words') AS entry
+  ORDER BY insight.rowid, entry.key;
 `,
 ];
 
@@ -147,7 +172,9 @@ function migrate(db: SqliteDatabase, readOnly: boolean): void {
     // A reader brings nothing up to date: an empty file is not a store, and an older one waits for a run to migrate it.
     if (tables === 0) throw new InsightsStoreError("this file is not an insights store (it is empty)");
     if (version < INSIGHTS_DB_VERSION) {
-      throw new InsightsStoreError(`this insights store is at version ${version} and this build reads ${INSIGHTS_DB_VERSION}; any 'codegraph explain' run on it migrates it`);
+      throw new InsightsStoreError(
+        `this insights store is at version ${version} and this build reads ${INSIGHTS_DB_VERSION}: a reader never migrates it, and any 'codegraph explain' on it does, '--dry-run' included, which calls no model`,
+      );
     }
   }
   if (version > INSIGHTS_DB_VERSION) {
@@ -242,6 +269,8 @@ export function sqliteInsightsStore(db: SqliteDatabase, options: StoreOpenOption
   const insertInsight = db.prepare(`INSERT OR REPLACE INTO insight (${INSIGHT_COLUMNS}, run_id) VALUES (${"?, ".repeat(18)}?)`);
   const insertFailure = db.prepare(`INSERT OR REPLACE INTO failure (${FAILURE_COLUMNS}, run_id) VALUES (${"?, ".repeat(14)}?)`);
   const deleteFailure = db.prepare("DELETE FROM failure WHERE id = ?");
+  const deleteWords = db.prepare("DELETE FROM insight_word WHERE id = ?");
+  const insertWord = db.prepare("INSERT INTO insight_word (id, entry) VALUES (?, ?)");
   const getMeta = db.prepare("SELECT value FROM meta WHERE key = ?");
   const setMeta = db.prepare("INSERT OR REPLACE INTO meta (key, value) VALUES (?, ?)");
   const dropMeta = db.prepare("DELETE FROM meta WHERE key = ?");
@@ -263,9 +292,11 @@ export function sqliteInsightsStore(db: SqliteDatabase, options: StoreOpenOption
     return typeof value === "string" ? (JSON.parse(value) as unknown) : undefined;
   };
 
-  /** Validated BEFORE the first bind, so a bad member aborts its unit with nothing written. */
+  /** Validated BEFORE the first bind, so a bad member aborts its unit with nothing written. Its words go with it. */
   const writeRecord = (input: InsightRecord, run: number | null): void => {
     const r = InsightRecord.parse(input);
+    // First: the replaced record's words must not outlive it, and the row they reference is about to be replaced.
+    deleteWords.run(r.id);
     insertInsight.run(
       nullable(r.id), LEVEL_RANK[r.level], nullable(r.kind), nullable(r.name), nullable(r.file),
       nullable(r.key?.lang), nullable(r.key?.module), nullable(r.key?.symbol), nullable(r.key?.disambiguator),
@@ -276,6 +307,7 @@ export function sqliteInsightsStore(db: SqliteDatabase, options: StoreOpenOption
       JSON.stringify(r.block),
       run,
     );
+    for (const { meaning: _meaning, ...listed } of r.block.words ?? []) insertWord.run(r.id, JSON.stringify(listed));
   };
 
   const writeFailure = (input: FailureRecord, run: number | null): void => {
@@ -485,7 +517,7 @@ export function sqliteInsightsStore(db: SqliteDatabase, options: StoreOpenOption
 
     importFile(file: InsightsFile) {
       transaction(() => {
-        db.exec("DELETE FROM insight; DELETE FROM failure;");
+        db.exec("DELETE FROM insight_word; DELETE FROM insight; DELETE FROM failure;");
         for (const record of file.records) writeRecord(record, null);
         for (const failure of file.failures) writeFailure(failure, null);
         setMeta.run("header", JSON.stringify(InsightsHeader.parse(file.header)));

packages/insights/src/store.ts

@@ -18,8 +18,8 @@ import type { InsightsFile } from "./sidecar.js";
  * does not carry, and it is declared losable.
  */
 
-/** Bumped with every migration appended to the adapter's ladder — never regenerated. */
-export const INSIGHTS_DB_VERSION = 1;
+/** Bumped with every migration appended to the adapter's ladder — never regenerated. 2: `insight_word` (M19c). */
+export const INSIGHTS_DB_VERSION = 2;
 
 /** `PRAGMA application_id`: "CGI1". A SQLite file that is not ours is refused, not migrated. */
 export const INSIGHTS_APPLICATION_ID = 0x43474931;