[jgrodziski]

Second step of milestone M19c in PLAN.md §20 (word cloud / glossary, #6). Stacked on #35: review that first. This PR's diff is the four commits after #35.

The insights store moves to version 2. A new table, insight_word, holds one row per word a record defines, indexed by word and by record id, so the readers in PR 11 (insights --word, /glossary.json…) look a word up through an index instead of scanning every block.

The table (packages/insights)

• One row per entry of a block's words, in block order. The row keeps the entry as JSON (entry); word, concern, sense and confidence are generated columns over it, as concept already is over insight.block. A column string holding U+0000 reads back cut short through node:sqlite, so a side-car that decodes would otherwise fail its round trip.
• in stays inside entry: it is a reserved word in SQL, and nothing filters on it.
• No meaning. It was a third of the table. A reader that shows it reads the record's block.
• Kept in step with the records. A put deletes the record's rows, then writes them, in the same transaction as the record. --import rebuilds the table. Nothing is exported from it: the side-car round trip is unchanged.

The migration from version 1

INSIGHTS_DB_VERSION moves from 1 to 2. The migration creates the table and fills it from the words already stored. PLAN expected nothing to fill, but #35's build writes words into a store still at version 1. A migration that fails leaves the file at version 1, untouched.

The test is a real version-1 file: a sqlite3 .dump of a store #35's build wrote (test/fixtures/store-v1.sql), with that build's export and the script that wrote it. It is a text dump because .gitignore excludes *.db. Migrated, the store keeps its records, failures, ledger and export stamp, and exports the version-1 build's bytes.

Worth a look in review: readers still never migrate. PLAN's M16c rule is kept: codegraph insights and serve refuse a store that is not at this build's version. So after upgrading, insights refuses every existing store, and the serve panel shows no explanations, until any codegraph explain runs on it. The message says so, and that --dry-run, which calls no model, migrates it; a CLI test checks that it does. The alternative is to let a reader read a version-1 store, which would answer the same (no prompt-1 record has words), but that changes a documented rule.

Docs

• docs/insights.md gains a "Version 2" paragraph, and its readers paragraph gives the remedy for an older store; docs/cli.md likewise.
• The --insights help and the serve reference say an unmigrated store shows no explanations.
• PLAN §20.3 gets an "as built" paragraph for the store; §20.4's store item is checked.

Verification

• pnpm -r build and typecheck are green, eslint is clean on every changed file, pnpm run gen:schemas leaves schemas/ unchanged, and the Hugo site builds.
• Tests:
  • insights 141 (5 new: a property that the table is every record's words in order, rebuilt by an import; a put replacing a record's words; the migration, the reader's refusal and a failed migration on the version-1 file);
  • cli 1,005 (2 new: insights refuses the version-1 file and leaves its bytes; explain --dry-run migrates it without a call);
  • vocabulary 332, core 544, viz 143, navigator-ui 105, city 94, extractor-typescript 66, scm 56, navigator 40, llm 34;
  • analyzer passes 613 of 614: the Node 26 node:sqlite warning test noted on feat(core): language lexicons on profiles (M19a) #26.
• Deliberate mutations. I broke the store 8 times, one change at a time; the tests caught all 8 (words not deleted, not written, written in reverse, kept across an import, not filled by the migration, the hint dropped from the message, meaning kept by the migration, and by a put).
• The single executable. ./build.sh --ts --sea passes its smoke checks.
• Scale, on a synthetic side-car with Broadleaf's counts (28,206 records, 131,190 words): import 1.5 s; export byte-identical; insight_word 22.1 MB of a 90.9 MB store (indexes included); WHERE word = 'order' 0.5 ms; migrating a version-1 store that size 0.6 s.

Still owed for M19c:

• PLAN's check on the real Broadleaf side-car (import, counts, identical export): there is none on this machine yet.
• A live run on the Java fixture, and the per-word cost measured from it.
• The readers (PR 11).

Part of #6.

🤖 Generated with Claude Code