Skip to content

feat(insights): the word readers (M19c) - #37

Open
jgrodziski wants to merge 3 commits into
feat/m19c-insight-wordfrom
feat/m19c-word-readers
Open

jgrodziski wants to merge 3 commits into
feat/m19c-insight-wordfrom
feat/m19c-word-readers

Conversation

@jgrodziski

Copy link
Copy Markdown
Contributor

Third step of milestone M19c in PLAN.md §20 (word cloud / glossary, #6). Stacked on #36 (itself on #35): review those first. This PR's diff is the three commits after #36.

codegraph insights gains four readers of the words the model defined: what a word means wherever the code spells it, a module's glossary, the words that mean different things in different places, and how many of each module's words are domain or technical. They read the store alone, like the rest of the command.

The four readers (codegraph insights)

  • --word order: every explained entity whose names spell the word, grouped by module. Each module's own glossary entry comes first, then the module, its types and its operations. One tab-separated line per entry: module, level, id, concern, confidence, sense, and the identifiers the word is in.
  • --glossary MODULE_ID: a module's glossary whole: word, concern, sense, meaning, and its types' senses where they disagree.
  • --polysemy: the words a module flagged as polysemous, and the words two modules' glossaries define differently, every module's entry side by side, the most senses first.
  • --concerns: per module, how many distinct words its records define, and how many the model called domain, technical and ambiguous.

Each has a --json form (codegraph.insightsWord/1, …Glossary/1, …Polysemy/1, …Concerns/1).

From the CLI test's store (tabs shown as aligned spaces):

$ codegraph insights --word order
java:com.acme.billing  glossary  java:com.acme.billing          domain         a purchase order
java:com.acme.billing  type      java:com.acme.billing/Invoice  domain    0.75 a purchase order      orderRef
java:com.acme.order    glossary  java:com.acme.order            domain         a customer's request  polysemous: Order "a customer's request" · SortSpec "a sequence"
java:com.acme.order    module    java:com.acme.order            domain    0.80 the ordering business com.acme.order
java:com.acme.order    type      java:com.acme.order/Order      domain    0.90 a customer's request  Order
java:com.acme.order    type      java:com.acme.order/SortSpec   technical 0.60 a sequence            sortOrder

What a reader says about a prompt-1 record

PLAN asked for "a reader's answer for a record explained under prompt 1". A record a model wrote without words predates them; one no model wrote is a template. wordsAbsence(record) tells the two apart, and:

  • --id says so on the record ("words: not defined — explained under prompt 1…", or "words: none — templated…");
  • --glossary on such a module says why it has none;
  • the summary counts them, and every word answer notes on stderr how many records it cannot see.

The store (packages/insights)

  • Four projections on the port. wordHits(word) reads the insight_word index joined to each record's envelope. glossaryEntries(word?) reads the module blocks' glossary through json_each. concernRows() and withoutWords() round them out. stats() gains withoutWords.
  • glossary.ts builds the reports from them: wordReport, polysemyReport, concernsReport, plus wordsAbsence and moduleOf.
  • Exact on any string. The index holds what SQLite's ->> made of a word; for a lone surrogate that is not what the binding sends, so such a word is looked up by a scan. Either way a result is kept only if it equals the word once parsed back from JSON. A property over arbitrary side-cars (binary strings included) checks all four projections against the blocks.

Worth a look in review:

  • Grouped by natural key, not by the model. insights never opens the model, so a record is placed in its module by its key: core renders the module id from lang and module, and groups compare through naturalKeyIndex. The id is never parsed. PLAN's "module → type → operation" is therefore by level within a module: an operation is not nested under its own type, which only the model knows. A record without a key goes under -.
  • No meaning in --word. The table doesn't hold it (PR 10); --glossary and --id show it.
  • "Different senses" ignores case and spacing. Two modules' entries differ when their concern or sense differs. An exact comparison would list nearly every word two modules define, since the wording varies from call to call.
  • A word judged two ways in one module counts under both in --concerns; the columns can add up to more than the word count.

Docs

  • docs/cli.md: the synopsis, the four flags, the JSON kinds, and what the answers say about prompt-1 records.
  • docs/insights.md: a "words' readers" paragraph.
  • PLAN §20.3: an "as built" note for the readers. §20.4 notes what is in: the four readers and the prompt-1 answer.

Verification

  • pnpm -r build and typecheck are green, eslint is clean on every changed file, and pnpm run gen:schemas leaves schemas/ unchanged. Nothing on the website changes: it has no insights page.
  • Tests:
    • insights 152 (11 new: a property that the four projections answer what the blocks say for any side-car, binary strings included; side-car order after a run writes out of order; a lone-surrogate word; and the reports, on a hand-built two-module store);
    • cli 1,014 (9 new: each reader in text and JSON, the empty answers, the prompt-1 answers in --id, --glossary and the summary, one question per invocation);
    • vocabulary 332, core 544, viz 143, navigator-ui 105, city 94, extractor-typescript 66, scm 56, navigator 40, llm 34;
    • analyzer passes 613 of 614: the Node 26 node:sqlite warning test noted on feat(core): language lexicons on profiles (M19a) #26.
  • Deliberate mutations. I broke the code 24 times, one change at a time. The first pass caught 21; I added tests for the three it missed, and all 24 are now caught:
    • removing either sort went unnoticed, because an imported store's rows already come out in side-car order. A run writes in walk order, so a test now writes out of order;
    • comparing senses without their concern went unnoticed, because no fixture word had two modules agreeing on the sense but not the concern. One now does.
  • Speed. Through the CLI, on the synthetic store at Broadleaf's scale from feat(insights): the store indexes the words (M19c) #36, Node's start included: --word order (549 entries, 19 glossary entries) 0.46 s, --polysemy 0.48 s, --concerns 0.54 s, --glossary 0.30 s.
  • The single executable. ./build.sh --ts --sea passes its smoke checks.

Still owed for M19c:

  • vocabulary --drift-insights; /glossary.json with the Vocabulary tab and Navigate panel entries; docs/vocabulary.md and the README.
  • The real Broadleaf side-car checks and the --word order timing on a re-explained Broadleaf store.
  • A live run on the Java fixture, and the per-word cost measured from it.

Part of #6.

🤖 Generated with Claude Code

Jérémie Grodziski and others added 3 commits October 27, 2026 22:57
Layer 3 of PLAN §20.3, as pure functions over the store port.

The port gains four projections: wordHits(word) through the
insight_word index, glossaryEntries(word?) over the module blocks,
concernRows() and withoutWords(); stats() counts the records a model
wrote without words (prompt 1). glossary.ts rolls them up:
wordReport, polysemyReport, concernsReport, wordsAbsence (prompt 1's
or a template's) and moduleOf. A record is placed in its module by
its natural key, rendered by core and grouped on naturalKeyIndex;
the id is never parsed.

Lookups are exact on any string: a word with a lone surrogate is not
what the binding sends for the index's TEXT, so it is found by a
scan, and JS equality decides either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codegraph insights --word WORD, --glossary MODULE_ID, --polysemy and
--concerns: tab-separated like the command's other lists (the module
first, so the grouping survives sort), each with a --json kind. A
record explained under prompt 1 is answered as such: --id says so,
--glossary on such a module says why it has none, and the summary and
every word answer count what they cannot see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/cli.md: the four flags, their JSON kinds and the prompt-1
answers. docs/insights.md: the readers over the port. PLAN §20.3:
as built; §20.4 notes the readers and the prompt-1 answer as in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
codegraph b94cafd Commit Preview URL

Branch Preview URL
Oct 28 2026, 03:34 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant