diff --git a/PLAN.md b/PLAN.md index 50edbfc..28b7801 100644 --- a/PLAN.md +++ b/PLAN.md @@ -5510,6 +5510,38 @@ and `confidence` are generated columns over it. The decisions: `WHERE word = 'order'` takes 0.5 ms, and the migration's backfill 0.6 s. The real Broadleaf side-car is owed. +**As built — the store's readers (2026-10-27).** `codegraph insights --word`, +`--glossary`, `--polysemy` and `--concerns`, over `glossary.ts` in +`@codegraph/insights` and four projections of the port (`wordHits`, +`glossaryEntries`, `concernRows`, `withoutWords`). The decisions: + +- **The module comes from the natural key.** `insights` opens the store alone, + so a record is placed by its key: core renders the module's id from `lang` + and `module`, groups compare through `naturalKeyIndex`, and the id is never + parsed. "Grouped module → type → operation" is by level within each module: + an operation is not nested under its type, which only the model knows. +- **Tab-separated, like the command's other lists**, the module first so the + grouping survives `sort`. A module's glossary entry heads its group. The + entries show no `meaning`: the projection does not hold it, and `--glossary` + and `--id` read it from the block. +- **Polysemy across modules** is two entries that differ in concern or in + sense, case and spacing aside: the wording varies from call to call, and an + exact comparison would list every word two modules define. The most senses + first. +- **Concerns** count distinct words per concern, so a word two units of one + module judged differently counts under both. +- **The prompt-1 answer** (`wordsAbsence`): a record a model wrote without + `words` is prompt 1's, one no model wrote is a template's. `--id` says which, + `--glossary` on such a module says why it has none, and the summary and every + word answer count what they cannot see. +- **Lookup is exact for any string.** The `word` index holds what `->>` made of + the word; a lone surrogate's bytes are not the binding's, so such a word is + looked for by a scan, and JS equality decides either way. +- **Measured** through the CLI on the synthetic store at Broadleaf's scale: + `--word order` (549 entries, 19 glossary entries) 0.46 s, `--polysemy` + 0.48 s, `--concerns` 0.54 s, Node's start included. The re-explained + Broadleaf store is owed, as above. + ### 20.4 Milestone split **M19a — the lexicon, the split, the artifact** (closes the profile half of #19) @@ -5582,11 +5614,13 @@ and `confidence` are generated columns over it. The decisions: answer for a record explained under prompt `"1"`; on the fake client and live on the Java fixture; the §20.3 estimate replaced by the Broadleaf measurement. *(In: `--terms`, the estimate's per-word - figures, the fake-client run. Owed: the measurement, live runs, the - prompt-1 answer.)* + figures, the fake-client run, the prompt-1 answer. Owed: the + measurement, live runs.)* - [ ] Readers: `insights --word`, `--glossary`, `--polysemy`, `--concerns`; `vocabulary --drift-insights`; `/glossary.json`; the entries in the - Vocabulary tab and in the Navigate panel as inferences. + Vocabulary tab and in the Navigate panel as inferences. *(In: the four + `insights` readers. Owed: `--drift-insights`, `/glossary.json`, the tab + and the panel.)* - [ ] `docs/vocabulary.md` (the idioms per profile, the classes, the weights, the artifact, the three layers, the readers, the drift reports); README names the feature; `docs/cli.md` the command and the diff --git a/docs/cli.md b/docs/cli.md index 2fcebdb..97589da 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -87,6 +87,7 @@ codegraph explain [model.jsonl] [--src DIR] [--out FILE] [--dry-run] codegraph insights [model.jsonl] [--store FILE] [--json] [--id ID | --failures | --runs | + --word WORD | --glossary MODULE_ID | --polysemy | --concerns | --level operation|type|module --concept NAME --min-confidence X --max-confidence X --limit N] @@ -327,13 +328,36 @@ invocation: - `--id ID` — one explanation whole: envelope, description, block. An id that was asked for and **failed** is answered with the reason, exit `3`; an id the store has never heard of is a usage error; +- `--word WORD` — every explained entity whose names spell the word (looked up + lowercase, as `vocabulary` splits names), grouped by module: one + tab-separated line per entry — the module, `glossary` or the entity's level, + the id, the concern, the confidence, the sense, and the identifiers the word + is in. Each module's own glossary entry comes first (its last cell lists the + types' senses when the module flagged the word polysemous), then the module, + its types, its operations. A record is placed by its natural key; one + without a key is under `-`. The meaning is not shown: `--glossary` and `--id` + show it; +- `--glossary MODULE_ID` — a module's glossary whole, one line per word: the + word, the concern, sense and meaning the module reconciled from its types, + and the types' senses where they disagree; +- `--polysemy` — the words a module flagged, and the words two modules' + glossaries define differently (in concern or sense, case and spacing aside): + every module's entry for each, side by side, the most senses first. No + model call judges them across the corpus: a human does; +- `--concerns` — per module: how many distinct words its records define, and + how many the model called `domain`, `technical` and `ambiguous` (a word two + units judged differently counts under both), the most-worded module first; - `--failures` — the units still owed, with the reason each failed; - `--runs` — the ledger, oldest first: when, which models, what each run explained and spent. `--json` makes each answer one kind-tagged document (`codegraph.insightsSummary/1`, -`…List/1`, `…Failures/1`, `…Runs/1`; `--id --json` is the record exactly as the -side-car carries it). The command writes nothing: the store is opened as a +`…List/1`, `…Word/1`, `…Glossary/1`, `…Polysemy/1`, `…Concerns/1`, `…Failures/1`, +`…Runs/1`; `--id --json` is the record exactly as the side-car carries it). +A record a model explained under prompt 1 defines no words, so the word +answers cannot see it: they say on stderr how many such records there are, +the summary counts them, `--id` says so on the record, and `--glossary` on +such a module says why it has none. The next `explain` re-explains them. The command writes nothing: the store is opened as a reader, a missing one is a usage error rather than a file that appears, and one written by a newer codegraph — or a SQLite file that is not an insights store — is named and left alone. One an older build wrote is refused and left alone too: diff --git a/docs/insights.md b/docs/insights.md index e46d7fb..072e1f1 100644 --- a/docs/insights.md +++ b/docs/insights.md @@ -279,6 +279,26 @@ everything else on that page is extracted from source, and this is not. Operations are not shown: a navigator member node carries no entity id, and ids are never constructed. +**The words' readers (M19c).** `glossary.ts` rolls the records' `words` and +the modules' `glossary` up, over four more projections of the port: +`wordHits(word)` (the `insight_word` index, joined to each record's envelope), +`glossaryEntries(word?)` (each module block's `glossary`, through `json_each` +over the module records alone), `concernRows()` (each module, word and concern +once) and `withoutWords()`. A record is placed in its module by its natural +key: core renders the module's id from the key's `lang` and `module`, and the +groups are compared through `naturalKeyIndex`. The id is never parsed; a record +without a key is placed nowhere, and still answered. The index holds what +`->>` made of a word, which for a lone surrogate (no identifier spells one; a +hand-edited side-car might) is not what the binding sends, so such a word is +looked for by a scan. Either way, what is returned is what equals the word +once parsed back from JSON. A record a model wrote under prompt 1 has no +`words`: `wordsAbsence(record)` says so (`prompt-1`, or `templated` for a record +no model wrote), and every report counts those records, since it cannot see +them. `codegraph insights --word`, `--glossary`, `--polysemy` and `--concerns` +print them (cli.md). On the synthetic store of Broadleaf's size, +`--word order` prints its 549 entries and 19 glossary entries in under half a +second, Node's start included. + `@codegraph/insights` stays pure: `store.ts` is a port, and `store-sqlite.ts` is handed an OPEN database, so the path, the file and `node:sqlite` itself (one import site, in the analyzer's `store/sqlite.ts`) stay with the CLI. diff --git a/packages/cli/src/args.ts b/packages/cli/src/args.ts index c68c252..14d3965 100644 --- a/packages/cli/src/args.ts +++ b/packages/cli/src/args.ts @@ -758,7 +758,7 @@ export type InsightLevel = (typeof INSIGHT_LEVELS)[number]; */ export const INSIGHTS_SPEC: CommandSpec = { name: "insights", - summary: "Query what `explain` has written: a summary, a list, one explanation, the run ledger.", + summary: "Query what `explain` has written: a summary, a list, one explanation, the words and their glossary, the run ledger.", positional: { name: "model.jsonl", describe: "The model whose insights store (.insights.db, beside it) to read. The model itself is not opened.", @@ -812,6 +812,35 @@ export const INSIGHTS_SPEC: CommandSpec = { placeholder: "N", integer: true, }, + { + name: "word", + type: "string", + describe: + "List every explained entity whose names spell this word (split as 'codegraph vocabulary' splits names: lowercase), " + + "by module: the module's glossary entry, then the module, its types, its operations — each with the concern, " + + "confidence and sense the model gave the word there, and the identifiers it is in. One index lookup.", + placeholder: "WORD", + }, + { + name: "glossary", + type: "string", + describe: + "Print a module's glossary whole: each word its types spell, with the concern, sense and meaning the module " + + "reconciled from them, and the types' own senses where they disagree.", + placeholder: "MODULE_ID", + }, + { + name: "polysemy", + type: "boolean", + describe: + "List the words a module flagged as meaning different things to its types, and the words two modules' glossaries " + + "define differently: every module's entry, side by side, the most senses first.", + }, + { + name: "concerns", + type: "boolean", + describe: "Per module: how many distinct words its records define, and how many the model judged domain, technical, ambiguous.", + }, { name: "failures", type: "boolean", @@ -1314,7 +1343,7 @@ export interface ExplainOptions extends ModelInputOptions, ViewOptions, CacheOpt } /** What `insights` prints: exactly one of these, decided by the flags. */ -export type InsightsMode = "summary" | "list" | "one" | "failures" | "runs"; +export type InsightsMode = "summary" | "list" | "one" | "word" | "glossary" | "polysemy" | "concerns" | "failures" | "runs"; export interface InsightsOptions { /** The model path the store sits beside; never opened. */ @@ -1323,6 +1352,10 @@ export interface InsightsOptions { readonly store: string | undefined; readonly mode: InsightsMode; readonly id: string | undefined; + /** `--word WORD`, looked up lowercase. */ + readonly word: string | undefined; + /** `--glossary MODULE_ID`. */ + readonly glossary: string | undefined; readonly level: InsightLevel | undefined; readonly concept: string | undefined; readonly minConfidence: number | undefined; @@ -2013,18 +2046,30 @@ export function parseInvocation(argv: readonly string[]): Invocation { } case "insights": { const id = stringOf(values, "id"); + const word = stringOf(values, "word"); + const glossary = stringOf(values, "glossary"); const filters = ["level", "concept", "min-confidence", "max-confidence", "limit"].filter((name) => stringOf(values, name) !== undefined); - // One question per invocation: the four shapes print different things. - const asked = [id === undefined ? undefined : "--id", flagOf(values, "failures") ? "--failures" : undefined, flagOf(values, "runs") ? "--runs" : undefined, filters.length === 0 ? undefined : `--${filters[0] ?? ""}`].filter( - (flag): flag is string => flag !== undefined, - ); + // One question per invocation: each shape prints a different thing. + const questions: readonly (readonly [InsightsMode, string | undefined])[] = [ + ["one", id === undefined ? undefined : "--id"], + ["word", word === undefined ? undefined : "--word"], + ["glossary", glossary === undefined ? undefined : "--glossary"], + ["polysemy", flagOf(values, "polysemy") ? "--polysemy" : undefined], + ["concerns", flagOf(values, "concerns") ? "--concerns" : undefined], + ["failures", flagOf(values, "failures") ? "--failures" : undefined], + ["runs", flagOf(values, "runs") ? "--runs" : undefined], + ["list", filters[0] === undefined ? undefined : `--${filters[0]}`], + ]; + const asked = questions.filter(([, flag]) => flag !== undefined); if (asked.length > 1) { throw new UsageError( - `${asked.join(" and ")} ask different questions`, - "Pass one of: --id ID (one explanation), a list filter (--level, --concept, --min/--max-confidence), --failures, --runs — or none, for the summary.", + `${asked.map(([, flag]) => flag).join(" and ")} ask different questions`, + "Pass one of: --id ID (one explanation), a list filter (--level, --concept, --min/--max-confidence), --word WORD, " + + "--glossary MODULE_ID, --polysemy, --concerns, --failures, --runs — or none, for the summary.", ); } - const mode: InsightsMode = id !== undefined ? "one" : flagOf(values, "failures") ? "failures" : flagOf(values, "runs") ? "runs" : filters.length > 0 ? "list" : "summary"; + if (word?.trim() === "") throw new UsageError("--word needs a word", "A word as 'codegraph vocabulary' lists it, e.g. --word order."); + const mode: InsightsMode = asked[0]?.[0] ?? "summary"; return { kind: "run", command: "insights", @@ -2033,6 +2078,8 @@ export function parseInvocation(argv: readonly string[]): Invocation { store: stringOf(values, "store"), mode, id, + word, + glossary, level: stringOf(values, "level") as InsightLevel | undefined, concept: stringOf(values, "concept"), minConfidence: confidenceOf(values, "min-confidence"), diff --git a/packages/cli/src/commands/insights.ts b/packages/cli/src/commands/insights.ts index 2de34a5..989916c 100644 --- a/packages/cli/src/commands/insights.ts +++ b/packages/cli/src/commands/insights.ts @@ -2,9 +2,15 @@ import { existsSync } from "node:fs"; import { loadSqlite } from "@codegraph/analyzer"; import { InsightsStoreError, + WORD_CONCERNS, answerInsight, + concernsReport, + polysemyReport, sqliteInsightsStore, + wordReport, + wordsAbsence, type FailureRecord, + type GlossaryEntry, type InsightRecord, type InsightRow, type InsightsStore, @@ -29,8 +35,10 @@ import { storePathForModel } from "../insights-paths.js"; * * ONE QUESTION PER INVOCATION, each with its own shape: the summary (no flag), * a list of ROWS — envelope and description, never blocks — one explanation - * whole (`--id`), what is owed (`--failures`), the ledger (`--runs`). stdout is - * the answer; `--json` makes it one kind-tagged document. + * whole (`--id`), the words (`--word`, `--glossary`, `--polysemy`, + * `--concerns`: M19c's readers, `glossary.ts`), what is owed (`--failures`), + * the ledger (`--runs`). stdout is the answer; `--json` makes it one kind-tagged + * document. */ export function insightsCommand(options: InsightsOptions, io: IoSink): ExitCode { const path = options.store ?? storePathForModel(options.model); @@ -43,6 +51,14 @@ export function insightsCommand(options: InsightsOptions, io: IoSink): ExitCode return list(store, options, io); case "one": return one(store, options.id ?? "", path, options, io); + case "word": + return word(store, options.word ?? "", options, io); + case "glossary": + return glossary(store, options.glossary ?? "", path, options, io); + case "polysemy": + return polysemy(store, options, io); + case "concerns": + return concerns(store, options, io); case "failures": return failures(store, options, io); case "runs": @@ -96,6 +112,9 @@ function summary(store: InsightsStore, path: string, options: InsightsOptions, i lines.push(stats.failures === 0 ? " nothing owed: every unit asked for has a record" : ` ${plural(stats.failures, "failure")} owed — 'codegraph insights --failures' lists them, 'codegraph explain --retry-failed' redoes them`); lines.push(` these records cost ${stats.usage.promptTokens} prompt + ${stats.usage.completionTokens} completion tokens${usd(stats.usage.cost)} (their own usage; failed and replaced calls are in the ledger only)`); lines.push(` ledger: ${ledgerTotal(ledger)}`); + if (stats.withoutWords > 0) { + lines.push(` ${stats.withoutWords} explained under prompt 1, before words were asked: the next 'codegraph explain' re-explains them`); + } const open = ledger.filter((run) => run.finishedAt === undefined); if (open.length > 0) lines.push(` ${plural(open.length, "run")} never finished (still going, or killed): its records are kept`); outLine(io, lines.join("\n")); @@ -161,14 +180,21 @@ function one(store: InsightsStore, id: string, path: string, options: InsightsOp } } +const ABSENT_WORDS = { + "prompt-1": " words: not defined — explained under prompt 1, before words were asked; the next 'codegraph explain' re-explains it", + templated: " words: none — templated, no model judged it", +} as const; + function recordText(record: InsightRecord): string { const { description, confidence, ...rest } = record.block; const concept = typeConcept(record); const usage = record.usage === undefined ? undefined : `${record.usage.promptTokens} + ${record.usage.completionTokens} tokens${usd(record.usage.cost)}`; + const absence = wordsAbsence(record); return [ record.id, ` ${[`${record.level} (${record.kind})`, concept, `confidence ${confidence.toFixed(2)}`].filter((part) => part !== undefined).join(" · ")}`, ` ${[record.file, record.origin, record.model, usage].filter((part) => part !== undefined).join(" · ")}`, + ...(absence === undefined ? [] : [ABSENT_WORDS[absence]]), "", description, "", @@ -185,6 +211,86 @@ function failureReason(failure: FailureRecord): string { return `${failure.reason.kind}${failure.reason.status === undefined ? "" : ` ${failure.reason.status}`}: ${failure.reason.message}`; } +/** What every word answer is blind to, on stderr, where a pipe does not read it. */ +function noteWithoutWords(count: number, io: IoSink): void { + if (count === 0) return; + errLine(io, `note: ${plural(count, "record")} a model wrote under prompt 1 define${count === 1 ? "s" : ""} no words; the next 'codegraph explain' re-explains ${count === 1 ? "it" : "them"}.`); +} + +/** A cell for a placed-nowhere module: a record without a natural key. */ +const moduleCell = (module: string | undefined): string => module ?? "-"; + +/** The types' senses of a word their module flagged; an empty cell when they agree. */ +function polysemyCell(entry: GlossaryEntry): string { + return entry.polysemous ? `polysemous: ${entry.senses.map((s) => `${s.in} "${s.sense}"`).join(" · ")}` : ""; +} + +/** One tab-separated line per entry, the module first so the grouping survives `sort`: module, what, id, concern, confidence, sense, and the identifiers (or the flag). */ +function word(store: InsightsStore, asked: string, options: InsightsOptions, io: IoSink): ExitCode { + const report = wordReport(store, asked); + if (options.json) { + outLine(io, JSON.stringify({ kind: "codegraph.insightsWord/1", ...report })); + return EXIT.OK; + } + const lines = report.modules.flatMap((group) => [ + ...group.glossary.map((g) => [moduleCell(group.module), "glossary", g.id, g.entry.concern, "", g.entry.sense, polysemyCell(g.entry)].join("\t")), + ...group.entries.map((e) => [moduleCell(group.module), e.level, e.id, e.entry.concern, e.entry.confidence.toFixed(2), e.entry.sense, e.entry.in.join(", ")].join("\t")), + ]); + if (lines.length === 0) errLine(io, `no explained entity defines '${report.word}'.`); + else outLine(io, lines.join("\n")); + noteWithoutWords(report.withoutWords, io); + return EXIT.OK; +} + +function glossary(store: InsightsStore, id: string, path: string, options: InsightsOptions, io: IoSink): ExitCode { + const answer = answerInsight(store, id); + if (answer.status !== "explained") return one(store, id, path, options, io); + const record = answer.record; + if (record.level !== "module") { + throw new UsageError(`${id} is a ${record.level}: a glossary is a module's`, `'codegraph insights --id ${id}' shows the words its names spell.`); + } + const entries = record.block.glossary; + if (entries === undefined) { + // A module is never templated; one without a glossary was explained before glossaries were asked. + const absence = wordsAbsence(record) ?? "prompt-1"; + if (options.json) outLine(io, JSON.stringify({ kind: "codegraph.insightsGlossary/1", id, glossary: null, absence })); + else errLine(io, `${id} was explained under prompt 1, which asked for no glossary: the next 'codegraph explain' re-explains it.`); + return EXIT.OK; + } + if (options.json) { + outLine(io, JSON.stringify({ kind: "codegraph.insightsGlossary/1", id, name: record.name, model: record.model, glossary: entries })); + return EXIT.OK; + } + if (entries.length === 0) errLine(io, `${id}'s glossary is empty: its types spell no word to reconcile.`); + else outLine(io, entries.map((e) => [e.word, e.concern, e.sense, e.meaning, polysemyCell(e)].join("\t")).join("\n")); + return EXIT.OK; +} + +/** One line per module's entry, grouped by word: word, module, concern, sense, the flag. */ +function polysemy(store: InsightsStore, options: InsightsOptions, io: IoSink): ExitCode { + const report = polysemyReport(store); + if (options.json) { + outLine(io, JSON.stringify({ kind: "codegraph.insightsPolysemy/1", ...report })); + return EXIT.OK; + } + if (report.words.length === 0) errLine(io, "no word has two senses: no module flagged one, and no two modules' glossaries define one differently."); + else outLine(io, report.words.flatMap((w) => w.entries.map((g) => [w.word, g.id, g.entry.concern, g.entry.sense, polysemyCell(g.entry)].join("\t"))).join("\n")); + noteWithoutWords(report.withoutWords, io); + return EXIT.OK; +} + +function concerns(store: InsightsStore, options: InsightsOptions, io: IoSink): ExitCode { + const report = concernsReport(store); + if (options.json) { + outLine(io, JSON.stringify({ kind: "codegraph.insightsConcerns/1", ...report })); + return EXIT.OK; + } + if (report.modules.length === 0) errLine(io, "no explained entity defines a word."); + else outLine(io, report.modules.map((m) => [moduleCell(m.module), plural(m.words, "word"), ...WORD_CONCERNS.map((c) => `${m.concerns[c]} ${c}`)].join("\t")).join("\n")); + noteWithoutWords(report.withoutWords, io); + return EXIT.OK; +} + function failures(store: InsightsStore, options: InsightsOptions, io: IoSink): ExitCode { const owed = store.failures(); if (options.json) { diff --git a/packages/cli/test/insights.test.ts b/packages/cli/test/insights.test.ts index ac9c545..a2c64d8 100644 --- a/packages/cli/test/insights.test.ts +++ b/packages/cli/test/insights.test.ts @@ -13,9 +13,13 @@ import { encodeInsightsToString, sqliteInsightsStore, type FailureRecord, + type GlossaryEntry, type InsightRecord, type InsightsEof, type InsightsHeader, + type RecordKey, + type WordConcern, + type WordEntry, } from "@codegraph/insights"; import { EXIT } from "../src/exit.js"; import { captureIo } from "../src/io.js"; @@ -88,6 +92,8 @@ describe("codegraph insights: what was bought, at a glance", () => { expect(out).toMatch(/1 failure owed/u); expect(out).toMatch(/these records cost 2400 prompt \+ 600 completion tokens, \$0\.0040/u); expect(out).toMatch(/ledger: 1 run, 4 calls, 2400 prompt \+ 600 completion tokens, \$0\.0041/u); + // Three were explained before words were asked; the template never is. + expect(out).toMatch(/3 explained under prompt 1, before words were asked: the next 'codegraph explain' re-explains them/u); }); it("--json is one kind-tagged document with the same facts", () => { @@ -237,3 +243,143 @@ describe("codegraph insights: no store, or not ours", () => { } }); }); + +describe("codegraph insights: the words of the names, as the model judged them (M19c)", () => { + const key = (module: string, symbol = "", disambiguator?: string): RecordKey => ({ lang: "java", module, symbol, ...(disambiguator === undefined ? {} : { disambiguator }) }); + const word = (w: string, concern: WordConcern, sense: string, ins: string[], confidence: number): WordEntry => ({ word: w, in: ins, concern, sense, meaning: `${sense}, here.`, confidence }); + const gloss = (w: string, concern: WordConcern, sense: string, senses: GlossaryEntry["senses"] = []): GlossaryEntry => ({ word: w, concern, sense, meaning: `What ${w} means here.`, polysemous: senses.length > 0, senses }); + const ORDERING = "java:com.acme.order"; + const BILLING = "java:com.acme.billing"; + const MODULE_BLOCK = { name: "M", description: "A module.", apis: [], spis: [], dependsOn: [], concepts: [], boundedContextHint: null, sharedKernelHint: null, ubiquitousLanguage: [], confidence: 0.6 }; + const worded = (record: InsightRecord, k: RecordKey, words: WordEntry[]): InsightRecord => ({ ...record, key: k, block: { ...record.block, words } }) as InsightRecord; + const module_ = (id: string, k: RecordKey, words: WordEntry[], glossary?: GlossaryEntry[]): InsightRecord => + ({ t: "i", id, key: k, level: "module", kind: "package", name: k.module, origin: "llm", fingerprint: FP, model: "strong/model", block: { ...MODULE_BLOCK, words, ...(glossary === undefined ? {} : { glossary }) } }) as InsightRecord; + const WORDED = [ + module_(ORDERING, key("com.acme.order"), [word("order", "domain", "the ordering business", ["com.acme.order"], 0.8)], [ + gloss("order", "domain", "a customer's request", [{ in: "Order", sense: "a customer's request" }, { in: "SortSpec", sense: "a sequence" }]), + gloss("lines", "domain", "the items ordered"), + ]), + worded(type(ORDER, "aggregate", 0.9), key("com.acme.order", "Order"), [word("order", "domain", "a customer's request", ["Order"], 0.9), word("lines", "domain", "the items ordered", ["lines", "addLine"], 0.85)]), + worded(type("java:com.acme.order/SortSpec", "valueType", 0.7), key("com.acme.order", "SortSpec"), [word("order", "technical", "a sequence", ["sortOrder"], 0.6)]), + worded(operation(`${ORDER}.total()`), key("com.acme.order", "Order.total", "()"), [word("total", "domain", "the sum", ["total"], 0.7)]), + module_(BILLING, key("com.acme.billing"), [word("billing", "domain", "invoicing", ["com.acme.billing"], 0.8)], [gloss("order", "domain", "a purchase order")]), + worded(type("java:com.acme.billing/Invoice", "entity", 0.8), key("com.acme.billing", "Invoice"), [word("order", "domain", "a purchase order", ["orderRef"], 0.75)]), + // Explained before words were asked. + { ...module_("java:com.acme.legacy", key("com.acme.legacy"), []), block: MODULE_BLOCK } as InsightRecord, + { ...type("java:com.acme.legacy/Old", "entity", 0.5), key: key("com.acme.legacy", "Old") } as InsightRecord, + ]; + + let wdir: string; + let wmodel: string; + beforeAll(() => { + wdir = mkdtempSync(join(tmpdir(), "insights-words-")); + wmodel = join(wdir, "model.jsonl"); + writeFileSync(wmodel, "not a model\n"); + const store = sqliteInsightsStore(loadSqlite().open(join(wdir, "model.insights.db"))); + store.importFile(decodeInsights(encodeInsightsToString(HEADER, WORDED, eof(WORDED.length)))); + store.close(); + }); + afterAll(() => rmSync(wdir, { recursive: true, force: true })); + const words = (...argv: string[]) => { + const io = captureIo(); + const code = runSync(["insights", wmodel, ...argv], io); + return { code, out: io.stdout(), err: io.stderr() }; + }; + // Only the final newline goes: a row may end in an empty cell. + const rows = (out: string): string[] => out.replace(/\n$/u, "").split("\n"); + const PROMPT_1 = "note: 2 records a model wrote under prompt 1 define no words; the next 'codegraph explain' re-explains them."; + + it("--word: every entity whose names spell it, by module — the module's glossary entry first, then the module, its types, its operations", () => { + const { code, out, err } = words("--word", "Order"); + expect(code).toBe(EXIT.OK); + expect(rows(out)).toEqual([ + `${BILLING}\tglossary\t${BILLING}\tdomain\t\ta purchase order\t`, + `${BILLING}\ttype\tjava:com.acme.billing/Invoice\tdomain\t0.75\ta purchase order\torderRef`, + `${ORDERING}\tglossary\t${ORDERING}\tdomain\t\ta customer's request\tpolysemous: Order "a customer's request" · SortSpec "a sequence"`, + `${ORDERING}\tmodule\t${ORDERING}\tdomain\t0.80\tthe ordering business\tcom.acme.order`, + `${ORDERING}\ttype\t${ORDER}\tdomain\t0.90\ta customer's request\tOrder`, + `${ORDERING}\ttype\tjava:com.acme.order/SortSpec\ttechnical\t0.60\ta sequence\tsortOrder`, + ]); + // What it cannot see, said once, where a pipe does not read it. + expect(err.trim()).toBe(PROMPT_1); + }); + + it("--word --json is the report: the word as looked up, by module, entries without their meaning", () => { + const doc = JSON.parse(words("--word", "lines", "--json").out) as { kind: string; word: string; withoutWords: number; modules: { module: string; glossary: { entry: GlossaryEntry }[]; entries: { id: string; entry: object }[] }[] }; + expect(doc.kind).toBe("codegraph.insightsWord/1"); + expect(doc.word).toBe("lines"); + expect(doc.withoutWords).toBe(2); + expect(doc.modules.map((m) => m.module)).toEqual([ORDERING]); + expect(doc.modules[0]?.glossary[0]?.entry.meaning).toBe("What lines means here."); + expect(doc.modules[0]?.entries[0]).toMatchObject({ id: ORDER, entry: { word: "lines", in: ["lines", "addLine"], concern: "domain", sense: "the items ordered", confidence: 0.85 } }); + expect(doc.modules[0]?.entries[0]?.entry).not.toHaveProperty("meaning"); + }); + + it("--word that nothing defines says so on stderr, and prints nothing", () => { + const { code, out, err } = words("--word", "zebra"); + expect(code).toBe(EXIT.OK); + expect(out).toBe(""); + expect(err).toMatch(/no explained entity defines 'zebra'/u); + expect(err).toContain(PROMPT_1); + }); + + it("--glossary: a module's whole — word, concern, sense, meaning, and the types' senses when they disagree", () => { + const { code, out } = words("--glossary", ORDERING); + expect(code).toBe(EXIT.OK); + expect(rows(out)).toEqual([ + `order\tdomain\ta customer's request\tWhat order means here.\tpolysemous: Order "a customer's request" · SortSpec "a sequence"`, + `lines\tdomain\tthe items ordered\tWhat lines means here.\t`, + ]); + const doc = JSON.parse(words("--glossary", ORDERING, "--json").out) as { kind: string; id: string; glossary: GlossaryEntry[] }; + expect(doc).toMatchObject({ kind: "codegraph.insightsGlossary/1", id: ORDERING, name: "com.acme.order", model: "strong/model" }); + expect(doc.glossary.map((g) => g.word)).toEqual(["order", "lines"]); + }); + + it("--glossary of a module explained under prompt 1 says why there is none; of a type, of an unknown id, it is a usage error", () => { + const old = words("--glossary", "java:com.acme.legacy"); + expect(old.code).toBe(EXIT.OK); + expect(old.out).toBe(""); + expect(old.err).toMatch(/java:com\.acme\.legacy was explained under prompt 1, which asked for no glossary: the next 'codegraph explain' re-explains it/u); + expect(JSON.parse(words("--glossary", "java:com.acme.legacy", "--json").out)).toEqual({ kind: "codegraph.insightsGlossary/1", id: "java:com.acme.legacy", glossary: null, absence: "prompt-1" }); + const typed = words("--glossary", ORDER); + expect(typed.code).toBe(EXIT.USAGE); + expect(typed.err).toMatch(/is a type: a glossary is a module's.*--id/su); + expect(words("--glossary", "java:nope").code).toBe(EXIT.USAGE); + }); + + it("--polysemy: every module's entry for a word a module flagged, or that two modules' glossaries define differently, side by side", () => { + const { code, out } = words("--polysemy"); + expect(code).toBe(EXIT.OK); + expect(rows(out)).toEqual([ + `order\t${BILLING}\tdomain\ta purchase order\t`, + `order\t${ORDERING}\tdomain\ta customer's request\tpolysemous: Order "a customer's request" · SortSpec "a sequence"`, + ]); + const doc = JSON.parse(words("--polysemy", "--json").out) as { kind: string; words: { word: string; flagged: boolean; acrossModules: boolean; senses: number }[] }; + expect(doc.kind).toBe("codegraph.insightsPolysemy/1"); + expect(doc.words.map(({ word: w, flagged, acrossModules, senses }) => ({ w, flagged, acrossModules, senses }))).toEqual([{ w: "order", flagged: true, acrossModules: true, senses: 3 }]); + }); + + it("--concerns: per module, its distinct words and how many the model called domain, technical, ambiguous", () => { + const { code, out } = words("--concerns"); + expect(code).toBe(EXIT.OK); + expect(rows(out)).toEqual([ + `${ORDERING}\t3 words\t3 domain\t1 technical\t0 ambiguous`, + `${BILLING}\t2 words\t2 domain\t0 technical\t0 ambiguous`, + ]); + const doc = JSON.parse(words("--concerns", "--json").out) as { kind: string; modules: unknown[] }; + expect(doc.kind).toBe("codegraph.insightsConcerns/1"); + expect(doc.modules).toHaveLength(2); + }); + + it("one question per invocation, and a word to look up", () => { + expect(words("--word", "order", "--concerns").code).toBe(EXIT.USAGE); + expect(words("--polysemy", "--level", "type").code).toBe(EXIT.USAGE); + expect(words("--word", "").code).toBe(EXIT.USAGE); + }); + + it("--id says why a record has no words: prompt 1's, or a template's", () => { + expect(words("--id", "java:com.acme.legacy/Old").out).toMatch(/words: not defined — explained under prompt 1, before words were asked; the next 'codegraph explain' re-explains it/u); + expect(insights("--id", `${ORDER}.getId()`).out).toMatch(/words: none — templated, no model judged it/u); + expect(words("--id", ORDER).out).not.toMatch(/words: (not defined|none)/u); + }); +}); diff --git a/packages/insights/src/glossary.ts b/packages/insights/src/glossary.ts new file mode 100644 index 0000000..d2d1cd4 --- /dev/null +++ b/packages/insights/src/glossary.ts @@ -0,0 +1,165 @@ +import { naturalKeyIndex, naturalKeyIssues, renderId, type NaturalKey } from "@codegraph/core"; +import { WORD_CONCERNS, type InsightRecord, type Level, type RecordKey, type WordConcern } from "./schema.js"; +import type { GlossaryHit, InsightsStore, WordHit } from "./store.js"; + +/** + * THE READERS OF THE WORDS (PLAN.md §20.3, layer 3): what the records' `words` + * and the modules' `glossary` say, rolled up over the store. None calls a + * model; none opens the model either — a record is placed in its module by its + * natural key, rendered by core, never by parsing its id. + * + * A record a model wrote under prompt 1 defines no word, so every report also + * says how many such records it cannot see (`withoutWords`). + */ + +/** Why a record carries no `words`: a model explained it under prompt 1, before words were asked; or no model did (a template). */ +export type WordsAbsence = "prompt-1" | "templated"; + +export function wordsAbsence(record: InsightRecord): WordsAbsence | undefined { + if (record.block.words !== undefined) return undefined; + return record.origin === "template" ? "templated" : "prompt-1"; +} + +/** The rendered id of the module a key names; undefined without a key, or for one core cannot render (a hand-edited side-car's). */ +export function moduleOf(key: RecordKey | undefined): string | undefined { + const module = moduleKey(key); + return module === undefined ? undefined : renderId(module); +} + +function moduleKey(key: RecordKey | undefined): NaturalKey | undefined { + if (key === undefined) return undefined; + // MM-1: a module names itself, its symbol empty. + const module: NaturalKey = { lang: key.lang, module: key.module, symbol: "" }; + return naturalKeyIssues(module).length === 0 ? module : undefined; +} + +const byCodeUnits = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0); + +/** Module ids in code-unit order, the records without one last. */ +const byModule = (a: string | undefined, b: string | undefined): number => + a === undefined ? (b === undefined ? 0 : 1) : b === undefined ? -1 : byCodeUnits(a, b); + +interface ModuleGroup { + /** `naturalKeyIndex`, the comparable form: the rendered id is for display (MM-1). */ + readonly index: string | undefined; + readonly module: string | undefined; + readonly items: T[]; +} + +function groupByModule(items: readonly T[], keyOf: (item: T) => RecordKey | undefined): Map> { + const groups = new Map>(); + for (const item of items) { + const key = moduleKey(keyOf(item)); + const index = key === undefined ? undefined : naturalKeyIndex(key); + let group = groups.get(index); + if (group === undefined) { + group = { index, module: key === undefined ? undefined : renderId(key), items: [] }; + groups.set(index, group); + } + group.items.push(item); + } + return groups; +} + +export interface WordModule { + /** The module's rendered id; undefined gathers the records whose natural key is missing or cannot render. */ + readonly module: string | undefined; + /** The module's own reconciliation of the word, when its glossary has one. */ + readonly glossary: readonly GlossaryHit[]; + /** The module itself, then its types, then its operations; by id within a level. */ + readonly entries: readonly WordHit[]; +} + +export interface WordReport { + /** As the splitter spells it: lowercase. */ + readonly word: string; + readonly modules: readonly WordModule[]; + readonly withoutWords: number; +} + +const TOP_DOWN: Readonly> = { module: 0, type: 1, operation: 2 }; + +/** `insights --word`: every entity whose names spell the word, by module, from one index lookup. */ +export function wordReport(store: InsightsStore, word: string): WordReport { + // `toLowerCase`, as the splitter: locale-independent. + const spelled = word.toLowerCase(); + const entries = groupByModule(store.wordHits(spelled), (hit) => hit.key); + const glossaries = groupByModule(store.glossaryEntries(spelled), (hit) => hit.key); + const modules: WordModule[] = []; + for (const group of entries.values()) { + const sorted = [...group.items].sort((a, b) => TOP_DOWN[a.level] - TOP_DOWN[b.level] || byCodeUnits(a.id, b.id)); + modules.push({ module: group.module, glossary: glossaries.get(group.index)?.items ?? [], entries: sorted }); + } + // A glossary entry with no entity behind it here (its types answered without the word) is still the module's. + for (const group of glossaries.values()) if (!entries.has(group.index)) modules.push({ module: group.module, glossary: group.items, entries: [] }); + return { word: spelled, modules: modules.sort((a, b) => byModule(a.module, b.module)), withoutWords: store.withoutWords() }; +} + +export interface PolysemousWord { + readonly word: string; + /** A module's glossary flags it: that module's types disagree on it. */ + readonly flagged: boolean; + /** The modules' entries give it more than one sense. */ + readonly acrossModules: boolean; + /** Distinct senses over the entries and the senses a flag lists. */ + readonly senses: number; + /** Every module's entry for the word, by module id: side by side, for a human to judge. */ + readonly entries: readonly GlossaryHit[]; +} + +export interface PolysemyReport { + /** The most senses first, then by word. */ + readonly words: readonly PolysemousWord[]; + readonly withoutWords: number; +} + +/** Two senses are one when they differ only in case and spacing: the model's wording varies, its meaning is what is compared. */ +const sense = (text: string): string => text.trim().replace(/\s+/gu, " ").toLowerCase(); + +/** `insights --polysemy`: the corpus has no call to judge a word across modules, so this lays the modules' senses side by side. */ +export function polysemyReport(store: InsightsStore): PolysemyReport { + const byWord = new Map(); + for (const hit of store.glossaryEntries()) { + const hits = byWord.get(hit.entry.word) ?? []; + hits.push(hit); + byWord.set(hit.entry.word, hits); + } + const words: PolysemousWord[] = []; + for (const [word, entries] of byWord) { + const flagged = entries.some((hit) => hit.entry.polysemous); + const judged = new Set(entries.map((hit) => `${hit.entry.concern}\u0000${sense(hit.entry.sense)}`)); + const senses = new Set(entries.flatMap((hit) => [hit.entry.sense, ...hit.entry.senses.map((s) => s.sense)].map(sense))); + if (flagged || judged.size > 1) words.push({ word, flagged, acrossModules: judged.size > 1, senses: senses.size, entries }); + } + words.sort((a, b) => b.senses - a.senses || byCodeUnits(a.word, b.word)); + return { words, withoutWords: store.withoutWords() }; +} + +export interface ModuleConcerns { + readonly module: string | undefined; + /** Distinct words its records define. */ + readonly words: number; + /** Distinct words given each concern: a word two units judged differently counts under both. */ + readonly concerns: Readonly>; +} + +export interface ConcernsReport { + /** The most words first, then by module id; the records without a natural key last. */ + readonly modules: readonly ModuleConcerns[]; + readonly withoutWords: number; +} + +/** `insights --concerns`: per module, the cognitive-load proxy of PLAN.md §20.3 split by what the model judged each word to be about. */ +export function concernsReport(store: InsightsStore): ConcernsReport { + const groups = groupByModule(store.concernRows(), (row) => + row.lang === undefined || row.module === undefined ? undefined : { lang: row.lang, module: row.module, symbol: "" }, + ); + const modules = [...groups.values()].map((group): ModuleConcerns => { + const concerns = Object.fromEntries(WORD_CONCERNS.map((concern) => [concern, 0])) as Record; + for (const row of group.items) concerns[row.concern] += 1; + return { module: group.module, words: new Set(group.items.map((row) => row.word)).size, concerns }; + }); + const unplaced = (m: ModuleConcerns): number => (m.module === undefined ? 1 : 0); + modules.sort((a, b) => unplaced(a) - unplaced(b) || b.words - a.words || byModule(a.module, b.module)); + return { modules, withoutWords: store.withoutWords() }; +} diff --git a/packages/insights/src/index.ts b/packages/insights/src/index.ts index 1444461..b28c38c 100644 --- a/packages/insights/src/index.ts +++ b/packages/insights/src/index.ts @@ -21,6 +21,7 @@ * sidecar.ts the `.insights.jsonl` file — the store's export * store.ts the insights store's port: `.insights.db`, never a cache * store-sqlite.ts its SQLite adapter, over a database the caller opened + * glossary.ts the readers of the words: by word, polysemy, concerns (M19c) */ export * from "./ddd.js"; export * from "./schema.js"; @@ -37,3 +38,4 @@ export * from "./run.js"; export * from "./sidecar.js"; export * from "./store.js"; export * from "./store-sqlite.js"; +export * from "./glossary.js"; diff --git a/packages/insights/src/store-sqlite.ts b/packages/insights/src/store-sqlite.ts index 345a5d3..f95968e 100644 --- a/packages/insights/src/store-sqlite.ts +++ b/packages/insights/src/store-sqlite.ts @@ -5,7 +5,9 @@ import { InsightsEof, InsightsHeader, LEVELS, + type GlossaryEntry, type Level, + type WordConcern, type WordEntry, } from "./schema.js"; import { conceptLabel, type RecordSummary } from "./records.js"; @@ -14,17 +16,21 @@ import { INSIGHTS_APPLICATION_ID, INSIGHTS_DB_VERSION, InsightsStoreError, + type ConcernRow, type ExportStamp, + type GlossaryHit, type InsightQuery, type InsightRow, type InsightStats, type InsightsStore, + type ListedWord, type OpenRun, type RunEnd, type RunHandle, type RunRow, type RunStart, type StoreOpenOptions, + type WordHit, } from "./store.js"; /** @@ -60,6 +66,13 @@ import { * index lookup rather than a scan of every block. It is written with its * record, in the same transaction, and rebuilt by an import: the block stays * the one copy that counts, and nothing is exported from the projection. + * + * ITS READERS NARROW IN SQL AND DECIDE IN JS. A word is looked up through the + * `word` index, which holds what `->>` made of it: a lone surrogate (no + * identifier spells one, a hand-edited side-car might) is not the bytes the + * binding sends for it, so such a word is looked for by a scan instead. Either + * way, and for the glossaries, what is returned is what equals the word once + * parsed back from JSON — never what SQLite's text comparison let through. */ /** Index i takes a store from version i to i+1. Append; never edit a shipped entry. */ @@ -440,9 +453,57 @@ export function sqliteInsightsStore(db: SqliteDatabase, options: StoreOpenOption byOrigin, byConcept, usage: { promptTokens: Number(spent?.["p"] ?? 0), completionTokens: Number(spent?.["c"] ?? 0), ...(cost === undefined ? {} : { cost }) }, + withoutWords: withoutWords(), }; }; + // A template has no words by design (no model judged it): only a record a model wrote is owed them. + const selectWithoutWords = db.prepare("SELECT count(*) AS n FROM insight WHERE origin = 'llm' AND block -> '$.words' IS NULL"); + const withoutWords = (): number => Number(selectWithoutWords.get()?.["n"] ?? 0); + + const HIT_SQL = + "SELECT w.entry AS entry, i.id AS id, i.level AS level, i.kind AS kind, i.name AS name, i.model AS model, i.key_lang, i.key_module, i.key_symbol, i.key_disambiguator FROM insight_word w JOIN insight i ON i.id = w.id"; + const hitsOfWord = db.prepare(`${HIT_SQL} WHERE w.word = ? ORDER BY w.rowid`); + const everyHit = db.prepare(`${HIT_SQL} ORDER BY w.rowid`); + const wordHits = (word: string): WordHit[] => { + const rows = /\p{Surrogate}/u.test(word) ? everyHit.iterate() : hitsOfWord.iterate(word); + const hits: WordHit[] = []; + for (const row of rows) { + const entry = JSON.parse(row["entry"] as string) as ListedWord; + if (entry.word !== word) continue; + const level = LEVELS[Number(row["level"])] as Level; + hits.push({ id: row["id"] as string, level, kind: row["kind"] as string, name: text(row, "name"), key: keyOf(row), model: text(row, "model"), entry }); + } + // Side-car order; the sort is stable, so a record's entries stay in block order (rowid). + return hits.sort((a, b) => LEVEL_RANK[a.level] - LEVEL_RANK[b.level] || (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)); + }; + + const selectGlossaries = db.prepare( + "SELECT g.value AS entry, i.id AS id, i.name AS name, i.model AS model, i.key_lang, i.key_module, i.key_symbol, i.key_disambiguator FROM insight i, json_each(i.block, '$.glossary') AS g WHERE i.level = ? ORDER BY i.rowid, g.key", + ); + const glossaryEntries = (word?: string): GlossaryHit[] => { + const hits: GlossaryHit[] = []; + for (const row of selectGlossaries.iterate(LEVEL_RANK.module)) { + const entry = JSON.parse(row["entry"] as string) as GlossaryEntry; + if (word !== undefined && entry.word !== word) continue; + hits.push({ id: row["id"] as string, key: keyOf(row), name: text(row, "name"), model: text(row, "model"), entry }); + } + return hits.sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)); + }; + + // `->`: the word as JSON text, which keeps what a TEXT column would cut short. + const selectConcerns = db.prepare( + "SELECT DISTINCT i.key_lang, i.key_module, w.entry -> '$.word' AS word, w.concern AS concern FROM insight_word w JOIN insight i ON i.id = w.id", + ); + const concernRows = (): ConcernRow[] => { + const rows = new Map(); + for (const row of selectConcerns.iterate()) { + const out: ConcernRow = { lang: text(row, "key_lang"), module: text(row, "key_module"), word: JSON.parse(row["word"] as string) as string, concern: row["concern"] as WordConcern }; + rows.set(JSON.stringify([out.lang ?? null, out.module ?? null, out.word, out.concern]), out); + } + return [...rows.values()]; + }; + const runs = (): RunRow[] => db.prepare("SELECT * FROM run ORDER BY id").all().map((row) => { const cost = number(row, "cost"); @@ -488,6 +549,10 @@ export function sqliteInsightsStore(db: SqliteDatabase, options: StoreOpenOption failures, query, stats, + wordHits, + glossaryEntries, + concernRows, + withoutWords, runs, fingerprints() { diff --git a/packages/insights/src/store.ts b/packages/insights/src/store.ts index dd34f8d..eafe047 100644 --- a/packages/insights/src/store.ts +++ b/packages/insights/src/store.ts @@ -1,4 +1,4 @@ -import type { FailureRecord, InsightRecord, InsightsEof, InsightsHeader, Level, Origin, RecordUsage, WordEntry } from "./schema.js"; +import type { FailureRecord, GlossaryEntry, InsightRecord, InsightsEof, InsightsHeader, Level, Origin, RecordKey, RecordUsage, WordConcern, WordEntry } from "./schema.js"; import type { RecordSummary } from "./records.js"; import type { InsightsFile } from "./sidecar.js"; @@ -108,6 +108,40 @@ export interface InsightStats { * a replaced record's and a failed call's tokens are in the ledger only. */ readonly usage: RecordUsage; + /** Records a model wrote with no `words`: explained under prompt 1, before words were asked (M19c). */ + readonly withoutWords: number; +} + +/** A word entry as `insight_word` holds it: all but `meaning`, which a reader that shows it reads from the block. */ +export type ListedWord = Omit; + +/** One record's entry for a word, with what a reader shows of the record (M19c). */ +export interface WordHit { + readonly id: string; + readonly level: Level; + readonly kind: string; + readonly name: string | undefined; + /** Where the record sits; a reader groups by it, never by parsing the id. */ + readonly key: RecordKey | undefined; + readonly model: string | undefined; + readonly entry: ListedWord; +} + +/** One entry of a module record's `glossary`. */ +export interface GlossaryHit { + readonly id: string; + readonly key: RecordKey | undefined; + readonly name: string | undefined; + readonly model: string | undefined; + readonly entry: GlossaryEntry; +} + +/** A word and a concern some record gave it, with that record's module (its natural key's). */ +export interface ConcernRow { + readonly lang: string | undefined; + readonly module: string | undefined; + readonly word: string; + readonly concern: WordConcern; } /** One line of the ledger: what the side-car's single trailer could never say. */ @@ -155,6 +189,14 @@ export interface InsightsStore { /** Rows matching a reader's filter, in side-car order. */ query(query: InsightQuery): InsightRow[]; stats(): InsightStats; + /** Every record's entry for one word, in side-car order then block order: an index lookup (M19c). */ + wordHits(word: string): WordHit[]; + /** The module records' glossary entries, in side-car order then block order; `word` keeps that word's. */ + glossaryEntries(word?: string): GlossaryHit[]; + /** Each (module, word, concern) the records' words hold, once; in no promised order. */ + concernRows(): ConcernRow[]; + /** `stats().withoutWords` alone. */ + withoutWords(): number; /** The ledger, oldest first. */ runs(): RunRow[]; exported(): ExportStamp | undefined; diff --git a/packages/insights/test/glossary.test.ts b/packages/insights/test/glossary.test.ts new file mode 100644 index 0000000..d348451 --- /dev/null +++ b/packages/insights/test/glossary.test.ts @@ -0,0 +1,191 @@ +import { describe, expect, it } from "vitest"; +import { loadSqlite } from "@codegraph/analyzer"; +import { + INSIGHTS_GENERATOR, + INSIGHTS_KIND, + INSIGHTS_METAMODEL, + PROMPT_VERSION, + type GlossaryEntry, + type InsightRecord, + type InsightsHeader, + type Level, + type ModuleBlock, + type OperationBlock, + type RecordKey, + type TypeBlock, + type WordConcern, + type WordEntry, +} from "../src/schema.js"; +import { decodeInsights, encodeInsightsToString } from "../src/sidecar.js"; +import { sqliteInsightsStore } from "../src/store-sqlite.js"; +import { concernsReport, moduleOf, polysemyReport, wordReport, wordsAbsence } from "../src/glossary.js"; + +/** + * LAYER 3 OF PLAN.md §20.3: the readers of the words, over a real store. Two + * modules that both speak `order`, one meaning a customer's request (and, in + * one type, a sequence), the other a purchase order; a record a model wrote + * under prompt 1; a template; and a record without a natural key. + */ + +const HEADER: InsightsHeader = { + t: "header", kind: INSIGHTS_KIND, generatedBy: INSIGHTS_GENERATOR, promptVersion: PROMPT_VERSION, metamodel: INSIGHTS_METAMODEL, + models: { leaf: "m", rollup: "m" }, provider: "openrouter", depth: 1, + source: { paths: ["model.jsonl"], langs: ["java"], view: { name: "all", filters: [] } }, +}; +const OPERATION: OperationBlock = { + name: "f", description: "Does f.", safe: true, idempotent: null, owner: "entity", handlesCommand: null, + emits: [], preconditions: [], postconditions: [], invariantsEnforced: [], usesSpi: [], domainTerms: [], confidence: 0.5, +}; +const TYPE: TypeBlock = { + name: "T", description: "A type.", concept: "aggregate", eventKind: null, interfaceRole: null, aggregateRoot: true, + containedIn: null, syncPattern: null, identity: "id", fields: [], invariants: [], stateMachine: null, relatesTo: [], + exposes: [], dependsOn: [], domainTerms: [], confidence: 0.9, +}; +const MODULE: ModuleBlock = { + name: "M", description: "A module.", apis: [], spis: [], dependsOn: [], concepts: [], boundedContextHint: null, + sharedKernelHint: null, ubiquitousLanguage: [], confidence: 0.4, +}; + +const key = (module: string, symbol = "", disambiguator?: string): RecordKey => ({ lang: "java", module, symbol, ...(disambiguator === undefined ? {} : { disambiguator }) }); +const word = (w: string, concern: WordConcern, sense: string, ins: string[] = [w], confidence = 0.8): WordEntry => ({ word: w, in: ins, concern, sense, meaning: `${sense}, here.`, confidence }); +const gloss = (w: string, concern: WordConcern, sense: string, senses: GlossaryEntry["senses"] = []): GlossaryEntry => ({ word: w, concern, sense, meaning: `${sense}, reconciled.`, polysemous: senses.length > 0, senses }); +const KINDS: Record = { operation: "method", type: "class", module: "package" }; +const record = (id: string, level: Level, k: RecordKey | undefined, block: object, extra: Partial = {}): InsightRecord => + ({ t: "i", id, ...(k === undefined ? {} : { key: k }), level, kind: KINDS[level], name: id.slice(id.lastIndexOf("/") + 1), origin: "llm", block, fingerprint: "a".repeat(64), model: "m", ...extra }) as InsightRecord; + +const ORDERING = "java:com.acme.order"; +const BILLING = "java:com.acme.billing"; +const RECORDS: InsightRecord[] = [ + record(ORDERING, "module", key("com.acme.order"), { + ...MODULE, + words: [word("order", "domain", "the ordering business", ["com.acme.order"])], + glossary: [ + gloss("order", "domain", "a customer's request", [{ in: "Order", sense: "a customer's request" }, { in: "SortSpec", sense: "a sequence" }]), + gloss("line", "domain", "an order line", [{ in: "Order", sense: "an order line" }, { in: "Text", sense: "a line of text" }]), + gloss("total", "domain", "the sum"), + gloss("price", "domain", "a unit price"), + gloss("service", "technical", "a layer of the code"), + ], + }), + record(`${ORDERING}/Order`, "type", key("com.acme.order", "Order"), { + ...TYPE, + words: [word("order", "domain", "a customer's request", ["Order"], 0.9), word("line", "domain", "an order line", ["lines"]), word("total", "domain", "the sum")], + }), + record(`${ORDERING}/SortSpec`, "type", key("com.acme.order", "SortSpec"), { ...TYPE, words: [word("order", "technical", "a sequence", ["sortOrder"], 0.6)] }), + record(`${ORDERING}/Order.place(Order)`, "operation", key("com.acme.order", "Order.place", "(Order)"), { + ...OPERATION, + words: [word("place", "domain", "to submit"), word("order", "ambiguous", "the request, or its rank", ["order"], 0.5)], + }), + record(`${ORDERING}/Order.getId()`, "operation", key("com.acme.order", "Order.getId", "()"), OPERATION, { origin: "template" }), + record(BILLING, "module", key("com.acme.billing"), { + ...MODULE, + words: [word("billing", "domain", "invoicing")], + glossary: [gloss("order", "domain", "a purchase order"), gloss("total", "domain", "The Sum "), gloss("price", "domain", "an amount due"), gloss("service", "domain", "a layer of the code")], + }), + record(`${BILLING}/Invoice`, "type", key("com.acme.billing", "Invoice"), { ...TYPE, words: [word("invoice", "domain", "a bill"), word("order", "domain", "a purchase order", ["orderRef"])] }), + // Explained under prompt 1, before words were asked. + record("java:com.acme.legacy/Old", "type", key("com.acme.legacy", "Old"), TYPE), + // No natural key, and more words than any module: placed nowhere, and still last. + record("loose/Thing", "type", undefined, { ...TYPE, words: ["order", "a", "b", "c", "d"].map((w) => word(w, "domain", `a loose ${w}`)) }), +]; + +const seeded = () => { + const store = sqliteInsightsStore(loadSqlite().open(":memory:")); + store.importFile(decodeInsights(encodeInsightsToString(HEADER, RECORDS, { + t: "eof", counts: { records: RECORDS.length, llm: RECORDS.length - 1, template: 1, reused: 0, failed: 0 }, usage: { promptTokens: 1, completionTokens: 1 }, generatedAt: "2026-10-27T00:00:00.000Z", + }))); + return store; +}; + +describe("wordReport: every entity whose names spell a word, by module, with the module's glossary entry", () => { + it("groups by the records' module (their natural key's), the module itself first, then its types, then its operations", () => { + const report = wordReport(seeded(), "order"); + expect(report.word).toBe("order"); + expect(report.modules.map((m) => [m.module, m.glossary.map((g) => `${g.id}: ${g.entry.sense}`), m.entries.map((e) => `${e.level} ${e.id} ${e.entry.concern}`)])).toEqual([ + [BILLING, [`${BILLING}: a purchase order`], [`type ${BILLING}/Invoice domain`]], + [ORDERING, [`${ORDERING}: a customer's request`], [ + `module ${ORDERING} domain`, + `type ${ORDERING}/Order domain`, + `type ${ORDERING}/SortSpec technical`, + `operation ${ORDERING}/Order.place(Order) ambiguous`, + ]], + // A record without a natural key is placed nowhere, and still answered. + [undefined, [], ["type loose/Thing domain"]], + ]); + }); + + it("an entry is what the projection holds: the identifiers it is in, concern, sense, confidence — its meaning is in the block", () => { + const ordering = wordReport(seeded(), "order").modules[1]; + expect(ordering?.entries[1]).toEqual({ + id: `${ORDERING}/Order`, level: "type", kind: "class", name: "Order", key: key("com.acme.order", "Order"), model: "m", + entry: { word: "order", in: ["Order"], concern: "domain", sense: "a customer's request", confidence: 0.9 }, + }); + expect(ordering?.glossary[0]?.entry).toEqual(RECORDS[0]?.level === "module" ? RECORDS[0].block.glossary?.[0] : undefined); + }); + + it("looks a word up as the splitter spells it — lowercase — and says how many records cannot answer at all", () => { + const store = seeded(); + expect(wordReport(store, "ORDER")).toEqual(wordReport(store, "order")); + expect(wordReport(store, "order").withoutWords).toBe(1); + expect(wordReport(store, "nothing")).toEqual({ word: "nothing", modules: [], withoutWords: 1 }); + }); + + it("keeps a module's glossary entry for a word none of its records' answers define", () => { + // `price` is in both glossaries (their types were asked it) and in no record's words. + expect(wordReport(seeded(), "price").modules.map((m) => [m.module, m.glossary.map((g) => g.entry.sense), m.entries.length])).toEqual([ + [BILLING, ["an amount due"], 0], + [ORDERING, ["a unit price"], 0], + ]); + }); +}); + +describe("polysemyReport: the words a module flagged, and the words its modules' senses differ on", () => { + it("lays every module's entry for such a word side by side, the most contested first", () => { + const { words, withoutWords } = polysemyReport(seeded()); + expect(words.map((w) => ({ ...w, entries: w.entries.map((e) => `${e.id}: ${e.entry.sense}`) }))).toEqual([ + // A customer's request and a sequence (the flag's), a purchase order (billing's): three senses. + { word: "order", flagged: true, acrossModules: true, senses: 3, entries: [`${BILLING}: a purchase order`, `${ORDERING}: a customer's request`] }, + // Flagged by its one module, whose types disagree. + { word: "line", flagged: true, acrossModules: false, senses: 2, entries: [`${ORDERING}: an order line`] }, + { word: "price", flagged: false, acrossModules: true, senses: 2, entries: [`${BILLING}: an amount due`, `${ORDERING}: a unit price`] }, + // One sense, judged domain here and technical there: a disagreement all the same. + { word: "service", flagged: false, acrossModules: true, senses: 1, entries: [`${BILLING}: a layer of the code`, `${ORDERING}: a layer of the code`] }, + ]); + // "the sum" and "The Sum " are one sense: case and spacing aside. A word one module defines once is not listed. + expect(withoutWords).toBe(1); + }); +}); + +describe("concernsReport: per module, how many of its words the model called domain, technical, ambiguous", () => { + it("counts each word once per concern it was given, the most-worded module first, the records without a key last", () => { + expect(concernsReport(seeded())).toEqual({ + modules: [ + // order is domain in Order, technical in SortSpec, ambiguous in place(): it counts under all three. + { module: ORDERING, words: 4, concerns: { domain: 4, technical: 1, ambiguous: 1 } }, + { module: BILLING, words: 3, concerns: { domain: 3, technical: 0, ambiguous: 0 } }, + { module: undefined, words: 5, concerns: { domain: 5, technical: 0, ambiguous: 0 } }, + ], + withoutWords: 1, + }); + }); +}); + +describe("a reader's answer for a record without words (PLAN.md §20.3)", () => { + it("is prompt 1's when a model wrote it, a template's when none did, and nothing when it has words — none included", () => { + const byId = new Map(RECORDS.map((r) => [r.id, r])); + expect(wordsAbsence(byId.get("java:com.acme.legacy/Old") as InsightRecord)).toBe("prompt-1"); + expect(wordsAbsence(byId.get(`${ORDERING}/Order.getId()`) as InsightRecord)).toBe("templated"); + expect(wordsAbsence(byId.get(`${ORDERING}/Order`) as InsightRecord)).toBeUndefined(); + expect(wordsAbsence({ ...(byId.get(`${ORDERING}/Order`) as InsightRecord), block: { ...TYPE, words: [] } } as InsightRecord)).toBeUndefined(); + }); +}); + +describe("moduleOf: where a record sits, rendered from its natural key — never parsed from its id", () => { + it("is the module's id for any key of it, and undefined without a key or for one that cannot render", () => { + expect(moduleOf(key("com.acme.order", "Order.place", "(Order)"))).toBe(ORDERING); + expect(moduleOf(key("com.acme.order"))).toBe(ORDERING); + expect(moduleOf(undefined)).toBeUndefined(); + // `/` is reserved in a key's components: a hand-edited side-car's key is placed nowhere rather than mis-rendered. + expect(moduleOf({ lang: "java", module: "a/b", symbol: "C" })).toBeUndefined(); + }); +}); diff --git a/packages/insights/test/store.test.ts b/packages/insights/test/store.test.ts index d854b29..172a6bf 100644 --- a/packages/insights/test/store.test.ts +++ b/packages/insights/test/store.test.ts @@ -395,6 +395,8 @@ describe("a reader's questions (M16c): projections, never blocks — and a store byOrigin: { llm: 5, template: 1 }, byConcept: { aggregate: 1, entity: 1, valueType: 1 }, usage: { promptTokens: 0, completionTokens: 0 }, + // A model wrote five of them and none has words: prompt 1's. The template is no model's. + withoutWords: 5, }); // What the records that exist cost, from their own usage — it survives an import, the ledger does not. const paid = memory(); @@ -712,6 +714,59 @@ describe("version 2 (M19c): insight_word, a projection of every record's words", }); }); +describe("the word readers' projections (M19c): what the blocks say, never more", () => { + const hit = (r: InsightRecord, entry: WordEntry) => ({ id: r.id, level: r.level, kind: r.kind, name: r.name, key: r.key, model: r.model, entry: listed(entry) }); + const asSet = (rows: readonly T[]): string[] => rows.map((row) => JSON.stringify(row)).sort(); + + it("for any side-car, exact on any string: one word's entries, the glossaries, each concern once, the records without words", () => { + fc.assert( + fc.property(fileArb, (file) => { + const store = memory(); + const { records } = decodeInsights(file); + store.importFile(decodeInsights(file)); + const spelled = new Set(records.flatMap((r) => (r.block.words ?? []).map((w) => w.word))); + // Side-car order, then block order. The words are any strings, a lone surrogate (which the index cannot match) included. + for (const word of [...spelled, "\u{1F4E6}never"]) { + expect(store.wordHits(word)).toEqual(records.flatMap((r) => (r.block.words ?? []).filter((w) => w.word === word).map((w) => hit(r, w)))); + } + const glossed = records.flatMap((r) => (r.level === "module" ? (r.block.glossary ?? []).map((entry) => ({ id: r.id, key: r.key, name: r.name, model: r.model, entry })) : [])); + expect(store.glossaryEntries()).toEqual(glossed); + for (const word of new Set(glossed.map((g) => g.entry.word))) expect(store.glossaryEntries(word)).toEqual(glossed.filter((g) => g.entry.word === word)); + const concerns = records.flatMap((r) => (r.block.words ?? []).map((w) => ({ lang: r.key?.lang, module: r.key?.module, word: w.word, concern: w.concern }))); + expect(asSet(store.concernRows())).toEqual([...new Set(asSet(concerns))]); + expect(store.withoutWords()).toBe(records.filter((r) => r.origin === "llm" && r.block.words === undefined).length); + store.close(); + }), + { numRuns: 60 }, + ); + }); + + it("answers in side-car order whatever order a run wrote the records in", () => { + const store = memory(); + const run = store.beginRun({ header: HEADER, startedAt: "t" }); + const glossary = [{ word: "order", concern: "domain" as const, sense: "a request", meaning: "m", polysemous: false, senses: [] }]; + // A walk writes bottom-up and in its own order: rowids say nothing about the export's. + store.putUnit(run, "m/b", [{ t: "i", id: "m/b", level: "module", kind: "package", origin: "llm", fingerprint: FP, block: { ...MODULE, words: [ORDER], glossary } }]); + store.putUnit(run, "t/B", [{ ...type("t/B"), block: { ...TYPE, words: [ORDER] } } as InsightRecord]); + store.putUnit(run, "m/a", [{ t: "i", id: "m/a", level: "module", kind: "package", origin: "llm", fingerprint: FP, block: { ...MODULE, words: [ORDER], glossary } }]); + store.putUnit(run, "t/A", [{ ...type("t/A"), block: { ...TYPE, words: [ORDER] } } as InsightRecord]); + store.putUnit(run, "o/z", [op("o/z", { block: { ...OPERATION, words: [ORDER] } })]); + expect(store.wordHits("order").map((h) => h.id)).toEqual(["o/z", "t/A", "t/B", "m/a", "m/b"]); + expect(store.glossaryEntries().map((g) => g.id)).toEqual(["m/a", "m/b"]); + store.close(); + }); + + it("a word no identifier can spell (a lone surrogate) is found exactly, and only it", () => { + const store = memory(); + const odd = { ...ORDER, word: "\ud800" }; + store.importFile(decodeInsights(encodeInsightsToString(HEADER, [{ ...type("t/Order"), block: { ...TYPE, words: [ORDER, LINES, odd] } } as InsightRecord], eof(1)))); + expect(store.wordHits("order").map((h) => h.entry.word)).toEqual(["order"]); + expect(store.wordHits("\ud800").map((h) => h.entry.word)).toEqual(["\ud800"]); + expect(store.wordHits("\udc00")).toEqual([]); + store.close(); + }); +}); + describe("insightsStorePathFor", () => { it("sits beside the side-car it exports", () => { expect(insightsStorePathFor("/x/model.insights.jsonl")).toBe("/x/model.insights.db");