diff --git a/CLAUDE.md b/CLAUDE.md index 9f010de..a48d1f0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -241,6 +241,8 @@ node scripts/homebrew/render.mjs --version X.Y.Z --sums SHA256SUMS [--npm-sha256 # role "pass" (codegraph-metrics): claims no files, runs ` --src ` after the extractor ./bin/codegraph city model.jsonl --layout --out city.json # the artifacts alone ./bin/codegraph navigator model.jsonl --out navigator.json +./bin/codegraph vocabulary model.jsonl [--level module|type] [--scope ID] [--drift FILE] [--svg FILE] # the names' words +./bin/codegraph serve model.jsonl --height vocabulary:order --height-scale sqrt # where one word lives, in the city ./bin/codegraph explain model.jsonl --src DIR --dry-run # the walk plan, no call ./bin/codegraph explain model.jsonl --src DIR --estimate --price-in 0.10 --price-out 0.60 # tokens in/out + cost, no call OPENROUTER_API_KEY=… ./bin/codegraph explain model.jsonl --src DIR [--max-calls N] [--yes] diff --git a/PLAN.md b/PLAN.md index a430aa0..bd59725 100644 --- a/PLAN.md +++ b/PLAN.md @@ -4884,7 +4884,7 @@ Seven principles, locked up front: 5. **Scope membership is the fold's.** A type's words are the words of everything that folds into its building (`containingType`), a module's likewise (`containingModule`), the corpus's every kept entity — the same - membership the city uses, so a `vocabulary:order` channel colours the + membership the city uses, so a `vocabulary:order` channel measures the building whose words were counted. Attachment is NOT followed, as the fold does not: a Ruby reopening speaks in the file it is written in. 6. **Deterministic bytes.** Terms interned and sorted, scopes in id order, @@ -5279,13 +5279,33 @@ its owner, `Category`, in Navigate; a page with no carriers route keeps the counts and says so. **The city channel**: `vocabulary:` as a metric source for `--height`, -`--color` and `--carry` — the term's weight in the building's type scope, +`--footprint` and `--carry` — the term's weight in the building's type scope, `undefined` where the type does not speak it, so an unmeasured building is floored and labelled, never zeroed. Wired through the CLI's `metric(name)` hook exactly as `history:` is (`history-metrics.ts`); `@codegraph/city` is -unchanged. `codegraph serve model.jsonl --color vocabulary:order` is "where +unchanged. `codegraph serve model.jsonl --height vocabulary:order` is "where does `order` live" — the issue's city link, and the second demo image. +**As built** (`vocabulary-metric.ts`): a HEIGHT, not a colour — this plan +first said `--color`, but the city's one colour channel is the framework's +categorical `role`, and a weight is a magnitude. A type whose names never say +the word weighs **0**, not `undefined`: its names were read and zero is what +they say, and `undefined` there would have printed "N unmeasured" on every +such city for a measured fact. `undefined` stays for a building with no type +scope — a stub, whose names the vocabulary never reads. The word goes +through the splitter (`Order` is `order`; `OrderLine`, two words, is a usage +error); the classes two languages give one text add up, the metric being +about the text; a word no name in the view says is a usage error rather than +a flat city that looks like an answer. `serve` and the app job hand `cityOf` +the artifact they serve; `city` counts one only when a `vocabulary:` metric +is named. On Broadleaf 627 of 3 305 types say `order`, the tallest +`OrderServiceImpl` (203), the towers in `core.order.service` and +`core.offer`: `docs/vocabulary-city-broadleaf.png` is the City tab at +`--height-scale sqrt` (the root lifts a type that says the word once off the +floor: 589 of the 627 rise above 5 of 40 units, against 92 on a linear +scale), arrows hidden and the tower selected so its panel shows the number. The viewer's help no longer says a +taller tower "holds more code": height is whatever the legend names. + ### 20.3 The glossary: words in the explanation (M19c) What a model is asked, and OF WHAT. The deterministic part knows that @@ -5470,11 +5490,11 @@ never looks like a fact. - [x] The Vocabulary tab: cloud, chips, table, carriers, reveal; the literal-pinning and types-only boundary tests; screenshots on Broadleaf reviewed at user-facing sizes. -- [ ] `vocabulary:` metric source through the `metric(name)` hook; - `--color vocabulary:order` on Broadleaf reviewed as a screenshot. -- [ ] The demo images: the corpus cloud of a public repository as SVG +- [x] `vocabulary:` metric source through the `metric(name)` hook; + `--height vocabulary:order` on Broadleaf reviewed as a screenshot. +- [x] The demo images: the corpus cloud of a public repository as SVG (`docs/vocabulary-.svg` — Broadleaf's, committed with `--svg`) - and the city coloured by one term, with the README section that shows + and the city raised by one term, with the README section that shows them — the issue's fourth acceptance criterion. **M19c — the glossary: words in the explanation** @@ -5595,7 +5615,7 @@ green; the coverage table and the cost filled in this section. | M18b | Ruby extractor — model | ✅ (2026-10-13, worktree `ruby-extractor`) `field` (an instance variable, its `attr_*`, a Struct member: one entity `@x`) and `constant` (keyed by full path, with its literal `value`); `Struct.new` / `Data.define` / `Class.new` / `Module.new` assigned to a constant as types, a `Struct.new` superclass as a `generated` inheritance; constant references resolved lexically then through the ancestors; the core table extended with superclasses, mixins and methods (RBS 4.2.0: 915 constants, 23 mixins, 2428 methods) so lookup crosses from the corpus into the VM's classes; the invocation tiers of §19.4 — `declared` where the receiver's class is written, `derived` for typed locals, typed instance variables and the sole-library-ancestor attribution, the language resolved with no edge, untyped receivers dropped and counted; `access` with read/write, `throws`, comments, `sloc` + `cyclomatic`, literals; the Ruby 1.8 require idioms, `load`, `autoload`, shebang scripts; `--explain-dropped`, every drop with its line → `fixtures/ruby/expected/model.jsonl` (27 files, 154 entities / 136 edges; 152 sites, 77.0 % resolved) byte-identical to core's encoder and profile-valid with zero issues; `ruby` in the metrics pass and the fifth `model.metrics.jsonl` (86 of 88 joined, `sloc = ploc` on all 86); 101 cargo tests (90 unit, 11 contract) + 24 core gate tests; the fixture in the analyzer, city, navigator, CLI `validate` and metrics suites; `validate`, `analyze`, `navigator`, `city --layout` and `serve` run on it, City and Navigate screenshots reviewed | | M18c | Ruby extractor — audit + oracles + distribution | ✅ (2026-10-13, worktree `ruby-extractor`; CI not yet run) rake 13.2.1, rubygems 3.6.2, Mastodon 4.3.0 and rails 8.0.2 audited `validate`-clean (100 / 794 / 2 541 / 3 334 files; 77.8 / 54.4 / 31.2 / 57.0 % of sites resolved, Mastodon's `app/`+`lib/` 62.1 %; rails in 4.7 s). Seven defects found and fixed test-first: a library namespace the corpus reopens below dropped as `namespace-only` (592 `Rails.*` in Mastodon); class methods given by an `included` hook unseen (now read where literal — plain Ruby and ActiveSupport::Concern — and `derived`); a library mixin ignored as a class-side candidate (`validates` charged to ActiveRecord::Base); a per-type `dynamic` flag clouding both sides; a compact declaration's qualifier not read; a constant qualified by a subclass stubbed below ``; `--explain-dropped` not saying what decided a drop. Oracles: Packwerk 3.3.1 over Mastodon — 4 289 of its 4 335 cross-file references in the model (98.9 %), the rest the wrapper rule; rake booted under its own tests — 1 006 of 1 012 declared methods held by the VM on the same line, 1 533 of 2 829 observed calls already edges. The trace tool (`trace/codegraph_trace.rb`, one file) and `--trace`: `generated` `traitUsage`, `dynamic-candidate` invocations with `candidates`, no entity added — exercised end to end in a container and `validate`-clean. `fixtures/ruby` → 31 files, 181 entities / 153 edges, 177 sites, 76.3 %; metrics 106 of 109 joined. 119 cargo tests (107 unit, 12 contract) + 27 core gate tests. Distribution wired: `ruby-native` on four Unix runners, the release loop, `Formula/codegraph-ruby.rb`, the catalogue row; `docs/ruby-extractor.md` | | M19a | Vocabulary — lexicon, split, artifact | ✅ (2026-10-26, five stacked PRs from #26, not yet merged; Broadleaf 76 310 entities → 3 381 terms in 0.29 s, 2.9 MB, its top ten a commerce vocabulary after 13 words joined the shared list) `Profile.lexicon` on the eleven profiles (keywords, conventional names, affixes, separators — the profile half of #19), METAMODEL §5 row, printed by `profiles --lang`; `@codegraph/vocabulary`: one idiom-parametrized splitter with the acronym rule on Unicode categories, four term classes as tags (never filters), role weights recorded in the artifact, scopes = the fold's containers; `codegraph vocabulary` report, `--level`/`--scope`/`--class`/`--drift FILE`; five `vocabulary.json` snapshots byte-identical, closure and determinism properties; the coverage table of §20.1 filled on fixtures, Broadleaf, codegraph, Mastodon, Plausible | -| M19b | Vocabulary — cloud, tab, city channel | deterministic spiral layout with an injected measurer and the no-overlap property; `--svg` with the fixture snapshot; `serve` builds `/vocabulary.json` and answers `/carriers.json?scope=&term=`; the sixth navigator tab (scope follows the selection, chips, table, carriers → reveal), literals pinned, types-only import; `vocabulary:` metric source through the CLI hook, city unchanged; the two demo images under `docs/` and the README section — the issue's fourth criterion | +| M19b | Vocabulary — cloud, tab, city channel | ✅ (2026-10-26, three stacked PRs from #31, not yet merged; Broadleaf's cloud and its city raised by `order` are the demo images) deterministic spiral layout with an injected measurer and the no-overlap property; `--svg` with the fixture snapshot; `serve` builds `/vocabulary.json` and answers `/carriers.json?scope=&term=`; the sixth navigator tab (scope follows the selection, chips, table, carriers → reveal), literals pinned, types-only import; `vocabulary:` metric source through the CLI hook, city unchanged; the two demo images under `docs/` and the README section — the issue's fourth criterion | | M19c | Vocabulary — glossary in the explanation | `words` on every insight block — the words of the unit's name, its parameters' and (for a type) its fields', each with `in`, `concern` (domain / technical / ambiguous), `sense`, `meaning`, in that entity's context, asked in the SAME call that explains it; `glossary` on the module block reconciling its types' entries with `polysemous`; `PROMPT_VERSION` 2, the words `.optional()` in the record schema so every existing side-car round-trips byte-identical — no new record type, no format change; the store at version 2 with the `insight_word` projection (rebuilt on `--import`); readers `insights --word/--glossary/--polysemy/--concerns`, `vocabulary --drift-insights`, `/glossary.json`, the entries shown as inferences in both panels; the Broadleaf re-explanation priced by `--estimate` and measured; `docs/vocabulary.md` | ## 22. Decisions made in this plan (deltas vs. the design doc) @@ -5711,7 +5731,7 @@ green; the coverage table and the cost filled in this section. | The idiom is profile data (M19, #19) | `Profile.lexicon`: keywords, conventional member names, technical additions, separators, prefixes, suffixes, the C# interface prefix, file-named modules; one splitter reads it | invariant 8: a profile is specifiable without being implemented; the algorithm knows no language, and a language without an extractor still gets its idiom | | Term classes are tags (M19) | every token kept and tagged `domain / technical / keyword / short`; a consumer filters; the technical list is data with its digest in the artifact | "not shown" must stay distinguishable from "not there" (the §3.8 absence rule applied to words); drift needs the technical words; the list is contestable by construction, so it is overridable and named | | Vocabulary weights (M19) | by role from traits — type and module 8, attribute 4, operation 3, parameter and local 1 — recorded in the artifact beside the raw occurrence count | a count without its scheme is not a fact (the city's `bindings` rule); the issue asks that a class name outweigh a buried variable, and nouns outweigh verbs whose prefix is technical | -| Vocabulary scopes (M19) | a scope's words are those of every entity whose fold container it is (`containingType`/`containingModule`); attachment not followed; operations on demand, not in the artifact | the same membership the city uses, so `vocabulary:` colours the building whose words were counted; the navigator addresses types and modules only | +| Vocabulary scopes (M19) | a scope's words are those of every entity whose fold container it is (`containingType`/`containingModule`); attachment not followed; operations on demand, not in the artifact | the same membership the city uses, so `vocabulary:` measures the building whose words were counted; the navigator addresses types and modules only | | Cloud layout (M19b) | pure deterministic geometry in the package — weight-ordered spiral, axis-aligned boxes, an injected text measurer with a monospace default; no `d3-cloud` | the demo image is an SVG the CLI writes and a fixture snapshots, so the layout must run in Node and give the same bytes; the browser passes `measureText` into the same function | | Carriers are a lookup (M19b) | the artifact carries counts per (scope, term); the carrying entities come from `/carriers.json?scope=&term=` over the model `serve` holds | lists per term per scope would multiply the artifact for evidence a page asks for one word at a time; a static host degrades to no evidence rather than a bigger file (the `/insight.json` precedent) | | Glossary tied to the entity (M19c) | each explained unit's block carries `words`: the words its name (and its parameters' or fields' names) is made of, with sense and concern in that entity's context, asked in the explanation call itself; the module block reconciles its types' words into `glossary`; scopes and the corpus roll up from the records | a meaning belongs to the entity that uses the word, and the context pack of that entity is the one place the pipeline has its source, facts and dependencies' explanations together; a scope has a vocabulary, not a meaning — the two earlier drafts (a standalone command over a term table; a second request per scope) asked at the wrong altitude | diff --git a/README.md b/README.md index 1ae3531..273bebc 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ Codegraph turns source code into a dependency model you can query, draw as a have a language model explain bottom-up. It does this without building the code: the extractors run on sources alone (no classpath, jars, etc.). -Four questions, four commands: +Five questions, five commands: | Question | Command | What you get | |---|---|---| @@ -37,6 +37,7 @@ Four questions, four commands: | What exactly depends on what? | `codegraph serve` → Navigate tab | a browsable tree with every incoming and outgoing dependency, the member that carries it, and the source line that proves it | | How did it get this way? | `codegraph scm` / `replay` | churn, hotspots, ownership, co-change, and a city whose timeline scrubs the years | | What does it mean? | `codegraph explain` | one plain-language explanation per method, class and package, written leaves-first so every summary rests on already-explained parts | +| What language does it speak? | `codegraph vocabulary` / `serve` → Vocabulary tab | the words of every name, split per language idiom and weighted by what carries them: a word cloud per package or class, drift against your glossary, one word across the city | Plus the plumbing you need to trust the answers: `validate` for the model, `analyze` for coupling and cycles, `export` for DOT, CSV, JSON and PlantUML. @@ -204,6 +205,53 @@ and red for the edges whose cut would break a cycle. Hover a building for its raw numbers. Unmeasured metrics are drawn at the minimum and labelled as unmeasured rather than faked. +### Read its vocabulary + +Every name in the model is split into words the way its language writes +names (`OrderItemImpl` → `order`, `item`, `impl`), and each word weighs what +carries it: 8 for a class or a package, 4 for a field, 3 for a method, 1 for +a parameter or a local. Words are tagged `domain`, `technical`, `keyword` or +`short`, never dropped. Only names are read, never source, and no language +model is called. + +```bash +codegraph vocabulary broadleaf.jsonl # the 40 heaviest domain words +codegraph vocabulary broadleaf.jsonl --top 80 --svg cloud.svg # and the cloud, as an SVG image +codegraph vocabulary broadleaf.jsonl --level module # per package, most distinct words first +codegraph vocabulary broadleaf.jsonl --drift glossary.txt # the business terms no name says; exit 3 +codegraph serve broadleaf.jsonl --height vocabulary:order --height-scale sqrt # where does `order` live? +``` + +

+ The word cloud of BroadleafCommerce: order, item, customer, offer, product, admin, group, fulfillment, category and payment are the largest of 80 domain words. +

+ +

+ The 80 heaviest domain words of + BroadleafCommerce's + names: order item customer offer product admin group fulfillment + category payment… A word's area is its weight; the legend states the + weights, the view and the technical list it was counted under. +

+ +

+ BroadleafCommerce's code city with each building raised by the weight of the word order in its names: the order service and offer packages stand as towers, most other types stay flat, and the selected OrderServiceImpl shows vocabulary:order 203. +

+ +

+ The same corpus as a city, each class raised by how much its names say + order (--height vocabulary:order --height-scale sqrt). + The towers are core.order.service and core.offer; + flat tiles never say the word. The selected tower is + OrderServiceImpl, weight 203. +

+ +On `serve`'s Vocabulary tab, the cloud and its table follow the package or +class selected in Navigate, and clicking a word lists the names that carry it, +each one click from its node. + ### Mine the history ```bash diff --git a/docs/cli.md b/docs/cli.md index 3e580a5..818dde0 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -229,6 +229,11 @@ With the change-history side-car `codegraph-metrics --history` writes beside the model (or `--history-metrics FILE`), `history:` reads the type's own blame and `history-file:` its file's history (`--height history-file:churn_recent`; docs/metrics.md §4). +`vocabulary:` is the word's weight in the names of the type and its +members — the Vocabulary tab's type scope, under the default weights — so +`--height vocabulary:order` shows where `order` lives. A type whose names +never say the word weighs 0; a stub, whose names the vocabulary never reads, +is unmeasured; a word no name in the view says is a usage error. Unmeasured metrics are drawn at the channel minimum and labelled unmeasured. ### `navigator` @@ -255,7 +260,10 @@ and where, and the domain words no term contains. `--out FILE` writes the root of the weight, colour by class and nothing else, a legend stating the scope, weights, view and technical list, the same bytes on every run. [`vocabulary-broadleaf.svg`](vocabulary-broadleaf.svg) is BroadleafCommerce's, -drawn with `--top 80`. +drawn with `--top 80`. One word across the city is the `city` metric +`vocabulary:` (above): +[`vocabulary-city-broadleaf.png`](vocabulary-city-broadleaf.png) is +BroadleafCommerce's city with `--height vocabulary:order`. ### `domain-facts` One JSON artifact (`codegraph.domainFacts/1`) with one dossier per corpus diff --git a/docs/vocabulary-city-broadleaf.png b/docs/vocabulary-city-broadleaf.png new file mode 100644 index 0000000..390be67 Binary files /dev/null and b/docs/vocabulary-city-broadleaf.png differ diff --git a/packages/cli/src/app/jobs.ts b/packages/cli/src/app/jobs.ts index 5e2c5cb..57a67bd 100644 --- a/packages/cli/src/app/jobs.ts +++ b/packages/cli/src/app/jobs.ts @@ -367,10 +367,11 @@ export function createJobRunner(options: JobRunnerOptions): JobRunner { const view = resolveView(options.build); const named = options.build.name === undefined ? {} : { name: options.build.name }; const navigatorModel = buildNavigator(graph, { view, ...named }); - const cityModel = layoutCity(cityOf(graph, options.build)); + const vocabularyModel = buildVocabulary(graph, { view, ...named }); + const cityModel = layoutCity(cityOf(graph, options.build, undefined, vocabularyModel)); navigator = navigatorToJsonString(navigatorModel); city = cityToJsonString(cityModel); - vocabulary = vocabularyToJsonString(buildVocabulary(graph, { view, ...named })); + vocabulary = vocabularyToJsonString(vocabularyModel); if (!source.clean) errLine(io, `job: warning — the model is not clean; the page was built anyway.`); record = { kind: PROJECT_KIND, diff --git a/packages/cli/src/args.ts b/packages/cli/src/args.ts index 86da1d5..65208b6 100644 --- a/packages/cli/src/args.ts +++ b/packages/cli/src/args.ts @@ -256,7 +256,8 @@ function metricHelp(channel: string): string { return ( `Metric driving ${channel}. Built in: ${metricNames().join(", ")}. ` + `Open forms: ${METRIC_PREFIXES.map((form) => `${form.prefix}`).join(", ")}. ` + - `With a change-history side-car: history: (the type's own), history-file: (its file).` + `With a change-history side-car: history: (the type's own), history-file: (its file). ` + + `From the names: vocabulary: (the word's weight in the type's names).` ); } diff --git a/packages/cli/src/commands/city.ts b/packages/cli/src/commands/city.ts index 80f0933..031801c 100644 --- a/packages/cli/src/commands/city.ts +++ b/packages/cli/src/commands/city.ts @@ -9,11 +9,13 @@ import { type CityModel, type MetricSource, } from "@codegraph/city"; +import { buildVocabulary, type VocabularyArtifact } from "@codegraph/vocabulary"; import type { CityBuildOptions, CityOptions } from "../args.js"; import { UsageError, type ExitCode } from "../exit.js"; import { historyMetric, loadHistoryJoin, type HistoryJoin } from "../history-metrics.js"; import { errLine, errLines, type IoSink } from "../io.js"; import { loadExitCode, loadModelFiles } from "../load.js"; +import { vocabularyMetric } from "../vocabulary-metric.js"; import { resolveView } from "../view.js"; /** @@ -69,18 +71,26 @@ export function cityCommand(options: CityOptions, io: IoSink): ExitCode { * The city of a graph under the CLI's flags — shared with `serve`. `history` * is the model's change-history side-car, joined (history-metrics.ts): it * gives the `history:`/`history-file:` metric names their sources. + * `vocabulary` gives `vocabulary:` its counts (vocabulary-metric.ts): + * the one a caller already built under the same view, else counted here + * when such a metric is named. */ export function cityOf( graph: ReturnType, options: CityBuildOptions, history?: HistoryJoin, + vocabulary?: VocabularyArtifact, ): CityModel { - const metric = (name: string): string | MetricSource => historyMetric(name, history) ?? name; + const view = resolveView(options); + let counted = vocabulary; + const words = (): VocabularyArtifact => (counted ??= buildVocabulary(graph, { view })); + const metric = (name: string): string | MetricSource => + historyMetric(name, history) ?? vocabularyMetric(name, words) ?? name; try { const framework = options.framework === undefined ? undefined : FRAMEWORK_PROFILES[options.framework]; return buildCity(graph, { - view: resolveView(options), + view, ...(options.name === undefined ? {} : { name: options.name }), ...(framework === undefined ? {} : { framework }), height: { metric: metric(options.height), scale: options.heightScale }, diff --git a/packages/cli/src/commands/serve.ts b/packages/cli/src/commands/serve.ts index 2c6c36d..4795534 100644 --- a/packages/cli/src/commands/serve.ts +++ b/packages/cli/src/commands/serve.ts @@ -88,11 +88,11 @@ export function serveCommand(options: ServeOptions, io: IoSink, deps: ServeDeps view, ...(options.name === undefined ? {} : { name: options.name }), }); - // The viewer refuses a city with no placement, so the layout is not optional. - const history = loadHistoryJoin(options.historyMetrics, options.models); - const city = layoutCity(cityOf(graph, options, history)); const counting = { view, ...(options.name === undefined ? {} : { name: options.name }) }; const vocabulary = buildVocabulary(graph, counting); + // The viewer refuses a city with no placement, so the layout is not optional. + const history = loadHistoryJoin(options.historyMetrics, options.models); + const city = layoutCity(cityOf(graph, options, history, vocabulary)); // One "not clean" line for the page, then each artifact's own caveats. const built = { clean: true, paths: source.paths }; diff --git a/packages/cli/src/vocabulary-metric.ts b/packages/cli/src/vocabulary-metric.ts new file mode 100644 index 0000000..e6dcec4 --- /dev/null +++ b/packages/cli/src/vocabulary-metric.ts @@ -0,0 +1,61 @@ +import type { MetricSource } from "@codegraph/city"; +import { splitIdentifier, type VocabularyArtifact } from "@codegraph/vocabulary"; +import { UsageError } from "./exit.js"; + +/** + * THE CITY'S VOCABULARY CHANNEL (PLAN.md §20.2): `vocabulary:` measures + * a building by the word's weight in its type's names — the type and every + * member folded into it, the Vocabulary tab's type scope — so the tab and the + * city read one number, and `--height vocabulary:order` shows where `order` + * lives. + * + * A type whose names never say the word weighs 0, a measured zero; a building + * with no type scope (a stub: its name is another corpus's word) is + * unmeasured. The metric is about the text, so the classes two languages give + * it add up. The artifact is counted only when such a metric is named. + */ + +export const VOCABULARY_PREFIX = "vocabulary:"; + +export function vocabularyMetric(name: string, vocabulary: () => VocabularyArtifact): MetricSource | undefined { + if (!name.startsWith(VOCABULARY_PREFIX)) return undefined; + const typed = name.slice(VOCABULARY_PREFIX.length); + // The splitter's own rule, so `Order` is `order` and `OrderLine` is two words. + const { tokens } = splitIdentifier(typed, "", undefined); + const [word] = tokens; + if (word === undefined || tokens.length > 1) { + throw new UsageError( + tokens.length > 1 + ? `${name} names ${tokens.length} words (${tokens.join(", ")}); a metric measures one` + : `${name} names no word`, + "Write one word after vocabulary:, e.g. vocabulary:order.", + ); + } + + const artifact = vocabulary(); + const said = new Set(); + artifact.terms.forEach((term, index) => { + if (term.text === word) said.add(index); + }); + if (said.size === 0) { + throw new UsageError( + `${name}: no name in view ${artifact.view.name} says "${word}"`, + "Run 'codegraph vocabulary ' for the words it does say.", + ); + } + const byType = new Map(); + for (const scope of artifact.scopes) { + if (scope.level !== "type" || scope.id === undefined) continue; + let weight = 0; + for (const [term, termWeight] of scope.terms) if (said.has(term)) weight += termWeight; + byType.set(scope.id, weight); + } + return { + name, + unit: "weighted occurrences", + describe: + `Weight of "${word}" in the names of the type and its members, each occurrence weighted by ` + + `its name's role, as codegraph vocabulary counts it. A stub's names are not counted.`, + value: ({ node }) => byType.get(node.id), + }; +} diff --git a/packages/cli/test/serve-command.test.ts b/packages/cli/test/serve-command.test.ts index 0786b69..e241356 100644 --- a/packages/cli/test/serve-command.test.ts +++ b/packages/cli/test/serve-command.test.ts @@ -153,6 +153,28 @@ describe("serve: three artifacts, one server", () => { expect(city.bindings.find((binding) => binding.channel === "height")?.metric).toBe("methods"); }); + // "Where does order live": the city's building and the tab's type scope + // are one count, so the City tab and the Vocabulary tab cannot disagree. + it("measures vocabulary: by the served vocabulary's type scopes", () => { + const { started } = serveTo({ height: "vocabulary:order" }); + const routes = started[0]?.routes ?? {}; + const vocabulary = parse<{ + terms: { text: string }[]; + scopes: { level: string; id?: string; terms: [number, number][] }[]; + }>(routes[VOCABULARY_ROUTE]); + const city = parse<{ buildings: { id: string; isStub: boolean; metrics: Record }[] }>( + routes[CITY_ROUTE], + ); + const order = vocabulary.terms.findIndex((term) => term.text === "order"); + const types = vocabulary.scopes.filter((scope) => scope.level === "type"); + expect(types.length).toBe(city.buildings.filter((building) => !building.isStub).length); + for (const scope of types) { + const building = city.buildings.find((candidate) => candidate.id === scope.id); + const row = scope.terms.find(([term]) => term === order); + expect(building?.metrics["vocabulary:order"], scope.id).toBe(row?.[1] ?? 0); + } + }); + it("fails BEFORE loading models when the frontend is not built", () => { const io = captureIo(); expect(() => diff --git a/packages/cli/test/vocabulary-metric.test.ts b/packages/cli/test/vocabulary-metric.test.ts new file mode 100644 index 0000000..8bdba7f --- /dev/null +++ b/packages/cli/test/vocabulary-metric.test.ts @@ -0,0 +1,182 @@ +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import { buildGraph } from "@codegraph/analyzer"; +import type { MetricContext } from "@codegraph/city"; +import { buildVocabulary, type VocabularyArtifact } from "@codegraph/vocabulary"; +import type { CityOptions } from "../src/args.js"; +import { cityCommand, cityOf } from "../src/commands/city.js"; +import { EXIT, UsageError } from "../src/exit.js"; +import { captureIo } from "../src/io.js"; +import { loadModelFiles } from "../src/load.js"; +import { vocabularyMetric } from "../src/vocabulary-metric.js"; + +/** + * The city's vocabulary channel (PLAN.md §20.2): `vocabulary:` measures + * a building by the word's weight in its type's names — the Vocabulary tab's + * type scope, so the tab and the city read one number. The Java fixture's + * `order` is counted by hand in fixtures/java/README.md: `Order` and + * `AbstractOrder` are their class names (8 each); `OrderService` is its name, + * `bill`'s parameter and `archive`'s local (8 + 1 + 1); `Notifications` is + * four `order` parameters (4 × 1). + */ +const FIXTURE = fileURLToPath(new URL("../../../fixtures/java/expected/model.jsonl", import.meta.url)); + +interface Building { + readonly id: string; + readonly isStub: boolean; + readonly metrics: Record; +} + +interface CityJson { + readonly bindings: { channel: string; metric: string; unit: string; describe: string; unmeasured: number }[]; + readonly buildings: Building[]; +} + +function cityOptions(overrides: Partial = {}): CityOptions { + return { + models: [FIXTURE], + height: "loc", + heightScale: "linear", + footprint: "members", + footprintScale: "sqrt", + carry: [], + name: undefined, + framework: undefined, + internalOnly: false, + declaredOnly: false, + layout: false, + out: undefined, + ...overrides, + }; +} + +function city(overrides: Partial): { json: CityJson; stderr: string } { + const io = captureIo(); + expect(cityCommand(cityOptions(overrides), io)).toBe(EXIT.OK); + return { json: JSON.parse(io.stdout()) as CityJson, stderr: io.stderr() }; +} + +/** The corpus's buildings that measure above zero, by short name. */ +function spoken(json: CityJson, metric: string): Record { + return Object.fromEntries( + json.buildings + .filter((building) => !building.isStub && building.metrics[metric] !== 0) + .map((building) => [building.id.slice(building.id.indexOf("/") + 1), building.metrics[metric]]), + ); +} + +describe("vocabulary:, the city's vocabulary channel", () => { + it("measures a building by the word's weight in its type's names, the counting the fixture's README does by hand", () => { + const { json } = city({ height: "vocabulary:order" }); + expect(spoken(json, "vocabulary:order")).toEqual({ AbstractOrder: 8, Notifications: 4, Order: 8, OrderService: 10 }); + const height = json.bindings.find((binding) => binding.channel === "height"); + expect(height?.metric).toBe("vocabulary:order"); + expect(height?.unit).toBe("weighted occurrences"); + expect(height?.describe).toContain("order"); + }); + + // A type whose names never say the word was read and weighs 0: a measured + // zero, not a missing measurement. + it("weighs 0 on a type whose names never say it, and leaves a stub unmeasured", () => { + const { json, stderr } = city({ height: "vocabulary:order" }); + const corpus = json.buildings.filter((building) => !building.isStub); + const stubs = json.buildings.filter((building) => building.isStub); + expect(corpus.filter((building) => building.metrics["vocabulary:order"] === 0)).toHaveLength(corpus.length - 4); + expect(stubs.length).toBeGreaterThan(0); + for (const stub of stubs) expect(stub.metrics["vocabulary:order"], stub.id).toBeNull(); + expect(json.bindings.find((binding) => binding.channel === "height")?.unmeasured).toBe(stubs.length); + expect(stderr).toContain(`height is unmeasured on ${stubs.length} buildings (metric vocabulary:order)`); + expect(city({ height: "vocabulary:order", internalOnly: true }).json.bindings[0]?.unmeasured).toBe(0); + }); + + it("drives the footprint and the carried metrics too", () => { + const { json } = city({ footprint: "vocabulary:order", carry: ["vocabulary:basket"] }); + expect(spoken(json, "vocabulary:order")).toEqual({ AbstractOrder: 8, Notifications: 4, Order: 8, OrderService: 10 }); + expect(spoken(json, "vocabulary:basket")["Basket"]).toBeGreaterThan(0); + }); + + it("reads the word as the splitter reads a name: case folded, one word", () => { + expect(spoken(city({ height: "vocabulary:Order" }).json, "vocabulary:Order")).toEqual({ + AbstractOrder: 8, + Notifications: 4, + Order: 8, + OrderService: 10, + }); + expect(() => city({ height: "vocabulary:OrderService" })).toThrow(/2 words \(order, service\)/); + expect(() => city({ height: "vocabulary:" })).toThrow(UsageError); + expect(() => city({ height: "vocabulary:42" })).toThrow(UsageError); + }); + + // A typo would otherwise draw a flat city that looks like an answer. + it("refuses a word no name in the view says", () => { + expect(() => city({ height: "vocabulary:zebra" })).toThrow(/no name in view all says "zebra"/); + }); + + it("reads the artifact a caller hands in rather than counting again", () => { + const graph = buildGraph(loadModelFiles([FIXTURE]).union); + const counted = buildVocabulary(graph); + const order = counted.terms.findIndex((term) => term.text === "order"); + const doctored: VocabularyArtifact = { + ...counted, + scopes: counted.scopes.map((scope) => + scope.id === "java:com.acme.order/Money" ? { ...scope, terms: [[order, 99, 1, 1]] } : scope, + ), + }; + const built = cityOf(graph, { ...cityOptions(), height: "vocabulary:order" }, undefined, doctored); + expect(built.buildings.find((building) => building.id === "java:com.acme.order/Money")?.metrics["vocabulary:order"]).toBe(99); + }); + + it("counts nothing unless a vocabulary: metric is named", () => { + const never = (): VocabularyArtifact => { + throw new Error("counted"); + }; + expect(vocabularyMetric("loc", never)).toBeUndefined(); + expect(vocabularyMetric("history:commits_long", never)).toBeUndefined(); + }); +}); + +describe("vocabularyMetric over an artifact", () => { + const ROLES = { module: 0, type: 0, operation: 0, attribute: 0, parameter: 0, local: 0, other: 0 }; + const scope = (id: string, terms: [number, number, number, number][]) => ({ + level: "type" as const, + id, + name: id, + kind: "class", + parent: 0, + entities: 1, + roles: ROLES, + skipped: { unnamed: 0, conventional: 0, numeric: 0 }, + distinct: terms.length, + distinctDomain: 0, + weight: 0, + occurrences: 0, + terms, + }); + const artifact = { + kind: "codegraph.vocabulary/1", + terms: [ + { text: "final", class: "domain", modules: 1, types: 1 }, + { text: "final", class: "keyword", modules: 1, types: 1 }, + { text: "ledger", class: "domain", modules: 1, types: 1 }, + ], + scopes: [ + { level: "corpus", name: "mixed", terms: [] }, + scope("cobol:FINAL", [ + [0, 8, 1, 1], + [1, 3, 1, 1], + ]), + scope("cobol:LEDGER", [[2, 8, 1, 1]]), + ], + view: { name: "all", filters: [] }, + } as unknown as VocabularyArtifact; + const at = (id: string): MetricContext => ({ node: { id } }) as unknown as MetricContext; + + // The metric is about the word on the building, not the class a language + // gives it: two classes of one text add up. + it("adds up a text two languages class apart", () => { + const source = vocabularyMetric("vocabulary:final", () => artifact); + expect(source?.value(at("cobol:FINAL"))).toBe(11); + expect(source?.value(at("cobol:LEDGER"))).toBe(0); + expect(source?.value(at("java:Unread"))).toBeUndefined(); + }); +}); diff --git a/packages/viz/src/scene/help.ts b/packages/viz/src/scene/help.ts index 6931ccb..0719d3a 100644 --- a/packages/viz/src/scene/help.ts +++ b/packages/viz/src/scene/help.ts @@ -29,7 +29,7 @@ export function helpModel(city: CityLayout | null): HelpModel { "This is your code as a city: every district is a module (a package), " + "every building is a type declared there. Building dimensions follow " + "the metrics named in the legend below — a taller tower holds more " + - "code; nothing is decorative.", + "of its height metric; nothing is decorative.", "A washed-out building is a stub: an external type the model only " + "glimpsed through its uses. Nested plates are nested modules, " + "exactly as the model declares them.", diff --git a/packages/viz/test/help.test.ts b/packages/viz/test/help.test.ts index 7de8f3a..7f15f9b 100644 --- a/packages/viz/test/help.test.ts +++ b/packages/viz/test/help.test.ts @@ -31,6 +31,16 @@ describe("helpModel", () => { expect(text).toContain("Reset view"); }); + // Height is whatever metric was bound (loc, fanIn, vocabulary:order…): the + // prose defers to the legend instead of naming one. + it("claims no metric of its own for a building's size", () => { + const text = helpModel(makeCity()) + .sections.flatMap((section) => section.paragraphs) + .join(" "); + expect(text).toContain("named in the legend"); + expect(text).not.toMatch(/more code|lines of code/); + }); + it("embeds the artifact-derived legend when a city is loaded", () => { const help = helpModel(makeCity()); expect(help.legend.some((entry) => entry.label === "height = loc (linear)")).toBe(true); diff --git a/website/content/docs/reference/city-metrics.md b/website/content/docs/reference/city-metrics.md index ff5dd93..49996ad 100644 --- a/website/content/docs/reference/city-metrics.md +++ b/website/content/docs/reference/city-metrics.md @@ -47,6 +47,18 @@ With a change-history side-car beside the model — what [`codegraph-metrics --h `codegraph serve model.jsonl --height history-file:churn_recent --carry history:risk_score` raises a building by how much its file changed in the last 90 days and carries its type's risk rank. The side-car is read from `.history-metrics.jsonl` beside each model, or from `--history-metrics FILE`. Asking for a form with no side-car, or for a measure at a level it lacks (`churn_long` exists per file only), is a usage error that says what to run or type instead — never a silent zero. +## Vocabulary form + +`vocabulary:` reads the words of the model's names, as [`codegraph vocabulary`](/docs/reference/cli/vocabulary/) counts them under its default weights, wherever a metric name is accepted: + +| Form | Reads | Unit | +|---|---|---| +| `vocabulary:` | the word's weight in the names of the type and of everything folded into it: the Vocabulary tab's type scope | weighted occurrences | + +`codegraph serve model.jsonl --height vocabulary:order` raises each building by how much its names say `order`, so the City tab and the Vocabulary tab read one number. The word is read as the splitter reads a name: `Order` is `order`, and `OrderLine` is two words, which is a usage error. When two languages class one text differently, the two weights add up. + +A type whose names never say the word weighs **0**: its names were read, and that is a measurement. A stub is **unmeasured**, since the vocabulary never reads a name another codebase chose. A word no name in the view says is a usage error rather than a flat city, which would look like an answer. + ## "I do not know" is an answer A source returns `undefined`, never a substitute zero: a stub type has no anchor, so it has no line count, and a zero there would be a measurement nobody took. `sum:` likewise returns `undefined` — not `0` — when *no* member carries the key. diff --git a/website/content/docs/reference/cli/city.md b/website/content/docs/reference/cli/city.md index cd5284b..8d3acfc 100644 --- a/website/content/docs/reference/cli/city.md +++ b/website/content/docs/reference/cli/city.md @@ -23,9 +23,9 @@ codegraph city [model.jsonl...] [--height METRIC] [--height-scale , sum:. With a change-history side-car: history: (the type's own), history-file: (its file). | `loc` | +| `--height METRIC` | Metric driving building height. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). From the names: vocabulary: (the word's weight in the type's names). | `loc` | | `--height-scale ` | How height follows its metric. | `linear` | -| `--footprint METRIC` | Metric driving building footprint. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). | `members` | +| `--footprint METRIC` | Metric driving building footprint. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). From the names: vocabulary: (the word's weight in the type's names). | `members` | | `--footprint-scale ` | How the footprint SIDE follows its metric; sqrt makes the AREA proportional. | `sqrt` | | `--carry M1,M2` | Extra metrics to measure onto every building, comma-separated, bound to nothing. | — | | `--history-metrics FILE` | The change-history side-car the history:/history-file: metrics read (what 'codegraph-metrics --history' wrote). | `.history-metrics.jsonl` beside each model | diff --git a/website/content/docs/reference/cli/serve.md b/website/content/docs/reference/cli/serve.md index a6d38a3..784c6a6 100644 --- a/website/content/docs/reference/cli/serve.md +++ b/website/content/docs/reference/cli/serve.md @@ -25,9 +25,9 @@ codegraph serve --app [--data-dir DIR] [--extractors FILE] [--port N] [--host AD | Option | Meaning | Default | |---|---|---| | `--name STR` | Display name for the corpus in the page header; defaults to the basename of each model's root. | — | -| `--height METRIC` | Metric driving building height. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). | `loc` | +| `--height METRIC` | Metric driving building height. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). From the names: vocabulary: (the word's weight in the type's names). | `loc` | | `--height-scale ` | How height follows its metric. | `linear` | -| `--footprint METRIC` | Metric driving building footprint. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). | `members` | +| `--footprint METRIC` | Metric driving building footprint. Built in: degree, fanIn, fanOut, fields, loc, members, methods, one. Open forms: attribute:, sum:. With a change-history side-car: history: (the type's own), history-file: (its file). From the names: vocabulary: (the word's weight in the type's names). | `members` | | `--footprint-scale ` | How the footprint SIDE follows its metric; sqrt makes the AREA proportional. | `sqrt` | | `--carry M1,M2` | Extra metrics to measure onto every building, comma-separated, bound to nothing. | — | | `--history-metrics FILE` | The change-history side-car the history:/history-file: metrics read (what 'codegraph-metrics --history' wrote). | `.history-metrics.jsonl` beside each model | diff --git a/website/content/docs/reference/cli/vocabulary.md b/website/content/docs/reference/cli/vocabulary.md index 309d324..c16f6cb 100644 --- a/website/content/docs/reference/cli/vocabulary.md +++ b/website/content/docs/reference/cli/vocabulary.md @@ -55,6 +55,10 @@ The glossary is a plain text file, one business term per line; blank lines and ` [`serve`](/docs/reference/cli/serve/)'s **Vocabulary** tab runs the same layout in the browser, measured in the page's own monospace face, over the scope selected in Navigate, and lists the names that carry a clicked word. +## In the city + +One word across the whole city is the city metric `vocabulary:`: `codegraph serve model.jsonl --height vocabulary:order` raises each building by the word's weight in its type scope, the number this command reports under `--level type`. A type that never says the word stands at the floor; [City metrics](/docs/reference/city-metrics/#vocabulary-form) has the rules. + ## Exit codes `0` ok · `1` internal error (a bug) · `2` usage error · `3` findings. diff --git a/website/content/docs/tutorials/navigator.md b/website/content/docs/tutorials/navigator.md index ad0337f..5cce7ec 100644 --- a/website/content/docs/tutorials/navigator.md +++ b/website/content/docs/tutorials/navigator.md @@ -43,7 +43,7 @@ it any time. You will query it yourself in The header shows what was loaded: **1,855 nodes · 6,236 dependencies**. - + ## 2. Find the class @@ -143,11 +143,13 @@ Four meanings carry a hue in the navigator and nothing else does: A stub — an external type — is drawn muted and italic rather than in a colour of its own. It is a degraded fact, not a fifth category. -## 6. Use the other three tabs +## 6. Use the other tabs -The tabs across the top are **Navigate**, **Graph**, **Cycles** and -**Coupling**. Every row in the three report tabs leads back to Navigate, so a -finding is always one click from its evidence. +The tabs across the top are **Navigate**, **City**, **Graph**, **Cycles**, +**Coupling** and **Vocabulary**. City is the one you toured in +[Your first code city](/docs/tutorials/first-city/). Every row in the report +tabs leads back to Navigate, so a finding is always one click from its +evidence. **Graph.** A force-directed dependency graph (Cytoscape with fcose). Two modes, **Modules** and **Types**; node size follows fan-in, and labels appear by zoom @@ -177,6 +179,19 @@ different questions. A number without its definition is not a fact. +**Vocabulary.** The words of the selection's names, split the way Java +writes names. With `TypeAdapter` still selected, the tab counts the class and +everything written in it, **26 entities**, and shows **7 domain terms of +17**. The heaviest is `tree`, at weight **7**. `json`, `type` and `adapter` +weigh more but are classed *technical*; the **technical** chip shows them. +Click `tree` and the panel lists the three names that carry it: the methods +`fromJsonTree` and `toJsonTree` (3 each, an operation's weight) and the +parameter `jsonTree` (1). The breadcrumb reads **gson › com.google.gson › +TypeAdapter**; click **gson** for the whole corpus, whose heaviest words are +`number`, `write`, `date` and `constructor`. + + + ## 7. Stop the server `Ctrl-C`. As with the city, you can write the artifact to a file instead: @@ -197,8 +212,8 @@ That file is the frontend's only input. - A way to go from "something depends on this" to the file and line that proves it, in three clicks. - The habit of reading the provenance badge before believing a dependency. -- Four views over one model that cannot disagree with each other, because all - four were computed once by the analyzer and only rendered in the browser. +- Six views over one model that cannot disagree with each other, because all + six were computed once on the server and only rendered in the browser. ## Where to go next