How CodeCensus was built on CodeGraph-Bench, how every tool was scored, and every change we made after the first version, with the control that ties each change to the scores before it. Built 2026-10-12 to 2026-10-18. README.md has the results; this file has the reasons.
Two naming notes. In file and folder names, original means CodeGraph-Bench and extended means
CodeCensus. "The answers" are the symbols and edges the TypeScript compiler reports for a
repository, the rows every tool is scored against; the published answer files keep the names
CodeGraph-Bench's generator gives them.
Contents: 0 Pins · 1 CodeCensus's rules · 2 Reproducing CodeGraph-Bench · 3 CodeCensus's generator · 4 Rows per rule · 5 First review round · 6 Verification · 7 Scoring · 8 Step 1: machine-generated files · 9 Step 2: one scorer artifact · 10 Changes to Memtrace · 11 Step 3: declaration merges · 12 Scored graphs · 13 Rerun · 14 Path disclosure
| what | pin |
|---|---|
| CodeGraph-Bench | by Arpit Mishra, github.com/vyazen/CodeGraph-Bench @ 50b87e2102145b4963612f8fb2221d7f792c566d (2026-08-21); 26 files, sha256 in upstream/UPSTREAM.lock.json |
| Identifier restoration | EXT***REMOVED***S → EXTENDS, EXT***REMOVED***ED → EXTENDED, 13 files (CodeGraph-Bench's public history was scrubbed with a tool that replaced the text "END"; the published code does not parse) |
| CodeGraph-Bench's generator | its TypeScript generator, sha256 9adadf45… after restoration |
| CodeCensus's generator | sha256 3712aa99…, built by scripts/apply_edits.py apply |
| TypeScript | 6.0.3 from CodeGraph-Bench's bun.lock; 140 files checked (upstream/typescript-6.0.3.sha256) |
| BabylonJS | github.com/BabylonJS/Babylon.js @ 4efc049052ba4ee3e8ae36c8d2e874b17aac9244; 8,367 files walked |
| VS Code | github.com/microsoft/vscode @ 0102a4dbe6c84ca2f9bdd2e662a419ef07b491b3; 7,325 files walked |
| Target trees | CRLF working copies (Windows checkout, core.autocrlf=true); every file's sha256 in targets/<target>.manifest.tsv; target dependencies not installed |
| GitHub Linguist | github.com/github-linguist/linguist @ 73ee9a3b277f398c37ca4ab0dea0193849de6460, lib/linguist/generated.rb sha256 e5d73b4b… |
| Runtimes | CodeGraph-Bench's answers generated with bun 1.4.0 (Windows); their reproduction and CodeCensus's answers with bun 1.2.21 (Linux); recorded scoring runs on Linux with bun 1.2.21, except GitNexus on CodeGraph-Bench (Windows); the public rescore of every run on Windows with bun 1.4.0; the public Dockerfile pins bun 1.2.21 |
CodeCensus = CodeGraph-Bench's answers plus:
- Symbols for every named declaration at any depth, with CodeGraph-Bench's id and kind conventions:
- 1a
FunctionDeclarationwith a name (Function). - 1b
VariableDeclaration, Identifier name, initializer directly an arrow/function expression (Function). - 1c
ClassDeclarationand namedClassExpression(Class), members exactly as CodeGraph-Bench emits top-level class members; members of unnamed class expressions get id = member name. - 1d Object-literal members named by Identifier/StringLiteral/NumericLiteral: methods and
get:/set:accessors (Method), property assignments initialised directly with an arrow/function expression (Function); id = member name. - 1e
a.b = <arrow|function>(Function, id = rightmost property name). - 1f Interface / type alias / enum declarations nested in blocks (CodeGraph-Bench's kinds).
- Computed names are skipped where CodeGraph-Bench skips them.
- 1a
- CALLS: each call site goes to the innermost enclosing callable or class-member container. Anonymous callbacks, IIFEs and unnamed function expressions stay transparent. With no container the site goes to the File. Callee resolution is CodeGraph-Bench's.
- EXTENDS/IMPLEMENTS for every new class/interface symbol, using CodeGraph-Bench's logic.
- Module-level IMPORTS: CodeGraph-Bench's rows plus one row per module dependency the compiler
resolves (
importdeclaration,import x = require(), dynamicimport(),require()in JS).export … fromis excluded. - TS/JS files no project program loads are analysed in one extra inferred program (allowJs, checkJs off, CodeGraph-Bench's fallback options), under the same rules.
Everything else is unchanged: canonicalisation, the matcher, the (path, id) de-duplication, the
fidelity scorer, and how each tool's graph is converted.
Where the generator departs from the literal rule text (section 3, decisions 7 and 8):
- Ids. "id = member name" holds unless that id is already used in the file by a different
declared entity. Then the added row gets
<name>@L<line>(decision 7). - Module file for JS importers. A JS importer's module row names the JS file it loads, not the TypeScript sibling the compiler's lookup prefers (decision 8).
CodeGraph-Bench's answers were produced on 2026-10-12 on Windows by its unchanged generator (after
the identifier restoration) with TypeScript 6.0.3 under bun 1.4.0, against Windows checkouts of both
targets, with no target dependency installed. The entry point did what generator/run_generator.ts
does with original.
They were then reproduced on a Linux machine. Its target trees match the Windows ones file for file:
BabylonJS 8,367 files and VS Code 7,325, every sha256 identical (targets/<target>.manifest.tsv).
Both trees are CRLF.
| attempt | symbols | edges |
|---|---|---|
| ext4 copy, Linux path | row-identical | BJS 144 rows differ; VS 1,733 rows differ and has 7 extra (union-member targets and namespace-import paths) |
| + NTFS readdir order | byte-identical | rows equal; only 114 BJS / 1,542 VS target ids differ (absolute path inside namespace-import targets) |
| + Windows path spelling | byte-identical | byte-identical (both targets) |
Root causes and the environment fixes (the generator was not changed):
- File order.
walkRepoFilesdoes not sortreaddir, so the generator queries the checker in filesystem order. TypeScript hands out type ids lazily, so union-constituent order, and with it which declaration a union property access resolves to, depends on that order. NTFS returns upcase-collation order; ext4 returns hash order. Fix:generator/build_ntfs_order_tree.pycopies the sources to tmpfs, creating each directory's entries in reverse of the recorded Windows walk order (targets/<target>.walk-order.txt, written bygenerator/walk_dump.ts). tmpfs on that kernel returns entries in reverse creation order (checked with a probe); the walk on the copy is byte-identical to the Windows walk. - Absolute path in ids. TypeScript names a module symbol
"<absolute file name without extension>", and CodeGraph-Bench records that name as the target id of namespace imports. Fix: the tree is bind-mounted at/<windows repo path>and the generator getsrepoPath = "<windows repo path>"(forward slashes) with cwd/. TypeScript treatsC:/…as rooted, and the OS resolves it relative to/. The path is the one recorded in the published summary file of each answers asset (section 14). - TypeScript 6.0.3 is the copy pinned before any run, verified by per-file sha256.
The reproduction was rerun alongside CodeCensus's generation (2026-10-18 03:38 UTC) with the same result:
| file | rows | sha256 |
|---|---|---|
| BabylonJS symbols | 69,161 | d18c6dc7… (identical) |
| BabylonJS edges | 410,264 | 6b8cffec… (identical) |
| VS Code symbols | 134,488 | 0fbdaa08… (identical) |
| VS Code edges | 811,310 | cb451d7b… (identical) |
CodeCensus's generator starts as a copy of CodeGraph-Bench's. The version that produced the
published answers has sha256 3712aa99… (the first version, before the review round of section 5,
was 0f2b8eb4…). CodeGraph-Bench has no license, so this repository does not ship the file: it
ships upstream/codecensus-generator.edits.json, 31 anchored edits that hold only our own text
(736 lines or fragments; 51 further lines that repeat a CodeGraph-Bench line are copied by line
number; 359 CodeGraph-Bench characters are removed, identified by sha256). scripts/apply_edits.py apply builds the file from the downloaded CodeGraph-Bench generator and refuses to write anything
that is not byte-identical to 3712aa99….
- CodeGraph-Bench lines changed: 11. These are return types, an extra
programargument, capturing the CALLS rowspushEdgereturns, andresolveSymbolLocationalso returning the declaration it resolved. - Every other change is an addition, marked in the code with the rule item or fix it implements.
The generator's parts:
- Declaration bookkeeping.
visit()records each statement it handles. A declaration counts as already emitted only if CodeGraph-Bench's pass visited it. - Call bookkeeping.
visitCallSitesrecords, for each call site, the container CodeGraph-Bench credited and the rows it pushed. - The added pass (rules 1-4). One full-AST walk per file:
- emits 1a-1f symbols through CodeGraph-Bench's own
pushSym, with the same line conventions (a variable's lines come from its statement); - registers every container region: CodeGraph-Bench's functions, fn-variables and class members, plus all added callables and members;
- emits EXTENDS/IMPLEMENTS for new named classes and interfaces with CodeGraph-Bench's code,
including
[expr]→scoreable:falseand member-level IMPLEMENTS; - credits every call site to its innermost container;
- emits module rows (rule 4).
- emits 1a-1f symbols through CodeGraph-Bench's own
- Unique ids. Each added symbol's id is checked against the ids already used in its file:
every CodeGraph-Bench row, then the added rows in walk order.
- Collision with a different declared entity: the row becomes
<id>@L<startLine>, then@L<line>:<col>if needed, then#n. - Same entity (several declarations sharing one binder symbol, such as overload signatures or merged interface/enum/class declarations): the id is shared, exactly as CodeGraph-Bench handles overloads.
- Always distinct: variable bindings, object-literal members, assignments and re-declared
bodies (
function ftwice in JS). namestays the declared name, so the matcher's(path, name)lookup is unchanged.- A renamed class qualifies its members (
K@L19.m, parentK@L19) and its heritage rows. - Container regions, and therefore the caller id of every CALLS row, use the final id.
- After all rows exist, a row is re-pointed when it resolves to a renamed declaration and still names it by the base id: its target id becomes the final id, and the old value is kept in a field of its own. This covers CodeGraph-Bench rows too.
- Collision with a different declared entity: the row becomes
- Quoted names. A 1d member name is the key's value (
.text), so'predicate'ispredicate. - JS siblings. A rule-4 module row from a JS importer (
.js/.jsx/.mjs/.cjs) normally names the file the compiler resolved. When the compiler resolved a TypeScript file (.ts/.tsx/.mts/.ctsor a.d.ts), a same-stem JS sibling exists, and the specifier does not itself name a TS extension, the row names the JS sibling, and a separate field keeps the compiler's file. - Ordering. The added pass is deferred until every file of a program has had CodeGraph-Bench's pass, so the checker sees exactly CodeGraph-Bench's query sequence while its rows are produced.
- Output. All CodeGraph-Bench rows come first, in its order. Added rows are appended per project, then the rule-5 rows. Every added row carries its rule tag (rule-5 rows also name the rule-1 item they came from).
- Rule 5. After all projects, every walked file matching the scorer's TS/JS regex
(
/\.(tsx?|jsx?|mts|cts|mjs|cjs)$/i) that no project program returned fromgetSourceFileis a root of one extra program. The options are CodeGraph-Bench's fallback pluscheckJs:false. These files get CodeGraph-Bench's full pass followed by the added pass. - The summary file carries CodeGraph-Bench's fields plus per-rule row counts (including the unique-id and JS-sibling counters) and notes.
- CodeCensus is not a strict superset of CodeGraph-Bench.
- Every CodeGraph-Bench symbol row is byte-identical.
- Two kinds of CodeGraph-Bench edge row are changed in place; every other field stays identical
(
generator/check_additions.py):- Re-attributed (rule 2). CodeGraph-Bench credits calls inside object-literal methods and class-expression members to the enclosing declaration; rule 2 credits them to the member. Only the caller id changes; the row is tagged with rule 2 and keeps the old caller id. BJS 489, VS 8,905 (9,394 rows).
- Re-targeted (unique ids). The row resolves to a nested declaration whose id was made unique. Only the target id changes, and the row keeps the old one. BJS 43, VS 899 (8 VS rows are also re-attributed).
- Module rows (rule 4) carry the module file as target path and target id, the specifier as
target name, and
scoreable:false.- The module-level imports metric reads only the target path, so the rows count there.
scoreable:falsekeeps them out of the symbol-level IMPORTS denominator, where a file target has no symbol identity.- Only resolutions to an in-repo file get a row (
program.getResolvedModuleFromModuleSpecifier). Unresolved specifiers get no row (BJS 2,912 + 398, VS 3,244 + 452), so the scorer's unresolved-import budget is unchanged.
- Computed and quoted names. CodeGraph-Bench emits computed class-member names (for example
XRBody.[Symbol.iterator], 94 BJS / 118 VS rows) and quoted names viagetText()(252 VS rows). New class members (1c), whose rule says "exactly as CodeGraph-Bench", mirror that. Object-literal members (1d) skip computed names and, since the quoted-name fix, use the key's value without quotes. Computed-name function values are not symbols, so they stay transparent. - Literal rule 2 everywhere. Calls CodeGraph-Bench dropped in class decorators, heritage
clauses, static blocks, namespace-level statements and enum initialisers now go to the File, or
to the enclosing member/callable. Calls in property and accessor decorators go to that member.
Function values that are not CodeCensus symbols are transparent (
x = a || function(){},{[k]: () => …}, named function expressions passed as arguments). - Other edge types (USES_TYPE, ACCESSES, METHOD_OVERRIDES) are not emitted for new nested symbols, because the rule does not list them. Rule-5 files get CodeGraph-Bench's full pass, so those types appear there.
- Unnamed
ClassDeclaration(export default class {}) is not in the rule. It is transparent and gets no symbols; it was never encountered in either repo (generator notes). - Unique ids.
- Why. Rule 1's "id = name" plus the unchanged
(path, id)de-duplication merged distinct declarations into one answer row (finding A, section 5). A tool that emitted every declaration was charged a false positive for each extra one and for its calls; a tool that emitted one lost nothing. - What changes. Only the id of an added row that would repeat one used by a different entity
in the same file.
name, kind and lines are unchanged. - Kept. CodeGraph-Bench rows keep their ids, including its own collisions (163 BJS / 1,120 VS groups, untouched).
- Why. Rule 1's "id = name" plus the unchanged
- JS importers load the JS file. Applied only to rule-4 module rows.
- Declined: symbol-level rows. CALLS/ACCESSES rows from the same JS files that resolve into
the
.tssibling are left as the checker resolves them. Rule 2 fixes callee resolution to CodeGraph-Bench's, and CodeGraph-Bench itself has such rows: 272 VS CALLS, plus 17 CALLS and 1 ACCESSES via a.d.ts. Changing only the added and rule-5 rows (89 CALLS and 32 ACCESSES, plus 5 CALLS via a.d.ts) would make rows from one file disagree.
- Declined: symbol-level rows. CALLS/ACCESSES rows from the same JS files that resolve into
the
Decisions 7 and 8 are judgment calls; each is a small, isolated change in the generator.
From the per-rule row counts in the summary file. The counts are identical to the first version, because the fixes of section 5 change ids and targets, not rows.
| item | babylonjs | vscode |
|---|---|---|
| CodeGraph-Bench rows (symbols / edges) | 69,161 / 410,264 | 134,488 / 811,310 |
| 1a nested function declarations | 140 | 1,492 |
| 1b variable-bound functions | 918 | 1,497 |
| 1c classes + members (Class/Constructor/Method/Property) | 184 (4/9/49/122) | 4,194 (505/791/2,145/753) |
| 1d object-literal members (Function/Method) | 2,075 (1,836/239) | 5,708 (3,935/1,773) |
| 1e assignment-bound functions | 1,154 | 325 |
| 1f nested interface/alias/enum (+members) | 15 | 759 |
| 2 CALLS added | 13,982 | 26,327 |
| 2 CALLS re-attributed (caller id only) | 489 | 8,905 |
| 3 EXTENDS / IMPLEMENTS | 2 / 0 | 399 / 252 |
| 4 module IMPORTS rows | 25,910 | 48,310 |
| 5 uncovered TS/JS files (not loadable) | 246 (0) | 232 (0) |
| 5 symbols / edges from those files | 18,759 / 94,817 | 2,766 / 10,544 |
| CodeCensus total (symbols / edges) | 92,406 / 544,975 | 151,229 / 897,142 |
CALLS added under rule 2, by the container that received them:
| container | babylonjs | vscode |
|---|---|---|
| 1a | 824 | 9,102 |
| 1b | 4,424 | 5,793 |
| 1c | 12 | 1,481 |
| 1d | 2,464 | 8,164 |
| 1e | 3,493 | 831 |
| a container CodeGraph-Bench already had | 2,714 | 152 |
| File | 51 | 804 |
Re-attributed rows went to 1c members (BJS 55, VS 6,150) and 1d members (BJS 434, VS 2,755).
Unique-id and JS-sibling counters:
| babylonjs | vscode | |
|---|---|---|
| unique ids: added symbols renamed, covered files (1a/1b/1c/1d/1e/1f) | 1,249 (3/76/0/988/182/0) | 3,156 (58/154/941/1,906/70/27) |
| unique ids: added symbols renamed, rule-5 files | 3,786 | 56 |
| unique ids: rows re-targeted (CodeGraph-Bench rows / added rows) | 833 (43 / 790) | 956 (899 / 57) |
JS siblings: module rows re-pointed to the JS sibling (from TS source / from .d.ts) |
0 | 126 (115 / 11) |
Rows the scorer's (path, id) de-duplication still collapses into an earlier row:
| babylonjs | vscode | |
|---|---|---|
| first version, before the unique-id fix (1d / 1c / 1e / 1b) | 988 / 0 / 182 / 76 | 1,906 / 1,019 / 70 / 154 |
| published version (added rows) | 0 | 26: overload signatures and merged interfaces (1a 8, 1c 15, 1f 3) |
| rule-5 rows from CodeGraph-Bench's pass (its own behaviour) | 0 | 199 |
Every (path, id) group that still holds several start lines and whose first row is an added row
was classified. BabylonJS has none. VS Code has 118, none of them two distinct declarations: 105
consist only of rule-5 rows from CodeGraph-Bench's pass (its behaviour, applied to rule-5 files),
11 are overload signatures (for example getWindowById in src/vs/base/browser/dom.ts:52-54), and
2 are a repeated nested interface IWaitUntil and its members.
Rule 5 in BabylonJS is dominated by vendored minified bundles under
packages/tools/babylonServer/public/: ammo.js alone has 4,984 symbols / 40,575 edges, followed
by gltf_validator.js, the draco bundles, libktx.js and cannon.js. These bundles repeat short
names on one line; the unique-id fix makes each a distinct answer row (@L1:<col>), which accounts
for most of the 3,786 rule-5 renames. In VS Code the largest rule-5 files are jquery.d.ts,
src/vs/loader.js, marked.js and build/lib/*.js. Step 1 (section 8) skips the files GitHub's
rules call generated: 25 of these BabylonJS bundles and 70 VS Code rule-5 files (mostly the
compiled build/**/*.js).
Our own internal review of the first version raised four findings. All were reproduced on that
version before anything was changed; the counts below come from that reproduction, and the tool
counts from CodeGraph-Bench's unchanged matchNodes.
| # | finding | reproduced | verdict | after the fix |
|---|---|---|---|---|
| A (major) | same-name added declarations collapse under (path, id) |
distinct added declarations dropped: BJS 4,521 (1a 3, 1b 76, 1d 988, 1e 182, rule-5 3,272), VS 3,518 (1a 66, 1b 154, 1c 1,017, 1d 1,905, 1e 70, 1f 59, rule-5 247); every tool that emitted them was charged false positives for them | confirmed, fixed (unique ids) | dropped distinct declarations: 0 in both targets. VS still collapses 26 overload/merge rows (one entity each) and 199 rule-5 rows from CodeGraph-Bench's pass (its own behaviour) |
| B (minor) | CodeCensus is not a strict superset (9,394 rows re-attributed) | 489 + 8,905 rows; 3,056 of 6,931 scoreable VS re-attributed rows land on an ambiguous container | confirmed; disclosed (decision 1) | re-attributed rows on an ambiguous container: 0 (both). Plus 43 / 899 re-targeted CodeGraph-Bench rows, disclosed the same way |
| C (minor) | a JS importer resolves to the .ts sibling, not the .js Node loads |
VS rule-4 IMPORTS 63 (+10 via .d.ts), rule-5 IMPORTS 52 (+1); CALLS: CodeGraph-Bench 272, rule-5 rows from its pass 62, added 27 (+5 via .d.ts); ACCESSES rule-5 32. BJS 0 |
confirmed; module rows fixed (JS siblings); symbol-level rows declined (decision 8) | JS-importer module rows to a TS file with a JS sibling: 0 |
| D (minor) | quoted 1d names ('predicate') |
BJS 14 + 216 rule-5, VS 37 | confirmed, fixed (quoted names) | 0 quoted names among added rows |
-
generator/check_additions.py: CodeCensus = CodeGraph-Bench plus additions, except the two documented in-place changes.- Symbols: the first 69,161 / 134,488 lines are byte-identical to CodeGraph-Bench.
- Edges: 409,732 / 801,514 CodeGraph-Bench rows are unchanged. BJS: 489 changed only in the caller id and 43 only in the target id. VS: 8,897 changed only in the caller id, 891 only in the target id, and 8 in both.
- 0 rows changed otherwise, and 0 added rows lack a rule tag: the check passes for both targets.
-
Determinism. A second full run of CodeCensus's generator gave byte-identical symbol and edge files for both targets:
target symbols sha256 edges sha256 babylonjs 00ed95af…9e1087f2…vscode 084aaca9…099aa39f… -
Fixture.
fixture/repo/exercises every rule item, plussrc/dup.ts(repeatedhandler/predicatemembers, a quoted and a numeric key, two unnamed class expressions, a nestedhelper2shadowing a top-level one, nested overloads and two nestedclass K) andscripts/req.jswithlib.tsandlib.js(a JS importer with a TS sibling). Expected:handler,handler@L5,handler@L6;predicate,predicate@L5;run,run@L8;constructor,constructor@L8;helper2@L11with CodeGraph-Bench's rowouterA → helper2re-targeted to it;K,K@L19,K@L19.m; the overloadsovshare one id;req.js → scripts/lib.jswith the compiler'sscripts/lib.tskept alongside.check_additions.pypasses on it, and the only multi-line id is theovoverload group.fixture/run_fixture.ts+fixture/check_fixture.pyrerun it on the benchmark code this repository builds; the outputs equalfixture/expected/(checked 2026-10-18 on Windows with bun 1.4.0).
One scoring core for every tool. scoring/compute_scope.ts is our computeScope: it filters the
tool graph to TS/JS-source nodes and relations and calls CodeGraph-Bench's own depth and fidelity
scorers, which use its node-identity matcher (path, canonical name, kind, start line within ±2) and
its edge adjudicator. CodeGraph-Bench's code is imported at runtime from the downloaded copy;
nothing of it is copied or changed. computeScope asserts that the two scorers agree on every
comparable edge type.
- Primary scope: every answer row is in the denominator, whether or not the tool indexed the file. All headline numbers use it.
- Secondary scope: the answers filtered to the TS/JS files the tool indexed (coverage-conditional; reported in SCORES.json, not in the headline).
- CodeGraph-Bench:
computeScopeon CodeGraph-Bench's answers and the tool graph, nothing else. - CodeCensus: three steps run on both sides first (sections 8, 9 and 11), then the same
computeScope.
How the recorded runs were made. The recorded scores came from per-tool wrappers around the same
computeScope. Memtrace's wrapper also checked the receipts of the private capture (export
receipts, raw-source hashes and an allow-list of converter versions); the GitNexus and Graphify
wrappers carried a copy of computeScope whose only change is the tool name in checkGraph.
scoring/score.ts replaces them with one public scorer that gates on the published inputs only
(CodeGraph-Bench sources, answer hashes, generated-file lists, graph meta). It reproduces every
recorded score table of all four graphs (three tools, Graphify in two releases) on both benchmarks, cell for cell (596-620 numeric cells per
run, 18 runs including two controls; scripts/verify_results.py).
CodeGraph-Bench → CodeCensus before any scoring step, F1 %, primary scope, GitNexus and Graphify (module-level F1 is derived from the scorer's P and R):
| babylonjs | Function | Method | all symbols | CALLS | EXTENDS | IMPLEMENTS | IMPORTS module | all edges |
|---|---|---|---|---|---|---|---|---|
| gitnexus | 46.16 → 69.47 | 91.17 → 98.64 | 79.36 → 83.16 | 58.71 → 57.24 | 87.35 → 87.39 | 73.93 → 73.93 | 71.44 → 82.99 | 48.58 → 48.78 |
| graphify-0.9.27 | 69.49 → 50.45 | 90.54 → 91.91 | 64.73 → 61.76 | 45.20 → 38.03 | 98.42 → 98.46 | 98.68 → 98.68 | 90.66 → 99.06 | 67.58 → 59.78 |
| graphify-0.9.92 | 59.03 → 58.35 | 91.31 → 92.66 | 66.91 → 66.19 | 57.70 → 55.66 | 99.01 → 99.04 | 95.26 → 95.26 | 89.72 → 98.40 | 70.81 → 67.79 |
| vscode | Function | Method | all symbols | CALLS | EXTENDS | IMPLEMENTS | IMPORTS module | all edges |
|---|---|---|---|---|---|---|---|---|
| gitnexus | 61.22 → 87.72 | 90.86 → 94.65 | 77.48 → 81.14 | 45.78 → 49.10 | 75.28 → 73.01 | 75.56 → 75.80 | 94.37 → 94.51 | 39.10 → 42.00 |
| graphify-0.9.27 | 81.93 → 64.89 | 97.60 → 95.24 | 71.43 → 68.96 | 51.11 → 48.64 | 97.88 → 95.58 | 93.49 → 93.41 | 98.63 → 98.72 | 67.18 → 65.14 |
| graphify-0.9.92 | 79.35 → 68.16 | 98.27 → 95.94 | 76.37 → 74.12 | 67.36 → 64.97 | 97.90 → 95.61 | 94.02 → 93.93 | 98.60 → 98.69 | 75.20 → 73.52 |
Where the errors came from at that stage (CodeGraph-Bench's matcher and adjudicator rerun per row; every count matches SCORES.json):
- BabylonJS. Most of the drop came from rule 5 crediting the vendored minified bundles of section 4. That led to step 1 (section 8).
- Graphify and GitNexus. Their Function recall loss in covered files comes from 1b/1d/1e symbols they do not emit (Graphify 0.9.92 on VS misses 1d 3,908, 1b 1,492 and 1e 285; GitNexus on VS misses 1d 528, 1e 249, 1b 148 and 1a 127).
- Tools that emit every declaration gain under the unique-id fix; tools that emit one node per repeated name lose recall, because each distinct declaration is now its own answer row.
Why. Rule 5 analyses TS/JS files that no tsconfig program loads. In BabylonJS that pulls in
vendored, minified bundles (packages/tools/babylonServer/public/ammo.js and others). They are
machine-generated code, not code people wrote, and a benchmark of written source should not score
them, for any tool.
Decision. Use an independent, published standard: GitHub's own rules for spotting generated
code (the "generated" classification in GitHub Linguist), ported faithfully. Linguist's
vendored-code rules (vendor.yml) are not used, and no rule of our own is added. The step applies
identically to the answers and to every tool.
The port (linguist/linguist_generated.py; its body has sha256 15eec963…, the published file
prepends Linguist's MIT notice, linguist/check_port.py):
- All 74 checks of
Generated#generated?(generated.rb L53-127) are ported, in Ruby's evaluation order, with the original thresholds, extension sets and regexes. Every method cites its generated.rb line; a script verified the check list, its order and all 76 cited definition lines. - Checks that can fire on the TS/JS family:
.jsminified_files?(integer average line length above 110, throughmaybe_minified?,.js/.cssonly),has_source_map?(one of the last two lines matches^/[*/][#@] source(?:Mapping)?URL|sourceURL=),compiled_coffeescript?,generated_parser?(PEG.js),generated_javascript_protocol_buffer?,generated_apache_thrift?,generated_jison?,generated_haxe?;.tsgenerated_typescript_protocol_buffer?; path checks (node_modules?,__generated__/,.pnp.*and others). No check names.mjs/.cjs/.jsx/.tsx/.mts/.cts. - Ruby → Python: data are bytes (Linguist reads blobs as ASCII-8BIT, so line lengths are byte
counts);
linesisdata.split("\n", -1)and""gives[];^/$are line anchors (re.M),\w \d \s \bASCII (re.A);\Z→(?=\n?\Z);(?<=^|\s)→(?:^|(?<=\s));File.extnamereimplemented from Ruby's file.c; out-of-range indexes are nil, a method on nil raises. - Verified against GitHub's own tests:
linguist/test_port.pyreplaystest_generated.rb(test_check_generated,test_check_not_generated) and thegenerated?assertions oftest_blob.rb, with the same contract (name-only cases must not load data, data cases must). 189/189 pass (Python 3.12 Linux, Python 3.14 Windows; rechecked 2026-10-18 with the fileslinguist/fetch_linguist.pydownloads).
Classification (linguist/classify_generated.py): every blob of the git tree at each pinned
commit; name = repo-relative path, data = the committed blob, full length. BabylonJS has 8,367
blobs (5,020 TS/JS source), VS Code 7,326 (5,117). A second run is byte-identical.
| target | generated files | TS/JS source among them | first check, TS/JS files | other generated files |
|---|---|---|---|---|
| babylonjs | 27 | 25 (6,140,110 bytes) | minified_files? 24, has_source_map? 1 |
package-lock.json, babylon.uiControls.max.js.map |
| vscode | 76 | 71 (759,407 bytes) | has_source_map? 69 (the committed tsc output build/**/*.js), minified_files? 2 |
2 Cargo.lock, 2 minified auth.css, 1 fixture under node_modules/ |
Sensitivity checks, reported and not applied: 128 KiB LazyBlob truncation changes 0 files; working-
tree bytes instead of blobs change 0 files; .gitattributes linguist-generated (a repository
attribute, not a generated.rb rule) marks 4 non-TS/JS BabylonJS files and none in VS Code; no file
would make Ruby raise.
Applying the list (scoring/extended_scope.ts, applyExclusion): answer symbols in a generated
file are dropped, and so are answer edges whose source or target file is generated. Tool nodes in a
generated file are dropped, and so is every tool edge touching one; the graph's node and edge counts
are set to the filtered counts so checkGraph holds. Paths are normalised the way CodeGraph-Bench
does it. Control: --exclusion none runs the same code with an empty list.
Rows each side loses (the answer-side cut is the same for every tool;
linguist/lists/DROPPED-BY-RULE.json):
- BabylonJS answers: 14,081 of 92,406 symbols (Function 13,964) and 70,733 of 544,975 edges (CALLS 70,424, ACCESSES 307, EXTENDS 2), from 25 files. Every dropped row is a rule-5 row.
- VS Code answers: 1,325 of 151,229 symbols and 6,430 of 897,142 edges, from 71 files: rule-5 rows
(1,312 symbols; CALLS 5,154, ACCESSES 1,070, IMPORTS 61, EXTENDS 6, METHOD_OVERRIDES 1), 13 symbols
and 74 edges of CodeGraph-Bench rows from
build/azure-pipelines/upload-configuration.js(generated, but loaded by a tsconfig program), 64 rule-4 module IMPORTS rows (62 from written build scripts into the generatedbuild/lib/*.js), and 67 edges dropped only because their target file is generated.
What the step leaves in (linguist/lists/RESIDUAL-RULE5.json). GitHub's rules do not call
every vendored file generated. BabylonJS keeps rule-5 rows in 221 files (4,678 symbols, 24,084
edges), the largest being three more bundles under babylonServer/public/: gltf_validator.js
(2,432 / 8,368), cannon.js (419 / 2,510), basis_encoder.js (452 / 1,272). VS Code keeps rule-5
rows in 148 files (1,454 / 4,257), the largest jquery.d.ts, marked.js, src/vs/loader.js and
dompurify.js. Removing vendored files would need a second standard, such as Linguist's
vendor.yml; it is not applied.
Controls. When the step was added, --exclusion none reproduced the scores from before it
(primary and secondary deep-equal) for 8/8 tool × target pairs, and the answer-side cut was identical
across the tools of a target. A review round then found that the documentation misstated which
converter versions the Memtrace scoring wrappers accepted (no score was affected; all 24 runs were
redone and equal), and that a relayed summary printed "n/a" for metrics that were scored (the
deliverables were complete).
The decision rule, fixed before the classification: a miss of a real declaration that the compiler's answers correctly contain is fixed in the product; a miss caused by the answers or the scorer is fixed generically, for every tool, and every tool is rescored. Nothing is tuned to Memtrace's output.
Before the fixes of section 10, every Function/Method row Memtrace missed on CodeCensus was
classified with the unchanged step 1 and CodeGraph-Bench's matchNodes (its TP/FP/FN equal
SCORES.json for all 10 tool × target pairs). Each row got one primary cause, the first rule that
matches; the primaries sum to the scorer's misses.
| Memtrace misses | primary cause | rows (rule tags) | kind of miss |
|---|---|---|---|
| BJS Function 1,533 | P1: the file is indexed as a File node only | 1,532 (5/1d 689, 5/1e 449, 5/1a 243, 5/1b 151) | no node at all |
P5: a member-stored named function expression is named by its inner name (value: function flat() → flat) |
1 (1d) | node with another name | |
| BJS Method 1,786 | P1 | 1,759 (5/1d 1,727, 5/1c 32) | no node at all |
P2: the TypeScript grammar rejects protected abstract override readonly _options: T; and ends the class there |
23 (CodeGraph-Bench rows) | no node, or a misparsed node | |
P4: a quoted method name is not captured ("Program > ExpressionStatement"(node) {}) |
3 (1d) | no node at all | |
P3: an optional body-less class method is dropped as an overload signature (rewriteRootURL?(…): string;) |
1 (CodeGraph-Bench row) | no node at all | |
| VS Function 22 | P6: a CommonJS export is named exports.X / module.exports.X / module.exports |
17 (1e 14, 5/1e 3) | node with another name |
P5 (node_module._load = function load → load, exec: function noopTest, _proto.escape = function escape$1) |
4 | node with another name | |
| A1: the second same-name declaration on one line (scorer artifact) | 1 (1d) | node emitted, collapsed by the benchmark code | |
| VS Method 3 | A1 | 2 (1d) | node emitted, collapsed by the benchmark code |
P3 (protected renderActionBar?(…): void;) |
1 (CodeGraph-Bench row) | no node at all |
Total: 3,344 rows = 3,341 product misses (P1-P6) + 3 scorer artifacts (A1). 20 BabylonJS P1 rows also need a second fix (12 anonymous CommonJS exports need P6, 1 needs P5, 7 quoted methods need P4); all fixes shipped together.
- P1 in detail. All BabylonJS P1 rows are in three vendored files under
packages/tools/babylonServer/public/:gltf_validator.js(dart2js output in a browserify pack; 2,145 of its 2,432 names have 1-2 characters),cannon.js(a readable browserify pack) andbasis_encoder.js(readable emscripten glue). Their average line lengths are 27, 27 and 33 bytes, so GitHub's rules do not call them generated; Memtrace's own bundle detector did, through two signatures Linguist does not have (the browserify trailer and emscripten markers). - Verdict. Every P row is a real declaration that the answers correctly contain, so all are product misses. P5 and P6 name the binding by which callers reach the function, which is what rule 1d/1e and GitNexus name it.
- GitNexus on the same rows. It matches 1,480 of the 1,532 P1 Function rows and 1,752 of the 1,759 P1 Method rows (it indexes the three vendored files), misses the 23 P2 rows too (the same grammar) and the 3 P4 rows, and matches 13 of the 22 VS Function rows.
The scorer artifact (A1) and the fix. The unique-id fix gives two same-name, same-kind
declarations written on one line distinct ids ([{ dispose() { } }, { dispose() { } }] →
dispose@L106 and dispose@L106:58). CodeGraph-Bench's matcher collapses tool nodes that share
(path, canonicalName, kind, startLine), but answer rows only by (path, id), so the second row can
never be matched by any tool: emitting both declarations is charged a miss. Step 2
(applyIdentityDedupe) drops an answer row whose (normalised path, canonicalName, kind, startLine)
equals an earlier kept row's with a different id, and re-points every edge that names it. Unnamed,
[computed] and [anonymous] rows are not touched (the benchmark code never matches them). Size:
BabylonJS 0 rows; VS Code 3 rows; 0 edges re-pointed. Control: --identity none.
Rescore. Every tool graph scored at the time × 2 targets: --identity none reproduced the
scores from before step 2 for 10/10 pairs, and --exclusion none --identity none reproduced the
scores from before step 1 for 8/8. Effect: BabylonJS nothing; VS Code every tool Function misses −1
and Method misses −2 (TP and FP unchanged, because no tool could match those rows); F1 moved by at
most +0.0033 points.
We built CodeCensus after seeing where Memtrace lost on CodeGraph-Bench, and we fixed Memtrace against what CodeCensus showed. Weigh the CodeCensus results with that in mind; CodeGraph-Bench, which we did not change, is published beside it.
- P1: the bundle detector keeps only GitHub's own JavaScript checks (source map, minified). This
reverses an earlier product decision (readable browserify packs and emscripten glue were
indexed as File nodes only), for every repository, not only the benchmark targets. In BabylonJS it
brings back 3,842 nodes in the three vendored files, 1,483 Methods and 665 Functions of them with
1-2 character names (dart2js names such as
T,eC,$0). - P2: a TS/TSX file whose syntax tree has errors is re-parsed with the
overrideof eachabstract override [readonly] <field>blanked (byte offsets unchanged); the repaired tree is used only when it has fewer errors. - P3: a class method signature with no implementation of that name in the class body is a declared member.
- P4: quoted method names are captured, named by the text inside the quotes.
- P5, P6: a member-stored callable is named by its member (the inner name stays a private
alias);
exports.load = function () {}isload, an anonymousmodule.exports = function () {}isexports, and a named whole-module export keeps its written name. Cost: an exact lookup by the written inner name no longer finds 6 nodes in the two targets. - Later in the same round (section 11): the bundle rule was made exactly GitHub's
(
minified_files?thenhas_source_map?, behindmaybe_minified?; no size floor, no data-literal exemption,.mjs/.cjs/.jsxnever minified).
The cost on CodeGraph-Bench. CodeGraph-Bench has no rows for files no tsconfig loads, so every node P1 brings back counts as a false positive there. That is the main reason Memtrace's BabylonJS Function F1 on CodeGraph-Bench is 53.62, with 5,988 false positives. It is also why both benchmarks are shown side by side.
The product changes were tested with the product's own test suites (no new failures against the base) before the capture; those logs are internal and not published here.
Finding (review round after the fixes, major). The code that converts Memtrace's graph to the benchmark format collapsed a TypeScript class + interface declaration merge into one node, and only Memtrace's graph was collapsed. The GitNexus graphs hold the same 36 BabylonJS and 1 VS Code merge groups (Graphify none), and no scorer collapsed them.
What the answers hold. For each merge, CodeCensus's generator writes one row per declaration, all with the symbol's id, and the benchmark code collapses answer rows by (path, id), keeping the first. So the answers score ONE symbol per merge, at its first declaration; in all 37 groups that first row is the earliest declaration. Memtrace lists the earliest part first in every group; GitNexus lists the Class first, which is not the earliest in 34 of the 36 BabylonJS groups and in VS Code's one. "Keep the first part" is therefore not tool-neutral; "keep the earliest part" is, and equals what Memtrace's converter did.
Fix (applyDeclarationMerge). For every tool's graph, after step 1 and before computeScope:
top-level (parentId null) Class and Interface nodes of one name in one TS/JS file, with at least one
of each kind, are collapsed into the earliest part (smallest startLine, then input order); the other
parts are dropped, their children re-parented, their edges re-pointed; a new kept→kept edge is
dropped unless it is IMPLEMENTS; only the (from, to, type) collisions the re-pointing creates are
grouped. Control: --merge none. The Memtrace graph we scored is converted without any collapse of
its own, so the step is the only place a merge is counted once, and it is the same for every tool.
Rescore. Every tool's graph, both targets, each with the --merge none control. Every check
passed: each control equals the scores from before the step, cell for cell; the answer side is
identical for every tool; only the Class and Interface node kinds move (plus the edge cells listed);
no score cell is missing.
CodeCensus, --merge none → with the step |
merge groups (parts dropped) | Class TP/FP/FN F1 | Interface TP/FP/FN F1 | all symbols F1 | other cells that move |
|---|---|---|---|---|---|
| BJS Memtrace | 36 (Class 34, Interface 19) | 2680/35/0 99.35 → 2680/1/0 99.98 | 1952/29/0 99.26 → 1952/10/0 99.74 | 92.99 → 93.01 | IMPLEMENTS TP +32, FP −48, FN −32 (503/48/35 → 535/0/3); USES_TYPE TP +2, FP −2, FN −2 |
| BJS GitNexus | 36 (Class 34, Interface 2) | 2664/0/16 99.70 (unchanged) | 1945/41/7 98.78 → 1945/5/7 99.69 | 88.40 → 88.42 | CALLS FP −15, ACCESSES FP −17, METHOD_OVERRIDES FP −5 |
| BJS Graphify 0.9.27 / 0.9.92 | 0 | unchanged | unchanged | unchanged | none |
| VS Memtrace | 1 (Class 1) | 7493/7/0 99.95 → 7493/6/0 99.96 | 6559/24/0 (unchanged) | 88.12 | none |
| VS GitNexus | 1 (Class 1) | 6530/3/963 93.11 (unchanged) | 5942/370/617 92.33 → 5942/369/617 92.34 | 81.47 | none |
| VS Graphify 0.9.27 / 0.9.92 | 0 | unchanged | unchanged | unchanged | none |
BabylonJS Implements is the one row whose winner depends on this step: without it Memtrace scores
92.38 against Graphify 0.9.27's 98.68, and CodeCensus gives 17 of 18. No other row's winner changes
with --merge none (checked on the Memtrace and GitNexus controls; Graphify has no merges). The
controls are in results/controls/.
CodeGraph-Bench's scoring is not changed (it has no merge step). For comparison only, the same collapse applied to GitNexus on CodeGraph-Bench: BabylonJS Interface 1940/46/7 98.65 → 1940/10/7 99.56, all symbols 79.36 → 79.37 (CALLS FP −15, ACCESSES FP −17, METHOD_OVERRIDES FP −5); VS Code Interface FP −1, F1 91.51 both. On CodeGraph-Bench no tool is collapsed.
Other findings of that round. (2) Memtrace's bundle rule was not exactly GitHub's (a size floor, a data-literal exemption, extra extensions, CR handling, trailer parsing, check order); fixed in the product to GitHub's exact rule. Measured on every JS-family blob of both targets, the only files whose verdict changed are ones GitHub's rules call generated, so no CodeCensus score moves. (3) A missing product test was added. (4) The scorer's comment contradicted its behaviour; rewritten with this step's code change.
| tool | graph | conversion | scored |
|---|---|---|---|
| Memtrace | the build we tested, from 2026-10-18 (private branch, commit 82504401; it reports version 0.8.1), cold capture of each target |
our private conversion code (anonymous functions get a fixed placeholder name; no merge collapse) | 2026-10-18 |
| GitNexus | 1.6.11 (released 2026-09-04), exported read-only from its database | our export and conversion code; kinds mapped by CodeGraph-Bench's normalizeGitNexusKind / normalizeGitNexusEdgeType; resolved null for every edge (no verified native resolution field) |
CodeGraph-Bench 2026-10-16, CodeCensus 2026-10-18 |
| Graphify | graphifyy 0.9.27 and 0.9.92, graphify update --force, no network, no LLM options |
CodeGraph-Bench's own Graphify import code, unchanged (graphify/) |
CodeGraph-Bench 2026-10-17, CodeCensus 2026-10-18 |
The CodeCensus scores of all tools come from one batch with the final scorer (2026-10-18), and
Memtrace's CodeGraph-Bench scores from the same scorer. Every published score was recomputed on
2026-10-18 by scripts/score_all.sh with the public scorer from the published inputs (GitNexus from
the private copies of its graphs) and equals the recorded run cell for cell
(results/<benchmark>/<run>/recorded/).
bash scripts/setup.sh # download + verify CodeGraph-Bench, restore identifiers, build CodeCensus's generator, install TS 6.0.3
python3 -I scripts/fetch_data.py # the answers and graphs (release assets, sha256-checked) into data/
bash scripts/score_all.sh results-local # every tool x target x benchmark, plus the --merge none controls (GitNexus: set GITNEXUS_GRAPHS)
python3 -I scripts/verify_results.py results results-local # equal to the recorded scores, cell for cell
python3 -I scripts/make_results_table.py results-local # the README tables
# the fixture, the Linguist port, and (Linux, optional) the answers themselves
bun fixture/run_fixture.ts original fx/original && bun fixture/run_fixture.ts extended fx/extended && python3 -I fixture/check_fixture.py fx/original fx/extended
python3 -I linguist/check_port.py && python3 -I linguist/fetch_linguist.py && python3 -I linguist/test_port.py linguist/.cache/linguist-73ee9a3b linguist
bash scripts/fetch_targets.sh targets-src
bash generator/generate.sh original babylonjs targets-src/babylonjs && bash generator/generate.sh extended babylonjs targets-src/babylonjs
python3 -I generator/check_additions.py babylonjs
bash linguist/classify.sh targets-src/babylonjs targets-src/vscode lists-rebuiltTypeScript names a module symbol after its absolute file name, and CodeGraph-Bench's generator
records that name as the target id of namespace-import rows. CodeGraph-Bench's answers were
generated on Windows, so the absolute Windows path of the folder they were generated in appears in
114 BabylonJS and 1,542 VS Code edge rows of both benchmarks (CodeCensus keeps every CodeGraph-Bench
row). The summary file of each answers asset records it as repoPath, and its outDir field
records the output directory of the generation run. Symbol files and tool graphs carry no such
path. A byte-identical reproduction needs the same spelling (section 2), and every recorded hash and
score was computed on these bytes, so v1 publishes the files verbatim. A regeneration under a
neutral path would change those rows and every sha256 of the answers.
The Graphify graphs carry a similar artifact of their own: Graphify derives the id of some nodes from
the absolute path of the directory it indexed, so 2 nodes of the 0.9.27 VS Code graph (referenced by
430 edges) and 1 node of the 0.9.92 VS Code graph (1 edge) have ids built from the path of the run
machine's copy of the target. They are Graphify's output, were scored as they are, and are published
verbatim (release/MANIFEST.json, release/DISCLOSURES.json).