CodeCensus

Every symbol counted. CodeCensus is built on Arpit Mishra's CodeGraph-Bench: same repos, same commits, same scorer, and the same judge, the TypeScript compiler. It counts every function, method, class, call and import the compiler sees, at any depth and in every file, and skips machine-generated files (minified bundles, compiled output) using GitHub's own published rules for spotting generated code.

We reproduced CodeGraph-Bench byte for byte and scored three code-graph tools on it and on CodeCensus: Memtrace, GitNexus and Graphify. Graphify runs in two releases, 0.9.27 (the version CodeGraph-Bench is set up for) and the newer 0.9.92. Memtrace has the best F1 on 13 of 18 scores (9 metrics × 2 repos) on CodeGraph-Bench, and on 18 of 18 on CodeCensus.¹ One of the five scores that change hands, BabylonJS IMPLEMENTS, depends on a scoring step we added after seeing scores: when a class and an interface share a name, every tool's graph counts the pair once. Without that step CodeCensus gives 17 of 18.² Before release we reran the public scoring script on the published inputs. Every number below matches the recorded runs exactly (GitNexus from private copies of its graphs).³

What it measures

• Graph accuracy against the compiler. Precision, recall and F1 of a tool's symbols and relationships (calls, imports, extends, implements) against the answers: the symbols and edges the TypeScript 6.0.3 compiler reports, which each tool's graph is scored against. Targets are BabylonJS 4efc0490 and VS Code 0102a4db, with no dependencies installed.
• CodeGraph-Bench's own scorers. Matching (path, name, kind, start line ±2), edge adjudication and the depth and fidelity scorers are CodeGraph-Bench's, imported at runtime and unchanged.
• Not agent success. These are graph-accuracy scores, not coding-agent solve rates.

Two benchmarks, always shown together

	CodeGraph-Bench	CodeCensus
What it is	Arpit Mishra's benchmark at 50b87e21: the compiler's answers from its own generator, unchanged	CodeGraph-Bench's answers plus five rules we fixed before generating anything4
Reproduced	byte for byte, both repos (sha256 below)5	deterministic: two runs byte-identical
BabylonJS rows (symbols / edges)	69,161 / 410,264	92,406 / 544,975; 78,325 / 474,242 after the scoring steps
VS Code rows (symbols / edges)	134,488 / 811,310	151,229 / 897,142; 149,901 / 890,712 after the scoring steps
Counts	top-level declarations and their members, in the files the TypeScript project programs load	also nested functions, object-literal members, assigned functions, nested classes and types, compiler-resolved module imports, and TS/JS files no tsconfig loads
Scoring steps	none	the same for every tool: skip machine-generated files (GitHub's rules) on both sides; apply CodeGraph-Bench's own same-symbol rule to the answers too; count a merged TypeScript class + interface once6

Every CodeGraph-Bench symbol row is byte-identical in CodeCensus. 9,394 call rows are credited to the narrower container that rule 2 names and 942 point to a renamed target; every one is tagged and disclosed (METHOD.md §3).

Results

F1 %, scored against every answer row, bold = best. Precision, recall and TP/FP/FN for every cell: results/results.json and each run's SCORES.json. Memtrace is the build we tested, from 2026-10-18.

CodeGraph-Bench

BabylonJS	Memtrace	GitNexus 1.6.11	Graphify 0.9.27	Graphify 0.9.92
All symbols	86.81	79.36	64.73	66.91
Function	53.62	46.16	69.49	59.03
Method	96.13	91.17	90.54	91.31
CALLS	85.44	58.71	45.20	57.70
IMPORTS (symbol)	99.15	n/a7	92.18	87.68
IMPORTS (module)	91.35	71.44	90.66	89.72
EXTENDS	99.84	87.35	98.42	99.01
IMPLEMENTS	92.38	73.93	98.68	95.26
All edges	90.91	48.58	67.58	70.81

VS Code	Memtrace	GitNexus 1.6.11	Graphify 0.9.27	Graphify 0.9.92
All symbols	82.80	77.48	71.43	76.37
Function	63.65	61.22	81.93	79.35
Method	95.49	90.86	97.60	98.27
CALLS	82.61	45.78	51.11	67.36
IMPORTS (symbol)	98.43	n/a7	83.71	83.51
IMPORTS (module)	99.24	94.37	98.63	98.60
EXTENDS	95.67	75.28	97.88	97.90
IMPLEMENTS	98.58	75.56	93.49	94.02
All edges	88.66	39.10	67.18	75.20

CodeCensus

BabylonJS	Memtrace	GitNexus 1.6.11	Graphify 0.9.27	Graphify 0.9.92
All symbols	93.01	88.42	65.24	68.39
Function	100.00	89.32	65.06	65.94
Method	99.99	98.86	91.92	92.67
CALLS	82.61	61.79	42.16	58.71
IMPORTS (symbol)	99.42	n/a7	92.40	87.97
IMPORTS (module)	99.44	82.99	99.06	98.40
EXTENDS	99.87	87.39	98.46	99.04
IMPLEMENTS	99.72	73.93	98.68	95.26
All edges	88.90	51.39	64.72	71.39

VS Code	Memtrace	GitNexus 1.6.11	Graphify 0.9.27	Graphify 0.9.92
All symbols	88.12	81.47	69.35	74.52
Function	98.88	89.12	66.22	69.54
Method	99.92	94.80	95.39	96.09
CALLS	86.49	49.21	48.87	65.18
IMPORTS (symbol)	98.43	n/a7	83.71	83.51
IMPORTS (module)	99.52	94.64	98.85	98.83
EXTENDS	99.61	73.05	95.63	95.65
IMPLEMENTS	99.24	75.80	93.41	93.93
All edges	90.98	42.08	65.32	73.65

Where Memtrace does not win on CodeGraph-Bench

Four of the five come from real declarations or edges CodeGraph-Bench has no row for. The fifth, BabylonJS IMPLEMENTS, is a counting mismatch: the rows exist, and CodeCensus's merge step resolves it for every tool. The same Memtrace graphs win all five on CodeCensus.

Row	Memtrace	Best	Why	CodeCensus
BabylonJS Function	53.62	69.49 Graphify 0.9.27	recall 100%, but 5,988 Function nodes have no row: nested, object-literal and assigned functions, and functions in files no tsconfig loads (three readable vendored libraries among them)8	100.00
VS Code Function	63.65	81.93 Graphify 0.9.27	recall 99.94%; 7,819 Function nodes with no row	98.88
VS Code Method	95.49	98.27 Graphify 0.9.92	4,305 methods with no row (object literals, nested classes, unloaded files); 76 remain on CodeCensus	99.92
VS Code EXTENDS	95.67	97.90 Graphify 0.9.92	396 heritage edges with no row (nested classes, class expressions, unloaded files); 2 remain	99.61
BabylonJS IMPLEMENTS	92.38	98.68 Graphify 0.9.27	TypeScript class + interface merges: CodeGraph-Bench counts one symbol per merge, Memtrace's graph one node per part2	99.72

Why it isn't tilted toward Memtrace

• Same inputs for everyone. Same commits, same answers, same matcher, same scorer, same denominator (every answer row). The answer-side cut of each CodeCensus step is checked to be identical across tools.
• CodeGraph-Bench's scorers, with one repair. The scorers, matcher and adjudicator are CodeGraph-Bench's, downloaded and hash-checked at run time. The repair: two names (EXTENDS, EXTENDED) that a cleanup of its git history broke are put back. The generator changes are separate: CodeCensus's generator is built from CodeGraph-Bench's with 31 anchored edits (METHOD.md §3). CodeGraph-Bench's own test suite passes on the repaired copy: 106/106 (bun test in the downloaded copy under upstream/ after scripts/setup.sh; rechecked 2026-10-18 with bun 1.4.0).
• Scoring steps apply to every tool, or to none. Each CodeCensus step has an off switch (--exclusion none, --identity none, --merge none) that reproduced the scores from before the step, cell for cell, when it was added (METHOD.md); the --merge none controls are in results/controls/.
• An independent standard for "generated". GitHub's rules (generated? in GitHub Linguist), ported to Python and checked against GitHub's own tests (189/189); no rule of ours added, vendor.yml not used.
• Tools as shipped, no LLM. Graphify ran with no network (graphify update has no LLM options); GitNexus's receipts record zero model calls. Graphify's graphs are converted by CodeGraph-Bench's own Graphify import code, unchanged; GitNexus's node and edge kinds are mapped by CodeGraph-Bench's own mapping.
• Code pins. The scorer refuses changed CodeGraph-Bench sources, answer files or generated-file lists.

Reproduce

git clone https://github.com/syncable-dev/codecensus && cd codecensus
bash scripts/setup.sh                   # download CodeGraph-Bench @50b87e21, verify 26 files, restore, build CodeCensus's generator, TypeScript 6.0.3
python3 -I scripts/fetch_data.py        # 111 MB of answers and graphs, sha256-checked
bash scripts/score_all.sh results-local # ~3 min; GitNexus needs GITNEXUS_GRAPHS (see Limitations)
python3 -I scripts/verify_results.py results results-local   # every cell equals the published one

Needs bun ≥ 1.2.21 and Python ≥ 3.12, or docker build -t codecensus . (digest-pinned). The answers themselves, the generated-file lists, the fixture and the Graphify graphs can be rebuilt too: METHOD.md §13.

How CodeCensus's generator is built without copying CodeGraph-Bench. CodeGraph-Bench's code has no license, so this repository ships none of its files. scripts/setup.sh downloads it, and scripts/apply_edits.py applies upstream/codecensus-generator.edits.json: 31 edits anchored by line number and line hash, holding only our own code. The result must be byte-identical to the generator that produced CodeCensus's answers (sha256 3712aa99…) or nothing is written.

What's here

scripts/     setup, downloads (CodeGraph-Bench, data, targets), apply_edits, score_all, verify_results, tables, release build
scoring/     compute_scope.ts (the one scoring core), extended_scope.ts (the three CodeCensus steps), score.ts
upstream/    UPSTREAM.lock.json (26 file hashes), codecensus-generator.edits.json, TypeScript file list
generator/   generate.sh, run_generator.ts, NTFS-order tree, check_additions.py, compare_outputs.sh
linguist/    the port of GitHub's generated-code rules (MIT notice), classifier, test replay, lists/
graphify/    Graphify run + conversion scripts and pins
targets/     per-file manifests and walk orders of both target trees
fixture/     a small repo that exercises every rule, and the expected answers of both generators
results/     original/ (CodeGraph-Bench), extended/ (CodeCensus), controls/: SCORES, SUMMARY, SCORE-RECEIPT per tool x repo, plus recorded/
release/     MANIFEST.json: release assets with sha256 (the answers, Memtrace and Graphify graphs)

File and folder names use original for CodeGraph-Bench and extended for CodeCensus. The Memtrace run folders and data assets carry the internal name of the build we tested; those names are part of the published hashes, so they stay.

Limitations

• TypeScript only, two repositories. A Python version is in progress and not part of v1.
• We built CodeCensus after seeing where Memtrace lost on CodeGraph-Bench. The five rules were fixed in advance. Later fixes to the generator (unique ids, quoted names, JS siblings) and the three scoring steps came from our own review rounds. Each applies to every tool and has a control (METHOD.md §5-§11).
• We also fixed Memtrace against what CodeCensus showed (six fixes, listed in METHOD.md §10). So CodeCensus is not a held-out test for Memtrace. Neither is CodeGraph-Bench: we fixed Memtrace bugs it showed us too. One fix reverses an earlier product decision and costs Memtrace on CodeGraph-Bench, which is why both benchmarks are shown side by side.
• The Memtrace build we tested, from 2026-10-18, is unreleased (it reports version 0.8.1). Its graphs are published; the code that captures and converts them is private, so its graphs cannot be rebuilt from this repository.
• GitNexus graphs are not redistributed (PolyForm Noncommercial); only its scores are. The code we wrote to export its graphs is not cleaned up for release yet, so it is not in v1, and its cells can be checked only by rebuilding them with GitNexus 1.6.11 and an equivalent export. 1.6.11 (source 780cac78) is the same GitNexus build our Deep-SWE agent runs use. GitNexus 1.6.12 has been out since 2026-09-12 and was not rerun.
• Vendored but non-generated bundles stay in CodeCensus (gltf_validator.js, cannon.js, basis_encoder.js in BabylonJS); Linguist's vendor.yml is not applied.
• Two interpretation calls (unique ids for same-name declarations; JS importers load the JS sibling) are judgment calls, isolated in the generator (METHOD.md §3).
• A path inside the answer files. 114 BabylonJS and 1,542 VS Code edge rows carry the absolute Windows path of the folder CodeGraph-Bench's answers were generated in, because TypeScript names module symbols after absolute file names. Graphify likewise derives a few VS Code node ids from the directory it indexed (2 nodes in 0.9.27, 1 in 0.9.92). v1 publishes all files verbatim so every recorded hash matches (METHOD.md §14, release/MANIFEST.json).

Credits and licenses

Built on CodeGraph-Bench by Arpit Mishra (Vyazen): github.com/vyazen/CodeGraph-Bench @ 50b87e21, described in his write-up. CodeGraph-Bench has no license, so it is downloaded and verified, never vendored. CodeCensus is not affiliated with or endorsed by Arpit Mishra or Vyazen, and it is not CodeGraph-Bench.

Code: Apache-2.0. Data and results: CC BY 4.0. Third-party terms (GitHub Linguist MIT, TypeScript, Graphify, BabylonJS, VS Code, GitNexus): THIRD_PARTY_NOTICES.md.

Footnotes

1. Nine metrics (all symbols, Function, Method, CALLS, IMPORTS symbol and module level, EXTENDS, IMPLEMENTS, all edges) × two repositories = 18 scores; Memtrace has the best F1 on a score unless another tool's F1 is strictly higher (no score is tied). Computed by scripts/make_results_table.py from results/*/*/SUMMARY.json. ↩

2. On CodeCensus with --merge none (results/controls/) the cell is exactly the CodeGraph-Bench cell, TP 503 / FP 48 / FN 35; counting each merge once, for every tool, gives 535 / 0 / 3. ↩ ↩²

3. scripts/score_all.sh with the public scoring script, on the published answer files and graphs (GitNexus: private copies of its graphs), reproduced every recorded run cell for cell: 18 runs, 596-620 numeric cells each (scripts/verify_results.py). The recorded runs are kept under results/*/*/recorded/, with machine paths replaced by placeholders and every number untouched. ↩

4. 1. every named declaration at any depth; 2) each call credited to its innermost named container; 3) heritage for the new classes and interfaces; 4) one module-level IMPORTS row per dependency the compiler resolves; 5) TS/JS files no tsconfig program loads, in one extra program. METHOD.md §1.
   ↩

5. BabylonJS d18c6dc7… / 6b8cffec…, VS Code 0fbdaa08… / cb451d7b… (symbols / edges). A byte-identical rerun needed the original NTFS file order and Windows path spelling (METHOD.md §2). ↩

6. GitHub's rules (Linguist 73ee9a3b) call 27 BabylonJS and 76 VS Code files generated (minified bundles, committed tsc output). The same-symbol step removes 3 VS Code answer rows that no tool could ever match; the merge step collapses 36 BabylonJS and 1 VS Code class + interface merges in Memtrace's and GitNexus's graphs (Graphify has none). METHOD.md §8, §9, §11. ↩

7. GitNexus records file-to-file imports only, so it has no symbol-level IMPORTS row to score. Its module-level IMPORTS are scored. ↩ ↩² ↩³ ↩⁴

8. On CodeCensus the same graph's Function precision is 100.00% (BabylonJS) and 97.78% (VS Code). ↩