Repository navigation
fix(extract): receiver-qualify Go method QNs - #1913
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
c70025f to
12d50c0
Compare
|
Rebased onto current main (stacked on the rebased #1907). No semantic changes; full test leg green. Post-fix field numbers for this change are in the #1906 census comment (method-node recovery incl. generic receivers: #1906 (comment)). |
|
Reviewed the last commit. The bug is real and badly worth fixing, the shape of the fix is right, and there is one measurement I want before it lands. The problem is as bad as you say. Receiver-qualifying is the right cure, matching the Go interface-member and C++ out-of-line paths already in The measurement I wantYou describe this as a side effect, in one line:
I think that line is doing more work than it looks. So You measured the false edges removed. I would like the other half measured too, and there is already an instrument for it — the per-language CALLS census with its dumped per-edge sets. The Go leg of that is the right baseline. Concretely, what I am asking for is:
If the answer is "the LSP type-dispatch path already covers them, so the demotions are few and the removals are many", that settles it and the PR merges on the strength of the numbers rather than on the argument. If a meaningful number of true edges demote to import-distance guessing, that is worth knowing before this is in The Two smaller thingsThis shifts a baseline other PRs are being judged against. There is an open cluster of call-resolution PRs that we have decided to judge against one shared census precisely because whichever lands first makes the rest unreadable on their own numbers. A change to Go method QNs moves that baseline for the Go leg. Not an objection — just something I would rather sequence deliberately than discover afterwards. Your reindex note is right and needs to travel further than the PR. A QN-shape change means existing graphs pick up the new shape only on rebuild, and that belongs in the release notes, not only in #1909. I will make sure it gets there. Status
Thanks for the measurements you did bring — "20 methods → 2 nodes, 19 structs linked to one shared node" is what turns this from a plausible tidy-up into an obvious defect. |
12d50c0 to
a6b1a69
Compare
|
Rebased onto current main as part of the 1907→1913→1915→1936 stack; no source conflicts, test anchors re-anchored. Full scripts/test.sh green on the stack head. |
a6b1a69 to
dbba757
Compare
|
Here is the measurement — and it caught something real, so thank you for insisting on it. The census (orca-runtime-sensor, Go→Go CALLS, isolated one-shot indexes)
Per-edge migration of #1907's 929 What the first run of this census caughtMy first measurement showed Amended into this PR (it is the same one-formula claim): One upstream test adopted the new shape: On the baseline-shift point: agreed — these numbers are from the stack in its merge order (#1907 → #1913 → #1915 → #1936), so whichever leg lands first, the others' columns above stay the readable reference. Full scripts/test.sh green on the amended stack head; branch force-pushed with the rebase. |
|
Census received, and thank you for insisting on measuring rather than arguing it — it caught exactly the thing I was worried about. Where this stands: the result changes the shape of Go method qualified names ( Please hold any further rebase of the stack (#1907 → #1913 → #1915 → #1936) until that call is made. |
|
Thank you for holding the rebase while we measured this ourselves — here is the result, and the decision. We ran our own census on kubernetes/kubernetes @
Decision: the direction is accepted. Please rebase #1907 first, then #1913 onto main; #1915 and #1936 follow once those two are in. We'll take each through the normal gates as it goes green. Two things from the measurement, so you have the full picture:
The full dumps stay on our side if you want any of the sampled edges with file:line. |
Review follow-up for DeusData#1913 (asks from the 2026-09-28 review), plus one finding from the review of the updated head: - store.h: CBM_INDEX_FORMAT_VERSION 3 -> 4. Every Go method QN changes (pkg.method -> pkg.Recv.method), so an index built before this change and refreshed incrementally would mix flat and receiver-qualified method nodes, with the old fused nodes keeping their stale edges. The bump routes such an index through one full rebuild. - pipeline_go_method_caller_keeps_lsp_join: the by-name assertion would also pass for a binding onto the decoy other.Decoy.helper. The test now pins the edge by QN suffix on both ends (.svc.Repo.Install -> .svc.Repo.helper) with an lsp_* strategy, and asserts no edge to the decoy. - extract_defs.c: the receiver-qualified def QN is Go-only. The templ grammar is Go-derived and reaches the same receiver branch, but its call-scope QN (compute_func_qn) stays flat, so a templ method would get a def QN no call names and its calls would degrade to the File node. extract_templ_method_qn_matches_call_scope pins def QN == call scope. - pass_semantic.c: the check_go_class_implements comment no longer describes Go method QNs as flat. Refs DeusData#1909 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_fqn_compute_source_lang returns NULL only when the module QN cannot be built (arena exhaustion). The receiver-qualified branches then passed that NULL to "%s". Both sides now fall back to the flat QN together, so the def and its calls still agree. Allocation-failure path only; no change on any normal run. (Raised in our security review of the updated head.) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel resolver re-picks a multi-candidate registry match through try_field_type_hint. That helper capitalizes the receiver VARIABLE name and accepts any candidate whose QN contains it as a substring: `t.Run` on a *testing.T became `T` and bound a project `TBench.Run`. The sequential resolver has no such step. The Go selector guard treated field_type_hint as receiver-aware and kept it, so the same weak pick it drops in sequential mode survived in parallel mode under a new name. On caddy + prometheus, parallel and sequential CALLS diverged by 1,684 (this branch) and 5,560 (with receiver-qualified method QNs, which put type names into more QNs). At base the gap is 84. SCIP precision of Go field_type_hint is 0.016 at base. Go no longer takes the hint. The guard keeps field_type_hint on the shared weak list, and the fixture of both twins now pins the `t.Run` case. A typed field call (lrp_go_s8, `s.repo.Find()`) resolves through the Go LSP and is unaffected. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData#1913 makes Go method nodes receiver-qualified (<dir>.<Recv>.<Method>). The Go doc-link model still let every method claim the flat node <dir>.<name>: - doclink_go.c go_scan_method pushed a claim record; - doc_links_go.c go_add_claim put it in the flat table. So a type, function or value that shares its name with a method in its directory became a graph_gap ("lost its node to the method"), although the graph now gives it that node. Base reports the same graph_gap for this shape, because there the method really did take the node. So this completes DeusData#1913's QN scheme for one of its consumers; it is not a regression DeusData#1913 causes. - A method record now carries w `-`: it claims no flat name. - The resolver no longer enters methods into the flat table. Methods are found through their receiver (the member table), which was already the first lookup. - The flat fallback in go_method_node is now unreachable. It stays, with a comment saying so. Corpus, before = this branch's tree, par x2, r1 == r2: - terraform: graph_gap rows 9 -> 0, doc-link MENTIONS 528 -> 536 (+8, -0). All 8 new edges were judged correct: e.g. [AbsProviderConfig] now lands on the type, not on the method RootProviderConfig.AbsProviderConfig. - vault, prometheus: unchanged. - prometheus incremental == full for MENTIONS. - The 3 -> 4 index-format bump already forces a full rebuild, so no stored scope blob mixes the two meanings. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
) main's Go doc-link tests were written against flat Go method QNs, where `(A) Close` and `(B) Close` shared one node `alpha.Close`. That is the fused-node defect DeusData#1913 removes. The expectations move to the new scheme, and each changed assertion pins an edge onto, or a count of, the receiver-qualified node (production: the previous commit and DeusData#1913): - sources: RefM's source is p.pkg.S.M, not p.pkg.M. - forms: [Widget.Run] lands on alpha.Widget.Run, [beta.Thing.Do] on beta.Thing.Do. - plain_text, struct_fields: Run's doc is the source alpha.Widget.Run. - lookup_order: two alpha.Run sources and beta.Promoted named no node any more, so they held vacuously. They are now alpha.Widget.Run and beta.Embedded.Promoted. The alias links land on beta.Thing.Do and alpha.Deep10.End. - node_owner, the positive contract that replaces the fused case: - two Method nodes, alpha.A.Close and alpha.B.Close, and none named alpha.Close; - [A.Close] and [B.Close] are edges, not graph_gap rows; - each method's own [Target] is an edge from its own doc line (9 and 12), with no unresolved row; - the type Open keeps alpha.Open: [Open] is the type (no graph_gap), [B.Open] is the method, and UsesOpen has 2 targets. - scope_blob: - method records carry w `-`; - swapping two methods of one name no longer changes the blob; - a func/type pair swap still does (new case); - the k1-k3 records follow (a method neither keeps nor contests a name); - a new k4 keeps the "two on one line contest it" rule covered with a var and a const. - incremental_equals_full: - [b.T.M], [b.W.Name] and [b.V.Name] all resolve, and U.M's [Target] is an edge; - removing T.M breaks only [b.T.M]; - swapping V.Name and W.Name moves nothing and is now an ordinary closure repair instead of a forced full rebuild. - scope_delta_rules: - a method/function swap is LOCAL; - removing the method of such a pair is LOCAL with the name "Open" (a member left); - new func/type cases keep the GLOBAL keeper-change rules covered for swap and removal. - workers_agree: [p01.T3.Close] now resolves (exact edge, no row). The graph_gap reason stays covered by the named Shared assertion: one function declared by every file of a package, which is still a gap. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you, @ilyabrykau-orca. Receiver-qualified Go method QNs fix the fused-method bug at its root, and your census on this thread is what let us take the direction decision on numbers. As on #1907, we finished the review asks on your branch ourselves so you do not need another rebase. Your commit is unchanged; ours sit on top, signed off:
Your change is what turns the Go LSP's type dispatch into graph edges. On the 13 measured repos, the calls that scip-go confirms as correct How we measured the stackEach of these was indexed on 13 Go repos with the parallel resolver, and on caddy + prometheus with the sequential one as well:
Every repo was indexed twice per binary, and both runs were identical for every edge type. Precision and recall come from scip-go 0.2.7 as an independent oracle. Precision is counted only on the 6 repos where scip-go also records external references (on the other 7 it records none, so a call bound to the wrong project function would look correct there). Recall covers all 13.
Per repo,
The honest part is the recall trade. Overall recall goes from 0.925 to 0.873. The largest drops are vault (0.906 -> 0.792, −11.4 points) and terraform (−7.4 points). Most of what the guard removes was wrong, but not all of it. The true calls it loses are method calls on values whose receiver type the Go LSP does not infer: field chains ( One caveat on reading those numbers: #1913 changes what the oracle can match. Receiver-qualified QNs let scip-go map 20,912 more caller/callee pairs (237,806 vs 216,894), so recall only compares within one QN scheme. Winning those calls back is on us, not on you. Receiver typing in the Go LSP (field chains, method-return chains, One more change: the parallel resolver's field-type hint for GoRunning both resolvers side by side showed that the stack made them disagree. Here is CALLS from the parallel / sequential resolver:
The cause was ours, not yours. In the review asks we kept Receiver-qualified QNs (#1913) put type names into many more QNs, which made the hint fire far more often. The typed field call in The fix (
Both pipeline twins now assert that
No Go Thank you again for the care in this stack, and especially for chasing down the third QN site on your own. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> # Conflicts: # tests/test_registry.c
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two PDF doc-link tests named a Go method by its flat QN. With DeusData#1913 a Go method's QN carries its receiver, so the lookups found no node. The resolver is unchanged: for a Go method, rel(parent_class).name now equals the QN, so the method binds through its QN, like a member of any other language. - pdf_links_pipeline: page 1 names `server.Handler.Serve`; the edge target pkg.server.Serve becomes pkg.server.Handler.Serve. Same assertions: pdf_qn syntax, unique tier. - pdf_qn_method_through_owner, page 2: `time.Provider.Now` binds pkg.time.Provider.Now (was pkg.time.Now), syntax pdf_qn. - pdf_qn_method_through_owner, page 1: `time.Now` must not bind the method. The target moves to pkg.time.Provider.Now too. Against the old flat QN, no node exists any more, so the zero count would have passed vacuously. The page 2 assertion proves that the node exists. The ownerless-QN rule in doc_links_pdf.c stays: a templ method keeps the flat package QN while it carries parent_class (extract_defs.c only receiver-qualifies CBM_LANG_GO). Its comment no longer says that Go methods have no receiver. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in main 658639e (DeusData#2685, Rust generic-impl call scope). One conflict in tests/test_extraction.c: main's new F-R4 test and its helper sit right before the flag-exempt test, whose comment this branch rewrote (Go now sets is_method, so the guarded language is Rust). Both are kept: main's block, then this branch's comment. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in the base branch with main 658639e merged (DeusData#2685, Rust generic-impl call scope), so this branch merges cleanly into main again. No conflicts. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
CI caught two PDF doc-link tests ( |
Conflict in tests/test_pipeline.c: this branch and DeusData#2703 (on main) each registered two Go tests at the same place in the suite list; both pairs are kept. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…nto fix/go-method-receiver-qn Brings main 32445b2 in through the refreshed DeusData#1907. No conflicts. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you very much, @ilyabrykau-orca, for this stack, the clear issue write-ups behind it, and your patience while main kept moving underneath it. It is merged as 5a4af9c, which lands #1907 together with this PR. Before merging, we rebuilt both sides from clean and used SCIP-Go as the oracle. The rebuild reproduced your published figures:
A small recall cost for that much precision is the trade we want: fabricated selector edges and collapsed method nodes were misleading every Go call-graph query. Right before the merge, a clean local merge of current main with this branch also passed the 18 affected suites (1,830 tests). For users: this PR bumps the index format from 3 to 4, so existing indexes are rebuilt once after upgrading. That will be in the release notes. |
Brings the branch up to date with main (5a4af9c, the Go method receiver QN stack DeusData#1907/DeusData#1913). No conflicts; the Go changes and this branch's Elixir changes touch the same files in disjoint, language-gated regions. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings DeusData#1915 onto main 83109dd, after Stack A (DeusData#1907 and DeusData#1913) landed on main as 5a4af9c. This branch carried pre-rebase copies of the DeusData#1907 and DeusData#1913 commits (651ae3c, dbba757). Main holds their final versions (50ee111, da76194) plus the maintainer follow-ups made before Stack A merged, so every DeusData#1907/DeusData#1913 hunk resolves to main's side. The merged tree is main plus exactly this PR's own commit (adff62e): - internal/cbm/extract_defs.c, extract_unified.c, helpers.c, helpers.h: the init() hunks apply to main unchanged. - tests/test_extraction.c, tests/test_pipeline.c: main added newer Go tests at the anchors this PR used, so the two new tests and their RUN_TEST lines sit next to them. The test bodies are byte-identical. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`CBMRegisteredType.alias_of` is followed by the method lookup (go_lookup_field_or_method_depth), but only the per-file AST scan ever set it, and that scan only sees the file it walks. The def-driven registration that builds the cross-file registries had no alias target to copy, because neither CBMDefinition nor CBMLSPDef carried one. So a `type FeatureSet = bitmap` in one file was invisible to `reqs.Set()` in another. Since receiver-qualified method QNs (#1913), the call no longer resolves by name coincidence either, and it got no CALLS edge at all. The Go extractor now records the aliased type text of a `type A = B` (CBMDefinition.alias_target). pass_lsp_cross copies it into CBMLSPDef, and both cross-file registrars (cbm_run_go_lsp_cross and cbm_go_build_cross_registry) qualify it against the def's module and set alias_of. Only a bare identifier target is qualified, which is a type in the same package. A qualified, pointer, composite or generic target is left unset rather than guessed. A defined type `type A B` is a type_spec, so it never gets an alias target and keeps its own method set. The persisted LSP surface carries the new field ("al"), so SURFACE_CODEC_VERSION goes from 3 to 4: existing caches re-extract once, and an incremental run sees the same alias as a full one. The literal version-3 surfaces in the codec tests move to 4. Tests: golsp_crossfile_type_alias_method (+_prebuilt) check both registries, with a same-named decoy and a defined-type negative. pipeline_go_crossfile_type_alias_method (+_parallel) index the three-file package from the issue. Before the fix, Use had no CALLS edge to Set. Fixes #2033 Signed-off-by: Gaurav Dubey <gauravdubey0107@gmail.com>
What does this PR do?
Fixes #1909. Stacked on #1907 (shares the Go test scaffolding in
tests/test_extraction.c; the first commit here is #1907's — review only the last commit).Go concrete methods carried a flat package QN (
proj.pkg.method, receiver ignored), so every same-name method in a package collided on one QN and the graph upsert kept exactly one node — on the repo measured in #1909: 20Process/Namemethods → 2 nodes, 9Task()methods fused into one chimera node carrying all nine bodies' call edges, 19 structsDEFINES_METHOD-linked to a single shared node, and a_test.gomock outranking the production method in the dedupe tie-break. The upsert's own comment calls kind-disambiguated QNs "the real cure".The change — same shape as Go interface members and the C++ out-of-line method path directly below it in
extract_func_def:internal/cbm/extract_defs.c— when the receiver type resolves,def.qualified_name = parent_class + "." + name(proj.pkg.Storage.Close);go_receiver_type_namebecomes the sharedcbm_go_receiver_type_name(declared inhelpers.h, following thecbm_cpp_out_of_line_parent_classprecedent) so both sides of the contract use one formula.internal/cbm/extract_unified.c(compute_func_qn) — mirror branch for Gomethod_declaration, so method-body calls keep exact source attribution instead of degrading to thecalls_find_sourceFile-node fallback.pxc_build_lsp_defpasses the def QN andparent_class(→receiver_type) verbatim into the Go LSP registries, solsp_type_dispatch/lsp_embed_dispatch/ interface-satisfaction emissions follow the new QN automatically (alllrp_go_s*probes stay GREEN, untouched).check_go_class_implementsexplicitly supports class-qualified method QNs (its path (b) reconstructs<ClassQN>.<method>; its comment describes the flat QN as the anomaly).Side effect (also #1909):
resolve_same_module's exactmodule.namehash no longer matches concrete methods, which removes the confidence-0.9 false edges where an interface-typed call (pipeline.Process(event)) bound to an unrelated same-package method.Tests:
extract_go_method_receiver_qualified_qn— two same-name methods on different receivers get distinct, receiver-qualified QNs with matchingparent_class; free functions keep the flat QN.extract_go_no_filename_in_module_qnupdated: the method expectation becomesproj.myapp.db.Conn.Query(its actual contract — no filename segment in the QN — still asserted).scripts/test.shleg green (ASan+UBSan, "All tests passed"), including alllsp_resolution_probeGo scenarios and the fix(pipeline): suppress weak short-name matches for Go selector calls #1907 pipeline test, with zero probe changes.Note for maintainers: this changes extracted QNs for Go methods, so existing graphs need a reindex to pick up the new shape (flagged in #1909 per CONTRIBUTING's indexing-change rule).
Checklist
git commit -s) — required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
scripts/test.sh— full leg, ASan+UBSan, "All tests passed")git clang-format --diffclean on changed lines; clang-tidy/cppcheck via CI)