feat(proto): link a .proto declaration to the code generated from it - #1660
Open
ferrine wants to merge 5 commits into
Open
feat(proto): link a .proto declaration to the code generated from it#1660ferrine wants to merge 5 commits into
ferrine wants to merge 5 commits into
Conversation
`callers`, `callees` and `impact` each carried their own symbol filter,
comparing the user's query against the BARE node name only:
node.name === symbol || node.name.endsWith('.' + symbol)
That fails in two opposite directions in the same repository.
A bare name OVER-reports. Every same-named definition passes the filter
and their results are unioned under one heading — "Callers of group" can
list callers that belong to an entirely different `group`, with nothing
saying the name was ambiguous. Collisions cluster on short generic names
(`group`, `num`, `parse`), so this bites hardest exactly where the verbs
would otherwise be most useful.
A qualified name UNDER-reports. `Foo.Bar.baz` can never equal a bare
`baz`, so every candidate fails the filter, and the guarded fallback
takes whichever node full-text search ranked first — or reports "not
found" for a symbol that plainly exists. It only ever appeared to work
when FTS happened to return exactly one hit.
All three now resolve through graph/symbol-lookup, which the MCP tools
share, so a verb cannot drift from the matcher again:
- the exact-name index is consulted first and is authoritative. It is
complete and uncapped, whereas FTS ranks, truncates, and tokenises
`::` away — so resolution no longer depends on search ranking. FTS
stays as the fallback for the fuzzy cases it is good at.
- `matchesSymbol` gains a boundary-aligned suffix match under a
canonical separator. Splitting on every separator assumes no scope
component contains one, which is false for any language whose module
names are themselves dotted: the stored `A.B::c` can never equal the
split-and-rejoined `A::B::c`, so a precise query resolved to nothing.
- an ambiguous bare name still aggregates (an interface method and its
overrides are usually all wanted) but the union is now disclosed, with
the matched definitions named and a qualified spelling that narrows
it. `--json` gains a `targets` array and an `ambiguous` flag.
`matchesSymbol` moves out of mcp/tools.ts unchanged apart from the new
stage; the tool path delegates to it, so its existing coverage applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Bsi9EH64kMisnik4E1gf7
A `.proto` is the contract layer of a polyglot repository: one field is implemented again in every generated language, each of those sites is machine-written and must never be hand-edited, and none of it was visible to the graph. Indexing it makes a field a symbol you can ask questions about — and makes "what else moves when this changes" answerable from the one place the answer is actually written down. Extracted: messages (including nested), enums and their values, fields, `oneof` members, `map` types, services, `rpc`s (including `stream`), imports, and `reserved`. Names are protobuf's own fully-qualified names (`acme.reporting.v1.Measurement.observed_at`), which is both what a user would type and what a generator uses. Field types and rpc request / response messages become `references`, so `callers` on a message lists the fields and RPCs that depend on it. Two properties are modelled on purpose, because they are where protobuf's real defects live and neither survives a naive extraction: - THE TAG NUMBER IS PART OF A FIELD'S IDENTITY. Renaming a field at the same tag is wire-compatible; changing its TYPE at the same tag and name is a silent mis-decode that every single-language check passes. The tag is recorded as a marker, not just left in prose, so a check can read it without re-parsing the declaration. - `reserved` IS SEMANTIC. A retired number must never be re-used and a reader touching a reserved field is a defect, so reservations — numbers, ranges and names — are kept as symbols rather than discarded as syntax. They are deliberately not `field` nodes, so a reservation can never be mistaken for a live field. Implemented as a standalone scanner (proto-extractor.ts) rather than a vendored grammar, following the Liquid / Razor / MyBatis precedent. The IDL is small and effectively frozen, so the usual reason to want a grammar — tracking an evolving syntax surface — does not apply, and it avoids shipping another megabyte of wasm whose silent absence is its own failure mode. Comments are blanked offset-preserving before scanning, skipping string literals so a URL in an option value is not mistaken for a comment. Verified against upstream protos: descriptor.proto (proto2, groups, extensions, 17 reserved statements), pubsub.proto, struct.proto, timestamp.proto and grpc health.proto all parse with zero errors, and the message / enum / service / rpc counts match the files exactly. Depends on the qualified-lookup fix: without it a protobuf FQN is a dotted container name, which the old matcher could not match at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Bsi9EH64kMisnik4E1gf7
A `.proto` is authored once and implemented again in every language the project generates for. Each of those sites is machine-written and must never be hand-edited, which makes the contract the only place the shared shape exists — and, with no edges, the only place with no link to anything that implements it. That gap holds a family of defects that every tier's own gate passes by construction: a field decoded on one side but never read on another; a field whose meaning changed while its name and tag did not; a field the server stopped sending that a client still declares. Synthesis pass, gated on the project having protos: - a message / enum / service links to its generated type in each target language; - an `rpc` links to the GENERATED METHOD itself — every generator emits one, so this resolves at member level; - a field links to the generated type that declares it. Go struct fields, Python class annotations and TypeScript interface members are all deliberately not extracted as nodes (member-dense code would explode the graph), so there is no member-level peer to match. The declaring type is coarser but true — regenerating it IS what the change requires — and containment does not rescue the field either: dependents traverse INCOMING edges while `contains` points message → field, so a field's impact never climbs to its message. The match kind is recorded in the edge so the two are distinguishable. Edges point GENERATED → PROTO. Generated code is the dependent, and that is the direction impact analysis reads; emitted the other way the relationship is recorded but the question stays unanswered. Discovery is by convention, never configuration — every generator names its output after the proto (`foo_pb2.py`, `foo.pb.ex`, `foo.pb.go`, `foo_pb.ts`), and those conventions are published and stable. Precision comes from the peer-file join: a field named `id` is matched only inside the generated outputs of its OWN proto, never repo-wide, which is what makes a name that common safe to match at all. A file without a generator marker in its name is never treated as generated output, so hand-written code sitting beside a proto is not linked. Also maps `.pyi`, which was not indexed at all. Type stubs are real checked-in API surface, and for protobuf they are the only place per-field Python declarations exist — a modern `_pb2.py` is a serialized descriptor blob with no per-field symbols. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Bsi9EH64kMisnik4E1gf7
Generators for languages with dotted module names emit the FULL name as the declaration's own name — protobuf-elixir writes `defmodule Acme.V1.Interval`, so the node is named `Acme.V1.Interval` while the proto message it implements is `Interval`. The peer index only held each candidate under spellings of its own name, so a simple-name match never reached it and every such peer was invisible. Silently, too: a target language whose generator emits a bare name links normally, so the result looks like a working feature with one language's generator simply absent from the output rather than a matching bug. On a polyglot repo generating from the same protos into both a bare-name and a qualified-name language, roughly a third of the expected contract edges were missing on that basis alone. Index each candidate under its trailing segment as well as its own name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Bsi9EH64kMisnik4E1gf7
Whether a generator emits a symbol for each field is a fact about THAT generator — protoc-gen-js writes an accessor per field, protoc's Python stubs write only a class annotation that is not extracted as a node. The fallback to a member's declaring type was decided once for the whole declaration, across every language at once, so any language that DID resolve precisely suppressed the fallback for every language that could not, and those languages contributed no edge at all. The failure is silent and inverted: making one language's extraction better DELETES the coarser coverage of the others. Seen when Elixir's generated modules started yielding per-field symbols — every Python declaring-type edge in the graph disappeared in the same re-index, with nothing reporting it. Peers are now indexed and matched per target language, so each one falls back on its own evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Bsi9EH64kMisnik4E1gf7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch:
ferrine:feat/protobuf-contract-resolver→mainStacked on:
feat/protobuf-language(PR #4).Size: +644 over #4 · Tests:
__tests__/protobuf-contract-synthesizer.test.ts(14 new)Why
One
.protofield is implemented again in every language a project generatesfor, every one of those sites is machine-written, and the
.protois the onlyplace the shape is authored. Without an edge between them, a whole family of
defects has nowhere to show up — a field decoded on one side but never read on
another; a field whose meaning changed while its name and tag did not; a field
the server stopped sending that a client still declares. Each is green in every
single-language check and wrong at runtime.
What changed
New synthesis pass (
src/resolution/protobuf-contract-synthesizer.ts), gated onthe project having protos at all.
Discovery is by convention, not configuration. Every generator names its
output after the
.protoit came from (foo_pb2.py,foo.pb.ex,foo.pb.go,foo_pb.ts), and those conventions are published and stable.Precision comes from a peer-file gate. A field named
idis matched onlyinside the generated outputs of its own proto, never repo-wide — that scoping
is what makes matching a name as common as
idsafe. A proto with no generatedpeers produces nothing.
Edge direction is generated → proto: generated code is derived from the
contract, so it is the dependent, and that is the direction impact analysis
traverses. Emitted the other way the relationship is recorded but "what else
moves when I change this field" stays unanswered.
Each edge carries
match: 'symbol' | 'declaring-type'so a consumer can tell anexact peer from the coarser fallback, plus the tag and the generated language.
Two fixes folded in, both found on a real multi-language repo
names emit the full name as the declaration's own name — protobuf-elixir
writes
defmodule Acme.V1.Interval, so the node isAcme.V1.Intervalwhilethe message is
Interval. Matching the simple name only made every such peerinvisible, and silently: another language's generator emitting a bare name
links normally, so it looks like a working feature with one generator simply
absent. Candidates are now also indexed under their trailing segment.
generator emits a symbol per field is a fact about that generator —
protoc-gen-js writes an accessor per field, protoc's Python stubs write only a
class annotation, which is not extracted as a node. Deciding the fallback once
for the whole declaration let any language that resolved precisely suppress
the fallback for every language that could not, and those languages then
contributed no edge at all. The failure is silent and inverted: improving one
language's extraction deleted the coarser coverage of the others. There is a
regression test that fails with
expected undefinedon the old code.Notes for review
A no-op on any repo without
.protofiles, and on a repo with protos but nogenerated peers.