Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ an optional read-only Neo4j backend — selected by the *type* of the `backend=`
| Java | `CLDK.java(...)` (needs the `cldk[java]` extra) | `JCodeanalyzer` (the `codeanalyzer-java` 3.2.0 wheel's jar on its bundled JVM, subprocess, `-a 1..4` — no jar in this repo, no JDK download) | `JNeo4jBackend` (graphs emitted by ≥ 3.2.0, probed at attach) | `cldk/models/java/` (schema v2 mirror) + `projections.py` |
| Python | `CLDK.python(...)` | `PyCodeanalyzer` (in-process `codeanalyzer-python`) | `PyNeo4jBackend` | re-exported from `codeanalyzer-python` |
| TypeScript (+ JavaScript modules) | `CLDK.typescript(...)` | `TSCodeanalyzer` (`codeanalyzer-typescript` 1.6.0 binary from the wheel, subprocess; `-a 1..4`, but `--emit neo4j` takes no `-a` and is always full depth) | `TSNeo4jBackend` (graphs emitted by ≥ 1.5.2; older refused at attach) | `cldk/models/typescript/` (schema v2 mirror) |
| Go | `CLDK.go(...)` | `GoCodeanalyzer` (`cango` binary, subprocess, `-a 1..2` only — no L3/L4; `--analysis-schema 2` mandatory or it emits legacy v1; binary via `$CODEANALYZER_GO_BIN` or the `codeanalyzer-go` wheel — **no release cut yet, so the version pin is deferred**) | `GoNeo4jBackend` (lazy Cypher over the `GO_`-prefixed graph; **not** an eager rebuild — Go graphs reach enterprise scale) | `cldk/models/go/` (schema v2 mirror; **no `projections.py`** — out of SDK3 scope) |
| Go | `CLDK.go(...)` | `GoCodeanalyzer` (the `codeanalyzer-go` 0.3.0 wheel's `cango` binary, subprocess, `-a 1..4`; binary via `$CODEANALYZER_GO_BIN` or the wheel) | `GoNeo4jBackend` (lazy Cypher over the `GO_`-prefixed graph; **not** an eager rebuild — Go graphs reach enterprise scale) | `cldk/models/go/` (schema v2 mirror, L1–L4; **no `projections.py`** — out of SDK3 scope) |

**Java, since leg 3a (#310):** the models are an `extra="forbid"` mirror of canonical schema v2, so
a 1.x `analysis.json` (and a pre-3.0.1 Neo4j graph) is refused, not parsed; `get_call_graph()` keys
Expand Down Expand Up @@ -45,6 +45,43 @@ absence claim in the result; `max_paths` caps per pair, never per call. The sour
vocabulary is the caller's — no framework catalogue ships. Java shares the `formal_in` dependence
probe above, so `taint` refuses on the same analyses the four verbs do.

**All four languages, since #436:** the reachability walk every local backend's slices and flow
predicates run is **`commons.graphs.reach`**, lifted out of four byte-identical copies for the
reason that lifted `shortest_walks` in leg 2.5b — it is a graph algorithm over an adjacency of
strings and knows no language. Each backend's `_reach` is a one-liner over it, keeping only
`direction` as its own concern. The lift carried a fix: a node must enter the next frontier **once**,
and the spelling in all four copies appended it once per predecessor, so the frontier tracked path
count rather than node count — answers were always right (a set holds no duplicates), but an
unbounded walk did not terminate on a real application, and `depth=None` is the default on
`paths_between` / `flows_to_*` / `taint`. Measured on nats-server: 151,417,783 frontier expansions
at depth 8 against 670,450. **Known limitation, #446:** `shortest_walks` still cannot produce a
witness for a flow `flows_to_*` confirms on a large application — its replay prunes on BFS layer
boundaries but not toward the sink.

**Go, since #436 (the L3/L4 train):** the dataflow surface, with `PythonAnalysisBackend`'s names and
signatures — `get_cfg`/`get_cdg`/`get_ddg`, `resolve_callable`/`resolve_value`,
`slice_backward`/`slice_forward`/`backward_cone`, `reaches`/`paths_between`,
`flows_to_call`/`flows_to_argument`, `taint`, `get_external_symbols`, `get_degraded`, `describe`.
`taint` is concrete on `GoAnalysisBackend`, so neither backend owns the policy; the port-connectivity
gate is Java's, declared on the ABC and implemented on both. Both backends answer identically
including the miss paths — measured on a live graph (`test_go_neo4j_live.py`), which is where four
Cypher bugs surfaced that the local side could not see.

Four Go-specific traps live in **`cldk/analysis/go/ids.py`**, each a rule that holds for Python or
TypeScript and *not* for Go, so a shared helper would be confidently wrong rather than absent:
a body-node id must never be split on its first `@` (a stand-in's own id holds `/@external/` before
the vertex `@`; the naive split mis-owns every external vertex); a formal resolves by its own `of`,
never by ordinal into `parameters[]`, because a method's `formal_in:0` is the receiver; a global's
module half is slash-separated, so the spelling a Go programmer writes (`libflow.Last`) needs the
package-qualified name as a second match name; and `degraded[]` — **envelope** scope, beside
`application`, not on it — is the honesty signal, which voids every `exhausted` certification rather
than refusing (a returned path over an approximated graph is still a real may-flow). `cango`'s own
renames matter too: the Neo4j projection spells `of` as `var`, `parent` as `call_node`, and the CFG
relationship as `GO_CFG_NEXT`. The `GoType.kind` Literal carries all four kinds the analyzer emits
(`struct`/`interface`/`defined`/`alias`) — before #436 an `alias` failed the whole document, which
blocked even `-a 1`. `Diagnostic.code` gained `analysis_degraded` (language-neutral, closed Literal,
published contract).

The legacy `CLDK(language="<lang>").analysis(...)` entry still works as a compat shim. Adding a
language means a new factory method + facade + backend ABC/impl(s) + models + tests — **update this
table in the same change**.
Expand Down
47 changes: 47 additions & 0 deletions cldk/analysis/commons/graphs.py
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,53 @@ def flow_path(nodes: Sequence[SliceNode], edges: Sequence[Tuple[str, "str | None
return FlowPath(hops=[PathHop(frm=nodes[i], to=nodes[i + 1], via=via[rel], var=var, prov=list(prov or [])) for i, (rel, var, prov) in enumerate(edges)])


def reach(edges: Mapping[str, Mapping[str, Sequence[tuple]]], src: str, depth: "int | None") -> set:
"""The node ids reachable from ``src`` over ``edges`` in at most ``depth`` hops.

``edges`` is one direction of the ``{src: {dst: [label]}}`` adjacency each local backend
builds -- only the keys are read, so the labels are ignored here. ``depth`` of ``None`` is the
whole cone.

Level-by-level rather than a plain stack, because ``depth`` is a hop budget and a depth-first
walk cannot count hops without revisiting. Every local backend's slices and flow predicates go
through this, so "reachable" means one thing per language *and* across them.

**A node enters the next frontier once**, which is why ``seen`` is added to inside the level
rather than after it. The obvious spelling builds the level as a comprehension and updates
``seen`` afterwards::

nxt = [d for src in frontier for d in edges.get(src, ()) if d not in seen]
seen.update(nxt)

That is correct and unusable. ``seen`` does not yet contain the level being built, so a node
with *k* predecessors in the current frontier is appended *k* times and the next level expands
every copy: the frontier tracks the number of **paths**, not the number of nodes. A set cannot
hold duplicates, so the answer never changes -- which is exactly why it survived in four
separate copies of this function until an evaluation against a real application measured the
cost. On the ``memos`` corpus (208,293 source nodes, 743,154 edges), over 250 ports: identical
reachable sets at every depth, and 151,417,783 frontier expansions at depth 8 against 670,450
-- 83.65s against 0.11s -- while unbounded it did not terminate at all. ``depth=None`` is the
default on ``paths_between`` / ``flows_to_call`` / ``flows_to_argument`` / ``taint``, so the
unusable case was the default one (python-sdk#436).

Lifted out of the four language backends for :func:`shortest_walks`' reason, and with its
history as the argument: it is a graph algorithm over an adjacency of strings and knows no
language, and four copies were four places to drift -- which is how one bug came to live in
all of them.
"""
seen, frontier, hops = {src}, [src], 0
while frontier and (depth is None or hops < depth):
nxt = []
for node in frontier:
for d in edges.get(node, ()):
if d not in seen:
seen.add(d)
nxt.append(d)
frontier = nxt
hops += 1
return seen


def shortest_walks(
edges: Mapping[str, Mapping[str, Sequence[tuple]]],
src: str,
Expand Down
12 changes: 10 additions & 2 deletions cldk/analysis/commons/results.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ class Diagnostic(BaseModel):
resolver, not in the error path", so nothing in the SDK constructs a ``Diagnostic``
with suggestions or with ``code="did_you_mean"``; the codes actually emitted are
``file_not_in_graph``, ``module_scope``, ``module_source_unavailable``,
``entrypoint_report_unavailable``, and — since ``taint()`` — ``unresolved_dispatch``
and ``degenerate_pair``. The field and the code stay because both are part
``entrypoint_report_unavailable``, ``unresolved_dispatch`` and ``degenerate_pair``
(since ``taint()``), and ``analysis_degraded`` (since Go's L3/L4 train,
python-sdk#436). The field and the ``did_you_mean`` code stay because both are part
of a published model contract (``docs/agent-api-reference.md``) that a caller may
already destructure; removing either is a separate, breaking change.
"""
Expand All @@ -84,6 +85,13 @@ class Diagnostic(BaseModel):
"graph_schema_mismatch",
"entrypoint_report_unavailable",
"degenerate_pair",
# Part of the analysis was approximated, so an *absence* of flow is not a proof over it
# (python-sdk#436). Emitted by Go's ``taint()`` when ``get_degraded()`` is non-empty, and
# filed under every requested pair, which is what voids their ``exhausted`` certification.
# A language-neutral code rather than a Go-specific one: the condition it names — "the
# analyzer told us it fell back" — is not Go's alone, and a sibling that gains a
# degradation channel should report it under this name rather than minting a second.
"analysis_degraded",
]
message: str
suggestions: list[str] = []
Expand Down
Loading
Loading