From 6f3fdba92ae6c7e39179e424c66fce9b29797888 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Mon, 28 Sep 2026 20:49:30 +0200 Subject: [PATCH 1/9] fix(python): extract defs nested in a function body (#1548) Why The Python def walk stopped at every function_definition: it extracted the def and never looked at its body. Every def or class declared inside a function -- closures, decorator wrappers, the route handlers of a Flask or FastAPI app factory (`@app.route` inside create_app), local classes and their methods -- was absent from the graph. Blocks at module level (if / try / for / with) were transparent, so only function bodies were blind. The calls inside those defs were not re-attributed either: the call-scope walk named a nested def by its class or module only (`pkg.mod.nested`, a QN no node had), so the edges fell back to the file node, and inbound traces of the called function silently lost the caller. A def nested in a method got the method's own QN (`Tree.visit` for the `def visit` inside method Tree.visit), so its calls, recursion included, landed on the method. On an app-factory codebase this removes the whole HTTP layer. Fix - Extraction (extract_defs.c): after a Python def is extracted, its body is walked with the def's QN as the enclosing scope; methods' bodies are walked under the method QN. A nested def or class is qualified by the def it sits in -- pkg.mod.create_app.index, pkg.mod.Tree.visit.visit, pkg.mod.create_app.Settings.value -- the QN the Python LSP already gives the calls inside it (py_process_function). Decorated nested defs keep their decorators, so their route_path mints the Route and HANDLES. Lambdas and comprehensions hold no statements and stay as they are. - Call scope (extract_unified.c): for Python the nearest named scope, a def or a class, prefixes a nested def or class, so every call, usage and env read inside a nested def carries the nested def's QN. - A def nested in a function is a local of it (Python LEGB), so it must not become a global name: a Python symbol whose QN extends a Function/Method node is registered for exact lookups only, never in the short-name index (registry.c cbm_registry_add_for_language, used by the full, parallel and incremental registration sites alike). Without this, django's per-test `class MyModelAdmin` and every decorator's `def wrapper` became unique_name / suffix_match targets for unrelated calls. The incremental seed derives each stored file's language the way discovery does (name, then shebang), so it keeps the same symbols out. The import symbol-name fallback skips such locals too (pass_pkgmap.c), and the Python LSP no longer binds a def-local class module-wide by its short name (py_lsp.c). - A bare call now resolves lexically first (registry.c cbm_registry_resolve_lexical, strategy lexical_scope): the def or class bound in the nearest enclosing function, skipping class bodies, stopping at the module. The sequential and parallel call passes use it identically, and so does the handler lookup of a call-registered route: `app.add_url_rule("/ping", "ping", ping)` in create_app reaches the local ping. Proof (real code; every node and edge diffed, main 1f9b4db1 vs this) - pallets/flask examples/tutorial (flaskr, create_app factory): the nested `hello` handler is now create_app.hello with HANDLES to /hello; 4 nested defs/classes added, 0 nodes and 0 edges removed. - pallets/flask d73fa1c (tests declare handlers inside test functions): Function 532 -> 1068, Method 318 -> 381, Class 130 -> 209. The +678 equals the 678 unique nested def/class QNs Python's own ast finds. Handler HANDLES (decorators and add_url_rule) 32 -> 296. 0 nodes removed, 0 duplicate QNs. The 94 removed CALLS: 90 moved from the file node onto the nested def that makes the call, 4 replaced by the lexically bound local. - django fb11376 (local, 2,927 .py files): nodes 55,483 -> 60,052, Function 1,992 -> 3,027, Method 28,850 -> 29,491, Class 8,187 -> 11,073. 0 nodes removed and 0 duplicate QNs. Every added Function/Method/Class sits under a Function/Method of the same file, and they match ast's 4,567 nested QNs except for 5 methods declared in an `if` inside a local class body (class-body `if` is not walked for any class). CALLS 62,430 -> 63,815. Every added CALLS edge has a nested def at one end: 1,015 from nested defs, 1,219 lexical_scope into a local, and none from outside the local's function. Of the 849 removed: 507 moved from the file node onto the nested def, 334 replaced by the local the name binds, and 8 were false on main, all checked (e.g. the recursive `get_actions` nested in call_command had been suffix-matched to ModelAdmin.get_actions). HANDLES unchanged (135). - Samples checked against the source: AbstractBaseUser.check_password.setter -> set_password / save; View.as_view.view -> setup / dispatch; Apps.lazy_model_operation -> lazy_model_operation.apply_next_model (line 426); InstrumentedASGIRequest.META -> the local InstrumentedDict (line 279); flask test_provide_automatic_options_arg_disable.index -> GET /. Tests - extraction: extract_python_nested_def_qualified_by_enclosing_def, _calls_attributed_to_it, _decorated_nested_route_handler, _nested_class, _nested_def_named_like_its_method, _lambda_and_comprehension_stay_in_enclosing_def (control), _nested_def_env_access_attributed_to_it (all _issue1548). - pipeline: pipeline_python_app_factory_routes, _nested_def_is_function_ local, _local_classes_stay_in_their_def, _import_never_binds_a_local_def, _nested_defs_parallel (>= 50 files, CBM_WORKERS=4), _nested_def_local_after_incremental (all _issue1548). - registry: registry_python_function_local_is_exact_only, registry_resolve_lexical_walks_enclosing_defs (both _issue1548). The extraction and pipeline tests were RED on main 3/3 for the asserted reason and are green with the fix; the registry tests cover the new registry API. Each production piece was reverted on its own, and each revert turns at least one of these tests RED. Fixes #1548 Signed-off-by: Martin Vogel (cherry picked from commit 62fda43fd66200879db62a58903a32dab0dd1a20) --- internal/cbm/extract_defs.c | 66 ++++- internal/cbm/extract_unified.c | 33 ++- internal/cbm/lsp/py_lsp.c | 132 ++++++++-- src/pipeline/pass_calls.c | 22 +- src/pipeline/pass_parallel.c | 19 +- src/pipeline/pass_pkgmap.c | 32 +++ src/pipeline/pipeline.h | 17 +- src/pipeline/pipeline_incremental.c | 5 +- src/pipeline/pipeline_internal.h | 10 + src/pipeline/registry.c | 131 +++++++++- tests/test_extraction.c | 299 ++++++++++++++++++++++ tests/test_pipeline.c | 378 ++++++++++++++++++++++++++++ tests/test_registry.c | 92 +++++++ 13 files changed, 1192 insertions(+), 44 deletions(-) diff --git a/internal/cbm/extract_defs.c b/internal/cbm/extract_defs.c index 77e0bbca07..2b24de63f1 100644 --- a/internal/cbm/extract_defs.c +++ b/internal/cbm/extract_defs.c @@ -6624,11 +6624,16 @@ static void extract_func_def(CBMExtractCtx *ctx, TSNode node, const CBMLangSpec * scope. Note the call-scope side (compute_func_qn in extract_unified.c) is * NOT language-gated — it qualifies whenever a scope is pushed — so this gate * and the scope-push rule must move together, or a call QN names a def QN that - * was never minted and the edge is dropped at write. */ + * was never minted and the edge is dropped at write. + * + * Python: the only enclosing scope that reaches here is the def a nested + * def sits in (py_push_def_body), so `def index()` inside create_app is + * proj.mod.create_app.index. */ if (ctx->enclosing_class_qn && (ctx->language == CBM_LANG_CPP || ctx->language == CBM_LANG_CUDA || ctx->language == CBM_LANG_TYPESCRIPT || ctx->language == CBM_LANG_TSX || - ctx->language == CBM_LANG_ARKTS || ctx->language == CBM_LANG_NIX)) { + ctx->language == CBM_LANG_ARKTS || ctx->language == CBM_LANG_NIX || + ctx->language == CBM_LANG_PYTHON)) { def.qualified_name = cbm_arena_sprintf(a, "%s.%s", ctx->enclosing_class_qn, qn_name); } def.label = "Function"; @@ -11667,6 +11672,51 @@ static void push_class_body_children(TSNode node, const CBMLangSpec *spec, wd_st } } +/* Python: a def's body is a scope of its own, and a `def` or `class` statement + * in it is a definition like any other -- closures, decorator wrappers, the + * route handlers of an app factory (`@app.route` inside create_app), local + * classes. Walk the body with the def's QN as the enclosing scope, so a nested + * def is qualified by the def it sits in: proj.mod.create_app.index. That is + * the QN the Python LSP already gives the calls inside it (py_process_function) + * and the one the call-scope walk computes (extract_unified.c + * py_nearest_scope_qn). Only the body is walked: parameters, + * annotations and decorators hold expressions, never statements. A lambda or + * a comprehension holds no statements either, so neither becomes a def. */ +static void py_push_def_body(wd_stack_t *s, TSNode def_node, const char *def_qn) { + TSNode body = ts_node_child_by_field_name(def_node, TS_FIELD("body")); + if (def_qn && !ts_node_is_null(body)) { + wd_push(s, body, def_qn); + } +} + +/* Python methods are extracted by extract_class_methods, not by the walk, so + * their bodies are pushed here, under the QN push_method_def gave the method + * (class_qn.name), in source order. */ +static void py_push_method_bodies(CBMExtractCtx *ctx, TSNode class_node, const char *class_qn, + wd_stack_t *s) { + TSNode body = find_class_member_body(class_node, ctx->language); + if (ts_node_is_null(body) || !class_qn) { + return; + } + for (int i = (int)ts_node_named_child_count(body) - SKIP_CHAR; i >= 0; i--) { + TSNode method = ts_node_named_child(body, (uint32_t)i); + if (strcmp(ts_node_type(method), "decorated_definition") == 0) { + method = ts_node_child_by_field_name(method, TS_FIELD("definition")); + } + if (ts_node_is_null(method) || strcmp(ts_node_type(method), "function_definition") != 0) { + continue; + } + TSNode name_node = resolve_method_name(method, ctx->language); + char *name = + ts_node_is_null(name_node) + ? NULL + : cbm_func_name_node_text(ctx->arena, name_node, ctx->source, ctx->language); + if (name && name[0]) { + py_push_def_body(s, method, cbm_arena_sprintf(ctx->arena, "%s.%s", class_qn, name)); + } + } +} + // ASCII case-insensitive equality vs a lowercase literal (CFML attribute names // are case-insensitive, e.g. NAME / Name / name). static bool ascii_ci_equals(const char *s, const char *lower_lit) { @@ -12211,7 +12261,16 @@ static void walk_defs(CBMExtractCtx *ctx, TSNode root, const CBMLangSpec *spec, } if (cbm_kind_in_set(node, spec->function_node_types)) { if (!is_template_class_node(node, ctx->language)) { + int defs_before = ctx->result->defs.count; extract_func_def(ctx, node, spec); + if (ctx->language == CBM_LANG_PYTHON) { + if (ctx->result->defs.count > defs_before) { + py_push_def_body(&s, node, + ctx->result->defs.items[ctx->result->defs.count - SKIP_ONE] + .qualified_name); + } + continue; + } // Most languages stop here. JS/TS (and Wolfram) descend into the // function body so NESTED named definitions are also captured — // e.g. arrow methods of an object literal returned from a factory @@ -12268,6 +12327,9 @@ static void walk_defs(CBMExtractCtx *ctx, TSNode root, const CBMLangSpec *spec, extract_class_def(ctx, node, spec); const char *new_enclosing = compute_class_qn(ctx, node, frame.enclosing_class_qn); push_class_body_children(node, spec, &s, new_enclosing, ctx); + if (ctx->language == CBM_LANG_PYTHON && new_enclosing != frame.enclosing_class_qn) { + py_push_method_bodies(ctx, node, new_enclosing, &s); + } continue; } diff --git a/internal/cbm/extract_unified.c b/internal/cbm/extract_unified.c index 96e6ac865c..c7b10d7e2e 100644 --- a/internal/cbm/extract_unified.c +++ b/internal/cbm/extract_unified.c @@ -916,6 +916,21 @@ static const char *objectscript_get_method_qn(CBMExtractCtx *ctx, TSNode node, return NULL; } +/* Python: the nearest named scope around the node -- a def or a class. A def + * or class nested in a def is qualified by that def (proj.mod.create_app.index), + * exactly as extract_defs.c mints it; enclosing_class_qn alone would skip the + * def and hand a def nested in a method its method's QN. Lambdas push no named + * frame, so they stay transparent. */ +static const char *py_nearest_scope_qn(const WalkState *state) { + for (int i = state->scope_top - SKIP_ONE; i >= 0; i--) { + const CBMWalkScope *f = &state->scopes[i]; + if ((f->kind == SCOPE_FUNC || f->kind == SCOPE_CLASS) && f->qn) { + return f->qn; + } + } + return NULL; +} + // Compute function QN for scope tracking (mirrors cbm_enclosing_func_qn logic). static const char *compute_func_qn(CBMExtractCtx *ctx, TSNode node, const CBMLangSpec *spec, WalkState *state) { @@ -1107,8 +1122,10 @@ static const char *compute_func_qn(CBMExtractCtx *ctx, TSNode node, const CBMLan } } - if (state->enclosing_class_qn) { - return cbm_arena_sprintf(ctx->arena, "%s.%s", state->enclosing_class_qn, qn_name); + const char *outer_qn = + ctx->language == CBM_LANG_PYTHON ? py_nearest_scope_qn(state) : state->enclosing_class_qn; + if (outer_qn) { + return cbm_arena_sprintf(ctx->arena, "%s.%s", outer_qn, qn_name); } /* Java/Go: directory-based module so this enclosing-func QN matches the def * QN and the LSP caller_qn (the lsp_resolve join keys on exact equality). */ @@ -1162,9 +1179,15 @@ static const char *compute_class_qn(CBMExtractCtx *ctx, TSNode node, const WalkS /* Nested class: prefix with the enclosing class QN (Outer.Inner) so this * scope QN matches the def-side class QN (extract_defs.c compute_class_qn / - * extract_class_def), which the lsp_resolve join requires for nested types. */ - if (state && state->enclosing_class_qn) { - return cbm_arena_sprintf(ctx->arena, "%s.%s", state->enclosing_class_qn, name); + * extract_class_def), which the lsp_resolve join requires for nested types. + * Python: a class inside a def is qualified by the def (py_nearest_scope_qn). */ + const char *outer_qn = NULL; + if (state) { + outer_qn = ctx->language == CBM_LANG_PYTHON ? py_nearest_scope_qn(state) + : state->enclosing_class_qn; + } + if (outer_qn) { + return cbm_arena_sprintf(ctx->arena, "%s.%s", outer_qn, name); } /* Java/Go: directory-based module (see compute_func_qn). */ diff --git a/internal/cbm/lsp/py_lsp.c b/internal/cbm/lsp/py_lsp.c index 6af7dae40b..cbad720fa3 100644 --- a/internal/cbm/lsp/py_lsp.c +++ b/internal/cbm/lsp/py_lsp.c @@ -246,6 +246,9 @@ static void py_register_lambda(PyLSPContext *ctx, const char *name, TSNode lambd static void py_register_dict_literal(PyLSPContext *ctx, const char *name, TSNode dict_node); static TSNode py_lookup_lambda(PyLSPContext *ctx, const char *name); static const char *py_lookup_dict_dispatch(PyLSPContext *ctx, const char *var, const char *key); +static void py_process_function(PyLSPContext *ctx, TSNode func_node, const char *container_qn); +typedef enum { PY_LOCAL_DEFS_BIND, PY_LOCAL_DEFS_PROCESS } PyLocalDefsMode; +static void py_visit_local_defs(PyLSPContext *ctx, TSNode block, PyLocalDefsMode mode, int depth); /* ── Scope mutation wrappers ───────────────────────────────────────────── * @@ -4362,6 +4365,90 @@ static void py_bind_parameters(PyLSPContext *ctx, TSNode params) { } } +/* Bind a def or class statement of the current function body as a local of + * that function (#1548): its graph QN extends the function's + * (enclosing_func_qn.name), the QN extraction gives it. An undecorated def + * registered with that QN becomes an exact callable alias; anything else + * (decorated, overloaded, not registered) is a hard local shadow, so no call + * falls through to a same-named module symbol. A class is bound by its type + * when registered, else shadowed. A parameter of the same name is already a + * local and keeps its own binding. */ +static void py_bind_local_def(PyLSPContext *ctx, TSNode def, bool is_class) { + TSNode name_node = ts_node_child_by_field_name(def, "name", 4); + char *name = ts_node_is_null(name_node) ? NULL : py_node_text(ctx, name_node); + if (!name || !name[0] || !ctx->enclosing_func_qn) { + py_disable_callable_value_proof(ctx); + return; + } + if (cbm_scope_lookup_local(ctx->current_scope, name)) + return; + const char *qn = cbm_arena_sprintf(ctx->arena, "%s.%s", ctx->enclosing_func_qn, name); + if (!qn) { + py_disable_callable_value_proof(ctx); + py_scope_bind(ctx, name, cbm_type_unknown()); + return; + } + if (is_class) { + py_scope_bind(ctx, name, + cbm_registry_lookup_type(ctx->registry, qn) ? cbm_type_named(ctx->arena, qn) + : cbm_type_unknown()); + return; + } + const CBMRegisteredFunc *func = cbm_registry_lookup_func(ctx->registry, qn); + if (py_func_is_exact_callable_value(func)) { + py_scope_bind_callable(ctx, name, cbm_type_unknown(), func->qualified_name); + } else { + py_scope_bind(ctx, name, cbm_type_unknown()); + } +} + +/* Statements a function body nests further statements in. A def inside any of + * them is still a local of the function; a nested def, class or lambda body + * is a scope of its own and is never entered. */ +static bool py_is_statement_block(const char *kind) { + static const char *const kinds[] = { + "block", "if_statement", "elif_clause", "else_clause", + "for_statement", "while_statement", "try_statement", "except_clause", + "except_group_clause", "finally_clause", "with_statement", "match_statement", + "case_clause", NULL}; + for (int i = 0; kinds[i]; i++) { + if (strcmp(kind, kinds[i]) == 0) + return true; + } + return false; +} + +/* Visit the def/class statements of a function body, including those inside + * its compound statements: BIND binds each as a local (py_bind_local_def), + * PROCESS walks each nested def's own body (py_process_function). */ +static void py_visit_local_defs(PyLSPContext *ctx, TSNode block, PyLocalDefsMode mode, int depth) { + if (!ctx || ctx->eval_failure || ts_node_is_null(block) || depth >= cbm_lsp_max_walk_depth()) + return; + PyKids kids = py_kids(ctx->arena, block); + for (uint32_t i = 0; i < kids.n && !ctx->eval_failure; i++) { + TSNode c = py_kid(&kids, i); + const char *ck = ts_node_type(c); + TSNode def = c; + if (strcmp(ck, "decorated_definition") == 0) { + def = ts_node_child_by_field_name(c, "definition", 10); + if (ts_node_is_null(def)) + continue; + ck = ts_node_type(def); + } + if (strcmp(ck, "function_definition") == 0) { + if (mode == PY_LOCAL_DEFS_BIND) + py_bind_local_def(ctx, def, false); + else + py_process_function(ctx, def, ctx->enclosing_func_qn); + } else if (strcmp(ck, "class_definition") == 0) { + if (mode == PY_LOCAL_DEFS_BIND) + py_bind_local_def(ctx, def, true); + } else if (py_is_statement_block(ck)) { + py_visit_local_defs(ctx, c, mode, depth + 1); + } + } +} + static void py_process_function(PyLSPContext *ctx, TSNode func_node, const char *container_qn) { if (!ctx || ctx->eval_failure) return; @@ -4391,22 +4478,13 @@ static void py_process_function(PyLSPContext *ctx, TSNode func_node, const char TSNode body = ts_node_child_by_field_name(func_node, "body", 4); if (!ts_node_is_null(body)) { + /* Python resolves a bare name in the function's own scope first: every + * def/class statement of the body binds a local for the whole body + * (LEGB), whatever a module-level symbol of the same spelling says. */ + py_visit_local_defs(ctx, body, PY_LOCAL_DEFS_BIND, 0); py_resolve_calls_in(ctx, body); - // Also descend into nested function/class definitions in the body. - PyKids bk = py_kids(ctx->arena, body); - for (uint32_t i = 0; i < bk.n; i++) { - TSNode c = py_kid(&bk, i); - const char *ck = ts_node_type(c); - if (strcmp(ck, "function_definition") == 0) { - py_process_function(ctx, c, ctx->enclosing_func_qn); - } else if (strcmp(ck, "decorated_definition") == 0) { - TSNode def = ts_node_child_by_field_name(c, "definition", 10); - if (!ts_node_is_null(def) && - strcmp(ts_node_type(def), "function_definition") == 0) { - py_process_function(ctx, def, ctx->enclosing_func_qn); - } - } - } + // Also descend into nested function definitions in the body. + py_visit_local_defs(ctx, body, PY_LOCAL_DEFS_PROCESS, 0); } py_scope_restore(ctx, saved); @@ -4687,6 +4765,28 @@ static char *py_module_prefix(CBMArena *scratch, const char *module_qn, size_t l return prefix; } +/* A class declared inside a def -- django's per-test `class MyModelAdmin`, an + * app factory's local class, and anything nested in one -- is a local of that + * def, not a global of the module: bound here by its short name, every + * same-named local class of the file collapsed onto whichever registered last. + * Its QN extends a registered function's (proj.mod.Tests.test_x.MyModelAdmin), + * which tells it apart from a class nested in a class (proj.mod.Outer.Inner). + * Only prefixes longer than the module are probed. */ +static bool py_type_is_function_local(const PyLSPContext *ctx, const char *qn, size_t module_len) { + char scope[512]; + size_t len = strlen(qn); + if (len >= sizeof(scope)) + return false; + memcpy(scope, qn, len + 1); + for (char *dot = strrchr(scope, '.'); dot && (size_t)(dot - scope) > module_len; + dot = strrchr(scope, '.')) { + *dot = '\0'; + if (cbm_registry_lookup_func(ctx->registry, scope)) + return true; + } + return false; +} + /* A project registry can contain another file's class in the same logical * Python module. Keep those cross-file globals available, but do not prebind * classes declared by this source: their binding epoch belongs in the ordered @@ -4734,6 +4834,8 @@ static void py_bind_external_module_classes(PyLSPContext *ctx, const PyKids *rk) continue; if (package_scope && strchr(qn + prefix_len + 1, '.')) continue; /* submodule/nested classes are not package globals */ + if (py_type_is_function_local(ctx, qn, prefix_len)) + continue; /* a def-local class is no module global (#1548) */ py_scope_bind(ctx, name, cbm_type_named(ctx->arena, qn)); } if (prefix) diff --git a/src/pipeline/pass_calls.c b/src/pipeline/pass_calls.c index 1eb3bddff7..77fa14f4aa 100644 --- a/src/pipeline/pass_calls.c +++ b/src/pipeline/pass_calls.c @@ -587,9 +587,19 @@ static void handle_route_registration(cbm_pipeline_ctx_t *ctx, const CBMCall *ca esc_fa); cbm_gbuf_insert_edge(ctx->gbuf, source_node->id, route_id, "CALLS", props); if (call->second_arg_name != NULL && call->second_arg_name[0] != '\0') { - cbm_resolution_t hres = - cbm_registry_resolve_handler(ctx->registry, call->second_arg_name, module_qn, imp_keys, - imp_vals, imp_count, ctx->gbuf, lang); + /* Python: an app factory registers its local defs + * (`app.add_url_rule("/", "index", index)` inside create_app); only the + * lexical lookup sees a function's locals. MUST match pass_parallel.c. */ + cbm_resolution_t hres = {0}; + if (lang == CBM_LANG_PYTHON) { + hres = cbm_registry_resolve_lexical(ctx->registry, call->second_arg_name, + call->enclosing_func_qn); + } + if (!hres.qualified_name) { + hres = cbm_registry_resolve_handler(ctx->registry, call->second_arg_name, + module_qn, imp_keys, imp_vals, imp_count, + ctx->gbuf, lang); + } if (hres.qualified_name != NULL && hres.qualified_name[0] != '\0') { const cbm_gbuf_node_t *handler = cbm_gbuf_find_by_qn(ctx->gbuf, hres.qualified_name); if (handler != NULL) { @@ -967,8 +977,10 @@ static int resolve_single_call(cbm_pipeline_ctx_t *ctx, CBMCall *call, const CBM lang, lsp->strategy, lsp->callee_qn, ctx->project_name); cbm_resolution_t res = {0}; if (!rust_external && !php_vendor_call) { - res = cbm_registry_resolve_call_lang(ctx->registry, call->callee_name, module_qn, imp_keys, - imp_vals, imp_count, lang); + /* Python LEGB fallback first (MUST match pass_parallel.c). */ + res = cbm_registry_resolve_call_scoped(ctx->registry, call->callee_name, module_qn, + imp_keys, imp_vals, imp_count, lang, + call->enclosing_func_qn); /* Cross-language veto: a name-only guess never binds another * language's symbol (a JS app.get must not become Python's * builtins.dict.get). The vetoed answer becomes EMPTY here, before diff --git a/src/pipeline/pass_parallel.c b/src/pipeline/pass_parallel.c index 2c0a9023a5..501320dd65 100644 --- a/src/pipeline/pass_parallel.c +++ b/src/pipeline/pass_parallel.c @@ -2410,8 +2410,17 @@ static void emit_route_registration(cbm_gbuf_t *gbuf, const cbm_gbuf_node_t *sou esc_rp); cbm_gbuf_insert_edge(gbuf, source->id, rid, "CALLS", props); if (handler_ref && handler_ref[0] != '\0') { - cbm_resolution_t hres = cbm_registry_resolve_handler(registry, handler_ref, module_qn, ik, - iv, ic, main_gbuf, lang); + /* Python: a handler an app factory registers is usually its local def + * (`app.add_url_rule("/", "index", index)` inside create_app); only the + * lexical lookup sees a function's locals. MUST match pass_calls.c. */ + cbm_resolution_t hres = {0}; + if (lang == CBM_LANG_PYTHON) { + hres = cbm_registry_resolve_lexical(registry, handler_ref, call->enclosing_func_qn); + } + if (!hres.qualified_name) { + hres = cbm_registry_resolve_handler(registry, handler_ref, module_qn, ik, iv, ic, + main_gbuf, lang); + } if (hres.qualified_name && hres.qualified_name[0] != '\0') { const cbm_gbuf_node_t *h = cbm_gbuf_find_by_qn(main_gbuf, hres.qualified_name); if (h) { @@ -3267,8 +3276,10 @@ static void resolve_file_calls(resolve_ctx_t *rc, resolve_worker_state_t *ws, CB lang, lsp->strategy, lsp->callee_qn, rc->project_name); if ((!res.qualified_name || !res.qualified_name[0]) && !call->requires_lsp_resolution && !rust_external && !php_vendor_call) { - res = cbm_registry_resolve_call_lang(rc->registry, call->callee_name, module_qn, - imp_keys, imp_vals, imp_count, lang); + /* Python LEGB fallback first (MUST match pass_calls.c). */ + res = cbm_registry_resolve_call_scoped(rc->registry, call->callee_name, module_qn, + imp_keys, imp_vals, imp_count, lang, + call->enclosing_func_qn); } /* `p->open(fd)`: the registry does not split a callee on the arrow. * The object's type decides, as for the dot call handled further diff --git a/src/pipeline/pass_pkgmap.c b/src/pipeline/pass_pkgmap.c index badc27954e..e30a0c2312 100644 --- a/src/pipeline/pass_pkgmap.c +++ b/src/pipeline/pass_pkgmap.c @@ -1819,6 +1819,34 @@ bool cbm_import_symbol_fallback_allowed(CBMLanguage lang) { return lang != CBM_LANG_GO; } +/* A Python def or class declared inside a function body -- a closure, a + * decorator's `wrapper`, a test's local class -- is a local of that function: + * no import can name it. Its QN extends a Function/Method node's + * (proj.tests.mock_inputs.inner.wrapper.mock_getpass.getpass), which is how it + * is recognised; the symbol-name fallback bound `import getpass` to exactly + * that def. Mirrors cbm_registry_add_lang, which keeps the same + * symbols out of the call resolver's short-name index. */ +static bool import_candidate_is_python_local(const cbm_gbuf_t *gbuf, const cbm_gbuf_node_t *cand) { + const char *qn = cand->qualified_name; + const char *slash = cand->file_path ? strrchr(cand->file_path, '/') : NULL; + char scope[CBM_SZ_512]; + size_t len = qn ? strlen(qn) : 0; + if (!cand->file_path || len == 0 || len >= sizeof(scope) || + cbm_language_for_filename(slash ? slash + SKIP_ONE : cand->file_path) != CBM_LANG_PYTHON) { + return false; + } + memcpy(scope, qn, len + SKIP_ONE); + for (char *dot = strrchr(scope, '.'); dot; dot = strrchr(scope, '.')) { + *dot = '\0'; + const cbm_gbuf_node_t *outer = cbm_gbuf_find_by_qn(gbuf, scope); + if (outer && outer->label && + (strcmp(outer->label, "Function") == 0 || strcmp(outer->label, "Method") == 0)) { + return true; + } + } + return false; +} + static const char *path_leaf(const char *path) { const char *leaf = path; for (const char *p = path; p && *p; p++) { @@ -2516,6 +2544,10 @@ static const cbm_gbuf_node_t *resolve_import_strategies( !import_targetable_label(cand->label)) { continue; } + /* A Python def-local (#1548) is no importable name. */ + if (import_candidate_is_python_local(ctx->gbuf, cand)) { + continue; + } if (source_file_qn && cand->qualified_name && strcmp(cand->qualified_name, source_file_qn) == 0) { continue; /* self */ diff --git a/src/pipeline/pipeline.h b/src/pipeline/pipeline.h index b0ba5bc15b..e03c0dbe66 100644 --- a/src/pipeline/pipeline.h +++ b/src/pipeline/pipeline.h @@ -253,7 +253,11 @@ void cbm_registry_add(cbm_registry_t *r, const char *name, const char *qualified /* As cbm_registry_add, recording `lang`: the DETECTED language of the file * that defines the symbol (CBM_LANG_COUNT = unknown), which the cross-language - * veto (cbm_registry_name_guess_vetoed) checks against the caller. */ + * veto (cbm_registry_name_guess_vetoed) checks against the caller. A Python + * symbol declared inside a function body (see cbm_registry_qn_is_function_local) + * is registered for exact lookups only, never offered to the short-name + * strategies; every registration site seeds through this, so full and + * incremental indexes agree. */ void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qualified_name, const char *label, CBMLanguage lang); @@ -268,6 +272,17 @@ bool cbm_lang_resolution_compatible(CBMLanguage caller, CBMLanguage target); * CBM_LANG_COUNT for anything else. */ CBMLanguage cbm_registry_synthetic_path_language(const char *file_path); +/* True when a proper prefix of `qualified_name` is a registered Function or + * Method, i.e. the symbol was declared inside a function body. */ +bool cbm_registry_qn_is_function_local(const cbm_registry_t *r, const char *qualified_name); + +/* Python bare-name lexical lookup: the def or class named `callee_name` bound + * in the nearest function scope enclosing `scope_qn` (the caller's QN), + * skipping class bodies, stopping at the module. Strategy "lexical_scope"; + * qualified_name is NULL when no such local exists. */ +cbm_resolution_t cbm_registry_resolve_lexical(const cbm_registry_t *r, const char *callee_name, + const char *scope_qn); + /* Resolve a callee name using prioritized strategies. * import_map: NULL-terminated array of {local_name, resolved_qn} pairs, or NULL. * Returns result with qualified_name="" if unresolved. diff --git a/src/pipeline/pipeline_incremental.c b/src/pipeline/pipeline_incremental.c index 99df053e41..4d0d667f56 100644 --- a/src/pipeline/pipeline_incremental.c +++ b/src/pipeline/pipeline_incremental.c @@ -1327,7 +1327,10 @@ static bool incr_label_is_registry_symbol(const char *label) { /* Callback for cbm_gbuf_foreach_node: seed the registry with the existing * project's definition symbols so the resolver can match cross-file symbols * during incremental. Mirrors the full-index registry contents exactly so an - * incremental re-resolve picks the same nodes a full reindex would. */ + * incremental re-resolve picks the same nodes a full reindex would -- including + * a Python function's locals, which cbm_registry_add_lang keeps out of the + * short-name index there too (nodes load ORDER BY id and a file's defs are + * inserted parent first, so the enclosing function is seeded before its locals). */ /* Each symbol is seeded with its file's DETECTED language, exactly what the * full index recorded (pass_definitions.c / pass_parallel.c), so the cross- * language veto judges an incremental re-resolve the same way. Stored nodes carry no diff --git a/src/pipeline/pipeline_internal.h b/src/pipeline/pipeline_internal.h index fbae8044fb..0965e03980 100644 --- a/src/pipeline/pipeline_internal.h +++ b/src/pipeline/pipeline_internal.h @@ -260,6 +260,16 @@ cbm_resolution_t cbm_registry_resolve_call_lang(const cbm_registry_t *r, const c const char *module_qn, const char **import_map_keys, const char **import_map_vals, int import_map_count, CBMLanguage lang); +/* The registry step of BOTH call resolvers once the LSP has not answered: + * for Python, the def or class `callee_name` binds in the nearest def enclosing + * `scope_qn` (cbm_registry_resolve_lexical, "lexical_scope"); otherwise, or + * when there is no such local, cbm_registry_resolve_call_lang. */ +cbm_resolution_t cbm_registry_resolve_call_scoped(const cbm_registry_t *r, const char *callee_name, + const char *module_qn, + const char **import_map_keys, + const char **import_map_vals, + int import_map_count, CBMLanguage lang, + const char *scope_qn); #ifdef CBM_ENABLE_TEST_SEAMS void cbm_pxc_test_import_probe_fail_after(int successful_allocations); #endif diff --git a/src/pipeline/registry.c b/src/pipeline/registry.c index f53d1dd121..8a1122ddfe 100644 --- a/src/pipeline/registry.c +++ b/src/pipeline/registry.c @@ -62,6 +62,8 @@ const char *cbm_confidence_band(double score) { #define CONF_IMPORT_MAP_SUFFIX 0.85 /* Strategy 2: same_module — same file/package → high confidence */ #define CONF_SAME_MODULE 0.90 +/* Python lexical scope — a def bound in an enclosing function (LEGB) */ +#define CONF_LEXICAL_SCOPE 0.95 /* Strategy 3: unique_name — only one candidate project-wide */ #define CONF_UNIQUE_NAME 0.75 /* Strategy 4: suffix_match — multiple candidates, filtered */ @@ -1181,20 +1183,33 @@ static void index_under_name(cbm_registry_t *r, const char *key, const char *own } } -void cbm_registry_add(cbm_registry_t *r, const char *name, const char *qualified_name, - const char *label) { - cbm_registry_add_lang(r, name, qualified_name, label, CBM_LANG_COUNT); +static bool registry_label_is_callable(const char *label) { + return label && (strcmp(label, "Function") == 0 || strcmp(label, "Method") == 0); } -void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qualified_name, - const char *label, CBMLanguage lang) { - if (!r || !qualified_name || !label) { - return; +bool cbm_registry_qn_is_function_local(const cbm_registry_t *r, const char *qualified_name) { + char scope[CBM_SZ_512]; + size_t len = qualified_name ? strlen(qualified_name) : 0; + if (!r || len == 0 || len >= sizeof(scope)) { + return false; + } + memcpy(scope, qualified_name, len + SKIP_ONE); + for (char *dot = strrchr(scope, '.'); dot; dot = strrchr(scope, '.')) { + *dot = '\0'; + if (registry_label_is_callable(cbm_ht_get(r->exact, scope))) { + return true; + } } + return false; +} +/* Record `qualified_name` in the exact map. Returns the registry's owned copy + * of the QN, or NULL when it is already registered or cannot be recorded. */ +static const char *registry_add_exact(cbm_registry_t *r, const char *qualified_name, + const char *label) { /* Check for duplicate */ if (cbm_ht_get(r->exact, qualified_name)) { - return; + return NULL; } /* Intern the label (bounded set; linear scan is fine at this size). */ @@ -1211,14 +1226,45 @@ void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qual r->label_pool_n++; } if (!interned) { - return; /* pool exhausted (cannot happen with sane label sets) */ + return NULL; /* pool exhausted (cannot happen with sane label sets) */ } /* Store in exact map: QN → interned label. The key is the registry's ONE - * owned copy of the QN; by_name below borrows it (same lifetime) instead + * owned copy of the QN; by_name borrows it (same lifetime) instead * of a second strdup — this pair of copies was ~280 MB on the kernel. */ cbm_ht_set(r->exact, strdup(qualified_name), (void *)interned); - const char *owned_qn = cbm_ht_get_key(r->exact, qualified_name); + return cbm_ht_get_key(r->exact, qualified_name); +} + +void cbm_registry_add(cbm_registry_t *r, const char *name, const char *qualified_name, + const char *label) { + cbm_registry_add_lang(r, name, qualified_name, label, CBM_LANG_COUNT); +} + +/* Python nests defs: a def or class declared in a function body -- a closure, + * an app factory's route handler, a local class and its methods -- is a local + * of that function, and its QN extends the function's + * (proj.mod.create_app.index). No code outside the function can name it, so it + * stays out of the short-name index: the weak name strategies (unique_name, + * suffix_match) must not offer django's per-test `class MyModelAdmin` or a + * decorator's `def wrapper` to a caller in another function or file. The exact + * map still holds it, for LSP targets and cbm_registry_resolve_lexical. One + * file's defs register parent first, so the enclosing function is already in + * the exact map when its locals arrive. Gated to Python: probing every + * prefix of every symbol would cost a kernel index seconds for nothing. */ +void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qualified_name, + const char *label, CBMLanguage lang) { + if (!r || !qualified_name || !label) { + return; + } + if (lang == CBM_LANG_PYTHON && cbm_registry_qn_is_function_local(r, qualified_name)) { + (void)registry_add_exact(r, qualified_name, label); + return; + } + const char *owned_qn = registry_add_exact(r, qualified_name, label); + if (!owned_qn) { + return; + } /* Index the symbol under the QN's last dot segment, and additionally under * the name its caller passed when that segment carries a '#' fence. @@ -2123,6 +2169,69 @@ cbm_resolution_t cbm_registry_resolve_lineage(const cbm_registry_t *r, const cha import_map_count, module_qn ? strlen(module_qn) : 0, false); } +/* Python looks a bare name up in the calling def's own scope, then in each + * enclosing def's scope, and only then in the module and the builtins; a class + * body is never among them. So a bare `index()` in create_app, or `visit(...)` + * in a def nested in method Tree.visit, names the def bound in the nearest + * enclosing def -- create_app.index, Tree.visit.visit -- before anything a + * module-wide strategy would find. Walks scope_qn outward while its prefixes + * are registered defs or classes (skipping the classes) and stops at the + * module, which the other strategies own. Empty unless such a local exists. + * Uncached: the answer depends on the caller, not only on the name. */ +cbm_resolution_t cbm_registry_resolve_lexical(const cbm_registry_t *r, const char *callee_name, + const char *scope_qn) { + char scope[CBM_SZ_512]; + char candidate[CBM_SZ_512]; + size_t len = scope_qn ? strlen(scope_qn) : 0; + if (!r || !callee_name || !callee_name[0] || strchr(callee_name, '.') || + strstr(callee_name, "::") || len == 0 || len >= sizeof(scope)) { + return empty_result(); + } + memcpy(scope, scope_qn, len + SKIP_ONE); + for (;;) { + const char *label = cbm_ht_get(r->exact, scope); + if (!label) { + break; + } + if (registry_label_is_callable(label)) { + int n = snprintf(candidate, sizeof(candidate), "%s.%s", scope, callee_name); + const char *stored_key = + n > 0 && (size_t)n < sizeof(candidate) ? cbm_ht_get_key(r->exact, candidate) : NULL; + if (stored_key) { + return (cbm_resolution_t){stored_key, "lexical_scope", CONF_LEXICAL_SCOPE, + REG_RESOLVED}; + } + } + char *dot = strrchr(scope, '.'); + if (!dot) { + break; + } + *dot = '\0'; + } + return empty_result(); +} + +/* The registry step both call resolvers (pass_calls.c, pass_parallel.c) run + * once the LSP has not answered: a Python bare name bound in an enclosing def + * first (the LSP binds those locals itself; this covers the sites it did not + * resolve), then the language-aware chain. One entry point, so the two + * resolvers cannot diverge on the rule. */ +cbm_resolution_t cbm_registry_resolve_call_scoped(const cbm_registry_t *r, const char *callee_name, + const char *module_qn, + const char **import_map_keys, + const char **import_map_vals, + int import_map_count, CBMLanguage lang, + const char *scope_qn) { + if (lang == CBM_LANG_PYTHON) { + cbm_resolution_t lexical = cbm_registry_resolve_lexical(r, callee_name, scope_qn); + if (lexical.qualified_name && lexical.qualified_name[0]) { + return lexical; + } + } + return cbm_registry_resolve_call_lang(r, callee_name, module_qn, import_map_keys, + import_map_vals, import_map_count, lang); +} + /* ── Fuzzy Resolve ──────────────────────────────────────────────── */ /* Filter candidates by import reachability. Returns count of reachable. */ diff --git a/tests/test_extraction.c b/tests/test_extraction.c index e39d40a8ab..5f46a416d9 100644 --- a/tests/test_extraction.c +++ b/tests/test_extraction.c @@ -4723,6 +4723,298 @@ TEST(python_init_nested_module_qn) { PASS(); } +/* ── #1548: Python defs nested in a function body ─────────────────── */ + +/* How many defs carry exactly this QN. */ +static int pyn_def_count(const CBMFileResult *r, const char *qn) { + int n = 0; + for (int i = 0; i < r->defs.count; i++) { + const char *q = r->defs.items[i].qualified_name; + if (q && strcmp(q, qn) == 0) { + n++; + } + } + return n; +} + +/* The one def with this QN, or NULL when there is none (or more than one). */ +static const CBMDefinition *pyn_def(const CBMFileResult *r, const char *qn) { + if (pyn_def_count(r, qn) != 1) { + return NULL; + } + for (int i = 0; i < r->defs.count; i++) { + const char *q = r->defs.items[i].qualified_name; + if (q && strcmp(q, qn) == 0) { + return &r->defs.items[i]; + } + } + return NULL; +} + +/* Calls of `callee` whose enclosing function is exactly `caller_qn`. */ +static int pyn_calls(const CBMFileResult *r, const char *callee, const char *caller_qn) { + int n = 0; + for (int i = 0; i < r->calls.count; i++) { + const CBMCall *c = &r->calls.items[i]; + if (c->callee_name && strcmp(c->callee_name, callee) == 0 && c->enclosing_func_qn && + strcmp(c->enclosing_func_qn, caller_qn) == 0) { + n++; + } + } + return n; +} + +/* No def QN is minted twice: a nested def must not also surface as a + * module-level one, and walking a def body must not re-extract anything. */ +static bool pyn_defs_unique(const CBMFileResult *r) { + for (int i = 0; i < r->defs.count; i++) { + const char *q = r->defs.items[i].qualified_name; + if (q && pyn_def_count(r, q) != 1) { + return false; + } + } + return true; +} + +/* The issue's reproducer: `def nested()` inside `def factory()` was never + * extracted. It is now a Function qualified by the def it sits in, the same + * QN the Python LSP gives its calls. Module-level defs, and defs in a + * module-level `if` block, keep their QNs. */ +TEST(extract_python_nested_def_qualified_by_enclosing_def_issue1548) { + CBMFileResult *r = extract("def target():\n" + " return 0\n" + "\n" + "\n" + "def module_level():\n" + " return target()\n" + "\n" + "\n" + "if True:\n" + " def inside_if():\n" + " return target()\n" + "\n" + "\n" + "def factory():\n" + " def nested():\n" + " return target()\n" + " return nested\n" + "\n" + "\n" + "def outer():\n" + " def middle():\n" + " def inner():\n" + " return target()\n" + " return inner\n" + " return middle\n", + CBM_LANG_PYTHON, "t", "app.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + const CBMDefinition *nested = pyn_def(r, "t.app.factory.nested"); + ASSERT_NOT_NULL(nested); + ASSERT_STR_EQ(nested->label, "Function"); + ASSERT_EQ(nested->start_line, 15); + ASSERT_NOT_NULL(pyn_def(r, "t.app.outer.middle")); + ASSERT_NOT_NULL(pyn_def(r, "t.app.outer.middle.inner")); + /* Never minted flat, where it would collide with a module-level name. */ + ASSERT_EQ(pyn_def_count(r, "t.app.nested"), 0); + ASSERT_EQ(pyn_def_count(r, "t.app.inner"), 0); + /* Controls: unchanged. */ + ASSERT_NOT_NULL(pyn_def(r, "t.app.target")); + ASSERT_NOT_NULL(pyn_def(r, "t.app.module_level")); + ASSERT_NOT_NULL(pyn_def(r, "t.app.inside_if")); + ASSERT_NOT_NULL(pyn_def(r, "t.app.factory")); + ASSERT_TRUE(pyn_defs_unique(r)); + cbm_free_result(r); + PASS(); +} + +/* A call inside a nested def belongs to that def. It used to carry the flat + * `t.app.nested`, a QN no node has, so the edge was pinned on the file node: + * inbound traces of target() lost the caller. It is not the outer def's + * either. */ +TEST(extract_python_nested_def_calls_attributed_to_it_issue1548) { + CBMFileResult *r = extract("def target():\n" + " return 0\n" + "\n" + "def factory():\n" + " def nested():\n" + " return target()\n" + " return nested\n" + "\n" + "def outer():\n" + " def middle():\n" + " def inner():\n" + " return target()\n" + " return inner\n" + " return middle\n", + CBM_LANG_PYTHON, "t", "app.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + ASSERT_EQ(pyn_calls(r, "target", "t.app.factory.nested"), 1); + ASSERT_EQ(pyn_calls(r, "target", "t.app.outer.middle.inner"), 1); + ASSERT_EQ(pyn_calls(r, "target", "t.app.factory"), 0); + ASSERT_EQ(pyn_calls(r, "target", "t.app.nested"), 0); + ASSERT_EQ(pyn_calls(r, "target", "t.app.inner"), 0); + cbm_free_result(r); + PASS(); +} + +/* The app-factory shape: route handlers are decorated defs inside + * create_app. Each is extracted with its QN and carries the route its + * decorator declares, which is what mints the Route node and its HANDLES + * edge (pipeline_python_app_factory_routes_issue1548 checks the graph). */ +TEST(extract_python_decorated_nested_route_handler_issue1548) { + CBMFileResult *r = extract("from flask import Flask\n" + "\n" + "def create_app():\n" + " app = Flask(__name__)\n" + "\n" + " @app.route(\"/users/\")\n" + " def show_user(uid):\n" + " return render(uid)\n" + "\n" + " @app.get(\"/health\")\n" + " async def health():\n" + " return render(\"ok\")\n" + "\n" + " return app\n", + CBM_LANG_PYTHON, "t", "web.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + const CBMDefinition *show = pyn_def(r, "t.web.create_app.show_user"); + ASSERT_NOT_NULL(show); + ASSERT_STR_EQ(show->route_path, "/users/"); + ASSERT_NOT_NULL(show->decorators); + const CBMDefinition *health = pyn_def(r, "t.web.create_app.health"); + ASSERT_NOT_NULL(health); + ASSERT_STR_EQ(health->route_path, "/health"); + ASSERT_STR_EQ(health->route_method, "GET"); + ASSERT_EQ(pyn_calls(r, "render", "t.web.create_app.show_user"), 1); + ASSERT_EQ(pyn_calls(r, "render", "t.web.create_app.health"), 1); + ASSERT_TRUE(pyn_defs_unique(r)); + cbm_free_result(r); + PASS(); +} + +/* A class declared in a def is qualified by the def; its methods are Methods + * of it, and a call in such a method is the method's. */ +TEST(extract_python_nested_class_issue1548) { + CBMFileResult *r = extract("def create_app():\n" + " class Settings:\n" + " debug = False\n" + "\n" + " def value(self):\n" + " return load(self)\n" + "\n" + " return Settings().value()\n", + CBM_LANG_PYTHON, "t", "web.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + const CBMDefinition *cls = pyn_def(r, "t.web.create_app.Settings"); + ASSERT_NOT_NULL(cls); + ASSERT_STR_EQ(cls->label, "Class"); + const CBMDefinition *value = pyn_def(r, "t.web.create_app.Settings.value"); + ASSERT_NOT_NULL(value); + ASSERT_STR_EQ(value->label, "Method"); + ASSERT_STR_EQ(value->parent_class, "t.web.create_app.Settings"); + ASSERT_EQ(pyn_calls(r, "load", "t.web.create_app.Settings.value"), 1); + ASSERT_EQ(pyn_def_count(r, "t.web.Settings"), 0); + ASSERT_TRUE(pyn_defs_unique(r)); + cbm_free_result(r); + PASS(); +} + +/* A def nested in a method, named like the method (django's + * AbstractBaseUser.check_password holds a `def setter`; Tree.visit here holds + * a `def visit`). It used to share the method's QN, so its calls -- its + * recursion included -- were the method's. It is its own Function now: the + * recursive call and render() are its, only visit(node) is the method's, and + * the recursion seed flags the nested def. */ +TEST(extract_python_nested_def_named_like_its_method_issue1548) { + CBMFileResult *r = extract("class Tree:\n" + " def visit(self, node):\n" + " def visit(n):\n" + " if n:\n" + " visit(n.child)\n" + " render(n)\n" + "\n" + " visit(node)\n", + CBM_LANG_PYTHON, "t", "m.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + const CBMDefinition *method = pyn_def(r, "t.m.Tree.visit"); + ASSERT_NOT_NULL(method); + ASSERT_STR_EQ(method->label, "Method"); + const CBMDefinition *nested = pyn_def(r, "t.m.Tree.visit.visit"); + ASSERT_NOT_NULL(nested); + ASSERT_STR_EQ(nested->label, "Function"); + ASSERT_EQ(nested->start_line, 3); + ASSERT_EQ(pyn_calls(r, "visit", "t.m.Tree.visit.visit"), 1); + ASSERT_EQ(pyn_calls(r, "render", "t.m.Tree.visit.visit"), 1); + ASSERT_EQ(pyn_calls(r, "visit", "t.m.Tree.visit"), 1); + ASSERT_EQ(pyn_calls(r, "render", "t.m.Tree.visit"), 0); + ASSERT_TRUE(nested->is_recursive); + ASSERT_TRUE(pyn_defs_unique(r)); + cbm_free_result(r); + PASS(); +} + +/* Controls: a lambda and the comprehensions are expressions, not defs. They + * mint nothing, and the calls inside them stay the enclosing def's. */ +TEST(extract_python_lambda_and_comprehension_stay_in_enclosing_def_issue1548) { + CBMFileResult *r = extract("def work(items):\n" + " double = lambda x: scale(x)\n" + " squares = [scale(i) for i in items]\n" + " table = {k: scale(k) for k in items}\n" + " return double(squares), table\n", + CBM_LANG_PYTHON, "t", "c.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + ASSERT_NOT_NULL(pyn_def(r, "t.c.work")); + for (int i = 0; i < r->defs.count; i++) { + const char *q = r->defs.items[i].qualified_name; + ASSERT_FALSE(q && strncmp(q, "t.c.work.", strlen("t.c.work.")) == 0); + } + ASSERT_EQ(pyn_calls(r, "scale", "t.c.work"), 3); + ASSERT_TRUE(pyn_defs_unique(r)); + cbm_free_result(r); + PASS(); +} + +/* Records other than calls take their enclosing function from the same + * unified-walk scope frame as the calls. The nested def's env read carries its + * QN; the outer def's reads, before and after the nested def, keep the outer + * QN once its frame is popped. */ +TEST(extract_python_nested_def_env_access_attributed_to_it_issue1548) { + CBMFileResult *r = extract("import os\n" + "\n" + "def outer():\n" + " token = os.environ[\"OUTER_TOKEN\"]\n" + "\n" + " def inner():\n" + " return os.environ[\"INNER_TOKEN\"]\n" + "\n" + " return inner, token, os.environ[\"OUTER_AGAIN\"]\n", + CBM_LANG_PYTHON, "t", "e.py"); + ASSERT_NOT_NULL(r); + ASSERT_FALSE(r->has_error); + int seen = 0; + for (int i = 0; i < r->env_accesses.count; i++) { + const CBMEnvAccess *ea = &r->env_accesses.items[i]; + ASSERT_NOT_NULL(ea->enclosing_func_qn); + if (strcmp(ea->env_key, "INNER_TOKEN") == 0) { + ASSERT_STR_EQ(ea->enclosing_func_qn, "t.e.outer.inner"); + seen++; + } else { + ASSERT_STR_EQ(ea->enclosing_func_qn, "t.e.outer"); + seen++; + } + } + ASSERT_EQ(seen, 3); + cbm_free_result(r); + PASS(); +} + TEST(js_index_module_qn_not_collide_with_folder) { /* Same bug for JS/TS index.ts files */ CBMFileResult *r = extract("export function App() { return null; }\n", CBM_LANG_TYPESCRIPT, @@ -12551,6 +12843,13 @@ SUITE(extraction) { /* __init__.py / index.ts Module QN collision regression */ RUN_TEST(python_init_module_qn_not_collide_with_folder); RUN_TEST(python_init_nested_module_qn); + RUN_TEST(extract_python_nested_def_qualified_by_enclosing_def_issue1548); + RUN_TEST(extract_python_nested_def_calls_attributed_to_it_issue1548); + RUN_TEST(extract_python_decorated_nested_route_handler_issue1548); + RUN_TEST(extract_python_nested_class_issue1548); + RUN_TEST(extract_python_nested_def_named_like_its_method_issue1548); + RUN_TEST(extract_python_lambda_and_comprehension_stay_in_enclosing_def_issue1548); + RUN_TEST(extract_python_nested_def_env_access_attributed_to_it_issue1548); RUN_TEST(js_index_module_qn_not_collide_with_folder); RUN_TEST(python_regular_module_qn_unchanged); RUN_TEST(extract_java_method_annotations_issue382); diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index b52513acee..36ef5667ef 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -14188,6 +14188,378 @@ static void teardown_lang_repo(void) { g_lang_tmpdir[0] = '\0'; } +/* ── #1548: Python defs nested in a function body ─────────────────── */ + +/* The QN of `rel` in `project` ("web.create_app.health"). Route QNs + * ("__route__GET__/health") carry no project and are taken as they are. */ +static void pyn_qn(char *buf, size_t cap, const char *project, const char *rel) { + if (strncmp(rel, "__route__", strlen("__route__")) == 0) { + snprintf(buf, cap, "%s", rel); + } else { + snprintf(buf, cap, "%s.%s", project, rel); + } +} + +/* True iff an edge of `type` runs from node `src_rel` to node `tgt_rel`. */ +static bool pyn_edge(cbm_store_t *s, const char *project, const char *src_rel, const char *tgt_rel, + const char *type) { + char sq[512]; + char tq[512]; + pyn_qn(sq, sizeof(sq), project, src_rel); + pyn_qn(tq, sizeof(tq), project, tgt_rel); + cbm_node_t src = {0}; + cbm_node_t tgt = {0}; + bool found = false; + if (cbm_store_find_node_by_qn(s, project, sq, &src) == CBM_STORE_OK && + cbm_store_find_node_by_qn(s, project, tq, &tgt) == CBM_STORE_OK) { + cbm_edge_t *edges = NULL; + int ec = 0; + cbm_store_find_edges_by_source_type(s, src.id, type, &edges, &ec); + for (int i = 0; i < ec && !found; i++) { + found = edges[i].target_id == tgt.id; + } + if (edges) { + cbm_store_free_edges(edges, ec); + } + } + cbm_node_free_fields(&src); + cbm_node_free_fields(&tgt); + return found; +} + +/* The fixtures of the #1548 pipeline tests. */ +static const char pyn_web_py[] = "from flask import Flask\n" + "\n" + "def load_user(uid):\n" + " return uid\n" + "\n" + "def render(name):\n" + " return name\n" + "\n" + "def create_app():\n" + " app = Flask(__name__)\n" + "\n" + " @app.route(\"/users/\")\n" + " def show_user(uid):\n" + " return render(load_user(uid))\n" + "\n" + " @app.get(\"/health\")\n" + " def health():\n" + " return render(\"ok\")\n" + "\n" + " def ping():\n" + " return render(\"pong\")\n" + "\n" + " app.add_url_rule(\"/ping\", \"ping\", ping)\n" + " return app\n"; +/* A nested def named like its method, and a local helper whose name exists + * nowhere else -- b.py calls that name without binding it. */ +static const char pyn_a_py[] = "class Tree:\n" + " def visit(self, node):\n" + " def visit(n):\n" + " if n:\n" + " visit(n.child)\n" + "\n" + " visit(node)\n" + "\n" + "\n" + "def outer():\n" + " def helper_only_here():\n" + " return 1\n" + "\n" + " return helper_only_here()\n"; +static const char pyn_b_py[] = "def elsewhere():\n" + " return helper_only_here()\n"; +/* Two defs, each with its own local class of the same name. */ +static const char pyn_c_py[] = "def make_forms():\n" + " class LocalForm:\n" + " def clean(self):\n" + " return 1\n" + "\n" + " form = LocalForm()\n" + " return form.clean()\n" + "\n" + "\n" + "def make_other_forms():\n" + " class LocalForm:\n" + " def clean(self):\n" + " return 2\n" + "\n" + " form = LocalForm()\n" + " return form.clean()\n"; +/* A local def named like a stdlib function another module imports. */ +static const char pyn_x_py[] = "def outer():\n" + " def getpass(prompt):\n" + " return prompt\n" + "\n" + " return getpass(\"?\")\n"; +static const char pyn_y_py[] = "from getpass import getpass\n" + "\n" + "\n" + "def ask():\n" + " return getpass(\"password: \")\n"; + +/* The issue's headline: an app factory's route handlers are defs inside + * create_app. They were never extracted, so the HTTP layer vanished -- no + * HANDLES, and load_user()'s only caller was pinned on the file node. */ +TEST(pipeline_python_app_factory_routes_issue1548) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_factory_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "web.py", pyn_web_py); + + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/factory.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + const char *project = cbm_pipeline_project_name(p); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + + ASSERT_TRUE( + pyn_edge(s, project, "web.create_app.show_user", "__route__ANY__/users/{}", "HANDLES")); + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.health", "__route__GET__/health", "HANDLES")); + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.show_user", "web.load_user", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.health", "web.render", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "web.py.__file__", "web.load_user", "CALLS")); + /* A call-registered handler: add_url_rule names create_app's local ping. */ + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.ping", "__route__ANY__/ping", "HANDLES")); + + cbm_store_close(s); + cbm_pipeline_free(p); + th_rmtree(tmp); + PASS(); +} + +/* A def nested in a function is a local of it (Python LEGB). A bare call + * resolves to the local bound in the nearest enclosing function: visit(node) + * in Tree.visit reaches the nested visit, whose own recursion stays a + * self-call instead of landing on the method. And no caller outside the + * function can reach the local by name -- b.py's unbound helper_only_here() + * must not be matched to it by a short-name strategy. */ +TEST(pipeline_python_nested_def_is_function_local_issue1548) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_local_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "a.py", pyn_a_py); + write_temp_file(tmp, "b.py", pyn_b_py); + + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/local.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + const char *project = cbm_pipeline_project_name(p); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + + ASSERT_TRUE(pyn_edge(s, project, "a.Tree.visit", "a.Tree.visit.visit", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "a.Tree.visit.visit", "a.Tree.visit", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "a.outer", "a.outer.helper_only_here", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "b.elsewhere", "a.outer.helper_only_here", "CALLS")); + + cbm_store_close(s); + cbm_pipeline_free(p); + th_rmtree(tmp); + PASS(); +} + +/* Two defs each declare `class LocalForm`. Each constructor call reaches its + * own def's class, and a method call on the instance never reaches the other + * def's class: the Python LSP must not bind a local class module-wide by its + * short name (every same-named local class collapsed onto one). */ +TEST(pipeline_python_local_classes_stay_in_their_def_issue1548) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_localcls_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "c.py", pyn_c_py); + + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/localcls.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + const char *project = cbm_pipeline_project_name(p); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + + ASSERT_TRUE(pyn_edge(s, project, "c.make_forms", "c.make_forms.LocalForm", "CALLS")); + ASSERT_TRUE( + pyn_edge(s, project, "c.make_other_forms", "c.make_other_forms.LocalForm", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "c.make_forms", "c.make_other_forms.LocalForm", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "c.make_other_forms", "c.make_forms.LocalForm", "CALLS")); + ASSERT_FALSE( + pyn_edge(s, project, "c.make_forms", "c.make_other_forms.LocalForm.clean", "CALLS")); + ASSERT_FALSE( + pyn_edge(s, project, "c.make_other_forms", "c.make_forms.LocalForm.clean", "CALLS")); + + cbm_store_close(s); + cbm_pipeline_free(p); + th_rmtree(tmp); + PASS(); +} + +/* No import can name a function's local. The symbol-name import fallback + * bound `from getpass import getpass` (the stdlib module, not in the project) + * to x.outer's local `def getpass`, and ask()'s call followed the import. */ +TEST(pipeline_python_import_never_binds_a_local_def_issue1548) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_localimp_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "x.py", pyn_x_py); + write_temp_file(tmp, "y.py", pyn_y_py); + + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/localimp.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + const char *project = cbm_pipeline_project_name(p); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + + ASSERT_TRUE(pyn_edge(s, project, "x.outer", "x.outer.getpass", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "y.py.__file__", "x.outer.getpass", "IMPORTS")); + ASSERT_FALSE(pyn_edge(s, project, "y.ask", "x.outer.getpass", "CALLS")); + + cbm_store_close(s); + cbm_pipeline_free(p); + th_rmtree(tmp); + PASS(); +} + +/* The same four fixtures through the parallel path: >= 50 files and + * CBM_WORKERS=4 force pass_parallel.c and the cross-file Python LSP, which + * register, resolve and bind through their own code. */ +TEST(pipeline_python_nested_defs_parallel_issue1548) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_nested_par_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "web.py", pyn_web_py); + write_temp_file(tmp, "a.py", pyn_a_py); + write_temp_file(tmp, "b.py", pyn_b_py); + write_temp_file(tmp, "c.py", pyn_c_py); + write_temp_file(tmp, "x.py", pyn_x_py); + write_temp_file(tmp, "y.py", pyn_y_py); + for (int i = 0; i < 52; i++) { + char name[64]; + char body[128]; + snprintf(name, sizeof(name), "filler%d.py", i); + snprintf(body, sizeof(body), "def filler%d():\n return %d\n", i, i); + write_temp_file(tmp, name, body); + } + + char *old_workers = getenv("CBM_WORKERS"); + char *saved = old_workers ? strdup(old_workers) : NULL; + cbm_setenv("CBM_WORKERS", "4", 1); + + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/nested_par.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + const char *project = cbm_pipeline_project_name(p); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + + ASSERT_TRUE( + pyn_edge(s, project, "web.create_app.show_user", "__route__ANY__/users/{}", "HANDLES")); + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.health", "__route__GET__/health", "HANDLES")); + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.show_user", "web.load_user", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "web.create_app.ping", "__route__ANY__/ping", "HANDLES")); + ASSERT_TRUE(pyn_edge(s, project, "a.Tree.visit", "a.Tree.visit.visit", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "a.Tree.visit.visit", "a.Tree.visit", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "a.outer", "a.outer.helper_only_here", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "b.elsewhere", "a.outer.helper_only_here", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "c.make_forms", "c.make_forms.LocalForm", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "c.make_forms", "c.make_other_forms.LocalForm", "CALLS")); + ASSERT_FALSE( + pyn_edge(s, project, "c.make_forms", "c.make_other_forms.LocalForm.clean", "CALLS")); + ASSERT_FALSE( + pyn_edge(s, project, "c.make_other_forms", "c.make_forms.LocalForm.clean", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "x.outer", "x.outer.getpass", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "y.py.__file__", "x.outer.getpass", "IMPORTS")); + ASSERT_FALSE(pyn_edge(s, project, "y.ask", "x.outer.getpass", "CALLS")); + + cbm_store_close(s); + cbm_pipeline_free(p); + if (saved) { + cbm_setenv("CBM_WORKERS", saved, 1); + free(saved); + } else { + cbm_unsetenv("CBM_WORKERS"); + } + th_rmtree(tmp); + PASS(); +} + +/* An incremental re-index seeds its registry from the stored graph, not from + * extraction. It must keep a function's locals out of the short-name index + * exactly as the full index does: re-resolving the touched b.py must not bind + * its unbound helper_only_here() to a.outer's local. */ +TEST(pipeline_python_nested_def_local_after_incremental_issue1548) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_local_incr_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "a.py", pyn_a_py); + write_temp_file(tmp, "b.py", pyn_b_py); + /* An extensionless script is Python by its shebang: its locals are locals + * in the incremental seed too, not only in the full index. */ + write_temp_file(tmp, "tool", + "#!/usr/bin/env python3\n" + "def run():\n" + " def helper_in_tool():\n" + " return 1\n" + "\n" + " return helper_in_tool()\n"); + write_temp_file(tmp, "d.py", + "def elsewhere_too():\n" + " return helper_in_tool()\n"); + + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/local_incr.db", tmp); + cbm_pipeline_t *p1 = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p1); + ASSERT_EQ(cbm_pipeline_run(p1), 0); + cbm_pipeline_free(p1); + + char b_path[512]; + snprintf(b_path, sizeof(b_path), "%s/b.py", tmp); + ASSERT_EQ(th_append_file(b_path, "\n# touched: re-resolve b.py incrementally\n"), 0); + snprintf(b_path, sizeof(b_path), "%s/d.py", tmp); + ASSERT_EQ(th_append_file(b_path, "\n# touched: re-resolve d.py incrementally\n"), 0); + + cbm_pipeline_t *p2 = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p2); + ASSERT_EQ(cbm_pipeline_run(p2), 0); + const char *project = cbm_pipeline_project_name(p2); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + + ASSERT_TRUE(pyn_edge(s, project, "a.outer", "a.outer.helper_only_here", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "b.elsewhere", "a.outer.helper_only_here", "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, "tool.run", "tool.run.helper_in_tool", "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, "d.elsewhere_too", "tool.run.helper_in_tool", "CALLS")); + + cbm_store_close(s); + cbm_pipeline_free(p2); + th_rmtree(tmp); + PASS(); +} + TEST(pipeline_python_project) { /* Port of TestPipelinePythonProject */ const char *files[] = {"main.py", "utils.py"}; @@ -23681,6 +24053,12 @@ SUITE(pipeline) { RUN_TEST(pipeline_c_variants_on_sequential_and_parallel_paths_c1); RUN_TEST(pipeline_cpp_overloads_are_listed_as_variants_c1); RUN_TEST(pipeline_semantic_version_3_index_is_rebuilt_in_full_c1); + RUN_TEST(pipeline_python_app_factory_routes_issue1548); + RUN_TEST(pipeline_python_nested_def_is_function_local_issue1548); + RUN_TEST(pipeline_python_local_classes_stay_in_their_def_issue1548); + RUN_TEST(pipeline_python_import_never_binds_a_local_def_issue1548); + RUN_TEST(pipeline_python_nested_defs_parallel_issue1548); + RUN_TEST(pipeline_python_nested_def_local_after_incremental_issue1548); RUN_TEST(pipeline_header_include_target_is_independent_of_registration_order); RUN_TEST(pipeline_imports_multi_symbol_edges); RUN_TEST(pipeline_go_cross_package_call); diff --git a/tests/test_registry.c b/tests/test_registry.c index 45fd6db147..055de4d781 100644 --- a/tests/test_registry.c +++ b/tests/test_registry.c @@ -1962,6 +1962,96 @@ TEST(registry_python_scope_qualified_terminal_bucket_uncapped) { bool seeded = cbm_registry_size(r) == 302; cbm_registry_free(r); ASSERT_TRUE(shape && seeded && resolved); +/* #1548: a Python def or class declared in a function body is a local of it. + * It is registered for exact lookups but kept out of the short-name index, so + * no weak strategy offers it to a caller elsewhere. A class nested in a class + * (proj.m.Outer.Inner) is an attribute, not a local, and stays nameable; so + * does the same QN shape from another language. */ +TEST(registry_python_function_local_is_exact_only_issue1548) { + cbm_registry_t *r = cbm_registry_new(); + cbm_registry_add_lang(r, "create_app", "proj.web.create_app", "Function", + CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "index", "proj.web.create_app.index", "Function", + CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "Settings", "proj.web.create_app.Settings", "Class", + CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "value", "proj.web.create_app.Settings.value", "Method", + CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "Outer", "proj.web.Outer", "Class", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "Inner", "proj.web.Outer.Inner", "Class", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "f", "proj.ts.f", "Function", CBM_LANG_TYPESCRIPT); + cbm_registry_add_lang(r, "g", "proj.ts.f.g", "Function", CBM_LANG_TYPESCRIPT); + + static const char *const local[] = {"proj.web.create_app.index", "proj.web.create_app.Settings", + "proj.web.create_app.Settings.value"}; + for (int i = 0; i < 3; i++) { + ASSERT_TRUE(cbm_registry_exists(r, local[i])); + ASSERT_TRUE(cbm_registry_qn_is_function_local(r, local[i])); + const char *simple = strrchr(local[i], '.') + 1; + const char **out = NULL; + int count = -1; + cbm_registry_find_by_name(r, simple, &out, &count); + ASSERT_EQ(count, 0); + } + static const char *const nameable[][2] = {{"create_app", "proj.web.create_app"}, + {"Inner", "proj.web.Outer.Inner"}, + {"g", "proj.ts.f.g"}}; + for (int i = 0; i < 3; i++) { + const char **out = NULL; + int count = 0; + cbm_registry_find_by_name(r, nameable[i][0], &out, &count); + ASSERT_EQ(count, 1); + ASSERT_STR_EQ(out[0], nameable[i][1]); + } + ASSERT_FALSE(cbm_registry_qn_is_function_local(r, "proj.web.Outer.Inner")); + /* A weak strategy cannot reach the local; before, index() anywhere in + * the project resolved to it by unique_name. */ + cbm_resolution_t res = cbm_registry_resolve(r, "index", "proj.other", NULL, NULL, 0); + ASSERT_NULL(res.qualified_name); + cbm_registry_free(r); + PASS(); +} + +/* Python LEGB for bare names: the caller's own def first, then each enclosing + * def outward; a class body is not a scope its methods see; the module is left + * to the other strategies. A dotted callee is not a bare name. */ +TEST(registry_resolve_lexical_walks_enclosing_defs_issue1548) { + cbm_registry_t *r = cbm_registry_new(); + cbm_registry_add_lang(r, "Tree", "proj.m.Tree", "Class", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "visit", "proj.m.Tree.visit", "Method", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "visit", "proj.m.Tree.visit.visit", "Function", + CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "helper", "proj.m.Tree.helper", "Method", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "factory", "proj.m.factory", "Function", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "helper", "proj.m.factory.helper", "Function", + CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "Local", "proj.m.factory.Local", "Class", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "run", "proj.m.factory.Local.run", "Method", CBM_LANG_PYTHON); + cbm_registry_add_lang(r, "top", "proj.m.top", "Function", CBM_LANG_PYTHON); + + /* In method Tree.visit, `visit` is its nested def; so it is inside that + * nested def (its own recursion). */ + cbm_resolution_t res = cbm_registry_resolve_lexical(r, "visit", "proj.m.Tree.visit"); + ASSERT_STR_EQ(res.qualified_name, "proj.m.Tree.visit.visit"); + ASSERT_STR_EQ(res.strategy, "lexical_scope"); + res = cbm_registry_resolve_lexical(r, "visit", "proj.m.Tree.visit.visit"); + ASSERT_STR_EQ(res.qualified_name, "proj.m.Tree.visit.visit"); + /* The class body is skipped: Tree.helper is not a bare name in a method. */ + res = cbm_registry_resolve_lexical(r, "helper", "proj.m.Tree.visit.visit"); + ASSERT_NULL(res.qualified_name); + /* A method of a local class sees the def enclosing its class. */ + res = cbm_registry_resolve_lexical(r, "helper", "proj.m.factory.Local.run"); + ASSERT_STR_EQ(res.qualified_name, "proj.m.factory.helper"); + res = cbm_registry_resolve_lexical(r, "Local", "proj.m.factory"); + ASSERT_STR_EQ(res.qualified_name, "proj.m.factory.Local"); + /* Module-level names and dotted callees are not this strategy's. */ + res = cbm_registry_resolve_lexical(r, "top", "proj.m.factory"); + ASSERT_NULL(res.qualified_name); + res = cbm_registry_resolve_lexical(r, "self.helper", "proj.m.factory"); + ASSERT_NULL(res.qualified_name); + res = cbm_registry_resolve_lexical(r, "helper", "proj.m"); + ASSERT_NULL(res.qualified_name); + cbm_registry_free(r); PASS(); } @@ -2069,4 +2159,6 @@ SUITE(registry) { RUN_TEST(python_import_binding_contradicts_only_foreign_chains); RUN_TEST(python_external_base_contradicts_django_shapes); RUN_TEST(weak_call_guards_share_one_drop_list); + RUN_TEST(registry_python_function_local_is_exact_only_issue1548); + RUN_TEST(registry_resolve_lexical_walks_enclosing_defs_issue1548); } From 9cdae25c12f6190f362b38aadc9a68305afac969 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sat, 17 Oct 2026 22:52:43 +0200 Subject: [PATCH 2/9] fix(python): bind a function's nested defs in the LSP; lexical_scope after it Re-port onto current main (agreed with lane L7): - The Python LSP binds every def/class a function body defines as a local before walking the body (LEGB): an undecorated def as an exact callable alias to its nested QN, anything else as a hard shadow, so no call falls through to a same-named module symbol. Defs inside the body's compound statements count too, and their bodies are now walked. - Classes declared in a function are walked under the function's QN (py_process_class_in), so super()/self calls in their methods resolve. - cbm_registry_resolve_lexical moves AFTER the LSP override: both resolvers call one entry point, cbm_registry_resolve_call_scoped, at the same place (before the language-aware chain). It never uses the per-file resolve cache. - Main's cbm_registry_add_lang carries the def-local exclusion, so every registration site (full, parallel, incremental seed) applies it; the separate cbm_registry_add_for_language and the second incremental language derivation are dropped. - The pass_pkgmap.c def-local import skip is its own continue. Tests: local-vs-module shadowing in both orders, a decorated local, and a function-local class's super()/self calls, sequential and parallel. Signed-off-by: Martin Vogel --- internal/cbm/lsp/py_lsp.c | 16 +++- tests/test_pipeline.c | 155 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+), 2 deletions(-) diff --git a/internal/cbm/lsp/py_lsp.c b/internal/cbm/lsp/py_lsp.c index cbad720fa3..cfc5499d4c 100644 --- a/internal/cbm/lsp/py_lsp.c +++ b/internal/cbm/lsp/py_lsp.c @@ -247,6 +247,8 @@ static void py_register_dict_literal(PyLSPContext *ctx, const char *name, TSNode static TSNode py_lookup_lambda(PyLSPContext *ctx, const char *name); static const char *py_lookup_dict_dispatch(PyLSPContext *ctx, const char *var, const char *key); static void py_process_function(PyLSPContext *ctx, TSNode func_node, const char *container_qn); +static void py_process_class(PyLSPContext *ctx, TSNode class_node); +static void py_process_class_in(PyLSPContext *ctx, TSNode class_node, const char *container_qn); typedef enum { PY_LOCAL_DEFS_BIND, PY_LOCAL_DEFS_PROCESS } PyLocalDefsMode; static void py_visit_local_defs(PyLSPContext *ctx, TSNode block, PyLocalDefsMode mode, int depth); @@ -4443,6 +4445,8 @@ static void py_visit_local_defs(PyLSPContext *ctx, TSNode block, PyLocalDefsMode } else if (strcmp(ck, "class_definition") == 0) { if (mode == PY_LOCAL_DEFS_BIND) py_bind_local_def(ctx, def, true); + else + py_process_class_in(ctx, def, ctx->enclosing_func_qn); } else if (py_is_statement_block(ck)) { py_visit_local_defs(ctx, c, mode, depth + 1); } @@ -4614,7 +4618,10 @@ static void py_exec_class_definition(PyLSPContext *ctx, TSNode class_node) { py_scope_restore(ctx, saved_scope); } -static void py_process_class(PyLSPContext *ctx, TSNode class_node) { +/* Walk a class's methods with the class qualified under `container_qn`: the + * module for a module-level class, the enclosing def for a class declared in + * a function body (#1548: extraction gives it the def's QN as prefix). */ +static void py_process_class_in(PyLSPContext *ctx, TSNode class_node, const char *container_qn) { if (!ctx || ctx->eval_failure) return; TSNode name_node = ts_node_child_by_field_name(class_node, "name", 4); @@ -4625,7 +4632,8 @@ static void py_process_class(PyLSPContext *ctx, TSNode class_node) { return; const char *prev_class = ctx->enclosing_class_qn; - ctx->enclosing_class_qn = cbm_arena_sprintf(ctx->arena, "%s.%s", ctx->module_qn, cname); + ctx->enclosing_class_qn = cbm_arena_sprintf( + ctx->arena, "%s.%s", container_qn ? container_qn : ctx->module_qn, cname); TSNode body = ts_node_child_by_field_name(class_node, "body", 4); if (!ts_node_is_null(body)) { @@ -4699,6 +4707,10 @@ static void py_process_class(PyLSPContext *ctx, TSNode class_node) { ctx->enclosing_class_qn = prev_class; } +static void py_process_class(PyLSPContext *ctx, TSNode class_node) { + py_process_class_in(ctx, class_node, NULL); +} + /* Module bindings are replayed in source order before deferred function bodies * are analyzed. Class definitions always replace an earlier callable identity; * the registry decides whether the resulting class type is known precisely. */ diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index 36ef5667ef..3141ab32d9 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -14560,6 +14560,159 @@ TEST(pipeline_python_nested_def_local_after_incremental_issue1548) { PASS(); } +/* As pyn_edge, and the edge's properties also contain `fragment`. */ +static bool pyn_edge_with(cbm_store_t *s, const char *project, const char *src_rel, + const char *tgt_rel, const char *type, const char *fragment) { + char sq[512]; + char tq[512]; + pyn_qn(sq, sizeof(sq), project, src_rel); + pyn_qn(tq, sizeof(tq), project, tgt_rel); + cbm_node_t src = {0}; + cbm_node_t tgt = {0}; + bool found = false; + if (cbm_store_find_node_by_qn(s, project, sq, &src) == CBM_STORE_OK && + cbm_store_find_node_by_qn(s, project, tq, &tgt) == CBM_STORE_OK) { + cbm_edge_t *edges = NULL; + int ec = 0; + cbm_store_find_edges_by_source_type(s, src.id, type, &edges, &ec); + for (int i = 0; i < ec && !found; i++) { + found = edges[i].target_id == tgt.id && edges[i].properties_json && + strstr(edges[i].properties_json, fragment); + } + if (edges) { + cbm_store_free_edges(edges, ec); + } + } + cbm_node_free_fields(&src); + cbm_node_free_fields(&tgt); + return found; +} + +/* A local def shadows a module def of the same name (Python LEGB), in both + * orders of the two callers within one file: the registry's per-file resolve + * cache is keyed by the callee name only, so a caller-dependent answer must + * never come from it. The undecorated local is bound by the Python LSP itself + * (an lsp_* edge); a decorated one is no exact callable value, so the LSP + * shadows it and the registry's lexical_scope step binds it. */ +static const char pyn_shadow_local_first_py[] = "def helper():\n" + " return 0\n" + "\n" + "def uses_local():\n" + " def helper():\n" + " return 1\n" + " return helper()\n" + "\n" + "def uses_module():\n" + " return helper()\n" + "\n" + "def uses_decorated():\n" + " @staticmethod\n" + " def helper():\n" + " return 2\n" + " return helper()\n"; +static const char pyn_shadow_module_first_py[] = "def helper():\n" + " return 0\n" + "\n" + "def uses_module():\n" + " return helper()\n" + "\n" + "def uses_local():\n" + " if True:\n" + " def helper():\n" + " return 1\n" + " return helper()\n"; + +/* A class declared in a def is walked by the LSP under the def's QN: super() + * and self calls in its methods resolve through the class's own bases. */ +static const char pyn_local_class_methods_py[] = "class Base:\n" + " def greet(self):\n" + " return 1\n" + "\n" + "def factory():\n" + " class Local(Base):\n" + " def greet(self):\n" + " return super().greet()\n" + "\n" + " def other(self):\n" + " return self.greet()\n" + " return Local\n"; + +static int pyn_assert_shadowing(cbm_store_t *s, const char *project) { + ASSERT_TRUE(pyn_edge_with(s, project, "sc.factory.Local.greet", "sc.Base.greet", "CALLS", + "\"strategy\":\"lsp_super\"")); + ASSERT_TRUE(pyn_edge_with(s, project, "sc.factory.Local.other", "sc.factory.Local.greet", + "CALLS", "\"strategy\":\"lsp_")); + const char *files[] = {"sa", "sb"}; + for (int f = 0; f < 2; f++) { + char local_caller[64], local_def[64], module_caller[64], module_def[64]; + snprintf(local_caller, sizeof(local_caller), "%s.uses_local", files[f]); + snprintf(local_def, sizeof(local_def), "%s.uses_local.helper", files[f]); + snprintf(module_caller, sizeof(module_caller), "%s.uses_module", files[f]); + snprintf(module_def, sizeof(module_def), "%s.helper", files[f]); + ASSERT_TRUE(pyn_edge_with(s, project, local_caller, local_def, "CALLS", + "\"strategy\":\"lsp_")); + ASSERT_FALSE(pyn_edge(s, project, local_caller, module_def, "CALLS")); + ASSERT_TRUE(pyn_edge(s, project, module_caller, module_def, "CALLS")); + ASSERT_FALSE(pyn_edge(s, project, module_caller, local_def, "CALLS")); + } + ASSERT_TRUE(pyn_edge_with(s, project, "sa.uses_decorated", "sa.uses_decorated.helper", + "CALLS", "\"strategy\":\"lexical_scope\"")); + ASSERT_FALSE(pyn_edge(s, project, "sa.uses_decorated", "sa.helper", "CALLS")); + return 0; +} + +static int pyn_run_shadowing(int fillers) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_shadow_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "sa.py", pyn_shadow_local_first_py); + write_temp_file(tmp, "sb.py", pyn_shadow_module_first_py); + write_temp_file(tmp, "sc.py", pyn_local_class_methods_py); + for (int i = 0; i < fillers; i++) { + char name[64]; + char body[128]; + snprintf(name, sizeof(name), "filler%d.py", i); + snprintf(body, sizeof(body), "def filler%d():\n return %d\n", i, i); + write_temp_file(tmp, name, body); + } + char *old_workers = getenv("CBM_WORKERS"); + char *saved = old_workers ? strdup(old_workers) : NULL; + if (fillers > 0) { + cbm_setenv("CBM_WORKERS", "4", 1); + } + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/shadow.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + int rc = p ? cbm_pipeline_run(p) : -1; + cbm_store_t *s = rc == 0 ? cbm_store_open_path(db_path) : NULL; + int verdict = s ? pyn_assert_shadowing(s, cbm_pipeline_project_name(p)) : 1; + if (s) { + cbm_store_close(s); + } + if (p) { + cbm_pipeline_free(p); + } + if (saved) { + cbm_setenv("CBM_WORKERS", saved, 1); + free(saved); + } else { + cbm_unsetenv("CBM_WORKERS"); + } + th_rmtree(tmp); + ASSERT_EQ(rc, 0); + return verdict; +} + +TEST(pipeline_python_local_def_shadows_module_def_issue1548) { + return pyn_run_shadowing(0); +} + +TEST(pipeline_python_local_def_shadows_module_def_parallel_issue1548) { + return pyn_run_shadowing(52); +} + TEST(pipeline_python_project) { /* Port of TestPipelinePythonProject */ const char *files[] = {"main.py", "utils.py"}; @@ -24059,6 +24212,8 @@ SUITE(pipeline) { RUN_TEST(pipeline_python_import_never_binds_a_local_def_issue1548); RUN_TEST(pipeline_python_nested_defs_parallel_issue1548); RUN_TEST(pipeline_python_nested_def_local_after_incremental_issue1548); + RUN_TEST(pipeline_python_local_def_shadows_module_def_issue1548); + RUN_TEST(pipeline_python_local_def_shadows_module_def_parallel_issue1548); RUN_TEST(pipeline_header_include_target_is_independent_of_registration_order); RUN_TEST(pipeline_imports_multi_symbol_edges); RUN_TEST(pipeline_go_cross_package_call); From 6b1cd8aa0de7718c6ab10a0b2d84b15782f31f2c Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 03:02:50 +0200 Subject: [PATCH 3/9] style(python): clang-format the nested-def binding and lexical step Whitespace only; the token stream is unchanged. Signed-off-by: Martin Vogel --- internal/cbm/lsp/py_lsp.c | 23 ++++++++++++++++------- src/pipeline/pass_calls.c | 11 +++++------ 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/internal/cbm/lsp/py_lsp.c b/internal/cbm/lsp/py_lsp.c index cfc5499d4c..18384d82e0 100644 --- a/internal/cbm/lsp/py_lsp.c +++ b/internal/cbm/lsp/py_lsp.c @@ -4408,11 +4408,20 @@ static void py_bind_local_def(PyLSPContext *ctx, TSNode def, bool is_class) { * them is still a local of the function; a nested def, class or lambda body * is a scope of its own and is never entered. */ static bool py_is_statement_block(const char *kind) { - static const char *const kinds[] = { - "block", "if_statement", "elif_clause", "else_clause", - "for_statement", "while_statement", "try_statement", "except_clause", - "except_group_clause", "finally_clause", "with_statement", "match_statement", - "case_clause", NULL}; + static const char *const kinds[] = {"block", + "if_statement", + "elif_clause", + "else_clause", + "for_statement", + "while_statement", + "try_statement", + "except_clause", + "except_group_clause", + "finally_clause", + "with_statement", + "match_statement", + "case_clause", + NULL}; for (int i = 0; kinds[i]; i++) { if (strcmp(kind, kinds[i]) == 0) return true; @@ -4632,8 +4641,8 @@ static void py_process_class_in(PyLSPContext *ctx, TSNode class_node, const char return; const char *prev_class = ctx->enclosing_class_qn; - ctx->enclosing_class_qn = cbm_arena_sprintf( - ctx->arena, "%s.%s", container_qn ? container_qn : ctx->module_qn, cname); + ctx->enclosing_class_qn = + cbm_arena_sprintf(ctx->arena, "%s.%s", container_qn ? container_qn : ctx->module_qn, cname); TSNode body = ts_node_child_by_field_name(class_node, "body", 4); if (!ts_node_is_null(body)) { diff --git a/src/pipeline/pass_calls.c b/src/pipeline/pass_calls.c index 77fa14f4aa..80b2e69b84 100644 --- a/src/pipeline/pass_calls.c +++ b/src/pipeline/pass_calls.c @@ -596,9 +596,8 @@ static void handle_route_registration(cbm_pipeline_ctx_t *ctx, const CBMCall *ca call->enclosing_func_qn); } if (!hres.qualified_name) { - hres = cbm_registry_resolve_handler(ctx->registry, call->second_arg_name, - module_qn, imp_keys, imp_vals, imp_count, - ctx->gbuf, lang); + hres = cbm_registry_resolve_handler(ctx->registry, call->second_arg_name, module_qn, + imp_keys, imp_vals, imp_count, ctx->gbuf, lang); } if (hres.qualified_name != NULL && hres.qualified_name[0] != '\0') { const cbm_gbuf_node_t *handler = cbm_gbuf_find_by_qn(ctx->gbuf, hres.qualified_name); @@ -978,9 +977,9 @@ static int resolve_single_call(cbm_pipeline_ctx_t *ctx, CBMCall *call, const CBM cbm_resolution_t res = {0}; if (!rust_external && !php_vendor_call) { /* Python LEGB fallback first (MUST match pass_parallel.c). */ - res = cbm_registry_resolve_call_scoped(ctx->registry, call->callee_name, module_qn, - imp_keys, imp_vals, imp_count, lang, - call->enclosing_func_qn); + res = + cbm_registry_resolve_call_scoped(ctx->registry, call->callee_name, module_qn, imp_keys, + imp_vals, imp_count, lang, call->enclosing_func_qn); /* Cross-language veto: a name-only guess never binds another * language's symbol (a JS app.get must not become Python's * builtins.dict.get). The vetoed answer becomes EMPTY here, before From 9b5139d73da34fc882beabeae403c3eee94c56b6 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 03:41:31 +0200 Subject: [PATCH 4/9] fix(python): bound the function-local walks at the symbol's own file A package __init__.py can define a function whose QN equals a submodule's (poetry: layouts/__init__.py `def layout` is the QN prefix of every symbol in layouts/layout.py). The prefix walks that recognise a def declared inside a function body accepted that other file's function as the enclosing def, so class Layout and its methods read as locals of layout(): they left the short-name index and the import symbol fallback, and the answer depended on which file registered first (discovery order on a full index, node-id order on an incremental one). Every walk now stops at the symbol's own module: - registration (pass_definitions.c, pass_parallel.c, and the incremental seed) passes the defining file's symbol prefix length, cbm_pipeline_fqn_symbol_prefix_len, computed without allocating, to cbm_registry_add_lang_in_module; cbm_registry_add_lang keeps its signature and does no local detection; - cbm_registry_resolve_lexical stops at the caller's module QN; - the import fallback's local test only counts an enclosing def in the same file. Also moves the #1548 registry tests out of the test they had been inserted into. Tests: registry_python_local_bound_at_own_file_issue1548 (both registration orders), fqn_symbol_prefix_len_matches_fqn_compute, pipeline_python_init_function_shares_submodule_qn{,_parallel}_issue1548 (src/ layout: the import and calls onto Layout survive, then again after an incremental re-index). Signed-off-by: Martin Vogel --- src/pipeline/fqn.c | 24 +++++ src/pipeline/pass_calls.c | 2 +- src/pipeline/pass_definitions.c | 13 ++- src/pipeline/pass_parallel.c | 18 +++- src/pipeline/pass_pkgmap.c | 10 +- src/pipeline/pipeline.h | 41 +++++--- src/pipeline/pipeline_incremental.c | 17 +++- src/pipeline/registry.c | 75 ++++++++++++--- tests/test_pipeline.c | 99 +++++++++++++++++++ tests/test_registry.c | 144 +++++++++++++++++++++------- 10 files changed, 366 insertions(+), 77 deletions(-) diff --git a/src/pipeline/fqn.c b/src/pipeline/fqn.c index f391ce1e94..f4603dc807 100644 --- a/src/pipeline/fqn.c +++ b/src/pipeline/fqn.c @@ -355,6 +355,30 @@ char *cbm_pipeline_resolve_relative_import(const char *source_rel, const char *m return resolve_js_relative(buf, sizeof(buf), module_path); } +size_t cbm_pipeline_fqn_symbol_prefix_len(const char *project, const char *rel_path) { + if (!project) { + return 0; + } + char path[FQN_PATH_BUF]; + int n = snprintf(path, sizeof(path), "%s", rel_path ? rel_path : ""); + if (n < 0 || (size_t)n >= sizeof(path)) { + return 0; + } + cbm_normalize_path_sep(path); + strip_file_extension(path); + const char *segments[CBM_SZ_256]; + int seg_count = 0; + segments[seg_count++] = project; + seg_count += tokenize_path(path, segments + seg_count, FQN_MAX_PATH_SEGS); + /* Any symbol name: the prefix cbm_pipeline_fqn_compute puts before it. */ + strip_init_or_index(segments, &seg_count, "symbol"); + size_t total = 0; + for (int i = 0; i < seg_count; i++) { + total += strlen(segments[i]) + (i > 0 ? FQN_SEP_LEN : 0); + } + return total; +} + char *cbm_pipeline_fqn_folder(const char *project, const char *rel_dir) { if (!project) { return strdup(""); diff --git a/src/pipeline/pass_calls.c b/src/pipeline/pass_calls.c index 80b2e69b84..dfbb8f7fb3 100644 --- a/src/pipeline/pass_calls.c +++ b/src/pipeline/pass_calls.c @@ -593,7 +593,7 @@ static void handle_route_registration(cbm_pipeline_ctx_t *ctx, const CBMCall *ca cbm_resolution_t hres = {0}; if (lang == CBM_LANG_PYTHON) { hres = cbm_registry_resolve_lexical(ctx->registry, call->second_arg_name, - call->enclosing_func_qn); + call->enclosing_func_qn, module_qn); } if (!hres.qualified_name) { hres = cbm_registry_resolve_handler(ctx->registry, call->second_arg_name, module_qn, diff --git a/src/pipeline/pass_definitions.c b/src/pipeline/pass_definitions.c index aecd91d992..80bf0a8a96 100644 --- a/src/pipeline/pass_definitions.c +++ b/src/pipeline/pass_definitions.c @@ -416,7 +416,7 @@ static void pd_add_file_doc(const cbm_gbuf_node_t *file_node, const char *doc) { * `file_doc`, the file's own doc (passed with its first def only), goes on the * File node. */ static void process_def(cbm_pipeline_ctx_t *ctx, const CBMDefinition *def, const char *rel, - const char *file_doc, CBMLanguage lang) { + const char *file_doc, CBMLanguage lang, size_t module_prefix_len) { if (!def->qualified_name || !def->name) { return; } @@ -442,7 +442,8 @@ static void process_def(cbm_pipeline_ctx_t *ctx, const CBMDefinition *def, const * (SQL FROM/JOIN lineage). pass_parallel.c and pipeline_incremental.c seed * through the same predicate, so the three registries cannot diverge. */ if (node_id > 0 && cbm_label_is_registry_symbol(def->label)) { - cbm_registry_add_lang(ctx->registry, def->name, def->qualified_name, def->label, lang); + cbm_registry_add_lang_in_module(ctx->registry, def->name, def->qualified_name, def->label, + lang, module_prefix_len); } char *file_qn = cbm_pipeline_fqn_compute(ctx->project_name, rel, "__file__"); const cbm_gbuf_node_t *file_node = cbm_gbuf_find_by_qn(ctx->gbuf, file_qn); @@ -1019,10 +1020,14 @@ int cbm_pipeline_pass_definitions(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t "test_declarations"); } - /* Create nodes for each definition */ + /* Create nodes for each definition. A Python file's module QN bounds + * the registry's function-local detection to this file. */ const char *file_doc = result->module_doc; /* goes with the first def */ + size_t py_prefix_len = lang == CBM_LANG_PYTHON + ? cbm_pipeline_fqn_symbol_prefix_len(ctx->project_name, rel) + : 0; for (int d = 0; d < result->defs.count; d++) { - process_def(ctx, &result->defs.items[d], rel, file_doc, lang); + process_def(ctx, &result->defs.items[d], rel, file_doc, lang, py_prefix_len); file_doc = NULL; total_defs++; } diff --git a/src/pipeline/pass_parallel.c b/src/pipeline/pass_parallel.c index 501320dd65..f908de5904 100644 --- a/src/pipeline/pass_parallel.c +++ b/src/pipeline/pass_parallel.c @@ -1773,7 +1773,8 @@ int cbm_parallel_extract(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t *files, * once per file by the caller: computing the file QN and finding its node for * every definition was 700 k allocations and lookups on the Go corpus. */ static int register_and_link_def(cbm_pipeline_ctx_t *ctx, const CBMDefinition *def, - int64_t file_node_id, CBMLanguage lang, int *reg_entries) { + int64_t file_node_id, CBMLanguage lang, size_t module_prefix_len, + int *reg_entries) { int edges = 0; if (!def->name || !def->qualified_name || !def->label) { return 0; @@ -1781,7 +1782,8 @@ static int register_and_link_def(cbm_pipeline_ctx_t *ctx, const CBMDefinition *d /* Registry membership is defined ONCE by cbm_label_is_registry_symbol * (helpers.c) — see pass_definitions.c for the per-label rationale. */ if (cbm_label_is_registry_symbol(def->label)) { - cbm_registry_add_lang(ctx->registry, def->name, def->qualified_name, def->label, lang); + cbm_registry_add_lang_in_module(ctx->registry, def->name, def->qualified_name, def->label, + lang, module_prefix_len); (*reg_entries)++; } char marked[CBM_SZ_4K]; @@ -1988,9 +1990,14 @@ int cbm_build_registry_from_cache(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t int64_t file_node_id = file_node ? file_node->id : 0; free(file_qn); pp_add_file_doc(file_node, result->module_doc); + /* Bounds the function-local detection to this file (pass_definitions.c). */ + size_t py_prefix_len = files[i].language == CBM_LANG_PYTHON + ? cbm_pipeline_fqn_symbol_prefix_len(ctx->project_name, rel) + : 0; for (int d = 0; d < result->defs.count; d++) { - defines_edges += register_and_link_def(ctx, &result->defs.items[d], file_node_id, - files[i].language, ®_entries); + defines_edges += + register_and_link_def(ctx, &result->defs.items[d], file_node_id, + files[i].language, py_prefix_len, ®_entries); } } @@ -2415,7 +2422,8 @@ static void emit_route_registration(cbm_gbuf_t *gbuf, const cbm_gbuf_node_t *sou * lexical lookup sees a function's locals. MUST match pass_calls.c. */ cbm_resolution_t hres = {0}; if (lang == CBM_LANG_PYTHON) { - hres = cbm_registry_resolve_lexical(registry, handler_ref, call->enclosing_func_qn); + hres = cbm_registry_resolve_lexical(registry, handler_ref, call->enclosing_func_qn, + module_qn); } if (!hres.qualified_name) { hres = cbm_registry_resolve_handler(registry, handler_ref, module_qn, ik, iv, ic, diff --git a/src/pipeline/pass_pkgmap.c b/src/pipeline/pass_pkgmap.c index e30a0c2312..303b293b93 100644 --- a/src/pipeline/pass_pkgmap.c +++ b/src/pipeline/pass_pkgmap.c @@ -1824,8 +1824,11 @@ bool cbm_import_symbol_fallback_allowed(CBMLanguage lang) { * no import can name it. Its QN extends a Function/Method node's * (proj.tests.mock_inputs.inner.wrapper.mock_getpass.getpass), which is how it * is recognised; the symbol-name fallback bound `import getpass` to exactly - * that def. Mirrors cbm_registry_add_lang, which keeps the same - * symbols out of the call resolver's short-name index. */ + * that def. Only an enclosing def of the SAME file counts: a package + * __init__.py function can share a submodule's QN (poetry: layouts/__init__.py + * `def layout` vs layouts/layout.py `class Layout`). Mirrors + * cbm_registry_add_lang_in_module, which keeps the same symbols out of the call + * resolver's short-name index. */ static bool import_candidate_is_python_local(const cbm_gbuf_t *gbuf, const cbm_gbuf_node_t *cand) { const char *qn = cand->qualified_name; const char *slash = cand->file_path ? strrchr(cand->file_path, '/') : NULL; @@ -1839,7 +1842,8 @@ static bool import_candidate_is_python_local(const cbm_gbuf_t *gbuf, const cbm_g for (char *dot = strrchr(scope, '.'); dot; dot = strrchr(scope, '.')) { *dot = '\0'; const cbm_gbuf_node_t *outer = cbm_gbuf_find_by_qn(gbuf, scope); - if (outer && outer->label && + if (outer && outer->label && outer->file_path && + strcmp(outer->file_path, cand->file_path) == 0 && (strcmp(outer->label, "Function") == 0 || strcmp(outer->label, "Method") == 0)) { return true; } diff --git a/src/pipeline/pipeline.h b/src/pipeline/pipeline.h index e03c0dbe66..072674d8d4 100644 --- a/src/pipeline/pipeline.h +++ b/src/pipeline/pipeline.h @@ -200,6 +200,12 @@ char *cbm_pipeline_fqn_compute(const char *project, const char *rel_path, const /* Module QN: project.dir.parts (no name). Caller must free(). */ char *cbm_pipeline_fqn_module(const char *project, const char *rel_path); +/* Length of the prefix cbm_pipeline_fqn_compute(project, rel_path, name) puts + * before a symbol `name` (the joining dot excluded): the QN prefix every + * definition of that file carries, __init__ / index already dropped. Computed + * without allocating; 0 when unavailable. */ +size_t cbm_pipeline_fqn_symbol_prefix_len(const char *project, const char *rel_path); + /* Language-aware module QN. When `module_is_dir` is true (Java/Go package * semantics) the module is derived from the CONTAINING DIRECTORY (the filename * stem is dropped), so it agrees with the extraction-side def QNs; when false @@ -253,14 +259,24 @@ void cbm_registry_add(cbm_registry_t *r, const char *name, const char *qualified /* As cbm_registry_add, recording `lang`: the DETECTED language of the file * that defines the symbol (CBM_LANG_COUNT = unknown), which the cross-language - * veto (cbm_registry_name_guess_vetoed) checks against the caller. A Python - * symbol declared inside a function body (see cbm_registry_qn_is_function_local) - * is registered for exact lookups only, never offered to the short-name - * strategies; every registration site seeds through this, so full and - * incremental indexes agree. */ + * veto (cbm_registry_name_guess_vetoed) checks against the caller. Detects no + * function-local symbols: that needs the defining file's module QN, see + * cbm_registry_add_lang_in_module. */ void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qualified_name, const char *label, CBMLanguage lang); +/* As cbm_registry_add_lang. A Python symbol declared inside a function body of + * its own file (cbm_registry_qn_is_function_local, bounded by + * `module_prefix_len`: cbm_pipeline_fqn_symbol_prefix_len of the defining file) + * is registered for exact lookups only, never offered to the short-name + * strategies. The three graph + * registration sites (pass_definitions.c, pass_parallel.c, + * pipeline_incremental.c) all register through this, so full and incremental + * indexes agree. 0 = no local detection. */ +void cbm_registry_add_lang_in_module(cbm_registry_t *r, const char *name, + const char *qualified_name, const char *label, + CBMLanguage lang, size_t module_prefix_len); + /* True when code in `caller` can call a symbol defined in `target` by bare * name: same language, same interop family (JS/TS + script hosts, C/C++/CUDA/ * ObjC, JVM, .NET, ...; the table lives in registry.c), or Swift with the C @@ -272,16 +288,19 @@ bool cbm_lang_resolution_compatible(CBMLanguage caller, CBMLanguage target); * CBM_LANG_COUNT for anything else. */ CBMLanguage cbm_registry_synthetic_path_language(const char *file_path); -/* True when a proper prefix of `qualified_name` is a registered Function or - * Method, i.e. the symbol was declared inside a function body. */ -bool cbm_registry_qn_is_function_local(const cbm_registry_t *r, const char *qualified_name); +/* True when a proper prefix of `qualified_name` longer than + * `module_prefix_len` (cbm_pipeline_fqn_symbol_prefix_len of the defining file) + * is a registered Function or Method, i.e. the symbol was declared inside a + * function body of its own file. False when module_prefix_len is 0. */ +bool cbm_registry_qn_is_function_local(const cbm_registry_t *r, const char *qualified_name, + size_t module_prefix_len); /* Python bare-name lexical lookup: the def or class named `callee_name` bound * in the nearest function scope enclosing `scope_qn` (the caller's QN), - * skipping class bodies, stopping at the module. Strategy "lexical_scope"; - * qualified_name is NULL when no such local exists. */ + * skipping class bodies, stopping at the caller's module (`module_qn`). + * Strategy "lexical_scope"; qualified_name is NULL when no such local exists. */ cbm_resolution_t cbm_registry_resolve_lexical(const cbm_registry_t *r, const char *callee_name, - const char *scope_qn); + const char *scope_qn, const char *module_qn); /* Resolve a callee name using prioritized strategies. * import_map: NULL-terminated array of {local_name, resolved_qn} pairs, or NULL. diff --git a/src/pipeline/pipeline_incremental.c b/src/pipeline/pipeline_incremental.c index 4d0d667f56..a860b5e320 100644 --- a/src/pipeline/pipeline_incremental.c +++ b/src/pipeline/pipeline_incremental.c @@ -1340,6 +1340,7 @@ static bool incr_label_is_registry_symbol(const char *label) { typedef struct { cbm_registry_t *registry; CBMHashTable *lang_by_path; /* rel_path -> (CBMLanguage + 1); keys borrowed */ + const char *project; /* bounds Python function-local detection per file */ } registry_seed_t; static void registry_visitor(const cbm_gbuf_node_t *node, void *userdata) { @@ -1354,14 +1355,20 @@ static void registry_visitor(const cbm_gbuf_node_t *node, void *userdata) { lang = (CBMLanguage)((uintptr_t)v - SKIP_ONE); } } - cbm_registry_add_lang(seed->registry, node->name, node->qualified_name, node->label, lang); + /* Same per-file bound as the full index (pass_definitions.c). */ + size_t prefix_len = lang == CBM_LANG_PYTHON + ? cbm_pipeline_fqn_symbol_prefix_len(seed->project, node->file_path) + : 0; + cbm_registry_add_lang_in_module(seed->registry, node->name, node->qualified_name, node->label, + lang, prefix_len); } /* Seed the registry from every registry symbol in the graph buffer, with * languages taken from the discovered files (see registry_visitor). */ static void registry_seed_from_gbuf(cbm_registry_t *registry, const cbm_gbuf_t *gbuf, - const cbm_file_info_t *files, int file_count) { - registry_seed_t seed = {.registry = registry, .lang_by_path = NULL}; + const char *project, const cbm_file_info_t *files, + int file_count) { + registry_seed_t seed = {.registry = registry, .lang_by_path = NULL, .project = project}; if (files && file_count > 0) { seed.lang_by_path = cbm_ht_create((size_t)file_count * PAIR_LEN); } @@ -3305,7 +3312,7 @@ static int run_closure_delta(cbm_pipeline_t *p, const char *db_path, const char if (!registry) { goto out; } - registry_seed_from_gbuf(registry, gbuf, files, file_count); + registry_seed_from_gbuf(registry, gbuf, project, files, file_count); cbm_log_info("delta.preseed_done", "registry", itoa_buf(cbm_registry_size(registry)), "elapsed_ms", itoa_buf((int)elapsed_ms(t))); @@ -4155,7 +4162,7 @@ int cbm_pipeline_run_incremental(cbm_pipeline_t *p, const char *db_path, cbm_fil /* Step 3-5: Registry + extract + resolve */ cbm_registry_t *registry = cbm_registry_new(); cbm_clock_gettime(CLOCK_MONOTONIC, &t); - registry_seed_from_gbuf(registry, existing, files, file_count); + registry_seed_from_gbuf(registry, existing, project, files, file_count); cbm_log_info("incremental.registry_seed", "symbols", itoa_buf(cbm_registry_size(registry)), "elapsed_ms", itoa_buf((int)elapsed_ms(t))); diff --git a/src/pipeline/registry.c b/src/pipeline/registry.c index 8a1122ddfe..5dda868a4e 100644 --- a/src/pipeline/registry.c +++ b/src/pipeline/registry.c @@ -1187,14 +1187,45 @@ static bool registry_label_is_callable(const char *label) { return label && (strcmp(label, "Function") == 0 || strcmp(label, "Method") == 0); } -bool cbm_registry_qn_is_function_local(const cbm_registry_t *r, const char *qualified_name) { +/* Length of the QN prefix the defs of the caller's file carry: its module QN + * (cbm_pipeline_fqn_module), minus the trailing `__init__` / `index` segment + * that symbol QNs drop (pkg/__init__.py defines proj.pkg.f). 0 when there is + * no module QN or `qualified_name` does not extend that prefix. */ +static size_t python_def_prefix_len(const char *module_qn, const char *qualified_name) { + static const char *const dropped[] = {".__init__", ".index"}; + size_t n = module_qn ? strlen(module_qn) : 0; + for (size_t i = 0; i < sizeof(dropped) / sizeof(dropped[0]); i++) { + size_t k = strlen(dropped[i]); + if (n > k && strcmp(module_qn + n - k, dropped[i]) == 0) { + n -= k; + break; + } + } + if (n == 0 || !qualified_name || strncmp(qualified_name, module_qn, n) != 0 || + qualified_name[n] != '.') { + return 0; + } + return n; +} + +/* Whether `qualified_name` sits inside a def of its OWN file: only prefixes + * strictly longer than `module_prefix_len` (cbm_pipeline_fqn_symbol_prefix_len + * of the defining file) count. A package's __init__.py can define a function + * whose QN equals a submodule's (poetry: layouts/__init__.py `def layout` vs + * layouts/layout.py), so an unbounded walk read the submodule's classes as + * that function's locals -- and the answer depended on which file registered + * first. */ +bool cbm_registry_qn_is_function_local(const cbm_registry_t *r, const char *qualified_name, + size_t module_prefix_len) { char scope[CBM_SZ_512]; size_t len = qualified_name ? strlen(qualified_name) : 0; - if (!r || len == 0 || len >= sizeof(scope)) { + size_t floor = module_prefix_len; + if (!r || floor == 0 || len <= floor || len >= sizeof(scope) || qualified_name[floor] != '.') { return false; } memcpy(scope, qualified_name, len + SKIP_ONE); - for (char *dot = strrchr(scope, '.'); dot; dot = strrchr(scope, '.')) { + for (char *dot = strrchr(scope, '.'); dot && (size_t)(dot - scope) > floor; + dot = strrchr(scope, '.')) { *dot = '\0'; if (registry_label_is_callable(cbm_ht_get(r->exact, scope))) { return true; @@ -1238,7 +1269,12 @@ static const char *registry_add_exact(cbm_registry_t *r, const char *qualified_n void cbm_registry_add(cbm_registry_t *r, const char *name, const char *qualified_name, const char *label) { - cbm_registry_add_lang(r, name, qualified_name, label, CBM_LANG_COUNT); + cbm_registry_add_lang_in_module(r, name, qualified_name, label, CBM_LANG_COUNT, 0); +} + +void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qualified_name, + const char *label, CBMLanguage lang) { + cbm_registry_add_lang_in_module(r, name, qualified_name, label, lang, 0); } /* Python nests defs: a def or class declared in a function body -- a closure, @@ -1250,14 +1286,19 @@ void cbm_registry_add(cbm_registry_t *r, const char *name, const char *qualified * decorator's `def wrapper` to a caller in another function or file. The exact * map still holds it, for LSP targets and cbm_registry_resolve_lexical. One * file's defs register parent first, so the enclosing function is already in - * the exact map when its locals arrive. Gated to Python: probing every - * prefix of every symbol would cost a kernel index seconds for nothing. */ -void cbm_registry_add_lang(cbm_registry_t *r, const char *name, const char *qualified_name, - const char *label, CBMLanguage lang) { + * the exact map when its locals arrive; the walk never leaves the symbol's own + * file (`module_prefix_len`, cbm_pipeline_fqn_symbol_prefix_len of the defining + * file; 0 = no local detection). + * Gated to Python: probing every prefix of every symbol would cost a kernel + * index seconds for nothing. */ +void cbm_registry_add_lang_in_module(cbm_registry_t *r, const char *name, + const char *qualified_name, const char *label, + CBMLanguage lang, size_t module_prefix_len) { if (!r || !qualified_name || !label) { return; } - if (lang == CBM_LANG_PYTHON && cbm_registry_qn_is_function_local(r, qualified_name)) { + if (lang == CBM_LANG_PYTHON && + cbm_registry_qn_is_function_local(r, qualified_name, module_prefix_len)) { (void)registry_add_exact(r, qualified_name, label); return; } @@ -2176,19 +2217,22 @@ cbm_resolution_t cbm_registry_resolve_lineage(const cbm_registry_t *r, const cha * enclosing def -- create_app.index, Tree.visit.visit -- before anything a * module-wide strategy would find. Walks scope_qn outward while its prefixes * are registered defs or classes (skipping the classes) and stops at the - * module, which the other strategies own. Empty unless such a local exists. - * Uncached: the answer depends on the caller, not only on the name. */ + * caller's module (`module_qn`), which the other strategies own -- a + * same-QN function of a package __init__.py is never an enclosing scope. + * Empty unless such a local exists. Uncached: the answer depends on the + * caller, not only on the name. */ cbm_resolution_t cbm_registry_resolve_lexical(const cbm_registry_t *r, const char *callee_name, - const char *scope_qn) { + const char *scope_qn, const char *module_qn) { char scope[CBM_SZ_512]; char candidate[CBM_SZ_512]; size_t len = scope_qn ? strlen(scope_qn) : 0; + size_t floor = python_def_prefix_len(module_qn, scope_qn); if (!r || !callee_name || !callee_name[0] || strchr(callee_name, '.') || - strstr(callee_name, "::") || len == 0 || len >= sizeof(scope)) { + strstr(callee_name, "::") || len == 0 || len >= sizeof(scope) || floor == 0) { return empty_result(); } memcpy(scope, scope_qn, len + SKIP_ONE); - for (;;) { + while (strlen(scope) > floor) { const char *label = cbm_ht_get(r->exact, scope); if (!label) { break; @@ -2223,7 +2267,8 @@ cbm_resolution_t cbm_registry_resolve_call_scoped(const cbm_registry_t *r, const int import_map_count, CBMLanguage lang, const char *scope_qn) { if (lang == CBM_LANG_PYTHON) { - cbm_resolution_t lexical = cbm_registry_resolve_lexical(r, callee_name, scope_qn); + cbm_resolution_t lexical = + cbm_registry_resolve_lexical(r, callee_name, scope_qn, module_qn); if (lexical.qualified_name && lexical.qualified_name[0]) { return lexical; } diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index 3141ab32d9..2cb51cfd6e 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -14713,6 +14713,103 @@ TEST(pipeline_python_local_def_shadows_module_def_parallel_issue1548) { return pyn_run_shadowing(52); } +/* L7 review of #1548: a package __init__.py function can share a submodule's QN + * (poetry: layouts/__init__.py `def layout` is the QN prefix of every symbol in + * layouts/layout.py). The function-local detection is bounded at the symbol's + * own file, so class Layout stays a module-level, importable class. src/ layout + * without a manifest: the module resolver misses `rowpkg.layout`, so the import + * relies on the symbol-name fallback, which skips function locals -- unbounded, + * Layout read as a local of layout() and its import and calls vanished. Both + * resolvers; the second run re-indexes incrementally after an edit of + * layout.py, so the incremental registry seed is held to the same bound. */ +static const char pyn_layout_init_py[] = "def layout(name):\n" + " return name\n"; +static const char pyn_layout_mod_py[] = "class Layout:\n" + " def __init__(self, name):\n" + " self.name = name\n" + "\n" + " def create(self):\n" + " return self.name\n"; +static const char pyn_layout_app_py[] = "from rowpkg.layout import Layout\n" + "\n" + "def test_build():\n" + " return Layout(\"demo\").create()\n"; + +static int pyn_run_init_collision(int fillers) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_initqn_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "src/rowpkg/__init__.py", pyn_layout_init_py); + write_temp_file(tmp, "src/rowpkg/layout.py", pyn_layout_mod_py); + write_temp_file(tmp, "tests/test_app.py", pyn_layout_app_py); + for (int i = 0; i < fillers; i++) { + char name[64]; + char body[128]; + snprintf(name, sizeof(name), "filler%d.py", i); + snprintf(body, sizeof(body), "def filler%d():\n return %d\n", i, i); + write_temp_file(tmp, name, body); + } + char *old_workers = getenv("CBM_WORKERS"); + char *saved = old_workers ? strdup(old_workers) : NULL; + if (fillers > 0) { + cbm_setenv("CBM_WORKERS", "4", 1); + } + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/initqn.db", tmp); + int rc[2] = {-1, -1}; + bool imports[2] = {false, false}; + bool calls[2] = {false, false}; + bool method[2] = {false, false}; + for (int run = 0; run < 2; run++) { + if (run == 1) { + char path[512]; + snprintf(path, sizeof(path), "%s/src/rowpkg/layout.py", tmp); + th_append_file(path, "\n# edited: the second run re-indexes incrementally\n"); + } + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + rc[run] = p ? cbm_pipeline_run(p) : -1; + cbm_store_t *s = rc[run] == 0 ? cbm_store_open_path(db_path) : NULL; + if (s) { + const char *project = cbm_pipeline_project_name(p); + imports[run] = pyn_edge(s, project, "tests.test_app.py.__file__", + "src.rowpkg.layout.Layout", "IMPORTS"); + calls[run] = + pyn_edge(s, project, "tests.test_app.test_build", "src.rowpkg.layout.Layout", + "CALLS"); + method[run] = pyn_edge(s, project, "tests.test_app.test_build", + "src.rowpkg.layout.Layout.create", "CALLS"); + cbm_store_close(s); + } + if (p) { + cbm_pipeline_free(p); + } + } + if (saved) { + cbm_setenv("CBM_WORKERS", saved, 1); + free(saved); + } else { + cbm_unsetenv("CBM_WORKERS"); + } + th_rmtree(tmp); + for (int run = 0; run < 2; run++) { + ASSERT_EQ(rc[run], 0); + ASSERT_TRUE(imports[run]); /* the class stays importable */ + ASSERT_TRUE(calls[run]); /* Layout("demo") */ + ASSERT_TRUE(method[run]); /* .create() through the constructed type */ + } + PASS(); +} + +TEST(pipeline_python_init_function_shares_submodule_qn_issue1548) { + return pyn_run_init_collision(0); +} + +TEST(pipeline_python_init_function_shares_submodule_qn_parallel_issue1548) { + return pyn_run_init_collision(52); +} + TEST(pipeline_python_project) { /* Port of TestPipelinePythonProject */ const char *files[] = {"main.py", "utils.py"}; @@ -24214,6 +24311,8 @@ SUITE(pipeline) { RUN_TEST(pipeline_python_nested_def_local_after_incremental_issue1548); RUN_TEST(pipeline_python_local_def_shadows_module_def_issue1548); RUN_TEST(pipeline_python_local_def_shadows_module_def_parallel_issue1548); + RUN_TEST(pipeline_python_init_function_shares_submodule_qn_issue1548); + RUN_TEST(pipeline_python_init_function_shares_submodule_qn_parallel_issue1548); RUN_TEST(pipeline_header_include_target_is_independent_of_registration_order); RUN_TEST(pipeline_imports_multi_symbol_edges); RUN_TEST(pipeline_go_cross_package_call); diff --git a/tests/test_registry.c b/tests/test_registry.c index 055de4d781..0f3e805f2f 100644 --- a/tests/test_registry.c +++ b/tests/test_registry.c @@ -1962,6 +1962,9 @@ TEST(registry_python_scope_qualified_terminal_bucket_uncapped) { bool seeded = cbm_registry_size(r) == 302; cbm_registry_free(r); ASSERT_TRUE(shape && seeded && resolved); + PASS(); +} + /* #1548: a Python def or class declared in a function body is a local of it. * It is registered for exact lookups but kept out of the short-name index, so * no weak strategy offers it to a caller elsewhere. A class nested in a class @@ -1969,24 +1972,29 @@ TEST(registry_python_scope_qualified_terminal_bucket_uncapped) { * does the same QN shape from another language. */ TEST(registry_python_function_local_is_exact_only_issue1548) { cbm_registry_t *r = cbm_registry_new(); - cbm_registry_add_lang(r, "create_app", "proj.web.create_app", "Function", - CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "index", "proj.web.create_app.index", "Function", - CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "Settings", "proj.web.create_app.Settings", "Class", - CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "value", "proj.web.create_app.Settings.value", "Method", - CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "Outer", "proj.web.Outer", "Class", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "Inner", "proj.web.Outer.Inner", "Class", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "f", "proj.ts.f", "Function", CBM_LANG_TYPESCRIPT); - cbm_registry_add_lang(r, "g", "proj.ts.f.g", "Function", CBM_LANG_TYPESCRIPT); + const size_t web = strlen("proj.web"); /* symbol prefix of web.py */ + static const char *const defs[][3] = { + {"create_app", "proj.web.create_app", "Function"}, + {"index", "proj.web.create_app.index", "Function"}, + {"Settings", "proj.web.create_app.Settings", "Class"}, + {"value", "proj.web.create_app.Settings.value", "Method"}, + {"Outer", "proj.web.Outer", "Class"}, + {"Inner", "proj.web.Outer.Inner", "Class"}, + }; + for (size_t i = 0; i < sizeof(defs) / sizeof(defs[0]); i++) { + cbm_registry_add_lang_in_module(r, defs[i][0], defs[i][1], defs[i][2], CBM_LANG_PYTHON, + web); + } + cbm_registry_add_lang_in_module(r, "f", "proj.ts.f", "Function", CBM_LANG_TYPESCRIPT, + strlen("proj.ts")); + cbm_registry_add_lang_in_module(r, "g", "proj.ts.f.g", "Function", CBM_LANG_TYPESCRIPT, + strlen("proj.ts")); static const char *const local[] = {"proj.web.create_app.index", "proj.web.create_app.Settings", "proj.web.create_app.Settings.value"}; for (int i = 0; i < 3; i++) { ASSERT_TRUE(cbm_registry_exists(r, local[i])); - ASSERT_TRUE(cbm_registry_qn_is_function_local(r, local[i])); + ASSERT_TRUE(cbm_registry_qn_is_function_local(r, local[i], web)); const char *simple = strrchr(local[i], '.') + 1; const char **out = NULL; int count = -1; @@ -2003,7 +2011,9 @@ TEST(registry_python_function_local_is_exact_only_issue1548) { ASSERT_EQ(count, 1); ASSERT_STR_EQ(out[0], nameable[i][1]); } - ASSERT_FALSE(cbm_registry_qn_is_function_local(r, "proj.web.Outer.Inner")); + ASSERT_FALSE(cbm_registry_qn_is_function_local(r, "proj.web.Outer.Inner", web)); + /* No module prefix: no local detection. */ + ASSERT_FALSE(cbm_registry_qn_is_function_local(r, "proj.web.create_app.index", 0)); /* A weak strategy cannot reach the local; before, index() anywhere in * the project resolved to it by unique_name. */ cbm_resolution_t res = cbm_registry_resolve(r, "index", "proj.other", NULL, NULL, 0); @@ -2017,44 +2027,110 @@ TEST(registry_python_function_local_is_exact_only_issue1548) { * to the other strategies. A dotted callee is not a bare name. */ TEST(registry_resolve_lexical_walks_enclosing_defs_issue1548) { cbm_registry_t *r = cbm_registry_new(); - cbm_registry_add_lang(r, "Tree", "proj.m.Tree", "Class", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "visit", "proj.m.Tree.visit", "Method", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "visit", "proj.m.Tree.visit.visit", "Function", - CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "helper", "proj.m.Tree.helper", "Method", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "factory", "proj.m.factory", "Function", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "helper", "proj.m.factory.helper", "Function", - CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "Local", "proj.m.factory.Local", "Class", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "run", "proj.m.factory.Local.run", "Method", CBM_LANG_PYTHON); - cbm_registry_add_lang(r, "top", "proj.m.top", "Function", CBM_LANG_PYTHON); + static const char *const defs[][3] = { + {"Tree", "proj.m.Tree", "Class"}, + {"visit", "proj.m.Tree.visit", "Method"}, + {"visit", "proj.m.Tree.visit.visit", "Function"}, + {"helper", "proj.m.Tree.helper", "Method"}, + {"factory", "proj.m.factory", "Function"}, + {"helper", "proj.m.factory.helper", "Function"}, + {"Local", "proj.m.factory.Local", "Class"}, + {"run", "proj.m.factory.Local.run", "Method"}, + {"top", "proj.m.top", "Function"}, + }; + for (size_t i = 0; i < sizeof(defs) / sizeof(defs[0]); i++) { + cbm_registry_add_lang_in_module(r, defs[i][0], defs[i][1], defs[i][2], CBM_LANG_PYTHON, + strlen("proj.m")); + } /* In method Tree.visit, `visit` is its nested def; so it is inside that * nested def (its own recursion). */ - cbm_resolution_t res = cbm_registry_resolve_lexical(r, "visit", "proj.m.Tree.visit"); + cbm_resolution_t res = cbm_registry_resolve_lexical(r, "visit", "proj.m.Tree.visit", "proj.m"); ASSERT_STR_EQ(res.qualified_name, "proj.m.Tree.visit.visit"); ASSERT_STR_EQ(res.strategy, "lexical_scope"); - res = cbm_registry_resolve_lexical(r, "visit", "proj.m.Tree.visit.visit"); + res = cbm_registry_resolve_lexical(r, "visit", "proj.m.Tree.visit.visit", "proj.m"); ASSERT_STR_EQ(res.qualified_name, "proj.m.Tree.visit.visit"); /* The class body is skipped: Tree.helper is not a bare name in a method. */ - res = cbm_registry_resolve_lexical(r, "helper", "proj.m.Tree.visit.visit"); + res = cbm_registry_resolve_lexical(r, "helper", "proj.m.Tree.visit.visit", "proj.m"); ASSERT_NULL(res.qualified_name); /* A method of a local class sees the def enclosing its class. */ - res = cbm_registry_resolve_lexical(r, "helper", "proj.m.factory.Local.run"); + res = cbm_registry_resolve_lexical(r, "helper", "proj.m.factory.Local.run", "proj.m"); ASSERT_STR_EQ(res.qualified_name, "proj.m.factory.helper"); - res = cbm_registry_resolve_lexical(r, "Local", "proj.m.factory"); + res = cbm_registry_resolve_lexical(r, "Local", "proj.m.factory", "proj.m"); ASSERT_STR_EQ(res.qualified_name, "proj.m.factory.Local"); /* Module-level names and dotted callees are not this strategy's. */ - res = cbm_registry_resolve_lexical(r, "top", "proj.m.factory"); + res = cbm_registry_resolve_lexical(r, "top", "proj.m.factory", "proj.m"); ASSERT_NULL(res.qualified_name); - res = cbm_registry_resolve_lexical(r, "self.helper", "proj.m.factory"); + res = cbm_registry_resolve_lexical(r, "self.helper", "proj.m.factory", "proj.m"); ASSERT_NULL(res.qualified_name); - res = cbm_registry_resolve_lexical(r, "helper", "proj.m"); + res = cbm_registry_resolve_lexical(r, "helper", "proj.m", "proj.m"); ASSERT_NULL(res.qualified_name); cbm_registry_free(r); PASS(); } +/* A package __init__.py function can share a submodule's QN: poetry's + * layouts/__init__.py `def layout` is proj.pkg.layout, the QN prefix of every + * symbol in layouts/layout.py. Bounded at its own file, class Layout and its + * methods are no locals of that function, in either registration order (full + * index: discovery order; incremental: node-id order), and the lexical walk of + * a caller in layout.py never enters the __init__ function. */ +TEST(registry_python_local_bound_at_own_file_issue1548) { + const size_t init_prefix = strlen("proj.pkg"); /* pkg/__init__.py */ + const size_t mod_prefix = strlen("proj.pkg.layout"); /* pkg/layout.py */ + for (int order = 0; order < 2; order++) { + cbm_registry_t *r = cbm_registry_new(); + if (order == 0) { + cbm_registry_add_lang_in_module(r, "layout", "proj.pkg.layout", "Function", + CBM_LANG_PYTHON, init_prefix); + } + cbm_registry_add_lang_in_module(r, "Layout", "proj.pkg.layout.Layout", "Class", + CBM_LANG_PYTHON, mod_prefix); + cbm_registry_add_lang_in_module(r, "create", "proj.pkg.layout.Layout.create", "Method", + CBM_LANG_PYTHON, mod_prefix); + cbm_registry_add_lang_in_module(r, "make", "proj.pkg.layout.make", "Function", + CBM_LANG_PYTHON, mod_prefix); + if (order == 1) { + cbm_registry_add_lang_in_module(r, "layout", "proj.pkg.layout", "Function", + CBM_LANG_PYTHON, init_prefix); + } + static const char *const nameable[][2] = {{"Layout", "proj.pkg.layout.Layout"}, + {"create", "proj.pkg.layout.Layout.create"}, + {"make", "proj.pkg.layout.make"}}; + for (size_t i = 0; i < sizeof(nameable) / sizeof(nameable[0]); i++) { + const char **out = NULL; + int count = 0; + cbm_registry_find_by_name(r, nameable[i][0], &out, &count); + ASSERT_EQ(count, 1); + ASSERT_STR_EQ(out[0], nameable[i][1]); + ASSERT_FALSE(cbm_registry_qn_is_function_local(r, nameable[i][1], mod_prefix)); + } + /* `Layout()` in layout.py's make(): no lexical local; module strategies own it. */ + cbm_resolution_t res = + cbm_registry_resolve_lexical(r, "Layout", "proj.pkg.layout.make", "proj.pkg.layout"); + ASSERT_NULL(res.qualified_name); + cbm_registry_free(r); + } + PASS(); +} + +/* The prefix every def of a file carries, without allocating; the same as + * cbm_pipeline_fqn_compute(project, path, name) minus ".name". */ +TEST(fqn_symbol_prefix_len_matches_fqn_compute) { + static const char *const paths[] = {"pkg/layout.py", "pkg/__init__.py", "a.py", + "deep/x/y/mod.py", "web/index.py", "noext"}; + for (size_t i = 0; i < sizeof(paths) / sizeof(paths[0]); i++) { + char *qn = cbm_pipeline_fqn_compute("proj", paths[i], "sym"); + ASSERT_NOT_NULL(qn); + size_t want = strlen(qn) - strlen(".sym"); + free(qn); + ASSERT_EQ(cbm_pipeline_fqn_symbol_prefix_len("proj", paths[i]), want); + } + ASSERT_EQ(cbm_pipeline_fqn_symbol_prefix_len("proj", "pkg/__init__.py"), strlen("proj.pkg")); + ASSERT_EQ(cbm_pipeline_fqn_symbol_prefix_len(NULL, "a.py"), 0); + PASS(); +} + SUITE(registry) { RUN_TEST(python_symbol_scope_len_preserves_file_identity); RUN_TEST(registry_python_package_scope_is_explicit); @@ -2161,4 +2237,6 @@ SUITE(registry) { RUN_TEST(weak_call_guards_share_one_drop_list); RUN_TEST(registry_python_function_local_is_exact_only_issue1548); RUN_TEST(registry_resolve_lexical_walks_enclosing_defs_issue1548); + RUN_TEST(registry_python_local_bound_at_own_file_issue1548); + RUN_TEST(fqn_symbol_prefix_len_matches_fqn_compute); } From 0bab94ce1cf465e446f4ffb3a803269a9ec68854 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 04:08:24 +0200 Subject: [PATCH 5/9] test(python): a def under a branch is bound by the lexical step, not the LSP At a compound statement's join the Python LSP invalidates what the branch bound, so the local def under if True: is resolved by the registry's lexical_scope step; pin that producer instead of an lsp_ strategy. Found at the branch's first build. Signed-off-by: Martin Vogel --- tests/test_pipeline.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index 2cb51cfd6e..9f64812cb2 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -14642,15 +14642,21 @@ static int pyn_assert_shadowing(cbm_store_t *s, const char *project) { "\"strategy\":\"lsp_super\"")); ASSERT_TRUE(pyn_edge_with(s, project, "sc.factory.Local.other", "sc.factory.Local.greet", "CALLS", "\"strategy\":\"lsp_")); + /* sa's local def is a plain statement of the body: the LSP binds it as an + * exact callable. sb's sits under `if True:`; at that compound statement's + * join the LSP invalidates what the branch bound (the def may not exist on + * every path), so it emits nothing and the registry's lexical step binds + * the local -- still never the module def. */ const char *files[] = {"sa", "sb"}; + const char *local_strategy[] = {"\"strategy\":\"lsp_", "\"strategy\":\"lexical_scope\""}; for (int f = 0; f < 2; f++) { char local_caller[64], local_def[64], module_caller[64], module_def[64]; snprintf(local_caller, sizeof(local_caller), "%s.uses_local", files[f]); snprintf(local_def, sizeof(local_def), "%s.uses_local.helper", files[f]); snprintf(module_caller, sizeof(module_caller), "%s.uses_module", files[f]); snprintf(module_def, sizeof(module_def), "%s.helper", files[f]); - ASSERT_TRUE(pyn_edge_with(s, project, local_caller, local_def, "CALLS", - "\"strategy\":\"lsp_")); + ASSERT_TRUE( + pyn_edge_with(s, project, local_caller, local_def, "CALLS", local_strategy[f])); ASSERT_FALSE(pyn_edge(s, project, local_caller, module_def, "CALLS")); ASSERT_TRUE(pyn_edge(s, project, module_caller, module_def, "CALLS")); ASSERT_FALSE(pyn_edge(s, project, module_caller, local_def, "CALLS")); From 5bfa7b53bfc9cd8d648e2a09e45f2d8caf427df2 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 04:11:16 +0200 Subject: [PATCH 6/9] test(python): note that H1 may move the branch-def case to an lsp strategy Signed-off-by: Martin Vogel --- tests/test_pipeline.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index 9f64812cb2..25e467b994 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -14646,7 +14646,9 @@ static int pyn_assert_shadowing(cbm_store_t *s, const char *project) { * exact callable. sb's sits under `if True:`; at that compound statement's * join the LSP invalidates what the branch bound (the def may not exist on * every path), so it emits nothing and the registry's lexical step binds - * the local -- still never the module def. */ + * the local -- still never the module def. H1 (the LSP walks compound + * statements) may legitimately move sb to an lsp_* strategy; that PR + * must update this expectation deliberately. */ const char *files[] = {"sa", "sb"}; const char *local_strategy[] = {"\"strategy\":\"lsp_", "\"strategy\":\"lexical_scope\""}; for (int f = 0; f < 2; f++) { From d4beec3c5811d8165f34905158c10a22e0c9d403 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 04:48:27 +0200 Subject: [PATCH 7/9] test(python): local classes are extracted defs now (#1548) pylsp_nested_and_local_classbody_calls_join_extracted_caller asserted that a class declared in a function body is NOT a def, which is the behaviour #1548 fixes. Their class-body calls still join the enclosing function; that part is unchanged. Signed-off-by: Martin Vogel --- tests/test_py_lsp.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/test_py_lsp.c b/tests/test_py_lsp.c index 3ee2d8ad56..7a1c460d54 100644 --- a/tests/test_py_lsp.c +++ b/tests/test_py_lsp.c @@ -3629,12 +3629,16 @@ TEST(pylsp_nested_and_local_classbody_calls_join_extracted_caller) { {"field(35)", "test.main.Controller.build", "test.main.field"}, }; CBMFileResult *r = extract_py(source); - bool correct = pylsp_class_sites(r, source, sites, 7) && - !pylsp_module_has_def(r, "test.main.build.Local") && - !pylsp_module_has_def(r, "test.main.Controller.build.Local"); + /* #1548: a class declared in a function body is extracted, as a local of + * that function; its class-body calls still join the enclosing function + * (the sites above), never the local class. */ + bool sites_ok = pylsp_class_sites(r, source, sites, 7); + bool locals_extracted = r && pylsp_module_has_def(r, "test.main.build.Local") && + pylsp_module_has_def(r, "test.main.Controller.build.Local"); if (r) cbm_free_result(r); - ASSERT_TRUE(correct); + ASSERT_TRUE(sites_ok); + ASSERT_TRUE(locals_extracted); PASS(); } From eec3418fef18066f457b46e3a243ab3c66732430 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 06:18:09 +0200 Subject: [PATCH 8/9] fix(python): a test's nested helper keeps TESTS for its calls (#1548) Since #1548 a def declared inside a Python test function is extracted, so its calls start at the helper instead of the test FILE node, whose name passed the test-name check. 4,641 of 6,315 TESTS edges lost on the corpus were this shape. TESTS for a call from such a helper is now attributed to the nearest enclosing test function of the same file; a helper nested in a non-test function stays without TESTS, like a module-level helper. Signed-off-by: Martin Vogel --- src/pipeline/pass_tests.c | 51 +++++++++++++++++++- tests/test_pipeline.c | 97 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+), 2 deletions(-) diff --git a/src/pipeline/pass_tests.c b/src/pipeline/pass_tests.c index 9cade02a0d..a763016fb0 100644 --- a/src/pipeline/pass_tests.c +++ b/src/pipeline/pass_tests.c @@ -28,6 +28,7 @@ enum { #include "pipeline/pipeline.h" #include "pipeline/pipeline_internal.h" #include "graph_buffer/graph_buffer.h" +#include "discover/discover.h" #include "foundation/log.h" #include @@ -208,6 +209,48 @@ static char *test_to_prod_path(const char *test_path) { return NULL; } +/* #1548: a def declared inside a Python test function (a closure, a local + * helper, a method of a test-local class) is that test's own code, extracted + * as a node of its own. Its calls exercise the target on the test's behalf, so + * TESTS is attributed to the nearest enclosing test function of the same file + * (QN prefix walk). Before #1548 those calls hung on the test FILE node, whose + * name passes the test-name check. NULL when no enclosing test function exists + * (a helper nested in a non-test function stays without TESTS, like a + * module-level helper). */ +static const cbm_gbuf_node_t *enclosing_python_test_func(const cbm_gbuf_t *gb, + const cbm_gbuf_node_t *src) { + if (!src->qualified_name || !src->file_path || !src->label || + (strcmp(src->label, "Function") != 0 && strcmp(src->label, "Method") != 0)) { + return NULL; + } + const char *slash = strrchr(src->file_path, '/'); + if (cbm_language_for_filename(slash ? slash + SKIP_ONE : src->file_path) != CBM_LANG_PYTHON) { + return NULL; + } + char scope[CBM_SZ_512]; + size_t len = strlen(src->qualified_name); + if (len >= sizeof(scope)) { + return NULL; + } + memcpy(scope, src->qualified_name, len + SKIP_ONE); + for (char *dot = strrchr(scope, '.'); dot; dot = strrchr(scope, '.')) { + *dot = '\0'; + const cbm_gbuf_node_t *outer = cbm_gbuf_find_by_qn(gb, scope); + if (!outer) { + continue; + } + if (!outer->file_path || strcmp(outer->file_path, src->file_path) != 0) { + return NULL; /* left the file: no enclosing test function */ + } + if (outer->label && + (strcmp(outer->label, "Function") == 0 || strcmp(outer->label, "Method") == 0) && + cbm_is_test_func_name(outer->name)) { + return outer; + } + } + return NULL; +} + /* Create TESTS edges from test functions to production functions via CALLS edges. */ static int create_tests_edges(cbm_pipeline_ctx_t *ctx) { int count = 0; @@ -238,11 +281,15 @@ static int create_tests_edges(cbm_pipeline_ctx_t *ctx) { continue; } + const cbm_gbuf_node_t *test_src = src; if (!cbm_is_test_func_name(src->name)) { - continue; + test_src = enclosing_python_test_func(ctx->gbuf, src); + if (!test_src) { + continue; + } } - cbm_gbuf_insert_edge(ctx->gbuf, src->id, tgt->id, "TESTS", "{}"); + cbm_gbuf_insert_edge(ctx->gbuf, test_src->id, tgt->id, "TESTS", "{}"); count++; } return count; diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index 25e467b994..8403a3aed3 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -14818,6 +14818,101 @@ TEST(pipeline_python_init_function_shares_submodule_qn_parallel_issue1548) { return pyn_run_init_collision(52); } +/* #1548 keeps TESTS for calls made through a test's own nested helper: the + * call now starts at the extracted helper def (not at the test FILE node, as + * before #1548), so TESTS is attributed to the enclosing test function. A + * helper nested in a NON-test function stays without TESTS, like a module-level + * helper. Both resolvers. */ +static const char pyn_tests_prod_py[] = "def target(value):\n" + " return value\n" + "\n" + "def other_target(value):\n" + " return value\n"; +static const char pyn_tests_test_py[] = "from pkg.prod import target, other_target\n" + "\n" + "def test_through_nested_helper():\n" + " def check(v):\n" + " return target(v)\n" + " assert check(1) == 1\n" + "\n" + "class TestGroup:\n" + " def test_method_helper(self):\n" + " def probe():\n" + " return target(2)\n" + " assert probe() == 2\n" + "\n" + "def make_fixture():\n" + " def build():\n" + " return other_target(3)\n" + " return build\n"; + +static int pyn_run_nested_helper_tests(int fillers) { + char tmp[256]; + snprintf(tmp, sizeof(tmp), "/tmp/cbm_py_ntests_XXXXXX"); + if (!cbm_mkdtemp(tmp)) { + FAIL("tmpdir"); + } + write_temp_file(tmp, "pkg/__init__.py", ""); + write_temp_file(tmp, "pkg/prod.py", pyn_tests_prod_py); + write_temp_file(tmp, "tests/test_rows.py", pyn_tests_test_py); + for (int i = 0; i < fillers; i++) { + char name[64]; + char body[128]; + snprintf(name, sizeof(name), "filler%d.py", i); + snprintf(body, sizeof(body), "def filler%d():\n return %d\n", i, i); + write_temp_file(tmp, name, body); + } + char *old_workers = getenv("CBM_WORKERS"); + char *saved = old_workers ? strdup(old_workers) : NULL; + if (fillers > 0) { + cbm_setenv("CBM_WORKERS", "4", 1); + } + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/ntests.db", tmp); + cbm_pipeline_t *p = cbm_pipeline_new(tmp, db_path, CBM_MODE_FULL); + int rc = p ? cbm_pipeline_run(p) : -1; + cbm_store_t *s = rc == 0 ? cbm_store_open_path(db_path) : NULL; + bool fn_tests = false, method_tests = false, helper_calls = false, fixture_tests = true; + if (s) { + const char *project = cbm_pipeline_project_name(p); + helper_calls = pyn_edge(s, project, "tests.test_rows.test_through_nested_helper.check", + "pkg.prod.target", "CALLS"); + fn_tests = pyn_edge(s, project, "tests.test_rows.test_through_nested_helper", + "pkg.prod.target", "TESTS"); + method_tests = pyn_edge(s, project, "tests.test_rows.TestGroup.test_method_helper", + "pkg.prod.target", "TESTS"); + fixture_tests = pyn_edge(s, project, "tests.test_rows.make_fixture", + "pkg.prod.other_target", "TESTS") || + pyn_edge(s, project, "tests.test_rows.make_fixture.build", + "pkg.prod.other_target", "TESTS"); + cbm_store_close(s); + } + if (p) { + cbm_pipeline_free(p); + } + if (saved) { + cbm_setenv("CBM_WORKERS", saved, 1); + free(saved); + } else { + cbm_unsetenv("CBM_WORKERS"); + } + th_rmtree(tmp); + ASSERT_EQ(rc, 0); + ASSERT_TRUE(helper_calls); /* the call starts at the nested helper (#1548) */ + ASSERT_TRUE(fn_tests); /* ... and the enclosing test function still TESTS the target */ + ASSERT_TRUE(method_tests); /* the same inside a test method */ + ASSERT_FALSE(fixture_tests); /* a helper nested in a non-test function: no TESTS */ + PASS(); +} + +TEST(pipeline_python_nested_helper_keeps_tests_issue1548) { + return pyn_run_nested_helper_tests(0); +} + +TEST(pipeline_python_nested_helper_keeps_tests_parallel_issue1548) { + return pyn_run_nested_helper_tests(52); +} + TEST(pipeline_python_project) { /* Port of TestPipelinePythonProject */ const char *files[] = {"main.py", "utils.py"}; @@ -24321,6 +24416,8 @@ SUITE(pipeline) { RUN_TEST(pipeline_python_local_def_shadows_module_def_parallel_issue1548); RUN_TEST(pipeline_python_init_function_shares_submodule_qn_issue1548); RUN_TEST(pipeline_python_init_function_shares_submodule_qn_parallel_issue1548); + RUN_TEST(pipeline_python_nested_helper_keeps_tests_issue1548); + RUN_TEST(pipeline_python_nested_helper_keeps_tests_parallel_issue1548); RUN_TEST(pipeline_header_include_target_is_independent_of_registration_order); RUN_TEST(pipeline_imports_multi_symbol_edges); RUN_TEST(pipeline_go_cross_package_call); From 090580b3c95ba2ec58e7e09eb2858438dc827111 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sun, 18 Oct 2026 16:36:35 +0200 Subject: [PATCH 9/9] refactor(python): scope the nested-def counter to the Python branch cppcheck 2.20 variableScope on defs_before; same behaviour. Signed-off-by: Martin Vogel --- internal/cbm/extract_defs.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/internal/cbm/extract_defs.c b/internal/cbm/extract_defs.c index 2b24de63f1..6491a677ff 100644 --- a/internal/cbm/extract_defs.c +++ b/internal/cbm/extract_defs.c @@ -12261,9 +12261,9 @@ static void walk_defs(CBMExtractCtx *ctx, TSNode root, const CBMLangSpec *spec, } if (cbm_kind_in_set(node, spec->function_node_types)) { if (!is_template_class_node(node, ctx->language)) { - int defs_before = ctx->result->defs.count; - extract_func_def(ctx, node, spec); if (ctx->language == CBM_LANG_PYTHON) { + int defs_before = ctx->result->defs.count; + extract_func_def(ctx, node, spec); if (ctx->result->defs.count > defs_before) { py_push_def_body(&s, node, ctx->result->defs.items[ctx->result->defs.count - SKIP_ONE] @@ -12271,6 +12271,7 @@ static void walk_defs(CBMExtractCtx *ctx, TSNode root, const CBMLangSpec *spec, } continue; } + extract_func_def(ctx, node, spec); // Most languages stop here. JS/TS (and Wolfram) descend into the // function body so NESTED named definitions are also captured — // e.g. arrow methods of an object literal returned from a factory