Skip to content

refactor: move compile assembly, lint and planner state to native data - #1120

Merged
kvlonge merged 12 commits into
mainfrom
feat/chi599-batch-4
Oct 16, 2026
Merged

kvlonge merged 12 commits into
mainfrom
feat/chi599-batch-4

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Why

Compile assembly, rules and the planner still passed internal data through Python objects, attribute lookups and keyword dictionaries between native stages.

Changes

  • Compile inputs are attached through native calls; constants, attached schemas and references are read natively.
  • DSL headers are parsed natively for header rules, formatting and table-function arguments.
  • The lint pipeline works on native rows, and early SQL lint runs on a native thread that holds the GIL only for macro expansion.
  • Waiting for early SQL lint polls for signals, so Ctrl-C stops after the current file, including while it is being stopped.
  • Lint paths are resolved without the Windows verbatim prefix, keeping relative finding paths and cache keys.
  • Planner helpers take typed requests; phase results, planner runtime and warehouse state are native. A pruning step that never pruned is removed.

Verification

  • make test and make check-ci passed with all optional dependencies available.
  • Compiler differential: 371 projects, unchanged goldens. Planner differential: 11 projects matched.
  • Compile outputs and exit codes match the reference on synthetic and private compile-only projects.
  • New e2e test interrupts a compile during slow background lint; it fails without the fix.
  • Windows cross-check passed. Independent review findings (Ctrl-C delivery, Windows paths) fixed and re-verified.

…der constants natively

- Compile-input attachment and assembly call their native stages directly instead of
  through _native lookups, functools.partial and keyword dictionaries; the frontier
  produces native stages without Python stage callables.
- Declaration expansion renders constants with a native ConstantRendering instead of a
  per-model pathlib.Path, functools.partial and Python callback host.
- Lane module strings for compile, auditing, reference and shared-error types resolve
  through the host table.
- SQL test classification runs without the GIL on a bounded worker pool.
…ts natively

Declared relations, standalone pivot checks, native resource facts and cursor roles read a
native model input's attached schema and reference rows from its owner record instead of
building its SchemaModelEntry and CompileSqlReference projections; hand-built inputs still
read their record fields.
…e's own path

The schema facts of a model's header columns compare each column location against the
discovered model file's relative path object instead of a pathlib.Path built per model,
and skip the Python path comparison when the location carries that same object.
Dynamic-family indexes, model analysis rows and resource SQL checks read a native model
input's attached schema and a native audit's reference rows from the owner record instead
of building SchemaModelEntry and CompileSqlReference projections.
…atting

Header rules, header formatting and table-function argument names parse DSL header values
with the native header parser and compare them as their Python projections compare, instead
of calling parse_header_values through Python; the formatter no longer warms the Python
header cache.
…int on a native thread

Lint now scans headers, plans and prepares bodies, lints SQL and applies suppressions over
Rust rows, calling into Python only to expand user macros. Early SQL lint runs on a native
thread with an atomic stop instead of a Python executor and context-variable stop event, and
SQL rules consume its outcome, cache entries, keys and duplicate-literal hints as Rust rows,
building Finding records once at the end. Expansion inputs, relation keys, rules config and
compiled expansions are read natively; LintViolation, LintRunResult and SqlExpansionContext
remain projections for the Python-facing functions. Seam tests that patched the removed
native lookups now exercise the native paths.
Rust callers of model entry assembly, SQL resolution, cursor bounds and
inputs, change detection, relation contexts, source freshness, plan entry
results, migration planning, audit planning, SQL-test planning and scenario
relation plans now pass typed request structs instead of keyword dicts.
Python-facing wrappers parse keywords into the same structs; wrappers no
Python code calls are removed.
Change reconciliation, model entry results, plan output inputs and model,
column and old-name migration planning results are Rust structs passed
between phases instead of native records read back by attribute name. The
no-op execution-scope pruning step is gone; plan output metadata is
unchanged. Removes the unused build_plan_output, prune and reconcile entry
points and their records.
PlannerRuntime and PlannerWarehouseState are frozen native classes with
typed fields instead of generic native records. Planner phases read their
fields directly rather than by attribute name; Python still constructs and
reads them with the same keywords and attributes. The unused class-call
record default is removed.
@kvlonge
kvlonge enabled auto-merge (squash) October 16, 2026 15:19
@kvlonge
kvlonge merged commit 1ec45de into main Oct 16, 2026
40 checks passed
@kvlonge
kvlonge deleted the feat/chi599-batch-4 branch October 16, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant