Skip to content

refactor: digest rule facts and read declared columns natively - #1122

Merged
kvlonge merged 12 commits into
mainfrom
feat/chi599-batch-6
Oct 16, 2026
Merged

kvlonge merged 12 commits into
mainfrom
feat/chi599-batch-6

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

Custom-rule facts, declared columns, model analysis and early SQL lint still crossed into Python objects on every compile, and early lint copied every model's SQL.

Changes

  • Custom-rule fact digests are computed natively from the compiled project with a new deterministic, type-tagged encoding; the custom-rule cache version is bumped, so existing entries miss once.
  • Per-model rule inputs and declared columns are read from native facts; column facts are built from the parsed header through schema merge, enum resolution and migration checks.
  • Project assembly runs model analysis directly; the Python analysis request and result records are removed.
  • Early SQL lint and SQL rules reference the compile's SQL strings and convert one file at a time.
  • Windows read errors and case-insensitive model path lookups in rule facts match the previous behaviour.
  • Project Python (rules and their helpers, Python nodes, adapters, materializations, UDFs) loads from current source and never from cached bytecode, so same-size edits within one second are no longer missed.

Verification

  • make test and make check-ci passed with all optional dependencies available.
  • E2E compile, rules, format, plan and test suites passed (performance budgets excluded).
  • Compiler differential: 371 projects, unchanged goldens. Planner differential: 11 projects matched.
  • Compile outputs match the reference on synthetic and private compile-only projects.
  • Dense 10k peak RSS is well under the guard; Windows cross-check passed.
  • Independent review findings fixed; new tests cover type-tagged digests and stale-bytecode loading.

Recompute and digest recorded custom-rule facts from the native compiled
project in Rust instead of rebuilding Python fact views and encoding them
with orjson. Rule hosts digest live filesystem reads through the same native
encoder, project-tree file reads are native, and the custom-rule cache
runtime version is bumped so entries keyed on the old digests miss.
Read model names, project paths and materializations for custom-rule cache
rows, rule selection, run summaries and the native rules request through the
compiled model views directly instead of per-model attribute lookups.
Answer declared column rows, custom-rule SQL facts and semantic-check audit
metadata of compiled model views from the native facts and schema records
the compile already holds, instead of rebuilding schema entries and reading
each column back through Python attributes.
… in schema readers

Build each model's declared column facts (name, type, nullability,
description, migration source, audit definitions and location) from the
parsed header and the named schema while the column specs are built, instead
of reading every merged and enum-resolved column back through Python
attributes. Audit attachment, declared relation types and nullability,
relation keys and semantic-check audit metadata read these facts and touch a
column's Python spec only for its audits. Model input records keep the model
name, so readers no longer derive it from the file path.
Project assembly now starts, runs and finishes the native model analysis
session through Rust calls on typed request and result structs, instead of
building a NativeModelAnalysisRequest record and calling
analyze_native_model_sql and start_model_analysis_session through the
extension module. The request and result record classes and both module
functions are removed.

The analysis session tests now assemble real projects with a cold, a warm
and no analysis cache and compare those assemblies; their goldens are
unchanged. Tests that could only reach unreachable internal invariants
through the removed module seam are dropped; the duplicate-name invariant
stays covered by the native session tests.
…ront

Early SQL lint and SQL rules keep references to the compile's shared
authored and expanded SQL texts instead of copying every model's expansion
into owned strings before linting. A file's expansion is converted only when
that file is linted and dropped afterwards.
…lean

Model analysis converts its session request before running the session, so
the per-model request rows are freed instead of living through the run.
Declared column facts keep only what readers use: whether a column is
described, and the location of columns declared outside the model file.
Attached schemas no longer copy every column fact; the declared-relation
reader asks for them on its own.
@kvlonge
kvlonge enabled auto-merge (squash) October 16, 2026 19:19
@kvlonge
kvlonge merged commit 6f3bee6 into main Oct 16, 2026
40 checks passed
@kvlonge
kvlonge deleted the feat/chi599-batch-6 branch October 16, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant